Essential Knowledge for Windows Azure Storage

What this post is about

1. This post is dedicated to understanding how to leverage Windows Azure Storage for your applications.
2. Windows Azure currently has over 4 trillion objects stored
3. Today, the storage service processes an average of 270,000 requests per second, and reaches peaks of 880,000 requests per second. By the time you read this, it is likely much higher.
4. It is a powerful and flexible cloud-based storage system and service that can store a seemingly limitless amounts of data for any duration of time.
5. Any application that supports http can access data from anywhere, at any time, and you only pay for what you use
6. Azure Storage is used inside Microsoft for many purposes, such as social networking search, serving video, music and game content, managing medical records, and more.

What you get	How much
Compute	750 small compute hours per month
Web sites	10 web sites
Mobile services	10 mobile services
Relational database	1 SQL database
SQL reporting	100 hours per month
Storage	35GB with 50,000,000 storage transactions
Bandwidth	Unlimited inbound & 25GB outbound
CDN	20GB outbound with 50,000 transactions
Cache	128MB
Service bus	1,500 relay hours and 500,000 messages
Sign Up Link	Sign up right here

Prerequisites and Setup

1. These are just some of the pre-requisites when developing cloud applications
2. Azure is an open platform, meaning you can access storage from any application capable of http

A World Wide Network of Mega Data Centers

1. Azure is globally distributed
2. Statement on Auditing Standards (SAS) No. 70, is a widely recognized auditing standard developed by the American Institute of Certified Public Accountants (AICPA). It represents that a service organization has been through an in-depth audit of their control objectives and control activities, which often include controls over information technology and related processes. SAS 70 forces companies to demonstrate that they have adequate controls and safeguards when they host or process data belonging to their customers
3. ISO/IEC 27001 formally specifies a management system that is intended to bring information security under explicit management control, meeting specific requirements. Organizations that claim to have adopted ISO/IEC 27001 can therefore be formally audited and certified compliant .
4. The Federal Information Security Management Act of 2002 (FISMA) is a United States federal law enacted in 2002, which recognizes the importance of information security to the economic and national security interests of the United States. The act requires each federal agency to develop, document, and implement an agency-wide program to provide information security for the information and information systems. FISMA has brought attention within the federal government to cybersecurity. FISMA emphasizes risk-based policy for cost-effective security.
   • FISMA requires agency program officials, chief information officers, and inspectors general (IGs) to conduct annual reviews of the agency's information security program and report the results to Office of Management and Budget (OMB).

Why Windows Azure Storage Services?

1. We will discuss tables, blobs, and queues in depth
2. Regardless of the storage type, the data is replicated 3 times. You can even replicate data across data centers.
3. Cloud-based storage is important because it is centralized, supporting todays mobile applications
4. Azure storage can store almost any data type, both structured and unstructured, including relational data
5. The platform includes other services to make storage more effective, such data synchronization, content delivery networks.The Azure platform also lets you host cloud applications in the cloud, whether those applications are based on Linux or Windows.
6. There is also a REST-based API that makes it possible for any http-capable client of communicating with storage. There is also a .NET Framework based API that makes programming with C# or Visual Basic very easy

Supported Developer Platforms

1. Windows Azure is an open cloud platform that enables you to quickly build, deploy and manage applications across a global network of Microsoft-managed datacenters. You can build applications using any language, tool or framework.
2. More information is available here:
   • https://www.windowsazure.com/en-us/develop/overview/

The 4 Pillars of Azure Storage

1. Note the 4 types of storage
2. They cover many of the typical use cases for storage
3. There are also other storage types in preview mode, such as Windows Azure Hadoop.
   • You can learn more here: https://www.hadooponazure.com
   • Windows Azure HDInsight Preview
   • To gain the full value of Big Data you need a modern data platform that manages data of any type, whether structured or unstructured, and of any size. This preview of Windows Azure HDInsight enables you to get up and running with your own Apache Hadoop cluster in the cloud in just minutes.
   • Microsoft's end-to-end roadmap for Big Data embraces Apache Hadoop by distributing enterprise class, Hadoop-based solutions on both Windows Server and Windows Azure. To learn more about Microsoft's roadmap for Big Data see https://www.microsoft.com/bigdata/

Typical uses for Azure Storage

1. Many customers are leveraging Azure Storage in a wide variety of ways
2. There are very few use-cases where Azure Storage doesn't make sense.
3. If you factor in Windows Azure HDInsight, you also have Map/Reduce scenarios covered
   • Hive, Pig, Hadoop Common are supported

Understanding Storage Account Limits

1. If more storage is needed beyond 100 TBs, additional storage accounts can be opened. Azure storage is effectively limitless.
2. I think the biggest challenge when dealing with cloud storage, is the latency. In the world of cloud, this is something that cannot be overcome for any cloud vendor.

Storage Account Hierarchy

1. An account has a one to many relationship to containers, tables, and queues.
2. The one to many relationship also exists for containers, tables, and queues. For example, a container can have many blobs, etc.
3. The corresponding URL reflects the storage type as depicted by the red boxes

Windows Azure Storage Analytics

1. Windows Azure Storage Analytics performs logging and provides metrics data for a storage account. You can use this data to trace storage requests, analyze usage trends, and diagnose issues with your storage account. Windows Azure Storage Analytics works with Blobs, Tables and Queues. The Goal is to improve the design of your applications and their access patterns to Windows Azure Storage.
2. Analytics data consists of:
   • Logs
     • Provides a trace of executed requests for your storage accounts
   • Metrics
     • Provide summary of key capacity and request statistics for Blobs, Tables and Queues
     • Provides hourly aggregates of number of requests, average server side latency, average E2E latency, average bandwidth, total successful requests and total number of failures and more.
     • These request aggregates are provided at a service level and per API level for APIs requested in that hour.
   • Capacity information
     • Provides daily statistics for the space consumed by the service, number of containers and number of objects that are stored in the service. Note, this is currently only provided for the Windows Azure Blob service.
3. Logs and Metrics Accessible from the Internet. All Analytics Logs and Metrics data are stored in your user account and is accessible via normal Blob and Table REST APIs. The logs and metrics can be accessed from a service running in Windows Azure or directly over the Internet from any application that can send and receive HTTP/HTTPS requests.
4. Windows Azure Storage stores analytics data in the storage account.
5. The Log data is in blobs, the metrics in tables. Log data is stored as Windows Azure Blobs in a special blob container and metrics data is stored in special tables in Windows Azure Tables. This allows you to import the log data into various tools You can query metric data with LINQ because it is stored in Azure tables.
6. There is a Retention Policy to ease the management of this data. Azure provides the ability to set a retention policy that will automatically clean up your analytics blob and table data.

Storage Analytics: Collecting Logs and Metrics Data

1. A summary of how Storage Analytics works

Storage Analytics: Collecting Logs and Metrics Data

1. Note that this information is more global in nature. It operates at the storage account level.

Storage Analytics: Collecting Logs and Metrics Data

1. Note that this information operates at the storage type level. Storage analytics work with blobs, tables, and queues.

Conclusion

1. Future posts will cover these areas in more detail.

Thanks..
I appreciate that you took the time to read this post. I look forward to your comments.