Windows Azure Storage Overview

  • Article

I am at the Azure Firestarter event in Redmond today and just heard Brad Calder give a quick overview of Azure data.  Here are my notes; slides and sample code are to be posted later and I will update the post with them when they are.

  • Blobs
    • REST APIs
    • Can have a lease on the blob - allows for limiting access to the blob (used by drives)
    • To create a blob…
      • Use StorageCredentialsAccountAndKey to create the authentication object
      • Use CloudBlobClient to establish a connection using the authentication object and a URI to the blob store (from the portal)
      • Use CloudBlobContainer to create/access a container
      • Use CloudBlob to access/create a blob
    • Two types of blobs
      • Block blob - up to 200 GB
        • Targeted at streaming workloads (e.g. photos, images)
        • Can update blocks in whatever order (e.g. potentially mulitple streams)
      • Page blob - up to 1 TB
        • Targeted at random read/write workloads
        • Used for drives
        • Pages not stored are effectively initialized to all zeros.
          • Only charged for pages you actually store.
          • Can create a 100 GB blob, but write 1 MB to it - only charged for 1 MB of pages.
        • Page size == 512 bytes
          • Updates must be 512 byte aligned (up to 4 MB at a time)
          • Can read from any offset
        • ClearPages removes the content - not charged for cleared pages.
  • CDN
    • Storage account can be enabled for CDN.
    • Will get back a domain name to access blobs - can register a custom domain name via CDN.
    • Different from base domain used to access blobs directly - if you use the main storage account URL, will retrieve from blob store not using CDN.
    • To use CDN
      • Create a blob
      • When creating a blob, specify "TTL" - time to live in the CDN in seconds.
      • Reference the blob using the CDN URL and it will cache it in the nearest CDN store.
  • Signed URLs (Shared Access Signatures) for Blobs
    • Can give limited access to blobs without giving out your secret key.
    • Create a Shared Access Signature (SAS) that gives time-based access to the blob.
      • Specify start-time and end-time.
      • What resource-granularity (all blobs in a container, just one blob in the container)
      • Read/write/delete access permissions.
    • Give out URL with signature.
    • Signature is validated against a signed identifier. You can instantaneously revoke access to a signature issued by removing the signed identifier.
      • Can also store time range and permissions with the signed identifier rather than in the URL.
      • Can change them after issuing the URL and the signature is still valid in the URL.
  • Windows Azure Drive
    • Provides a durable NTFS drive using page blob storage.
      • Actually a formatted single-volume NTFS VHD up to 1 TB in size (same limit as page blob)
      • Can only be mounted by one VM instance at a time.
        • Note that each role instance runs on a VM, so only one role instance can mount a drive read/write
        • Could not have both a worker role and a web role mounting the same drive read/write
      • One VM instance can mount up to 16 drives.
    • Because a drive is just a page blob, can upload your VHD from a client.
    • An Azure instance mounts the blob
      • Obtains a lease
      • Specifies how much local disk storage to use for caching the page blob
    • APIs
      • CloudDriveInitializeCache - initalize how much local cache to use for the drive
      • CloudStorageAccount - to access the blob
      • Create a CloudDrive object using CreateCloudDrive specifying the URI to the page blob
      • Against CloudDrive…
        • Create to initialize it.
        • Mount to mount it - returns path on local file system and then access using normal NTFS APIs
        • Snapshot to create backups
          • Can mount snapshots as read-only VHDs
        • Unmount to unmount it.
    • Driver for mounting blobs only in the cloud - not on development fabric.
      • Instead, just use VHDs
  • Tables
    • Table can have billions of entities and terabytes of data.
    • Highly scalable.
    • WCF Data Services - LINQ or REST APIs
    • Table row has a partition key and a row key
      • Partition key:
        • controls granularity of locality (all entities with same partition key will be stored and cached together)
        • provides entity group transactions - as long as entities have same partition key, can do up to 100 insert/update/delete operations as a batch and will be atomic.
        • enable scalability - monitor usage patterns and use partition key to scale out across different servers based on partition keys
          • More granularity of partition key = better scalability options
          • Less granularity of partition key = better ability to do atomic operations across multiple rows (because all must have same partition key)
    • To create / use an entity
      • Create a .NET class modeling an entity
      • Specify the DataServiceKey attribute to tell WCF Data Services the primary keys (partitionkey, rowkey)
      • APIs
        • CloudTableClient - establish URI and account to access table store
        • TableServiceContext - get from CloudTableClient
        • Add entities using the context AddObject method specifying the table name and the class with the data for the new entity
          • SaveChangesWithRetries against context to save the object.
        • To Query… using LINQ with AsTableServiceQuery<xxxx> where xxx is the .NET class modeling the entity.
          • Manages continuation tokens for you
        • Then do a foreach and can use UpdateObject to update objects in the LINQ stream.
        • Use SaveChangesWithRetries
          • Add SaveChangesOptions.batch if < =100 records and all have same partition key - save as one batch.
          • If not, sends a transaction for each object.
    • Table tips
      • ServicePointManager.DefaultConnectionLimit = x (default .NET HTTP connections = 2)
      • Use SaveChangesWithRetries and AsTableServiceQuery to get best performance
      • Handle Conflict errors on inserts and NotFound errors on Delete
        • Can happen because of retries
      • Avoid append only write patterns based on partitionkey values
        • Can happen if partition key is based on timestamp.
        • If you keep appending, defeating scale out strategy of Azure
        • Make partition key be distributed not all in one area.
  • Queues
    • Provide reliable delivery of messages
    • Allow loosely coupled workflow between roles
      • Work gets loaded into a queue
      • Multiple workers consume the queue
    • When dequeuing a message, specify an "invisibility time" which leaves the message in the queue but makes it temporarily invisible to other workers
      • Allows for reliability.
    • APIs
      • Create a CloudQueueClient using account and credentials
      • Create a CloudQueue using the client and GetQueueReference - queue name
      • CreateIfNotExist to create it if not there
      • Create a CloudQueueMessage with content
      • Use CloudQueue.AddMessage to add it to the queue
      • Use CloudQueue.GetMessage to get it out (passing invisibility time)
    • Tips on Queues
      • Messages up to 8 KB in size
        • Put in a blob if more and send blob pointer as message
      • Remember that a message can be processed more than once.
        • Make that not be a problem - idempotent
      • Assume messages are not processed in any particular order
      • Queues can handle up to about 500 messages/second.
        • For higher throughput - batch items into a blob and send a message with reference to blob containing 10 work items.
        • Worker does 10 items at a time.
        • Increased throughput by 10x.
      • Use DequeueCount to remove "poison messages" that seem to be repeatedly crashing workers.
      • Monitor message count to increase/decrease worker instances using service management APIs.
    • Q: can you set priorities on queue messages?
      • A: No - would have to create different queues
    • Q: Are blobs stored within the EU complying with the EU privacy policies?
      • A: Microsoft has a standard privacy policy which we adhere to.