Astoria Design: payload formats

DB Blogs

The goal of Astoria is to make data available to loosely coupled systems for querying and manipulation. In order to do that we need to use protocols that define the interaction model between the producer and the consumer of that data, and of course we have to serialize the data in some form that all the involved parties understand. So protocols and formats are an important topic in our design process.

With that in mind, we’ve made sure that Astoria can handle a number of options from the system architecture perspective. Of course, each additional option has the potential of introducing added complexity to the system, and also adds to the overall development and testing time of the system, so we have to pick a small set to start with for the first release.

This entry focuses on formats. In general, introducing support for a format in Astoria means going through the exercise of mapping the data model used in Astoria to the different constructs of the target format. Astoria is largely an EDM-based system from the data model perspective, so one could see the problem as creating serialization forms for EDM-schematized data.

Key questions:

• Is this the right list of formats? Should we do less, more, or a different set?
• For each format, does the mapping to the underlying Astoria and EDM construct look reasonable?
• Format experts in each area: feel free to pick on details and add random thoughts…none of the aspects discussed here are set on stone

NOTE: we’ve discussed these formats to a ridiculous level of detail during our design meetings. Here I’m including the most relevant aspects and those that will show up immediately when looking at a request or response. Once we have cleaner specs we can post all the details if there is demand for them.

Multiple formats, one protocol (almost)

For the most part there is a single “protocol”, and by that I mean the set of HTTP headers for requests and responses, as well as the overall interaction model. In certain cases in order to make a format really look natural to clients we do need to introduce a format-specific protocol element, but we try to keep those to a minimum.

Also, any added protocol elements on top of HTTP need to be done so that unsophisticated agents can ignore a lot of that, do “plain HTTP” and still get by for the most part.

A full discussion on protocol details needs its own write-up, and it’ll come later on. The short story is that we leverage as much as HTTP as we can; HTTP verbs are used for the basic operations, and negotiation such as content-type and charsets are done in the standard way.

Now, with the almost-single protocol in place, the question comes to which formats should we do. Right now we’re thinking ATOM/APP, Web3S, and JSON. We need to define the basic requirements for any format used in Astoria, and then map those to each format we want to support. That’s what comes next.

Astoria requirements for formats

For each format we want to support in Astoria we need to introduce representations for the following constructs:

• A “top level” resource, which is returned when a client does a GET on the data service URI. We want to present a fully connected (hyperlinked) set of resources, so this resource is typically some sort of list of links to top-level resource containers.
• A representation for an “Entity”, which is what resources are in Astoria. This is typically a property bag.
• A representation for properties. Each property has a name and a value; the value can be a scalar, a structure (complex type in EDM parlance) or a link to a related entity or set of entities (an association in EDM).
• A representation for a “set of entities” and a “bag of entities”. This will typically be the same, but it’s worth noting that sometimes we return sets and sometimes we return bags. Each top-level resource container returns one of these; other sources of sets/bags are URI queries and the result of traversing an association when the other end is a set of entities.

These formats are used, in somewhat different contexts, for all Astoria operations, including retrieval, update and query over resources. Formats are as symmetric as we can make them for retrieval and update (you get from, and send to the server the same thing, except if very rare cases).

In general Astoria services don’t embed a lot of typing information in the payloads. Some formats include some “light” typing (e.g. JSON), but if a client wants strict typing then the metadata resource should be accessed and the complete typing information extracted from it (more about metadata resources in a future post).

ATOM/APP

This format follows the specification described in APP and ATOM (and it goes beyond formats to include protocol-related aspects). I resisted APP at first. My question was “why do you want to use a feed format to represent data that is a temporal feed”. After pretty much everybody inside/outside Microsoft and within the Astoria team pushed for it, I realized that I was missing an obvious aspect: the fact that it’s a feed format is completely irrelevant. The important characteristic about ATOM/APP is that everybody speaks it, and it’s increasingly becoming kind of a lingua franca of the web when it comes to data exchange between unrelated agents.

It turns out that many of the ATOM and APP concepts map pretty well to the way Astoria exposes data.

• Top-level resource: the “service document” defined by APP makes a great root resource for Astoria services, as it provides a clean, compliant way to list the links to each of the top-level resource containers in a given data service, presenting them as APP “collections”.
• Entities: Entities or resources are serialized as “entry” elements in an ATOM feed.
• Properties:
• Each scalar/structure property is turned into a custom property in the item, carrying a custom XML namespace prefix to separate them from standard ATOM elements
• Optionally we can map some of the properties to standard elements (e.g. title, author). I’m curious to know what folks think about this…should we do this at all? Should we do this by convention (e.g. by name) or by annotations in metadata?
• Finally, ATOM defines the concept of links in entries, which not only have a target URI but also a “rel” (relation) attribute that carries added semantics for the link. This maps nicely to the EDM model that underlies Astoria, where entities are linked together by named associations. Associations are turned into links, and the “rel” attribute says which association it is.
• Sets/bags of entities are mapped to APP “containers” which are represented as ATOM “feed” documents. This isn’t completely natural (they are not really feeds), but it works ok in practice.

From the perspective of manipulating data (i.e. adding, removing, modifying entries), APP follows HTTP very closely, which makes our life easy from the Astoria perspective, as the system is built from the ground up to map to HTTP operational semantics.

There are some caveats with mapping data to an ATOM feed, and to APP more in general to cover for updates, service documents and collections. Most of them have to do with certain required elements in ATOM that do not have a direct mapping in Astoria or EDM in general. For example “title” and “author” elements of “feed” are tricky to map. ATOM was the last protocol in our design list so it’s the least baked right now. Once we have all the rules for it we’ll write an ATOM/APP specific post to discuss the details.

The production code does not yet produce APP, so I can’t include an example right now. Once this is complete we’ll iterate again on the topic of APP and include samples.

JSON

The JSON (Javascript Object Notation) format has become popular lately mostly (my guess) because it makes it easy to consume results from services in Javascript environments, particularly inside web browsers. JSON is a simple format, with no prescription and no widely adopted standards on top of the base format. On one hand that means a fairly clean serialization form, on the other it means that we have to do a few tricks in order to accommodate our requirements.

In addition to be compliant, we tried to design the JSON format so that it’s friendly to their main consumers: javascript clients. What that means in practice is that whenever possible we structured the format so that the result of doing an “eval” of a given service response results in an object graph that is fairly natural and can be used directly from Javascript code without seeing any artifacts in it.

The JSON view of Astoria resources is fairly simple:

• Top-level resource: we return an object with a member for each top-level resource container and their links as values.
• Entities:  each entity/resource is mapped to a JSON object (associative array)
• In addition to the actual data, we usually want to include some instance-level metadata with each entity. At least we want the URI for the entity and its actual type. Since there are no namespaces in JSON to separate this from actual data, we use a property called “__metadata” that has these values in it.
• Properties: we map them to JSON object properties.
• If the property is a scalar, we use a JSON scalar. This is no easy task sometimes because of the small set of types and literal forms supported in JSON. For example, there is no Date/Time literal form, and there is no decimal (precise floating point) data type at all. For those that don’t map directly we generate their values as strings; the one exception is date/time, where we use the Microsoft ASP.NET AJAX (aka Atlas) literal form (see more about it here).
• If the property is a structure (e.g. an expanded related object, or a value of a complex type) we simply nest the object underneath the outer object
• If the property is a link we add an object with a single property called “__deferred”, indicating that there is deferred content for this property. The __deferred object has a single property called “uri” that indicates how to get the deferred data. We may generate links for associations and we’re also considering doing it for BLOBs so we don’t include them by default in the payload.
• Sets/bags: we use JSON arrays for these. One caveat is that we cannot attach metadata to the array itself, sometimes resulting in redundant instance-level metadata in entities.

To show an example I’ll use the typical Northwind sample database, and pick a single “Customer” entity; this response is a set like the one that would be returned from a top-level container:

[

    {

        __metadata: {

            uri: “Customers!\’ALFKI\'”,

            type: “NorthwindModel.Customer”

        },

        CustomerID: “ALFKI”,

        CompanyName: “Alfreds Futterkiste”,

        ContactName: “Maria Anders”,

        ContactTitle: “Sales Representative”,

        Address: “Obere Str. 57”,

        City: “Berlin”,

        Region: null,

        PostalCode: “12209”,

        Country: “Germany”,

        Phone: “030-0074321”,

        Fax: “030-0076545”,

        Orders: {

            __deferred: {

                uri: “Customers!\’ALFKI\’/Orders”

            }

        },

        CustomerDemographics: {

            __deferred: {

                uri: “Customers!\’ALFKI\’/CustomerDemographics”

            }

        }

    }

]

Web3S

In the first CTP release of Astoria we had a “plain” XML format we informally called POX for plain-old XML. The goal of this format was to introduce something that did not added extra concepts on top of Astoria/EDM and just presented the data in XML for easy parsing.

It turns out that at the same time the folks at the Live organization in Microsoft were working on a protocol for data exchange that would be used uniformly across many different services. That protocol is called Web3S and its extensively documented here, and there is also a FAQ here.

The obvious question is: should we still roll our own custom XML format or try to unify with Web3S. Since in this kind of thing less is often more, we’ve been working hard trying to unify them. What that means in practice is that Astoria would speak Web3S, and we would make some tweaks to Web3S so that it work well with Astoria services and clients.

Making Astoria work with Web3S is a bit tricky…the main issue is that Web3S models data as a big tree, where as Astoria uses a record-oriented view of the world (entities and associations to be precise). The mapping goes more or less as follows:

• Top-level resource: this one looked easy so we ended up postponing this a bit. We will list the set of top-level resource containers; what we haven’t decided yet is whether we’ll make it a Web3S element itself or not.
• Entities: in Web3S there are “elements”, which are a very generic, tree like thingy. We map entities to elements, which actually works well because nobody says that we cannot only generate elements that are a single level deep (i.e. not trees). To be somewhat more specific, we map entities to elements in a multi-element container, so every element that represents an entity has an ID (which we form using the key(s) of the entity).
• Properties:
• Scalar properties become children of entity elements. They are elements themselves but have no children, only a scalar value.
• Structures (EDM complex types) are represented as elements with nested elements.
• Links (associations) are mapped to elements with “deferred content”. Since Web3S has a clean and simple URI composition rule, clients can easily come up with the URI of any element, including deferred ones.
• Sets/bags: these are Web3S elements with multiple children element. In Web3S children of an element could be of any kind; Astoria does not use such generic construct and instead children of a given element are typically restricted to the ones that are of the type of the container or some subtype.

Here is an example of the typical Customer entity from the Northwind sample database in Web3S format; in this case it includes a wrapping Customers element, as in when the entity is returned as part of a set:

<?xml version=”1.0″ encoding=”utf-8″ standalone=”yes” ?>

<Customers xmlns:web3s=”Web3S:” xmlns:dw=”http://schemas.microsoft.com/ado/2007/08/dataweb” xml:base=”/Northwind.svc” xmlns=”Web3SBase:NorthwindModel“>

 <Customer>

<web3s:ID>’ALFKI'</web3s:ID>

<web3s:Uri>Customers/Customer(‘ALFKI’)</web3s:Uri>

<CustomerID>ALFKI</CustomerID>

<CompanyName>Alfreds Futterkiste</CompanyName>

<ContactName>Maria Anders</ContactName>

<ContactTitle>Sales Representative</ContactTitle>

<Address>Obere Str. 57</Address>

<City>Berlin</City>

<Region>-</Region>

<PostalCode>12209</PostalCode>

<Country>Germany</Country>

<Phone>030-0074321</Phone>

<Fax>030-0076545</Fax>

<Orders>

<web3s:DeferredContent />

</Orders>

<CustomerDemographics>

<web3s:DeferredContent />

</CustomerDemographics>

 </Customer>

</Customers>

What happened with RDF?

The May 2007 CTP also included support for RDF. While we got positive comments about the fact we supported it, we didn’t see any early user actually using it and we haven’t seen a particular popular scenario where RDF was a must-have. So we are thinking that we may not include RDF as a format in the first release of Astoria, and focus on the other 3 formats (which are already a bunch from the development/testing perspective).

My personal take is that while I understand how RDF fits in the picture of the semantic web and related tools, the semantic web goes well beyond a particular format. The point is to have well-defined, derivable semantics from services. I believe that Astoria does this independently of the format being used. That, combined with the fact that we didn’t see a strong demand for it, put RDF lower in our priority lists for formats.

Retrieval, query and update semantics

This post did not elaborate a lot on how payloads are actually used in retrieval and update operations. This is already a long post as it is, so let’s call this “the formats initial write-up” and stop it here. Future posts will discuss these topics and also dig into specific details of each particular format.

Pablo Castro
Technical Lead
Microsoft Corporation
http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

DB Blogs

Follow