Codename "Astoria": Data Services for the Web


The “data programmability” team at Microsoft is responsible for the various technologies developers use in applications to access and manipulate data. One of the topics we have been looking at lately is how “new” internet applications use data in the web environment. Project codename “Astoria” reflects our current thinking on the topic. Instead of telling “what we are going to build”, we decided to make the bits and infrastructure public for the development community to look at and give us feedback.


Today we are making a very early experimental release of Project Codename “Astoria”. This Community Tech Preview (CTP) release has a dual nature; we are shipping both CTP bits you can download, install and run on your own systems; we are also making available an experimental online service that we hope will help better understand the requirements and use cases of data interfaces in the web.


So, what does it do?


Astoria exposes “data services” that enable applications to access and manipulate data over regular HTTP connections, using URIs to identify pieces of information within the data service, and simple payload formats such as XML and JSON to represent the data exchanged between the client and the server.


What is it useful for?


There are a number of scenarios that we think are interesting for this technology. Most of the scenarios are related to the way new web applications are built.


If you look at AJAX-based web applications, one interesting aspect of the way they are delivered to the client is that typically the presentation and client-side behavior is delivered on the initial hit to a given web page, and then the code in that page (typically Javascript) turns back and fetches data as the user interacts with the user interface. This brings a strong separation between presentation and data. What is more, this means that now we require a server-side piece that the javascript code can “talk to”. Of course, you can always roll you own server-side entry points for data access; however, not only that is expensive but also it greatly reduces the chances to build tooling and user-interface controls that can work generically on any data access entry point into the server.


This separation is even stronger in rich internet applications built on technologies such as Silverlight and Flash. The code that is delivered to the client when a user hits the web application is pre-compiled and it contains no data at all other than any hardcoded information. Again, these applications talk back to their servers to retrieve data, manipulate it and push changes back into the service for storage.


Astoria data services are designed to address this space. They present data in a uniform way that can be consumed by any client that can connect over HTTP and parse XML or JSON. The uniform URI and payload format patterns mean that user-interface widgets can be built so they work against any particular shape of data (โ€œschemaโ€ if you will) that is exposed through the service.


Where is it?


The main site is here: http://astoria.mslivelabs.com


This document introduces the concepts and motivations for Astoria.


You can also see and interact with several sample data-sets that are already available on the experimental online service. Click on the “Online Service” link in the link bar.
 


Looking forward to hearing your feedback.


-pablo


 

Comments (37)

  1. You’ve been kicked (a good thing) – Trackback from DotNetKicks.com

  2. Diego says:

    Hey Pablo! Congratulations for your new blog and for Astoria! Now I know why you were missing lately ๐Ÿ˜‰

  3. Microsoft has just announced two new projects at MIX '07: Jasper and Astoria (codenames). Both tools

  4. CodeClimber says:

    MIX07 Announcements

  5. Pablo Castro has announced a very cool project , Microsoft Codename "Astoria": Data Services for the

  6. Jared says:

    Funny… I was in Astoria yesterday.

  7. rstrahl says:

    Pablo, this sounds great. I’ve been thinking for a while that this is one of the missing pieces in Microsoft’s Web strategy, especially in light of the coming Silverlight 1.1.

    I’ve been doing exactly what you mention – constantly rebuilding a data interface on the server and grabbing that data. It works well enough but there’s plenty of work to do to get this set up for every project.

    I noticed Julie mentioned that this what tied to the Entity framework? I hope this will be a bit more generic than that <s>…

  8. Where will the business rules go?

    What differs Astoria from WCF REST/POX/JSON endpoints that exposes EDM?

  9. Udi Dahan says:

    I guess that my thinking is similar to Patrik’s (above). My full writeup is here:

    http://udidahan.weblogs.us/2007/05/01/astoria-sdo-and-irrelevance/

    As a generic data fetching mechanism, I guess that it’s OK, but you seem to pushing the entire REST model which includes data updates – something that absolutely requires business rules.

    Also, how does this jive with the Service Data Objects work being done elsewhere (IBM et al.)?

    If all we’re really interested in is bringing data closer to the web client, why not just use XML files that are HTTP cached? That is probably the most scalable solution, designed for growth even to things like the Akamai network.

    I’d really like to see some more architecture/vision style things so that I can understand how this fits in an overal solution.

    Udi Dahan – The Software Simplist

    MVP Solutions Architect

  10. Well, this one snuck up on me. I hadn’t heard about this project before Mix, unlike the Silverlight &amp;

  11. Adrian says:

    Hey Pablo forgive me but am I right that "Astoria" is the long awaited  new technology called HTTP2SQL2XML ? ๐Ÿ˜‰

  12. Alex James says:

    I’ve been sifting throught the docs and have collected my thoughts here: http://www.base4.net/blog.aspx?ID=394

    Keen to here your answers to my questions ๐Ÿ˜‰

  13. Patrik: regarding where do the business rules go, this is the way I picture apps would expose their data services: for the subset of the data that is not heavy in semantics would just be exposed directly using Astoria URIs (using policies as needed to enforce read-only/read-write access, as well as auth/anonymous access requirements); for the other parts of the data that need business logic enforced, there are a few mechanisms, and you’d choose depending or your needs:

    a) you can use regular WCF service operations; these are nice when you want to take an arbitrary input and return an arbitrary output.

    b) you can use "data aware" WCF service operations, these are the ones that return ObjectQuery<T>. This are great when you want to provide controlled access to the data (e.g. restrict how much data you can see, apply custom policies, validate parameters, etc.). The great part about these ones is that they still get some of the URI patterns (e.g. you can sort, page, expand, etc.)

    c) if you are ok with flat access but want to control/validate/etc. the entities that enter and leave the system, there are "interceptors" (i’m looking for a better word ๐Ÿ™‚ that you can register; you can think of these as middle-tier triggers or something like that. Your code gets called when creating/updating/deleting entities as well as when sending them to the client, and you can make checks, modify the entity or abort the operation.

    As in how is this different from WCF entry points with EDM entities: this is actually a special form of WCF entry points that exposes data in a URI space automatically, without having to deal with data shaping, URI syntax for sorting/paging/filtering/etc. yourself.

    -pablo

  14. Udi: I just posted some comments regarding Patrik’s and your points around business logic. I agree that in most real apps, a large portion of the data in the system cannot be exposed "directly", and that is way Astoria supports various mechanisms for running code in the service, including WCF service operations, "data aware" service operations and also interceptors that run on both entity fetching and during entity updates.

    I am planning on writing a more elaborate "app architecture using Astoria" post at some point…let’s say Mix has kept us busy lately :), but I’ll get to it soon.

    -pablo

  15. Adrian:

    ๐Ÿ™‚

    We borrowed a lot from SQLXML. In fact, Andy Conrad, one of the folks in the Data Programmability team, used to be part of the SQLXML team. We learned a lot from SQLXML, and also the environment has changed since then. Astoria takes a lot of the lessons (good and bad) and is also designed to live in the current state of the web.

    There are some aspects such as instance-based updates to give it a more REST-like feel, using managed code in the middle tier for business rules, support for JSON, and others, that were necessary in my opinion and weren’t in SQLXML back then.

    I talk often with various folks that built SQLXML and exchange notes to make sure I "reuse" good ideas and avoid mistakes.

    -pablo

  16. At 1999/2000 people start to do queries with URL and QueryStrings to get XML from a data source, to make it possible to ask for specific data and get a result that was platform independent. Then later we got FOR XML added to SQL Server and also a way to run query directly against the database with URLs. We also got SOAP Web Services. Now we have WCF. But suddenly we are back again to the 1999/2000 where we do queries over URLโ€™s ๐Ÿ˜‰

  17. A few months ago, Alex Barnett excitedly told me about a project he was working on with Pablo Castro

  18. vikasgoyal77 says:

    Hi,

    are the above links down ?

    i am getting ‘host not found’

    regards,

    vikas

  19. rogerj says:

    Links to the video of Pablo’s XD006 session are at http://oakleafblog.blogspot.com/2007/05/mix07-session-videos-with-linq-ef-or.html

    More background on LINQ and Entity Framework related MIX07 sessions is at http://oakleafblog.blogspot.com/2007/04/linq-related-sessions-at-mix.html

    –rj

  20. Thanks for your answer Pablo. It does clear some clouds but I still don’t see where Astoria fits into the puzzle and excels over existing technology.

    I’m looking forward to your architecture post.

    [OT]:: Btw, can you confirm that you pulled EF so you could implement Model first as an approach in the tools?

  21. vikas: I just tested both the website and the online service and they all worked from here. Please try again if you have a chance…if this keeps happening let me know and I’ll take another look.

    -pablo

  22. vikasgoyal77 says:

    Thanks Pablo .. there seems to be some problem with one of the proxy in my corporate .. i am able to access using other.

    regards,

    vikas

  23. Dennis' Blog says:

    At MIX07 I’ve seen a presentation by Pablo Castro from the ADO.NET team on a new project codenamed "Astoria".

  24. Udi Dahan says:

    Pablo,

    The problem I have is with the generic "put" which leaves you with simplistic optimistic locking. I need a business level API for passing along the client’s intent.

    Check out Realistic Concurrency here:

    http://udidahan.weblogs.us/2007/01/22/realistic-concurrency/

  25. Last week at Mix07 Microsoft announced a project code named Astoria. The technology code named Astoria

  26. This was one of several projects that did not surface much at MIX due to the silverlight hype.. I have

  27. mike montagne says:

    Hypothetical Performance Question…

    Pablo,

    I would like to know if Astoria can solve a hypothetical data broadcasting performance issue. We have a client who needs to broadcast, virtually simultaneously, a small amount of data — typically a single integer — to as many as tens of thousands of client applications. The client applications need to register for the broadcast; wait; receive the data within the narrowest possible timespan; and react to the data immediately. We need to know what the "narrowest possible timespan" for such an Astoria broadcast would be… and of course whether Astoria is a candidate technology for such an implementation.

  28. Last week at MIX07 Microsoft announced a project code named Astoria. The technology code named Astoria

  29. Bashmohandes says:

    ASTORIA Data Services

  30. pabloc says:

    Mike,

    I don’t think that Astoria is the right tool for your scenario. Astoria would still do point-to-point communication with clients, and clients would have to poll for changes, both probably bad things given your requirements.

    If the setup you describe runs within a private network, not across the internet, then you could consider using a multi-cast based scheme to push data to client applications. This is very common in certain sectors of the financial industry, where you need to push small bits of data at a very high rate.

    -pablo

  31. Project Codename โ€œ Astoriaโ€ – http://astoria.mslivelabs.com . Microsoft Codename Astoria enables applications

  32. Quick update on what Ive been looking at recently.. (the on-going list of technolgies and things-to-learn

  33. Data says:

    The Astoria Team is happy to announce the availability of an add-on to the Astoria May CTP that contains

  34. Back in April, the Data Programmability team at Microsoft announced "Astoria": Data Services for the

  35. I’ve seen a few instances on posts to forums or blogs asking whether Astoria continues to be an experiment

  36. I&#39;ve seen a few instances on posts to forums or blogs asking whether Astoria continues to be an experiment

  37. Joe says:

    12 or 24 templates two start projects.And will there bee a way use office live work space and listas.