My tryst with Destiny, err… Integrated Storage

Hi, my name is Sanjay Anand. I run the Program Management team for WinFS.  I’ve been at Microsoft for almost 12 years now in which time I have been involved with networking, security, messaging, file systems, and of course integrated storage 🙂


A very very long time ago, my friend and I joined Microsoft to work on “next generation” Operating Systems – I was on Windows NT, he was on Cairo. Cairo was this sexy, glitzy, very futuristic attempt at building an integrated store – an object file system. This was the thing that would replace Windows NT the moment it shipped. I plodded along on the not-so-new NT, knowing that Cairo would come along and replace most of what we were working on. Then one day, Cairo disappeared and the machinery moved on to salvage parts of Cairo to deliver on the premium OS – Windows NT. At that time I didn’t think much of the whole thing except that it was perhaps far ahead of it’s time…

Webstore circa 1999:

A half-decade later, I encountered “Integrated Storage” for the second time while working on the Exchange Webstore. The premise was to build a single enterprise store that would have integrated manageability, multi-tier deployment, supported multiple protocols and data access stacks and a platform for app development. It was a file server, a web server, a corporate email server, internet email server, collaboration server, all rolled into one. The file server in particular was the piece I worked on. We actually shipped this product in Exchange 2000, but it was still-born. The technological underpinnings weren’t quite there – it lacked real transactions, had issues with scalability, was too mail-oriented. The win32 support was too hard – we mastered the entire file system in a single file, literally re-implementing NTFS within a file, with our own block management strategy, our own locking implementation, complex interactions with the cache manager – oh! It was a nightmare, but we got it limping along. A brave effort nonetheless and something we were to learn from eventually!

The client manifestation of this webstore was called the LIS (local information store) – a valiant attempt at building a client platform to match the Webstore on the server. Office Designer was the dev environment on the client and Outlook was the marquee application on the client to use that store. The beginnings of the Outlook offline store took root here. The challenges were two-fold: take the server store and “shrink” it to the client and provide a MAPI shim over an HTTP wire protocol for the sync. The performance never really matched the requirements of Outlook, the store was too general purpose… this went by the way side as well.

Fastforward to 2003:

Another half decade later, we launched with renewed energy towards the 5th attempt (and my second) at Integrated Storage – WinFS. A lot of factors were in our favor – the need for a storage platform was more acute than ever with digitally born data exploding every day, a lot of technological underpinnings were there for the taking: transactions in file systems, mature databases, sophisticated programming languages, but most importantly, a set of very motivated individuals who had deep experience in the various elements required to bring together this complex technology. We had folks with deep database experience from QO/QP to transactions, aces in programming languages,  folks with emmense experience in file systems, distributed systems, web services, hard core synchronization wonks, O/R mapping gurus, but above all, a fearless leader who was ready to take a real shot at this thing once and for all and who had the maturity and discipline to control the engineering of this innovation – Peter Spiro.

The project started with much fanfare – a who’s-who of very senior people passionately oversaw the vision, tons of internal teams hooked into the common schema effort – identity services, state management, windows help, natural languages, Windows shell, Pix, Media, etc. At one point we also had over 80 external ISV’s who were at various levels of commitment. These were heady times; we knew these were also unsustainable for any single platform to serve, but we also knew these engagements were essential to distilling the core nuggets of innovation to target for a v1. The Vista change occurred and while negative on the surface, from a project perspective it actually helped us take pause, distill our core value props and run really fast at hitting these. There are many ways where WinFS adds value, however I want to touch upon one that really appeals to me: a storage platform for application development.

Storage platform:

First off, we are about building a storage platform that enables applications to model their data in a much richer fashion, while still benefiting from common services one would expect from a file system: administration, sync, sharing, security, backup, drag-n-drop, move/copy, delete and yes, search. File systems don’t allow applications to model their data in anything richer than byte streams when it comes to interacting with other applications, Windows shell, etc. Databases allow you to model your data in richer ways by leveraging relational capabilities, however none of these automatically enable that data to be shared with the Windows Shell much as a file is. So, there you have it – we are the best of file systems (Integration and services) and databases (rich data modeling capabilities) rolled into one – hence Integrated Storage.

Now, lets move further and see how you can do much more if you actually share that rich data definition across applications. Let me take an example: reusable organization is an extremely powerful concept – something that application-specific tagging cannot even come close to achieving. Consider this scenario: at each of my daughter’s b’day celebrations at her school, we take in a poster of her pictures from previous years. I search for the pictures in one app (Digital Image Suite), auto-fix in another (Photoshop), print using HP’s custom album printing app. Each app has some notion of query-based searches, but no two share it. So, whenever I transition between applications, I end up duplicating my photos into folders since that’s the common denominator across applications. Then, I go through and delete these duplicates which I have collected over time, including copies of edited ones, ones I staved off to put on, post on spaces, email to my sister, the list goes on and on. Let me admit that one of the most difficult item types to delete is a photo – you can buy music again, re-write a document, but not a photo. So, I end up not deleting the photos many times… Yes storage is cheap, but this is ridiculous – and I am a reasonably tech savvy person… I recognize that there are several answers to this scenario: build a better single app to do the entire photo workflow; standardize on a common tag file format, etc. However, the same scenarios apply to music – my organization in Media Player is lost in iTunes. I use both: the first to stream the second for my iPod. In fact they apply to all forms of digital data.

The point is that the file system today is too primitive a substrate for these applications to work over and WinFS attempts to close this gap by raising the level of involvement in the application’s data: it’s a trade – the more apps tell the system about their data, the more the platform does for their types in terms of allowing an ecosystem of applications to interact with their content. The user organizes her content in one app – sets up a query to filter down the photos – “all 5-star photos of my daughter that I haven’t used earlier, grouped by year taken, ordered by resolution” and another app uses these queries to edit the set, etc. This is achievable with a rich underlying storage platform, just not today.

Now and beyond:

At this point, we have perhaps the best shot at this than ever before – we have robust file system support – perhaps the most optimal implementation one could hope to build. We have gone through a number of iterations on our data model so have higher confidence that we are closer to the eventual answer ;-). A great platform needs a great API and we have that with our alignment with ADO.Net vnext, we have solutions to hard questions around business logic, cross-store versioning, security, a strong synchronization story, the list goes on and on. There is a lot more to solve and gell, but this for me is by far closest to hanging together.

I am also a huge believer in learning from shipping and real world exposure. So, right now, it’s on to shipping this baby and getting it in your hands. As with any platform, the real proof lies in the universe of applications and how they build on that platform. Non-Microsoft applications, btw 😉

The dream continues…

Author: Sanjay Anand

Comments (13)

  1. lynn eriksen says:

    When can we expect a public beta to work with?

  2. David says:

    Wow, I would love to learn more about these past projects. Maybe, once WinFS shipped you or someone else can get clearance to tell that full story?

    Also: Yes, PLEASE ship something!!! Especially CTPs. I feel WinFS is the most dramatic of the original Longhorn pillars, by far. You need feedback on what you are doing as early as possible!

  3. Richard Rothery says:

    M. Sanjay Anaud,

    Hello I am an avid develloper mostly scripting for the web. I have been following the progress of WinFS for the past year or so and am as excited as you and the WinFS team.
    The visions you have are mine as well but more in terms of applying WinFS capabilities to WebSites. Evidently, even more common than Microsoft Os’s, website/pages, are on everybody’s displays today and this is our true “common” link. NT/XP is already in most of our common users and these users often visit websites. Thus I am working on implementing WinFS structure to my websites. Now I have not been able to track down much information as most APIs are of the compiled type and thus not practical for fast client side execution.
    Jumping to the future visions, imagine a website with let’s say a WinFS type search that for our sake just comprises a search of one domain. You type in for instance, Photo, bday, mydaughter, school, posters and you have all of those presented in a WinFS search type format based upon storage type and not limitted by ADO filing system objects etc. Now this becomes possible when MS OSs ship with a STORAGE filing system integrated which knows how to account for all the inforamtion already stored on each machine. You can see where I am going with all of this…..
    So when and where can I get a hold of someone/something that can integrate a WinFS store to a scripted webpage. ADO just doesn’t do enough for me and how about you.

    Great article btw :<]
    RicharD Rothery

  4. Eric says:


    There was a PDC talk on how ADO.Net is evolving to incorporate some the WinFS aspects. If you haven’t seen it already, it’s worth reading.

  5. G. T. says:

    Having all these things is nice, but isn’t all that delaying the file store release?

    At one point, one day, Windows and Linux will be perfect, since they will just do everything there is to do, and there will be nothing they can’t do.

    But until this day comes, all what I want it is NTFS fixed, I needed to put 100,000+ files in a single folder without affecting the Windows performance.

    NTFS is slower with every newly added file, and the Windows Explorer is just not designed to handle tons of files in a single folder, whoever wrote that application tested it with 10 files only (explorer.exe)

    As a beginning, just give me the chance to drop a folder with a million files instantly, I can drop one million records from a database in just no time (databases have that since ever), but it may take windows days to drop such a folder from the file system.

    And then take all the time of the world to develop whatever you are developing right now, but first, please fix the existing NTFS, search, versioning, and speed; I guess the first one is coming, the versioning is just a hack to the file system using shadow copy, and speed ??? who knows

  6. Richard Rothery says:


    Thanks for the heads up and yes I saw all the PDC videos. Actually after having seen Bill Gates at Mix06 I now understand that server-side developpment is to be accomplished by .NET technolgies and thus I better get hot into VB.NET and VC#. Atlas or WinFX is the predecessor to FS in my opinion and in the next 5 years or so the integration of websites with the OS will fuse closer and closer together. Thus we all need to get busy embracing the .NET platform and move into more robust methods of programming.
    I like the idea of the internet becoming a type of OS itslef. The future holds great things instore for the IT industry and certain at the development level.

    Best Regards,
    Richard Rothery

  7. Sahil Malik says:

    Uhhh hey .. no blog posts for a full month?

    This could mean, either the next community drop is right around the corner, or you guys gave up on the idea of WinFS again.

    C’mon man – break the suspense.

  8. nabegrave says:

    A full month? Maybe you mean a full 3 months 😉 If that is not indication of something big *cough*-beta2-*cough*  comming (I hope for WinHEC) then I don’t know what can be…

  9. Manish says:

    Nice Nehru ref.

  10. Как Microsoft видит развитие систем доступа к данным и чему Microsoft научилась