The Origin of LINQ to SQL


LINQ to SQL, possibly Microsoft’s first ORM to actually ship in ten years of trying, was never even supposed to exist. 


It started out as a humble Visual Studio project on my desktop machine way back in the fall of 2003, long before anyone heard about it, long before anyone even guessed what would come next, except for the readers of this blog, of course, since I used to post often with long obtuse and sometimes psychedelic meanderings that with the proper one-time pad to decrypt it you might have possibly guessed what was on my mind.  I needed that project to prepare for what was coming.  I needed expression trees and a basic SQL translator to get ready for the day when I would get my hands on the C# codebase and turn it into language that could actually do something interesting for a change; a language that would know how to query.  [Cue the FoxPro fans] 


Luckily, it didn’t take me long to get the basics up and running.  You see, it wasn’t the first time I’d slapped together an ORM or modified a language to add query capabilities; having already designed ObjectSpaces and parts of C-Omega so I was certainly up to the task.  Fortunately, it gets a lot easier the ‘next’ time you design and build something, especially if it was you that did the designing before and you have the opportunity to start over fresh.  I knew all the parts that I needed and which of those needed improvement from the last time around.  So you might say that LINQ to SQL started out as ObjectSpaces vNext, or vNextNext, since I had actually built three different working prototypes of ObjectSpaces before settling on the one we went with, and then another simpler one underlying C-Omega. 


Yet, I was not really setting out to create a whole full-blown ORM system.  I just needed something to kick the tires of LINQ with, a ‘mock’ LINQ provider if you will.  We all assumed at the time that eventually ObjectSpaces and even WinFS would adopt the patterns that we were devising.  So I could simply have just started with ObjectSpaces, and to tell you the truth I wanted to, but I also knew that LINQ would need something a little different.  You see, ObjectSpaces was originally targeted to come out long before even Whidbey; its original target was Everett (.Net 1.1).   That might not sound too crazy if you realize that Luca and I started the project a few years before even .Net 1.0 shipped. (Back then my manager asked me point blank what I thought came after ObjectSpaces; what was the next big thing? It only took me a moment to realize that integration into a programming language was it, getting the queries out of the strings, getting compile time type-checking back and intellisense and all that. Not that I’m all that brilliant or foresighted, we had been already working on fleshing out ideas for new database programming language so it was pretty much on my mind before I walked into his office.)  The big drawbacks of ObjectSpaces were its limited query capabilities (predicates only), its focus on queries in strings and the lack of strong-typing through generics.  Even with ObjectSpaces delayed and retargeted around Whidbey, they were already too far gone in their design to start thinking about drastic changes, and besides, whatever I was doing to the language wouldn’t affect them until long after they shipped; they’d get around to this ‘LINQ’ thing in the next version. 


So what I needed to feed LINQ was a language-integrated ObjectSpaces redesigned around generics and IEnumerables that processed relationally complete queries foremost via strongly-typed query trees generated by the compiler.  That’s what I’d hoped ObjectSpaces v2 would turn into eventually, but I needed something now to design and build LINQ against.


Why didn’t I start with WinFS?  After all, it was all the rage inside the SQL Server org at the time.  Unfortunately, it was the same story as with ObjectSpaces.  They were shipping before us.  We weren’t on their radar.  Their hubris was bigger than ours.  Not to mention my belief that WinFS was the biggest fiasco I’d ever bore witness to, but that’s another story. 


Yet, part of that story was the impetus to turn LINQ to SQL into an actual product.


The WinFS client API even started out as a complete copy of the ObjectSpaces codebase and had all the same limitations. It just had more political clout as it was being lead by a figure at a higher-point in the corporate org chart, and so it was positioned as part of a juggernaut that was making a massive internal land grab.  We on the outside used to refer to WinFS as the black hole, forever growing, sucking up all available resources, letting nothing escape and in the end producing nothing except possibly a gateway to an alternate reality.  Many of our friends and co-workers had already been sucked in, and the weekly reports and horror stories were not for the weak-of-heart. It eventually sucked up ObjectSpaces too and in the process killing it off so that in WinFS v2 it could all be ‘aligned’. 


At that point, those of us designing LINQ got a bit worried.  There were not too many in the developer division that believed in the mission of WinFS.  As a developer tool for the masses, something simple that targeted the lower end was paramount.  ObjectSpaces had been it, and now it was gone.  There was still some glimmer of possibility that WinFS v2 might eventually get it right and be useful as a general ORM tool.  But all hope of that was shot when WinFS was pulled out of Vista and its entire existence was put in doubt.  Had they immediately turned around and brought back ObjectSpaces, that might have worked, but in the intervening months ObjectSpaces had slipped past the point of no return for being part of .Net 2.0,  turnover within the SQL org was spiraling out of control, and most of the brain-trust that knew anything about ORM had already fled.


That’s when we realized we had no choice.  If LINQ was to succeed it needed some legs to stand on.  The ‘mock’ ORM I was building was shaping up to be a pretty good contender.  We had co-designed it in tandem with LINQ as part of the C# 3.0 design group and it really was a full-fledged implementation; we just never thought it was actually going to be a shipping product.  It was simply meant to act as a stand-in for products that now no longer existed. So, for the sake of LINQ and the customer in general, we took up the ORM torch officially, announcing our intention internally and starting the political nightmare that became my life for the next three years.


 

Comments (28)

  1. Not many Microsoft employees dare to publicly refer to WinFS as a black hole. Matt Warren does in this

  2. rogerj says:

    I’ll never forget Luca Bolognese’s tearful announcement at TechEd 2004 that ObjectSpaces had acquired a dependency on WinFS and thus was postponed to the Longhorn/Orcas timeframe. WinFS and the Microsoft Business Framework/Project Green were two stakes in ObjectSpace’s heart.

    I refered to WinFS as a "black hole" in a forthcoming (July) feature story for "Visual Studio Magazine" about LINQ to SQL: "Redmond’s first attempt to deliver an O/RM platform appeared briefly in Whidbey (VS 2005 Beta 1) as ObjectSpaces and later vanished into the black hole generated by the demise of WinFS. Judging by the ruckus that Whidby participants raised when ObjectSpaces disappeared, data-oriented .NET developers definitely are O/RM enthusiasts."

    See: http://oakleafblog.blogspot.com/2007/05/matt-warren-origin-of-linq-to-sql.html

    –rj

  3. public says:

    You got to elaborate  on the political nightmare in future posts!

  4. Matt Warren talks about what happened to ObjectSpaces, WinFS and why Microsoft is going with LINQ to

  5. rbirkby says:

    I love: LINQ expression trees, LINQ generator-like behavior

    I hate: Typing. IEnumerable<IEnumerable<IEnumerable<AnonymousType>>> just becomes too difficult to fathom. Bring back COmega * and + datatype identifiers.

  6. davidacoder says:

    I bet he referred to "preventing integration of LINQ to SQL into ADO Entities" by "political nightmare". If that was what he meant, congratulations for fighting this war and standing firm!!! Entities seems to have the same great project management skills that WinFS had and it is great to see the people that are commited to LINQ prevent a disaster this time around.

  7. You’ve been kicked (a good thing) – Trackback from DotNetKicks.com

  8. Lazy Coder says:

    "It’s an undeniable fact that Microsoft has fumbled the ball repeatedly when it comes to delivering an…

  9. mattwar says:

    Maybe ‘politcal nightmare’ was a bit over the top. I would have actually preferred a better integration story between the two products.

  10. Anders Borum says:

    Hello Matt,

    Thank you for all the work you and your team mates put in the the APIs. LINQ (in general) and LINQ to SQL is forever going to change the way we think and write code.

    I’ve recently held a 3½ hour internal session amongst developers in our company and the everybody agreed that LINQ is the best thing since sliced bread.

    I also followed the development of ObjectSpaces and WinFS. Even back then, it was quite easy to see the black hole grow in strengh – sucking light and life away. Thanks for standing up to the internal political cauldrun and truly trying to make a big difference for developers out there.

    Feel free to provide part 2 of the story btw. :-)

  11. Delphi Fan says:

    I bet it is because of Anders’ name rather than bravery of smart (but powerless) developer like Matt.

    After using Turbo Pascal and Delphi and C#, what can I say. C’mon, the guy knows more about developers’ needs than almost anyone else who is active today. I bet WinFS suffered because they didn’t have anyone like him. A handful of people can build big systems but even fewer can understand what developers want before launching a big project.

  12. Anders Borum says:

    Having met Anders in person I can say that he’s definately a guy that cares about developers. I’m sure the developers also want to improve life for all of us there – Anders may take a lot of the press, but I would seriously credit the entire team for LINQ.

    It really doesn’t matter who was responsible for pushing LINQ and related technologies such as LINQ to SQL – the techonologies just appeared at the right time in the right form.

    Still looking forward to part 2 of the story btw. :-)

  13. Matt has a very good post on the history of object relational framework inside Microsoft. He and I started

  14. Segnalo un post di Luca Bolognese, nostro connazionale e Lead PM di LINQ, che ripercorre gli ultimi 6

  15. anon says:

    The Business Portal SDK (a precursor to MBF) shipped an object-relational mapping framework back in 2003 as part of MBS’s Business Portal. It is even documented on MSDN.

  16. Some quick links about LINQ: Articles about extension methods by the Visual Basic team Third-party LINQ

  17. Some quick links about LINQ: Articles about extension methods by the Visual Basic team Third-party LINQ

  18. In search of a solution to how LINQ to SQL should be used in an N-tier application architecture with…

  19. Previously: Architecting Linq to SQL, part 9 End of the line This is intended to be the last part in

  20. Previously: Architecting Linq to SQL, part 9 End of the line This is intended to be the last part in

  21. Previously: Architecting Linq to SQL, part 9 End of the line This is intended to be the last part in

  22. While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

  23. While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

  24. Tracy.Bai says:

    Tobehonest,IhavetosaythatmynextprojectwilluseNHibernateforitspersistencetechnology…

  25. Articles says:

    Microsoft Kills LINQ to SQL? For the last couple of months, I&#39;ve been hearing all kinds of complains

  26. Articles says:

    For the last couple of months, I&#39;ve been hearing all kinds of complains, cries, nervous breakdowns