Transparency in the design process


Today marks the first day of engineering for the ADO.NET Entity Framework V2.0. V2 of the product is a combination of a continued investment in our greater data platform vision as well as a focus on new and expanded ORM scenarios. Some of the things that we are trying to do in this release are as follows:


  • Persistence Ignorance : We are looking at ways to introduce a full POCO solution for state management and interaction with the ObjectContext.

  • N-Tier Support : Today we support Data Contract serialization of entities or exposing entities via Astoria, in V2 we would like to expand to a DataSet like experience where one can remote graphs and changes to the graphs across the wire using standard WCF services.

  • Code-First : We want to enable a convention based, code only experience with EF wherein one may start with classes and opt-in to database generation and deployment. We expect that we would provide attributes and external mapping capabilities for people who wanted something beyond the convention based mapping.

  • TDD Scenarios: With the introduction of POCO classes some of the TDD scenarios get a lot easier, and we are looking at incorporating some other asks to better fill out the scenario, such as making our ObjectQuery<T> and other generated members of our context and classes virtual.

  • FK’s : Today we support bi-directional relationships, and we are looking at introducing model concepts to facilitate the definition of FK like experiences in the model or in one’s POCO classes.

  • Lazy Loading: Today we support explicit lazy loading (calling .Load), and we are looking at various options around LoadOptions as well as outright implicit lazy loading.

  • Query Tree Re-Writing: This allows framework develolpers to contextualy, vertically and horizontally filter query results.


We have been looking at various options to ensure that the design of Entity Framework V2 truly reflects the requirements of the day to day challenges that our developer community faces when building real-world applications and services. We would like to start by being as transparent as possible in the design process. The approach we take will be similar to what you have seen on Astoria Team blog (ADO.NET Data Services), but I wanted to briefly describe how we’ll go about it so you know exactly what you are looking at when reading one of our design-related posts.

How did we get here? Version 1 of the ADO.NET Entity Framework is set to ship as a part of Visual Studio 2008 SP1 and the .NET Framework 3.5 SP1. This release has been the result of a number of years of effort. During V1 the team was focused on implementing an initial set of scenarios as well as the groundwork necessary to continue to build out the Entity Framework in future versions.

Where are we now? Today marks the first day of engineering for ADO.NET Entity Framework V2.0. What this means in practice is that we have a team with developers, testers, program managers, etc., that we have regular design meetings, and that we have a timeline (or more accurately, that we’re required to have a timeline but of course we’re still working on it :).

Transparency in the design? Over the years Microsoft has been opening up the engineering processes incrementally. Long ago there were only betas, and that was the only chance to see and give feedback about a product before it shipped. Then we started to do Community Tech Previews (CTPs). CTPs enabled us to provide bits more often and gather feedback frequently. The goal with increasing the transparency of design is to take this one step further: we would like to enable folks that are interested in Entity Framework version 2 to follow the design topics as we discuss them, and have the opportunity to provide feedback right during the time where we are actively discussing a certain aspect and before we have made a final decision.

What exactly would we make visible? In short, our design process. To be more concrete, I’m not talking about some fancy set of specifications. What I mean is that as we go through the detailed design of the various aspects of Entity Framework V2, we would post to this blog a) the meeting notes from our design meetings (the team has a design meeting twice a week, plus a lot of impromptu hallway chats), and b) deeper write-ups of specific design challenges where we’d like folks to understand how we’re approaching a problem and provide a channel for deep, detailed feedback.

How transparent is transparent? We want to be completely clear about the scope of the information we are sharing. One of the things we need to learn both from the Microsoft side and from the community side is whether the model works within a practical set of restrictions. We will post as much of our discussions as it is practically possible. However, we have to make sure we don’t compromise the interests of Microsoft as a company. There are certain things that can range from ideas to specific implementation details that we could consider trade secrets, high-value Microsoft intellectual property or something along those lines. It *will* happen that in some cases we will not discuss a topic publicly, either for a certain term (e.g. until a proper IP protection mechanism is in place) or until we ship or ever. This is nothing new, but I haven’t seen folks from large companies discuss this explicitly before, so we wanted to make sure it is clear here.

About your feedback: We would love to hear your thoughts, be it comments, suggestions, ideas or anything else. However, in the end we are designing a commercial Microsoft product. So we’ll happily take your feedback but you need to understand that by providing us feedback in any form you are agreeing that we may use it to develop our product, that others may use it in connection with the product and that you will not be compensated for any of these things. We may incorporate ideas or make changes based on the comments you make, or we may make changes to the product that are indirectly influenced by discussions that we have with you and other folks in the community. Again, this is nothing new, but instead of having some fancy statement written in legal lingo we wanted to be upfront about this here in this first post on the topic. Of course our legal folks looked at this, and they were cool enough to understand that the informal nature of the process is what makes it work, and they let us get away with this statement in which I think we clearly delineate what will happen with whatever feedback you send our way.

We will start posting design notes and challenges soon and tweak the process as we go.


Tim Mallalieu
Program Manager, Entity Framework

Comments (58)

  1. Today marks the first day of engineering for the ADO.NET Entity Framework V2.0, and the team has been

  2. Serioga says:

    Great news!

    Will there be more bits for early download for MVPs?

  3. Entity Framework 2.0 development starts today

  4. i think this and this is good news: MS takes a second try to make the ADO.NET Entity Framework a really

  5. Martin Laufer says:

    Good news!

    What do You mean with "FK like experiences in the model"?

    What about payload attributes in relationships?

    What about n-ary relationships (n <> 2)?

    What about multi valued complex types?

    What about static class members?

    These are the things I would like to see in EF V2. No competitor (AFAIK) can provide such and realization of the former will make EF the "best in class".

    What about the conceptional asymmetry between fetching (LINQ-like) and storing (procedural)?

    What about the fact, that neither the class model nor the database schema can contain as much information as the conceptual model. So deriving the classes from the database (the only thing one has is a relation! hence the name) can’t succede in general (what are entities, what are relationships, what are complex types, what are enums, where to get the behaviour). And deriving the db schema from the classes (where to store the static members, how to map inheritance) can’t succede either without additional information.

    The missing info is stored in the conceptual model (CM)! The only way to infer either classes and db schema is deriving both from the CM! Deriving the artifacts is a PROJECTION!

    So please leave the straight (not to say trivial models, with convention over configuration) to "LINQ to SQL" and please provide a mature designer for the more complex scenarios, where one needs to store complex mapping which can’t be derived from either artifact.

    For updating the CM out of changed artifacts, please consider instrumenting the generated assemblies with the necessary info out of the CM and find a way to incorporate this info into the db (independent of vendor!). This may reduce the effect of the projection but leads to exposition of conceptional info to end users.

    Thanks for Your patience.

    Kind regards

    Martin

  6. Jorriss says:

    Of course this post leads to the question…When is EF v1 going to be released?

  7. Here is a raw cut and paste for our POCO 1-Pager. We are currently working through the design and have

  8. He comentado muchas veces que la velocidad de crucero que ha cogido Microsoft es imposible de seguir

  9. Nullable says:

    I realize that this has been said a few times already, but this is a great step in the right direction for the Entity Framework and Microsoft in general.

    I love the Astoria team blog, and now I have another favorite new blog to keep up with and hopefuly contribute to.

    P.S. -> Lazy Loading via configuration FTW!

  10. BlogCoward says:

    ADO.NET Entity Framework Vote of No Confidence

  11. unbornchikken says:

    Great news, great idea! Thank you!

    I think the most important thing is to make the designer features equal to engine features. I wont use stuff in my real world application if I have to write a tons of imperspicuous XML code to have these.

    A question: will EF v2 be part of .NET 4.0 or there will be an other .NET 3.5 SP release?

    Regards,

    Gabor Mezo

  12. mallalieut@hotmail.com says:

    General Responses:

    >> Early Bits

    Yes we will have early bits, outside of the CTP Rhythm. We are still working on the logistics. We will likely make these more broadly available via something like code gallery than restricting just to MVP’s. We need to close on the details before makign any full commitment.

    >> FK experiences

    We are working on being able to reason about 1-way references instead of bi-directional references. We will drop design notes on this once we have something people can throw darts at

    >> More relationships (n-ary, rel’ns with payload…)

    We have been looking at these, they were part of the original data model but cut due to schedule and priority. In the fullness of time we will add these. I am not sure that they will land in V2 as a lot of the focus will be on the types of things outlined in the post.

    >> assymetry between fetching and storing….

    Not sure what this meant.

    >> designer

    Yes we need to work on the designer. Yes, the transformation from the classes to the model and from the model to the store are projections. That is a key foundation upon which many of our services will be built in the future.

  13. lowendahl says:

    I would love to be able to apply cross-cutting concerns like validation and behavior to the objects returned from a query. Either by letting me hook the materialization and return objects configured by spring.net or the enterprise library or any other solution that would allow me to choose strategies for those cross-cutting concerns.

    I would also like a template based experience for the designer, where one could create code templates used to generate the classes from the c-model. Something similar to what Roger Alsing is doing for NPersist: http://rogeralsing.com/2008/06/19/caramel-alpha-source-code-is-public/

    That would be awesome :)

  14. I had intended to be happy simply being a signatory of ADO .NET Entity Framework Vote of No Confidence

  15. Peter Ritchie says:

    This is fabulous news.  Thanks!

  16. I made the mistake recently of reading some of the responses to the &#39;Entity Framework Vote of No

  17. jeffders says:

    >> Cross-cutting concerns like validation

    Adding more extensibility points to the object layer so that developers have more extensibility hooks is definitely an area we are looking into for v2.

    >> Designer Templates

    This is also something we are working towards. There are just too many code generation options to support without templates.

    Jeff Derstadt

    Developer

    Entity Framework Team

  18. If you’ve been paying attention to what the Entity Framework team has been saying. You will remember

  19. .Net -i arendajate kommuuni poolt on postitatud väga asjalik "Vote of no confidence" artikkel ADO.NET

  20. No início do ano eu postei um artigo chamado 2008 – O ano para alavancar sua carreira , que foi muito

  21. While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

  22. While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

  23. rogerj says:

    I see no mention of adding DML features to eSQL in v2. Having to execute INSERT, UPDATE, and DELETE commands out-of-band in the provider’s SQL dialect will discourage use of the EDM in non-OO applications.

    –rj

  24. Thanks for the feedback Roger.  Yes, we are aware of the value of this feature.  Unfortunately, as we look at the list of all the things that we need to do in v2 and relative priorities, it’s already pretty apparent that this will have to wait for v3.  DML support in both eSQL and LINQ is important, but it is a very large set of work.

    – Danny

  25. The ADO.NET Entity Framework is very strategic to Microsoft – but a) it is a V1.0 technology (although

  26. The Entity Framework’s provider model makes it possible for it to work over different database’s. The

  27. V1 of the Entity Framework allows you to use stored procedures in two main ways: Mapping Create, Update

  28. Evolving an API to support new requirements, like POCO, while maintaining backward compatibility is challenging.

  29. In V1 of the Entity Framework it is possible to annotate a schema using attributes declared in another

  30. Well, I’ve tried Ideablade’s DevForce EF, it is already much more advanced than EF v1 and probably more than EF v2 will be. How difficult can it be for big Microsoft to develop a real enterprise solution with code-generator, model designers, best practices and patterns… everything combined. MM.Net from Mere Mortals, CSLA.Net, DevForce from IdeaBlase, … They are all much more advanced than everything MS has to offer nowadays. And please, don’t talk me about EntLib 4. It doesn’t have the designers and felixibility 3rd party vendors has. You want a good example, look as XAF from DevExpress and where they are heading… I’m sorry, I don’t believe in this v2 so fast, it will take months before the ‘final’ version is released and it will be dropped in the next year or 2. Adopt CSLA.Net or XPO or so, make a superb generator, wizard thingy for it with perfect N-tier samples. Then we, developers, would have something to rely on for more than 2 or 3 years.

  31. One of the most painful omissions from the Entity Framework V1 was Model First, which basically means

  32. In this post Colin Meek and Diego Vega delve into some enhancements we are planning that to LINQ to Entities,

  33. In this post Colin Meek and Diego Vega delve into some enhancements we are planning that to LINQ to Entities,

  34. Three key things are missing from the entity framework for me, you can work around then with a bit of coding but for me these are important for a project to be expandable and for the EF to become very flexible.

    First, Model inheritance, you should be able to create models that inherit from other models and can have associations with objects from other models. This would be usefull for applications that use a common base database and build on this structure.

    Second, Xml datatype.

    Third, dynamic columns in tables.

  35. girolo says:

    It would be great if we could known what kind of WCF Service API would be choosen to re-attach a graph of entities in a 3-tiers dettached scenario. So we could already use this API with a custom implementation and switch to EFv2 implementation latter without changing the API.

    For example, is the API will looks like EntityBag<T> API ?

    It will also be great if the API would not be too "Microsoft" oriented but rather technology agnostic so we could use it in an interrop scenario between a JAVA Server/.NET Client OR .NET Server/ JAVA Client

  36. Moon says:

    I’m pretty sure you know very well that ,

  37. If you are reading this, you have probably heard by now about the so called impedance mismatch between

  38. The first version of Entity Framework provides convenient ways to load, manipulate and persist objects

  39. Unfortunately in the current version of the Entity Framework, which ships in .NET 3.5 SP1, we don’t make

  40. good news this blog.

    we waiting v2 early.

    good job.

  41. Today the Entity Framework, and more specifically the Entity Data Model, have a limited notion of Functions.

  42. A while back I wrote a post that introduced the concept of Computed Properties . Since that time we’ve

  43. When VS 2010 ships it will include some significant improvements to our code generation story for the

  44. Eric's Blog says:

    After a year of working with LINQ to SQL, I strongly belivev that LINQ to SQL and Entity Framework (EF)

  45. Background A number of months ago we asked whether Foreign Keys (FKs) in Conceptual and Object models

  46. Background One of the biggest pieces of feedback we received from the N-Tier Improvements for Entity

  47. Courtois says:

    a nice feature will be working with several edmx:

    entity1 map in in file1.edmx can inherit from entity2 map in file2.edmx.

    So you can work with several models !

  48. There are currently two ways to get Entity Framework apps up and running, we call these Database First

  49. Nabil Sedoud says:

    I make a test for this Walkthrough by using code only Context, and i receive this error "underlying connection was closed: The connection was closed unexpectedly"

    I think that there is an incompatobility when use T4 SelftTracking with CodeOnly approche.

    unfortunately for us, it’s an other limitation of EF

  50. Daniel Simmons says:

    Could you please share the code that you tried either here or by sending email to dsimmons@microsoft.com?

    These two features are not intended to be incompatible, and if there is some issue we will certainly fix it before final release.

    – Danny

  51. Nabil Sedoud says:

    Think you Daniel,

    I made some diagnostics, and I suspect that the problem lies in the client side when reading the response message. "HandleReturnMessage Void (System.Runtime.Remoting.Messaging.IMessage ….."

    I make the query to the  model directly from a console application and it works correctly.

    The service also works correctly in this case:

           public Post GetPostWithComments(int postID)

           {

               var builder = new ContextBuilder<BloggingStoreContext>();

               builder.RegisterKey((Entry c) => c.ID);

               builder.RegisterKey((Person p) => p.ID);

               builder.RegisterKey((Blog b) => b.ID);

               SqlConnection connection = new SqlConnection(@"Data Source=LABSQLEXPRESS;Initial Catalog=BloggingCode;Integrated Security=SSPI;MultipleActiveResultSets=True;");

               using (var context = builder.Create(connection))

               {

                   if (!context.DatabaseExists())

                       context.CreateDatabase();

                   Post post = context.Entries.OfType<Post>().Include("Comments").SingleOrDefault(e => e.ID == postID);

                   Post copy = new Post(); //Just for test succesfull returned object

                   copy.Title = post.Title;

                   return copy;

               }

           }

    but I receice an error "underlying connection was closed: The connection was closed unexpectedly" when make this test

           public Post GetPostWithComments(int postID)

           {

               var builder = new ContextBuilder<BloggingStoreContext>();

               builder.RegisterKey((Entry c) => c.ID);

               builder.RegisterKey((Person p) => p.ID);

               builder.RegisterKey((Blog b) => b.ID);

               SqlConnection connection = new SqlConnection(@"Data Source=LABSQLEXPRESS;Initial Catalog=BloggingCode;Integrated Security=SSPI;MultipleActiveResultSets=True;");

               using (var context = builder.Create(connection))

               {

                   if (!context.DatabaseExists())

                       context.CreateDatabase();

                  return context.Entries.OfType<Post>().Include("Comments").SingleOrDefault(e => e.ID == postID);

               }

           }

    I will send you the refactored files.

    Nabil

  52. Kevin says:

    It would be really good if xml entity mapping (as an option) for Code-Only is allowed.

  53. @Kevin,

    Can you elaborate a bit?  What do you mean by xml entity mapping?  Are you saying you want to pass some xml to a code-only method which describes the mapping, or are you saying you want to map from properties in the database to an xml property on your entity class, or are you saying you want to map from an xml property in the database to more than one regular property on your entity class, or something else?

    Thanks,

    Danny

  54. Thehim says:

    Hi Tim,

    Sorry, if this is not the right forum to ask this question, but still:

    Several papers and documentation on Entity Framework are describing how to generate the ‘first model’ or create the ‘first mapping’. I miss the discussion of how to handle an ‘application change’ or ‘the application change iteration’. Ongoing applications with long life cycles, several iterations, several customers, several dbms’s often include consecutive changes to the database which require changes to the entity model and changes to the mapping. Where are the ‘automatic’ or the ‘reversing?’ detecting of such changes addressed? Since it is not addressed, is it because EVERY application change is surrounded with a NEW database including data transfer from the old one?

    Thanks.

    Thehim.

  55. Diego Vega says:

    Hello Thehim,

    In general, using EF will not really force you to create a new database or to transfer the data for every change that you make. It is even possible in some cases to alter the database schema (i.e. using DDL commands or the tools provided by the database) and then adjust the storage model definition and the mapping specification without affecting the existing conceptual model, and vice-versa: make changes to the conceptual model and your classes that do not require changes in the database schema, just in the mapping.  Of course, some types of changes (i.e. add a new non-nullable column) will have to be propagated to the other side.

    Currently, the process of propagating the changes in most cases is not automatic, though. How you keep the database, the model and your entity types in sync depends heavily on how you started your application and how you want to make the changes:

    – Changes in the database can be propagated to the model:

    o Semi-automatically to the model using the “Updated model from the database” option in the EDM Designer (but this may overwrite your customization), or

    o Manually using the designer to edit the storage  and the mapping information

    o Manually editing the XML artifacts directly

    – Changes in the model can be propagated to the database:

    o Manually issuing DDL commands or using the tools provided by the database

    o Optionally, you can regenerate the database completely using Model-First tooling (but this will overwrite the exiting database)

    – Changes in the model can be propagated to the entity types:

    o Automatically if you are using any of the code-generation options

    o Manually if you have handwritten POCO classes

    – If you are using Code-First in the Feature CTP, you can propagate changes to the database:

    o Manually issuing DDL commands or using the tools provided by the database

    o Optionally, you can regenerate the database completely using ObjectContext.CreateDatabase (but this will overwrite the existing database)

    We are currently considering alternative designs for generating automatic migration scripts for some of these scenarios in future versions of EF. It would be great to hear more about how you would use this feature and what you would expect to be able to do.

    Thanks,

    Diego

  56. Thehim says:

    Thanks Diego,

    When delving more into the framework, I noticed the ‘Update from database’ feature and as this feature matures, I’m sure the basic ‘maintenance’ scenarios are covered.

    You’re asking for how to use a possible automatic migrations script feature in the future. All I can say is that if Entity Framework are trying also to cover the database domain and take the responsiblity of so called modify scripts, you’re in for quite a task. I’m used to a state of the art design tool that handles ‘change scenarios’ VERY seriously and indeed does so by still keeping NO knowledge whatsoever between code layer and database layer. When you open this ‘knowledge Pandoras Box’, you must NOT forget all constraint types, such as primary key, foreign key, index and ALL other references that a, say, column could have in database. When this column change, a lot of references/declarations must first be ‘shut down’ before the column can change and declarations must be re-declared again after. Not to mention renaming of a table to a temporary one and data transfer from this to a new version of the same table. There are so many sophisticated tasks that ‘an application change’ will demand. To conclude (and repeat myself); Starting up something for the first time is the easy part, it’s maintaining hereafter that has the real load. Design with this in mind; ‘How it looks today is quite different from how it looks in a year’ or ‘Always leave things ready for next iteration’. Think in recursive scenarios. Starting up is only the first iteration.

    And thanks again. Damn exiting subject.

  57. Thehim says:

    …sorry, should have been exciting, not exiting, huh?

  58. Mark Gordon says:

    We are up to CTP 5 and I still don't see much improvement from  early releases it is still clunky to work with.

    http://dotbloat.blogspot.com