[Discussion] Future Direction of WCF Data Services


WCF Data Services

Microsoft initially released WCF Data Services as an easy way to implement an OData service over queryable data sources. This made it very easy to expose a model backed by the Entity Framework as an OData Service, and included a data provider model for plugging in other types of queryable data sources. WCF Data Services abstracted away all of the details of HTTP, OData, and the various payload formats, and the data provider was responsible for executing the underlying queries and returning the raw results to be serialized in the appropriate format.

This framework worked well for exposing a core set of OData features over data sources that supported fairly rich query functionality, but had two major limitations. First, it required that the underlying data source support fairly rich query semantics. This made it difficult to implement over diverse sources of data such as data returned from a function invocation or collected from other services. Even for queryable sources, computing the best query to perform on the underlying store was often difficult because the query passed to the data provider lacked a lot of the context of the initial request. Second, because WCF Data Services was somewhat of a monolithic “black box”, services had no way to expose new features until supported for each feature was added to the WCF Data Services stack. Adding support to WCF Data Services often in turn required defining new interfaces that each data provider would then have to implement before the functionality was available to the service.

Web API OData Support

In order to give developers more control, we decided to invest in a better factored approach to building OData services using the Web API OData libraries. The Web API OData libraries, built on the popular model-view-controller (MVC) pattern, give the service developer much more control over how each request is processed by allowing the developer to implement custom controllers for each “route”.  The methods exposed by these controllers have all the necessary context to satisfy the request, regardless of where and how the data is obtained. And, the better factoring of Web API makes it easier to implement new functionality without waiting for one or more external components to add support the feature.

Giving the developer the ability to implement custom controllers does mean added code for the developers. Most of this code, though, is fairly boilerplate, and things like scaffolding in Web API controllers make it easy to generate default code for handling these requests.

Supporting OData Version 4.0

In implementing support for the new OData version 4.0 Standard, we started by building OData V4 complaint versions of the low-level libraries use by both WCF Data Services and Web API for parsing URLs and serializing/deserializing payloads,. At the same time we released a common OData Client for consuming OData version 4.0 compliant services.

In order to give developers the most flexibility to build on a core set of features, we next prioritized adding core OData version 4.0 support to Web API.

We didn’t make this decision lightly. Microsoft has a lot of services internally that are implemented using WCF Data Services. We are investing time in helping internal teams move to using Web API to expose their OData services. Through that process we are finding that the amount of work necessary varies depending on the level of investment the team has in WCF Data Services; teams that built on top of entity framework have the least amount of work to do, and teams that have already implemented custom data providers have incremental work to expose their queries through controller methods. So far, though, the migrations have gone well with the teams enjoying the benefits of more control in how requests are processed, and in the features they expose through their service.

So what becomes of WCF Data Services?

We have actually done the bulk of the work to implement the existing WCF Data Services functionality according to the version 4.0 protocol, but because of the monolithic nature of WCF Data Services it still wouldn’t be possible to take advantage of the many compelling new features of OData version 4.0. Most of the new features would require additional work across WCF Data Services, as well as each individual data provider, before they could be used by a service built with WCF Data Services.

Rather than be a bottleneck to the features that WCF Data Service developers want to expose in their services, we are looking at providing an OData 4.0 compatible version of WCF Data Services as Open Source. This option allows developers to extend and adapt the code as appropriate for their individual use, and leverage the community to evolve the functionality much faster than we could ourselves.

We would love feedback on this approach. How far should we take the WCF Data Services code before releasing it as Open Source? Is it better to release it sooner and have the Open Source community finish the work to make the basic functionality fully compliant to the OData 4.0 protocol specification, or should we wait until we’ve had a chance to clean up the code and make it more compliant with the new standard?

Thoughts and feedback genuinely appreciated.

Thanks,

Michael Pizzo
Principal Architect, Microsoft OData Team


Comments (71)

  1. Roopesh Shenoy says:

    Ok, I am confused – so are you suggesting that you will open source a version of WCF with OData v4.0 compatibility, but will not "officially support" it? Or are you suggesting that this is just a way to speed up overall compatability to the entire OData v4.0 spec, once some basic work is done by MS?

    Based on earlier blog posts from the OData team, I have reported on InfoQ that MS is going to stop investing in making WCF OData compliant. If that is mistaken, please let me know so I can post a correction. Also let me know whether this has been the plan all along and whether I somehow misunderstood it, or was it based on community feedback that you are revising your stand.

    Thanks. Be Awesome.

  2. Vaccano says:

    Thank you for giving real reasons for why you are making this change.

    I have two comments/questions:

    1. Migration Walk-throughs

    >>We are investing time in helping internal teams move to using Web API to expose their OData services.

    It would go a long way toward smoothing over the frustrations with this move if you could post a simple AND a very detailed "walk through" of how to move from a WCF Data Service to a Web API service.  

    Meaning one for a simple EF backed service AND one that has customizations in the resource pipeline, uses a different way to get the data than EF and in general uses some of the more advanced server side features.

    This would help show that you care about the difficulties this change is causing everyone who trusted that WCF DS was a "Keeper" technology.

    2. Road Map

    You really really need a road map of what you plan to do.

    This one is born of the deep frustration of the general "abaondonware" that is plaguing Microsoft as it moves from having enterprise application development tools to having consumer application development tools.

    Can we get some kind of commitment that you are "Sure" about Web API?  I would take a serious blow to my reputation if I try to get us to migrate to Web API only to find out in 2 years that our OData stack is being abandoned again.

  3. João Vitor says:

    Finish sooner and focus in the Web Api OData.

    Will be better for the .NET community if the web api odata get a better and faster improviments since it has more flexibility than the WCF Data Service.

  4. tnelis says:

    Statements like "just scaffold the boilerplate" do not inspire much confidence. Yes, Web API allows for more flexibility, but at the price of requiring application developer to do the work in its place. When it comes to esoteric backends, that flexibility is welcome, but how many developers are using these systems in contrast to those who use the Entity Framework?

    For the extremely common use-case of using EF as a backend, requiring the developer to manage that boilerplate is not an advantage, it's a burden. It goes against every basic code reuse principle. What if the backend changes its API, or what if there is a desire to change any bit of that boilerplate, for any reason? Why should the application developer care to update all of the boilerplate in all of its controllers, and why should it be duplicated many times in all application assemblies to begin with?

    It's not a "better factoring" for everyone.

    I'm not saying Web API cannot eventually ease the pain by supporting EF (and other backends) explicitly (read: directly, itself, not through application plumbing code generated by an IDE), but right now, it doesn't, so to be honest I find the sugar-coating a little difficult to accept. If Data Services fails as an open project, then ASP.NET Web API will most likely evolve a lot to fill that hole over the years. What I find worrying is that it seems so far away from being good enough that I wouldn't even be surprised if an entirely new project appeared a few years down the road, based on ASP.NET Web API and with the promise of "making it easy to implement OData with the Entity Framework for application developers".

    Projects will always naturally optimize for the common cases, whether we want it or not, and it's a good thing. The current mismatch should be fairly easy to see, though I would love to be proven wrong.

    Until that new project appears, or until ASP.NET Web API slowly morphs into that framework, application developers will most likely fill their needs by other means. Most likely by first creating extensible controllers that target EF and other backends directly, then by somehow routing requests for multiple entities to a single generic controller after realizing that every controller in the project contains a single class declaration with an empty body. There will always be frustration: Some entities need to be extended, but others don't, so some requests will have to be routed to the generic controller, but others not. Eventually, the controller abstraction might even be questioned, as developers will soon start to register event handlers to that generic "catch-all" controller.

    I can't pretend to be able to predict what will happen with such accuracy, but is this vision of the future really far off?

    Now, I'm excited about OData 4, and I would love to see Data Services thrive as an open project (particularly in the OWIN environment), but you can't start an open source community by implying that the project will be abandoned from the start.

    MSDN thread: https://tinyurl.com/lz9972q

  5. Bart says:

    First, a question, what will the support of the open source library be? Fully supported or just here's what we used to have, good luck?

    My preference is build it out to support v4.0 and clean up the code and then release it to open source. If there is a large burden on the community right out of the gate, it's less likely to gain traction and support. If it starts out though fully featured, then in my mind, people are more likely to utilize it and build it out where needed.

  6. Jeff says:

    I completely agree with tnelis. This is very sad news indeed. Web API makes me write all of the plumbing code that we've been trying to get away from, whereas WCF Data Services allowed me to VERY quickly stand up a web service to wrap my data.

    What the ASP.NET Web API team did is another unfortunate example of MS teams trying to one-up each other instead of working together. The ASP.NET team specifically has a long history of this behavior.

  7. john says:

    Very disapointing. I also agree with tnelis on this. We need a solution that is just as easy to implement as WCF DS. I tried to migrate to Web Api, but the huge amount of boilerplate just killed the plan. Your job is to help us write software and with this you are making it only harder. Provide a solution or gtfo.

  8. CNRice says:

    I don't think it really matters when you release the source code.  No one is going to care when the owner has stated it's dead.  It's a shame I had put a lot of time and investment into WCF Data Services.  I had a love/hate relationship with it.  While I thought it tried to do too much with trying to do CRUD, which should go to the web API.  It was great for the query, even if the code was so locked down.  I don't think the WCF Data Service server team cared that developers couldn't extend it, everything was pretty much internal, static, private.  Which goes to my hate side.

    I think going Web API is great for CRUD but lousy for querying.  I don't want to have to create a controller for every entity – seem ridicules.  I will be looking more at the web api odata to see where I might be able to change that.  The great thing about the Web API is they do allow a lot of flexibility in their framework which was greatly missing in the WCF Data Service.  But who knows if the WCF Data Service Server team wrote that for Web API I might be out of luck on that as well.

    So for me I will move on to Web API for query but it would have been better for the WCF Data Service to just scale it's goals down to just query and flexibility in the code base.   But since it was trying to be everything it was too bloated and locked down.

  9. AdamCaviness says:

    Why have you just informed us only days ago about this major shift while you were helping your own internal teams migrate?  I suppose I can understand it if you wanted to first ascertain if abandoning WCF Data Services was a viable option BUT why didn't you also consult with us as well, your paying customers (both literally and figuratively).  Mind you that it costs a great deal to keep up our MSDN subscriptions and deal with the server licensing, etc to develop on the MS stack.  It's ain't all free you know.  We are customers.  Not telling us until so recently is lame, epic lame.  Have you looked at the UserVoice answers MS has been giving that lead us to believe you were going to fully implement OData v4? data.uservoice.com/…/72027-wcf-data-services-feature-suggestions Just look down the list at the comments from Mark Stafford.  We felt like we were in good hands.  

    Vaccanoll is correct; you really need to provide us a simple and detailed sample of porting from WCF DS -> Web API.  And lastly, how pray tell are we to know you don't go head over heals over something else in two years and abandon Web API.  We use Microsoft because we are enterprise developers, whether we like it or not, that is what we are.  Not all of us are making twitter clients.

  10. Vaccano says:

    AdamCaviness has a good point.  There are a lot of features with Started and Planned on User Voice.  Is the team going to renege on those statements now?

  11. AdamCaviness says:

    I put my vote in for not releasing it sooner but instead getting it to a place you feel like the community can successfully pick it up and move on without a lot of pain.  In other words, treat us with the same respect you would treat another team "on the inside". This would include getting the EF6 provider production quality first (this has been a stink that you've ignored long enough). On another note, thank you tackling this subject in a dedicated post like you have as it shows you are reading our comments.  I gather however the team has committed 100% to this decision or you wouldn't have helped internal teams transition but I do hope you are still reconsidering. –crickets chirping–  I for one don't want Web API becoming more like WCF Data Services over time due to this decision. Also, I don't know what metrics you used to determine value between the two strategies.  I totally get that Web API has a more technically appealing direction (I also gotcha on the monolithic architecture) and I acknowledge it has more momentum but I would argue that momentum is cheap when it fades so quickly these days.  I think MS should forget trying to be cool and provide rock-solid solutions for longer hauls. That's a competitive advantage that you are throwing away as you head more and more consumer centric.  What other metrics are consulted in making this value comparison?  Certainly downloads and support tickets are not good indicators.  We all know Web API is going to have more "lookers" but the bigger implementations with hundreds of entities like ours seem better suited to WCF Data Services.

  12. Uffe says:

    Another issue with the chosen direction is that we end up being dependent on two different teams to supply the frameworks for us to build our OData services on.

    In the "old days" we had full functional releases everytime where everything fitted together on day one. Now we will have to wait for the Web API team to take new OData releases in before we can use new OData features or bug fixes.

    I have no faith in the Web API team prioritizing this issue very high and I fear we end up waiting a long time on the Web API team to catch up – history shows this to be correct. Currently OData V4 is on RTM version 6.1.0, but if you wan't to implement your service on Web API you'll first of all have to look at their nightly builds (in an alpha1 stage). Those nightly builds are not build on the latest and greatest 6.1.0 OData release and not even on the RTM 6.0.0. They still build on an old 6.0.0 alpha1.

    Web API is about much more than just OData i really think this will end up being a big issue, just the out of band EF provider issues.

    Currently i am seriously considering rolling our own HttpHandler based on ODataLib and EdmLib directly rather than moving to the bloated Web API.

  13. CNRice says:

    The more I look at this the more I think this is a mistake.  The great thing about WCF DataService was it's ability to read EF, besides the fact the EF 6 alpha is a joke and not ready for prime time.   From this link http://www.asp.net/…/working-with-entity-relations  I will need to create a controller for each of my entities 70+ and not only that create a method for each navigation between them and write the linq statement for each of the joins.   WOW great call on making my life and code a lot harder.

    If this is truly the way then some serious thought should be put into make this a lot more streamlined of just reading the EF as we have already defined everything there.  Now I have to write and maintain a bunch of code in Web API because it can't route into the navigations and read and react to the EDM.

    We are writing an ERP package and not some game or small application.  Clearly you guys have lost sight on the line of business developers if you think we should just scaffold this out and maintain all the relationships in Entity Framework and in Web API and keep them in sync as the application continues to grow.

  14. Nedelcho Nedelchev says:

    I also think, that this is a mistake. And i totally agree with previous comments.

    Clearly you guys have lost sight on the line of business developers.

  15. Michal Wozniak says:

    This seems like a complete mistake to me. In October you release an alpha that allows integration with EF6 (blogs.msdn.com/…/using-wcf-data-services-5-6-0-with-entity-framework-6.aspx), you ask the community to test it, you get it tested and you just leave it. You then release data services 5.6.1 with no possibility to acutally use EF 6 with (because the previous alpha has a hard reference to the previous version) and now just leave everything offering to switch to web api with no clear instructions how to do it. I am extremely dissapointed in the way how this "transition" is being proceeded.

    We used OData services as a replacement for Microsoft Sync – another abandoned technology. It is becoming extremely tiresome to go thorugh all that again after less than two years.

  16. CNRice says:

    Also I setup a Web API OData endpoint and it doesn't respond with field size limits like WCF Data Service did.  this will break our application because we would get metadata from the WCF Data Service and generate a EDM Model for client validation.   I understand I could do something like this http://www.strathweb.com/…/extending-your-asp-net-web-api-responses-with-useful-metadata  but it doesn't seem Web API is prime time either for OData.  

    I know MS wants people to adopt OData and now the people that have are being screwed over.  Really not happy with this decision or the thought process that went into it.  Maybe it's time to start looking at RDF from W3C.  I know it doesn't have a .net version support of it, but hey doesn't seem like OData does either.

  17. AdamCaviness says:

    For whatever it is worth I've made a User Voice suggestion 🙂

    data.uservoice.com/…/5705738-don-t-abandon-wcf-data-services

  18. Michael Pizzo - MSFT says:

    First of all, sincere thanks for taking the time to provide thoughtful feedback. I don't actually control the resources, but I will make sure these concerns are heard.

    In the meantime, some comments on specific feedback:

    First of all, it's clear that we could use real-life examples, and that reducing the amount of even boilerplate code for implementing a service would help, especially for models with a large number of entity sets.

    Question; what if we provided a handler that, given an EF ObjectContext, handled requests to the underlying context, much as WCF Data Services does today? You could use this for default behavior and register individual paths for custom handling. I would need to play around with this a little, but it seems like it could work, and provide a good balance between the simplicity of WCF Data Services and the flexibility/extensibility offered by Web API.  

    Several people expressed concern over our commitment to Web API; what happens if a new framework comes along for building services in .NET?

    Honestly, the OData team doesn't have much control over that. The web is constantly evolving, as are development patterns, practices and tools. The ASP.NET team is committed to providing the best development environment for that evolving landscape. Sometimes, like with Web API, that means defining new frameworks that facilitate those emerging patterns and practices.

    The OData team is committed to support OData on the premiere development environments/frameworks for .NET. Web API provided a new development framework for writing RESTful services, and we wanted to make sure that OData played in that space. Along the way we discovered that Web API solved some of the issues developers were encountering with WCF Data Services.

    Concentrating our efforts on Web API has allowed us to deliver core support for OData V4 in nightly builds days after the OASIS standard was ratified, and continue to add underlying support for new features at a more rapid pace.

    Again, thanks for the input. We believe in the power of making data open and accessible on the web with OData, and will take this feedback seriously in helping us understand how to balance and prioritize resources moving forward.

  19. John says:

    Mike,

    Thanks for your reply.  I think you're on the right path with your idea of a handler.  I think a large value of WCF Data Services is that you can simply expose your entity model over HTTP without practically writing any code.  I could be wrong, but I don't think most WCFDS users necessarily care about the underlying stack (WCF vs. WebAPI) as much as they care about the development story around starting with an EF model and using a product/tool that generates the code to expose it.  Whether it's a handler, a T4 template, a wizard, something that gets you from 0 to a working OData service without having to do much is a value add to a code base.

    One more question.  As many folks have pointed out, just curious why the client-side tooling/code generation story hasn't been taken to its logical conclusion of generating code for custom query operations and actions/functions from OData V3?  It seems like this would really make the dev story compelling.  I'm frankly surprised that no one in the community has done this (someone please set me straight if I'm missing something).

    Thanks,

    John

  20. Vaccano says:

    For those who are wanting a bit more “Why” behind the WCF Data Services abandonment, here is some Microsoft trends that explain it.

    Microsoft is in the middle of a “refocusing”.  They are dropping their previous identity of a company that creates tools and APIs to build enterprise apps.  They are putting all their efforts into becoming a company that creates tools and APIs to build consumer apps.  

    This is all because there is A LOT of money in the consumer app market.  And Microsoft has a very very small piece of that pie.  

    To try to capture more of this market, Microsoft has made the following changes:

        1. Redesigned Visual Studio to be more appealing to non-Microsoft stack developers

        2. Created Windows RT

        3. Retired Silverlight  to give resources to Windows RT

        4. Retired WPF to give resources to Windows RT

        5. Retired Windows Mobile and CF.NET to give resources to Windows Phone

        6. Retired WCF Data Services to give resources to Web API

    WCF Data Services is a bit too enterprise oriented.  What it does really well is hook up to a database to provide queryable, but controlled access.  This is something that businesses really need, but is less useful to consumer oriented apps.  

    They need something that can hook to twitter, facebook and other web sources.

    In the end, WCF Data Services is just another casualty in Microsoft’s pursuit of part of the pie that Google and Apple are getting most of right now.

    And if I take off my developer hat, I can see it is a good move for Microsoft.  Because despite the fact that it is all abandon-ware, WPF, Windows Mobile, Silverlight and WCF Data Services are all still the best options out there for creating enterprise software.

    I figure Microsoft has about 3-4 years before they completely lose their enterprise superiority in the tools and API space.  However, they will have lost developer confidence long before that.

    Never again will enterprise developers switch to a new framework because it is the “next generation” like we did with WPF.  That level of trust will be long gone before Microsoft turns back to looking at Enterprise tooling (if they ever do).

  21. CNRice says:

    Mike,

    Thank you for answering our comments.  I think a handler would be great if I don't have to write all the plumbing code.  I've always wanted to handle the edge cases and that's where WCF Data Services didn't work for us.  It was either WCF Data Services handles them all or none.  

    Currently we "hide" WCF Data Service behind a Web API Controller, that was made possible by the IDataServiceHost interface, to make the API seem seamless to the outside world.  It also gave us an insertion point to handle an odata request in the API controller before it hit WCF Data Service, for our edge cases.

    We never really used the query pipeline that was provided because it was always too limiting of what we wanted.  If the handler could give us hooks into the expressiontree it's creating and allowing us to modify them that would be awesome!  Right now we have to implement an IQueryable wrapper around EF to modify them.  Even then we can't do everything we want.

    For example I tried to implement Distinct by adding it via the expressiontree if it was part of the URI query options but Count ask for count on the entire entity and doesn't have the projection in the query so I couldn't get the distinct count of the projection.   This is just one example where WCF Data Services was hard to extend.  If there was a handler it should be thought of how to make it so parts of it are extensible or replaceable.  You guys have the IServiceProvider interface and it would be nice to use it like a Service Locator to be able to replace interfaces with our own, just like Web API has a IDependencyResolver.  You create the public interface and have a default implementation but give us the ability to replace that interface with our own with the IServiceProvider.  That is more what I look for instead of a pipeline.

    I would still need metadata that provides what WCF Data Service metadata provides today.   Please consider that when looking at the handler.  I know I only speak of the query side because that is all we use it for.  We always wanted more control on the CRUD and have been using MVC Web API for that already.  I thought about creating a T4 template that read the EDMX file to create the controllers and methods and the registration but I think an handler object would be better, plus once the EDMX file format changes the t4 template would need to be reviewed each time.

  22. CNRice says:

    I'll also add that if you are going to create a handler it would be nice for it to be open source right away.  I would still expect MS to own it and maintain it but I know I would give time towards making it better in anyway I can.  

    I would have done it with WCF Data Service Server if it was open sourced, but not now because I don't want to spend my limited free time on a project that is already dead from the owners view point.  I also know as an organization I can't just stay on WCF Data Service Server even if it's open sourced.  I could spend the time to make it work with EF 6 or 7 when it comes out but then OData v5 comes it would be too big of a change for us to undertake.  With no one overseeing the project it would be hard to get pull request and solid code out of it.

  23. tnelis says:

    @Michael Pizzo, Thank you for replying. To answer your question: yes, I also like the idea of a handler.

    In doing this, there's a chance you might experience a little identity crisis however (at least after a while) — would this still be ASP.NET Web API, or ASP.NET Data Services v6, or a new ASP.NET OData project? I think it's of relative importance because at first sight it doesn't seem like an API you simply "play around with" for a while. That is unless you want to risk it being seen as another feature slapped atop another as an afterthought. If you truly aren't underestimating the work, and if the way resources are allocated to that project ensures its long-term support, then I'm all for it. If it looks like a two-year experiment, we'll probably pass.

    By the way this joins what I've been rambling about earlier; the controller abstraction isn't all that appropriate in all use-cases, and it shows in that you immediately thought of dropping to a lower-level (IIS/ASP.NET HTTP handlers). Providing extensibility via custom controllers for specific paths only is a very welcome touch — but also consider the use-cases that interceptors addressed, which are considerably more streamlined for validation, filter and access control logic (among other use-cases).

    In the long run, it's a possibility that the OData team will be reinventing most of Data Services, driven by small feature requests over time. It's not necessarily a bad thing, we all wish we had a second chance at our designs, but it takes time and resources. While it's difficult to envision the future, I think it's worthwhile to have at least a couple of discussions about how the APIs and their underlying systems might evolve. This is to avoid yet another legacy-ridden product, and maybe skip a couple of steps.

    For example, and I know I mentioned this already, ASP.NET seems to be in the process of being slowly modularized and modules target OWIN APIs instead of IIS or internal ASP.NET-specific APIs. If this trend continues, this means that IIS/ASP.NET HTTP handlers might not be the best interface to use in the future (or even today). While this particular example only affects the setup of the components, it's still user-visible, and thus should be considered sooner rather than later (unless supporting both environments is easy enough, of course, but.. famous last words and all of that).

    I also strongly support open-sourcing such a component. Coming back to the resources point: you might want to consider the Outercurve Foundation and its "ASP.NET Gallery". Sounds like a perfect fit.

    To conclude, I think the community would be keen to provide constructive feedback on the component. It'll probably range from "reimplement DataService<T> using ASP.NET Web API components" to "let's reimagine everything" (let's use methods defined on the entity classes themselves to hook into the request!), but somewhere in the middle is a system that actually solves the problems of its users, and I believe you should at least take this whole mess as an opportunity to find it.

  24. Uffe says:

    I too would like to se a HttpHandler based option, but please do NOT tie it to EF as the only option for backend. We have a custom provider implementation with a runtime auto generated model of more than 400 entities. Our back end is not IQueryable, so I would need the same options for doing a custom provider implementation as WCF DS supports.

  25. Eric Berens says:

    "Question; what if we provided a handler that, given an EF ObjectContext, handled requests to the underlying context, much as WCF Data Services does today? You could use this for default behavior and register individual paths for custom handling. I would need to play around with this a little, but it seems like it could work, and provide a good balance between the simplicity of WCF Data Services and the flexibility/extensibility offered by Web API."

    Are we using the word "handler" more generically or are we talking about a HttpHandler? If the direction is moving towards WebAPI to provide implementation then I don't think a HttpHandler would be the best approach. If we are providing a drop in replacement for WCF DS in the WebAPI stack, then why wouldn't it be a combination of the ApiController and ODataController? I'm not sure what prevents anyone, if not Microsoft leading the effort, in creating a DataServiceController<TContext> where TContext : DbContext or something similar. The majority of the boilerplate code people are referring too would be abstracted away in this base class.

  26. CNRice says:

    I was assuming "handler" was a generic term and not a HttpHandler.  I was just looking for something to parse the request and generate the expression trees that are needed plus metadata.  But that does seem to be the core of WCF Data Services.  I also agree if this is just a short term bridge I wouldn't be happy about that unless it's built in a way that allows me to extend and replace parts like ASP.NET MVC or event WPF Prism application framework.

  27. Uffe says:

    Sorry, I think it was me accidentaly typing the word HttpHandler. Don't get to hung up on that – it could be any handler type of approach.

  28. Michael Pizzo - MSFT says:

    Thanks for the continued feedback; we really *are* listening, and so are others (see visualstudiomagazine.com/…/wcf-data-services-and-odata.aspx.)

    Yes; I was using "handler" in the generic sense, not necessarily HttpHandler (I was thinking something higher in the stack, building off of the untyped support we added to Web API/OData). Thanks for clarifying.

    Still very interested in hearing (both sides of) how much we should invest on bringing the WCF Data Service scenarios (and investments) into Web API, where we can advance more quickly, or continue to invest in moving both WCF Data Services and Web API forward independently. If we can address the issues with amount of code required (especially for large models) in Web API, and help people move their WCF investments to Web API, are we better off concentrating resources on one stack or splitting our resources between two?

  29. CNRice says:

    I for one would recommend investing in Web API, but like I pointed out above we already use Web API for CRUD and only use DS for queries so I might be bias.  If you can address the large amount of code I would need to write to support EF and it was on Web API to me that would be a win/win.

    Right now I really need EF 6 support and full metadata like I had with EF 5 – mainly field size and complex type support.   Is this going to be resolved in the release of DS?  Or would you shelf that release to work on this "handler"?  

  30. tnelis says:

    For those who didn't read the article: it's confirmed that WCF Data Services for OData 3 (current version) will be officially supported for some time. That's good news, it means whatever transition we will have to make won't have to be rushed.

    @Michael Pizzo: Definitely, one stack is better than two. I think our very own main concerns are: (1) How long until ASP.NET Web API gets to where it fills our needs and (2) How many iterations until it's considered stable?

    #1 matters less since we'll evidently be able to linger with Data Services for longer than you initially made it seem like. However, it looks like you're still brainstorming ideas, and there is no definitive confirmation that you will find resources to develop such a handler (thanks for clarifying by the way) and most importantly design its interfaces (both to applications and data providers).

    #2 is probably more important, as you can't reasonably make a long-term commitment to unstable and rapidly evolving technologies. And some kind of official commitment to an API (any API) is evidently what many people desire most here. My only fear is that this ends up being rushed (like the EF6 provider for DS) or that the required work is underestimated (again, mostly related to the design, patterns and interfaces, not so much the handler itself).

    For what it's worth, I think the response to the feedback has been very good so far. I hope it continues and, for my own sake, that these two concerns can be addressed in time.

  31. AdamCaviness says:

    First, I thank you Michael for your involvement in the comments!  This appears to be a refreshing change from what we didn't see in the alpha EF6 DS post comments.  Thank you.  I think you've proven that you are listening to our very real concerns.  My primary issue was how it was originally addressed and that for us, very real money (ie payroll) and of course time is involved in migrating to Web API at this point.  I agree with tnelis that it's good news that OData v3 will be officially supported.  Any improvements to the Web API story for large numbers of entities (>400) and expression tree manipulation would be appreciated and certainly removes some of the sting.  Currently we make much use of PredicateBuilder to support some pretty complex data security requirements with our own containment strategy.  I'm also happy that we weren't talking about an HttpHandler since we must support self-hosting like we currently do.

  32. CNRice says:

    I would like to get a better understanding of what "support" means.   Usually once software goes into maintenance support it's just security and major bug fixes but not new enhancements or making it work with newer technologies.  For example if EF 7 came out and it didn't work with DS would that be part of "support" or not?

    As far as the handler I was thinking would it be better or possible to actually make it work against an IEdmModel instead of object context?   People with EF could get the EDM Model and people with custom providers could create an EDM, they could even use the Web API teams ODataModelBuilder to help build the EDM.   If you implemented an interface to get queryroots it wouldn't be hard to make that work with EF.

    Something like (warning I just typed this in the comment and didn't check it in VS)

    public Interface IDSQueryRoot

    {

       IQueryable<TEntity> QueryRoot<TEntity>(object dataSource);

       IQueryable QueryRoot(Type entity, object dataSource);

    }

    If you used something like IServiceProvider to resolve the IDSQueryRoot the implementation for EF would be just the Set methods already;

    public class DSQueryRoot : IDSQueryRoot

    {

        public IQueryable<TEntity> QueryRoot<TEntity>(object dataSource)

       {

           return ((DbContext)dataSource).Set<TEntity>();

       }

       public IQueryable QueryRoot(Type entity, object dataSource)

       {

           return ((DbContext)dataSource).Set(enitty);

       }

    }

    people with custom providers would need to implement this for their custom providers.  Then you aren't tied directly to Entity Framework, which I think long term would be a long-term plus.  I think the only missing thing would be null projection or not but again if you make it so we can help build/extend the expression tree while they are being built this could be an addon like how EF does with the pluralization convention.

    I know you don't know me from Adam but if the team needed any help that I can provide for the handler I would be willing to help out.

  33. Alberto Silva, says:

    "Question; what if we provided a handler that, given an EF ObjectContext, handled requests to the underlying context, much as WCF Data Services does today? You could use this for default behavior and register individual paths for custom handling. I would need to play around with this a little, but it seems like it could work, and provide a good balance between the simplicity of WCF Data Services and the flexibility/extensibility offered by Web API.  "

    Yes, that would be great.

  34. Jim Wilcox says:

    This news is pretty tragic to me.  We have just rolled out several enterprise apps for Windows RT that rely on a WCF data service to their data.  To be honest, I am not sure where OData comes into play here, but we tried using Entity Framework and it just wasn't reliable enough.  The biggest issue was that the entities created with EF were not serializable on the Windows RT tablet.  You could serialize them fine but if you went to deserialize them and the entity contained a child collection, it would give an error.  The objects generated by the WCF would serialize and deserialize fine.  This is important because part of the requirement for Windows Store apps is that they be able to save the state of the program then recall it if the program is run again.  If you have connectivity at the time the program suspends, you can save the data to the server.  If you don't have connectivity, there is no other option other than to serialize what you are working on and when the program starts back up, deserialize it.  The other thing I like is that you can add the client for a WCF service very easily to your Windows Store app.  I am not sure what difficulty there will be in doing this in a WEB API environment.  All I know is that all I need is a simple way to securely pass data back and forth from a mobile app to our SQL server.  Please don't take this away from us.  I convinced my boss that using a Microsoft platform would ensure stability and continued support.  Please don't prove me wrong on this.

  35. Wilczek says:

    Hi,

    We are porting an existing Windows Phone 8.0 app to WP 8.1 using one of the universal app templates.

    However, the ODataLib 5.6.1 cannot be added to the Windows Phone 8.1 project. The error message I get is:

    "

    Could not install package 'System.Spatial 5.6.1'. You are trying to install this package into a project that targets 'WindowsPhoneApp,Version=v8.1', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

    "

    Will you release an updated version of the ODataLib (5.6.1) that can be used by WP8.1 universal apps? If yes, more or less when can we expect such an updated library?

    Thank you,

    Wilczek

  36. DotNetWise says:

    Open Source? Great news!! If so, add it to Github please!! Codeplex sucks and you all know it.

  37. Jeremy Caney says:

    Wow.  I take a week off work, and I completely miss the excitement!  I hope it's not too late to get my feedback in, although much of what I have to say has already been covered by tnelis, CNRice, AdamVaciness and John ; thank you all. Also, a big thank you to Michael Pizzo for communicating the direction: I am frustrated this didn't happen six months ago (e.g., with the EF6 provider thread), but I am happy it is happening now.

    As with others, my biggest concern is the amount of code required to achieve comprehensive coverage of complex models. The scaffolding is certainly a big help, but as tnelis points out in his first comment, this becomes impractical to maintain since generated code and customizations are (by default) combined.  What would be preferable is if this generated abstract classes (with the generated code) alongside a concrete class (where overrides could be placed); this would allow scaffolding to be regenerated without interfering with customizations.  Alternatively, this might be achieved through Eric Berens' proposed DataServiceController<TContext> base class, with the trade off of being runtime assessment of the model.  

    I also want to reiterate, as some others have said, that the issue here isn't necessarily the abandonment of WCF Data Services or the investment into ASP.NET Web API: the issue is the coverage gap.  I'd much rather Microsoft put all of their resources into one-OData-framework-to-rule-them-all, as opposed to having resources split between competing approaches.  But when it comes to OData support, Web API isn't quite there, and yet WCF Data Services has been left in limbo (first with the EF6 support and now with this post).  

    Last, if WCF Data Services is to remain the preferred approach for generating comprehensive coverage of large models, I like the idea of it moving to open source, but share concerns about it being prematurely migrated.  Having the library at a stable place is critical (e.g., EF6 support, basic OData V4 implementation), as is lining up appropriate sponsors (such as the Outercurve Foundation, as proposed by tnelis).  It may also be worth discussing this with IdeaBlade: they have an open source library called Breeze which includes similar support based on Web API and Entity Framework; there might be efficiencies to merging these efforts.  Still: if Microsoft is abandoning WCF Data Services I'd rather just see resources dedicated to bridging the feature gap with Web API.

    Thank you again for engaging the community on this matter, and not just lobbing this over the fence without further dialog. This goes a long way in rebuilding trust with us developers that have invested so much in the Microsoft data stack.  

  38. Wilczek says:

    Hi,

    Update: the ODataLib can be manually added to WP8.1 projects, so the issue seems to be a nuget issue.

    First one has to nuget the ODataLib for a Windows 8.1 app then add the portable libs manually to the WP8.1 app and everything will just work fine.

    Wilczek

  39. CNRice says:

    I'm going to look at "Typeless" support in the Web API OData.  Seems very promising on being flexible to replace WCF Data Services and to not have to write a ton of boiler plate code. Might be the basics to a "handler" already.  I'll post back any code or issues I find.

  40. CNRice says:

    I'm going to look at "Typeless" support in the Web API OData.  Seems very promising on being flexible to replace WCF Data Services and to not have to write a ton of boiler plate code. Might be the basics to a "handler" already.  I'll post back any code or issues I find.

  41. CNRice says:

    So far I have $metadata working.  I actually have a little bit more than that – entity sets generally working but I still need to refine that and make sure it will work as fully as I would like.   You can see what I have done so far for $metadata by going to http://www.nhail.com/…/move-from-wcf-dataservices-to-web-api.  I would be interested in anyone's thoughts on this attack plan.  I wouldn't say I'm far enough in to say it's going to work fully.

    I try to pull down the nightly build for Web API OData but couldn't get things to play nicely together so I just working with Web Api 2.1 which doesn't support OData v4 yet.  That is on my list but there is a lot on the list, it will be more important once Web Api 2.2 gets released.

    @Michael Pizzo what do you think about this approach for a "handler"?  It would be nice to get a better backing then just me in my spare time 🙂  But I feel it's still pretty early to try and open it up for everyone.

  42. CNRice says:

    Here's part 2 if anyone is interested http://www.nhail.com/…/move-from-wcf-dataservices-to-web-api-part-2 I got querying an entityset but not an entity working.  Seems like this approach has some hope.

  43. Michal Wozniak says:

    @CNRice – really nice work. There is an error on you blog that prevents from adding comments (ERROR: Unreadable CAPTCHA token file). Could you please create a repository with the code you are working on? It is  by far the most consistent approach to the WCF Data Services -> Web API migration

  44. tnelis says:

    @Michal Wozniak, CNRice: I agree, a code repo anywhere would go a long way.

  45. CNRice says:

    Fixed the error on my blog.  I'm working on Part 5 still getting the expression tree's all worked out, hopefully will have it done this week.   Any suggestions for the repository?  just want the project as a download link on the blog?

  46. Michal M. Wozniak says:

    @CNRice – checked, the comments work  now. I am leaving the repo up to you. If you like git then I would place it on GitHub, if you prefer svn then assembla has a free svn hosting that should be good enough. I would be happy to contribute to this effort – I believe others might also be willing to take some action as the current situation looks uncertain and sitting and waiting took too long.

    Really looking forward to see how's the odata request -> expression tree transition going on.

    Best regards

  47. CNRice says:

    Think I'll set it up on GitHub as I already have an account and have never used it yet.  Expression Tree's are all done I just need to get the OData payload done.   I had to refactor some of the code I previous posted as my assumptions were not correct but I'm feeling pretty good about it now.  I think I could have the payloads figured out tomorrow, if I get time to work on it.

  48. Michal M. Wozniak says:

    @CNRice, great – please give a shout what is the address of the repository

  49. CNRice says:

    I haven't put it up there just yet.  It will be @ github.com/…/WebAPIDataService  I'm creating unit test for the project now, before I was just manually testing with Fiddler.  Probably looking towards Friday.  Unless some time frees up for me – there is still a bit of payload to still get figured out.  Took awhile for me to figure out the self hosting of the Web API in a unit test project..

  50. CNRice says:

    I've posted the code up on github now.  Not saying it's 100% polished but it's a start.  I haven't written the final blog part on it as I just got done with the unit test and decided to send it on up.  From the unit test I noticed I'm missing complex type support in the query but can always add to it.

  51. tnelis says:

    @CNRice: Nice work!

  52. Dave says:

    What is the current path to getting WCF Data Services to work with EF 6.1? The alpha1 package from last October or something else?

  53. James Bailey says:

    WCF was simple to use and snap on to a project to quickly access data restfully. The asp.net web api needs this simplification. Writing custom code may be more configureable, but it's also way more costly to be maintained. I really think the community should be focused on writing less code, not more just so you can "configure" something.

    Most the development that I've done in the past 10 years is disposable after a 3-5 year life cycle. So turning a corner like this and abandoning WCF does not affect most people. The product that was developed will just die with the architecture choice of using WCF when it's life cycle ends. However, one client did invest 2 million into a WCF based odata system. They built a larger, more enterprise system. It cannot be reversed (for cost reasons) and the life cycle will not end for 10 years. Microsoft threw them under the bus. I represented the technology based on Microsoft's determination and confidence that it would be moving forward. Thanks for that. Very bad decision.

  54. tnelis says:

    @Dave: AFAIK, that's the latest, yes. If you look around you can find hacks to workaround the CurrentDataSource issue.

  55. CNRice says:

    The good news is I think I'm done with getting Web API to query like WCF Data Services.  You can read about it in my blog or download the code from github github.com/…/WebAPIDataService  Seems to be real stable.  I'm still going to make some changes.  The great news is I believe I can add batching pretty easy and Service Operations.  Also I believe I can get distinct to work!  Which would be awesome in my book.  I know @Michal M. Wozniak  has been testing it, found a couple of things I fixed.

    Really wish someone from Microsoft would comment on if WCF Data Service is going to support EF 6.  I would even give them this code if they wanted to move forward with their own handler.  

  56. CNRice says:

    Thought I would post here one last time.  We are using the Web API OData successfully in our application and for the query side it has been a direct replacement for us.  We were even able to do unit test against it by replacing the IQueryRoot interface to return backed mocked data!  Try that with WCF Data Services.  I still plan on expanding it because now we do have a requirement for batch querying and I'm still going to add distinct.   The code on github is using EF 6.1 and because a bug in EF 6.1 at work we had to go to 6.0.2 and with just one code change it worked as well.

  57. tnelis says:

    @CNRice; That's quite impressive. At this point I think it would be a good idea to contact the ASP.NET team directly, if only just to get an acknowledgement of the project. The reason is that they're supposedly still looking to do the same thing, and dump the work on a repository somewhere afterwards. If there is even a tiny chance that they would acknowledge your codebase publicly, it might attract more users and contributors, and I for one would much rather see someone like you lead this kind of project, rather than hope someone else will. Even if they get you or someone else to maintain their hypothetical project (if it exists), it would be a tremendous amount of redundant work. If I may, I would also suggest that you clear up licensing.

  58. CNRice says:

    @tnelis I would love to contact ASP.NET team directly, but I don't really know how to contact them directly.  What would you suggest?  Their forum or is there something better?

  59. tnelis says:

    @CharlesNRice Indeed the forum might not the best of places unless manage to get proper attention. I'd first try to get an email address somewhere, or maybe a twitter handle. If you can't find anything without getting all stalker-y, I'd suggest you ask around on the https://jabbr.net/ server. This might be your best bet, actually. There's also codeplex (aspnetwebstack.codeplex.com) or github (https://github.com/aspnet).

  60. RAbla says:

    We have realized the following:

    – The web api controller is very basic, and beneficial for small applications with few data entities.

    – The generated code causes more maintenance, and subjects systems to bugs, and its update is not very feasible.

    – There lots of generated files, a controller per entity, which makes it harder to manage.

    – There is no way to easily group entities under specified context.

    – Very repetitive when setting security permissions per entity, rather than context.

    – In practice few times one has to do custom changes on OData controller per entity.

    – The idea to use OData with forms is to help in RAD rapid application development, to work with simple forms, otherwise we simple use a basic controller.

    – There should be Web Api OData Context Controller, for users who want to use OData for RAD, or when creating something generic.

    Thank you, I hope you take this facts into considerations

  61. projectdd says:

    please go on wcf data service ,not stop on version 5.6.1

    don't say aspnet web api ,even if it better than. but please still wcf and wcf data service!. because we have needing.

  62. Toby says:

    Any chance of please releasing the WCF Data Services Entity Framework Provider it still in alpha on NuGet and causes packing issues. http://www.nuget.org/…/1.0.0-alpha2

    Thanks

    Toby

  63. AdamCaviness says:

    @tenlis @CharlesNRice, it turns out you can reach this team directly via odatafeedback@microsoft.com (I asked this question on another post) and their codeplex site where you make create a ticket is odata.codeplex.com/…/advanced (discovered via correspondence via that email address).

  64. Brian Kirkpatrick says:

    *** PLEASE READ MICROSOFT ODATA TEAM ***

    There were a few comments earlier by folks much smarter than I that requested [taking liberty to expand a bit] a "ROADMAP" posted on the repository site (or a link from there).  The feature sets should be listed with ESTIMATIONS of possible dot releases.  The "community" as you stated that are important to the evolution, can then pick off portions to work on in a bit of an organized manner.  An example of some success from OuterCurve with a Microsoft supported project Orchard (docs.orchardproject.net/…/feature-roadmap) uses the pattern with success.

  65. Nedelcho Stanev says:

    Seems like Microsoft finally decide to stop future development of WCF Data Services. As usual, they simply don't want to listen what customers expect. Just throwing away something, which fit perfect for enterprise. Guys, we understand that now you are concentrated on "Mobile first", but don't forget that MS got big money from companies, and they expect that you will continue to work on this !

  66. Michal M. Wozniak says:

    For those who still follow the comments in this blog entry: the old OData libraries have been updated and some problems with integration with Entity Framework 6 have been supposedly fixed: blogs.msdn.com/…/wcf-data-services-entity-framework-provider-is-updated-with-wcf-data-service-5-6-2.aspx

  67. CharlesNRice says:

    I don't know if anyone is still reading this or not.  I'm about to upgrade what I've done to OData V4.  I still only plan on doing query support.  Anyone still interested in this?  If so I'll put it on Github like the last one.

  68. RobDrawsky says:

    @CharlesNRice I've been hoping to see something more direct from MS on migrating WCF DS to WebApi. I am now at the place of having to move forward without them, so I am interested in your updates. Funny that you just posted today, as I was doing one last check on something official from MS before digging deeper into your code.

  69. tnelis says:

    @CharlesNRice: Still tracking the feed, yes. I'm still interested personally, though we've now successfully ended our OData-related development work. It's definitely a sad story, but I do hope we can take another shot at it in the future (I'll be the first to push for the move).

    If you're in a situation where you're able to keep it up, I'd love to see how your project evolves. Other than that, I'm not comfortable encouraging you without being able to *actually* contribute something (not even proper feedback), so I find myself in a rather awkward position. I want to say "yes, please do publish it", but I must follow with "but don't kill yourself over it".

    It looks like MSFT is in "wait&see" mode, and it's quite probable they won't go through with their promise of further developing and open-sourcing WCF-DS. If they won't provide a framework for your work (project management, governance, community, visibility, etc), you'd have to do it yourself, which is even more (non-technical) work for you (at least if you want the project to become more than a personal project). I think we'll all respect whatever decision you take at this point.

    In any case, thanks again!

  70. Adriaan Booysen says:

    Hi all.

    Thanks for all the comments, after 12 years working on the .Net framework and over 18 years working on MS technology, I am now actually getting peeved at MS.  

    Over the years I have figured out how to use the same code bed for exposing asmx and svc endpoints with my own custom ORM's and off late include the WCF DS with EF in the same config file of the same code base with different endpoint bindings and include different handlers to either sit on various levels within the hosted site/services to intercept authentication and various other "channel" driven work as per customer requirements.  

    The reason for most of this was to expose all the types of end-points to actually any type of client connecting to it allowing SOAP, REST and TCP bindings to anything from CF.Net, Mobile, Java, etc.

    Now if this continues, I will have to redo all the plumbing work, where OData will be separated from WCF, eg which is for WINDOWS COMMUNICATION FOUNDATION, not bloody ASP.Net and Wep API.  

    So why is the OData then being moved away from the WCF platform and what it was good for?

    What will be happening for various endpoint bindings, we still want to use SOAP, TCP, etc, not just REST with OData, or OData over http?

    Why waste the additional time to get Web API sorted if you could have extended the WCF DS with the same amount of resources being blown to fix up something that did not actually support it, eg Web API?

    In the past 18 years, I have only written enterprise grade applications, and honestly I believe MS is loosing the plot for any and all enterprise developers.

    Further more I have invested a lot of time in investigating how to write a dynamic EF model, based on little meta data stored in a repository (CLR), and (using DLR) dynamically exposing it over WCF Data Services by using the IL Generator to build the classes up and then dynamically consuming them through XAML bindings and by using the Dynamic Linq Libraries.  

    Sure it can be done with Expando, but we found that EAV typed structures, or any part thereof, was not suited for the amounts of data we where processing, we had to stay with traditional "wide" tables, which meant a lot of "dynamic" runtime generation.

    If you are going to go this route, please let me know how the following can be done through Web API:

    http://www.codeproject.com/…/Unknown-Class-Dynamically-Generated-at-runtime-for

    And for goodness sake, why stop WPF, you already killed WinForms and Silverlight, what are we going to get as the next flavour?  There is still a huge need for fat client apps (thin if you use WCF) out there, not everything can be done with ASP.Net/web, especially not in all enterprises!  We still have lots of applications out there that store data locally till it can find a web connection, we are in Africa!

    Sorry for all the rambling, but at 37 years of age (which means I have spend all my career in MS tech), I am eventually thinking about throwing in the towel and choose some other development platform.

    Cheers

  71. Luke Puplett says:

    I must first say that open-sourcing is a great idea, the best possible outcome from this situation.

    But I also empathize with Adriaan above, though I usually am in agreement when MS need to "move on". I've worked in MS tech for 16 years and I've seen technologies come and go. But the world changes, needs change.

    We're working in technology, not horticulture. I think we have to love change, else we're going to get pretty miserable before retirement.

    We can still use these technologies, people still use VB6, but MS can't keep servicing their huge legacy. Hand it over to the people and move on. If people really care, they'll take it on. Talking of which, the churn is nothing compared to strictly OSS frameworks! They come and go like buses.

    Though Astoria and OData are wonderful technologies, all the points in this blog post are true and converging on ASP.NET will be a better long term choice.

    Perhaps more needs to be done to make building OData data services from existing data sources as mind-blowingly productive as it was in WCF DS.