Standalone XQuery Implementation in .NET?

XQuery 1.0 and XPath 2.0 are now W3C Recommendations, thanks in part to the contributions of  several Microsoft employees over the years. An earlier draft of the XQuery specification is supported in SQL Server 2005 , and you can send an XQuery to the server using the ADO.NET that shipped in Visual Studio 2005.  These features are becoming widely used.    Now that the XQuery family of specifications is complete, it’s fair to ask what our implementation plans might be, 

We had been working on an implementation of XQuery in the .NET Framework that operated over standalone XmlDocuments several years ago, and showed the work in progress in the first beta of what became Visual Studio 2005. That was not shipped for several reasons .  The most compelling reason was that it was obvious in 2004 that the W3C Recommendation would not be complete before that product  release was frozen.  Another reason was that this seemed like it was just XSLT in a different skin, whereas what our customers really wanted was a more powerful implementation that provided advanced features such as in-memory indexes and support for mapping  parts of the query to the back-end server, including use of the XML datatype and XQuery provided by SQL server.    This would have been much more work, and would have required us to go far beyond what the W3C was standardizing – so our focus back then was to make sure the back end server support for XQuery was rock solid.  We have had occasional requests from our user community for a client side implementation of XQuery that operates over standalone XmlDocuments, but we see no clear groundswell of demand yet.

We very much wish to hear from our user community about their requirements that could be met with XSLT 2.0 and XQuery 1.0.  We announced last week that we are actively working on an XSLT 2.0 implementation. As with XLST 2.0, those needing an implementation of XQuery that runs in the .NET environment may wish to check out the Saxon open source project or the schema-aware commercial version produced by Saxonica  Likewise, those needing query and transformation features over standalone XML files could help us understand whether XSLT 2.0 meets your needs, or if you think XQuery is more suitable. 

We are very excited about the Language Integrated Query (LINQ) work that is our primary focus for “Orcas”.  In LINQ you will see “LINQ to XML” and VB-XML language extensions that provide functionality similar to XQuery, but with LINQ you can operate over Objects, SQL Databases and XML documents … and can create your own LINQ provider to support virtually any other type of data.  We are also incubating LINQ to XSD for XSD-based generation of strongly typed object models that support the LINQ programming model.

We do recognize, however, that some scenarios require a cross-platform standards based approach  and so while we finish up “Orcas” we will be listening to our customers to figure out how to blend of these technologies and create a roadmap for our future products.  Some have asked for an XQuery-powered in-memory database with indexes, query plans, an optimizer, etc.  Some have asked for something that might be called  “LINQ to XQuery” that would provide a LINQ-enabled front end for an XQuery-enabled DBMS, much like what LINQ to SQL provides for an RDBMS. You might have even more interesting ideas for how to combine our technology innovations and the W3C standards to provide high Windows productivity/performance and cross-platform interoperability. We can’t do all of this, but we can focus on the approach that meets the most real-world needs with the time and resources available. Now is a great time to have an influence on those plans, so please don’t hesitate to send your feedback to the XML Team contact page.

Comments (15)

  1. tzagotta says:

    In my opinion, now that Saxonica have XQuery available for .NET, I think this reduces the need to have support for XQuery on XML documents integrated into the .NET Framework. I haven’t studied their implementation, however. It is somewhat important that they integrated their XQuery with the existing XML services in .NET. If they didn’t do this, then it is not as useful as an implementation that is well-integrated.

    For example, we had an application last year that took a "tree" of objects, serialized them to XML, then applied an XQuery to change the data model, and finally applied an XSL transform to generate a text file with a particular file format. In our implementation, we had to format the XML DOM to a text representation to have the command-line Saxonica XQuery tool run, and then we had to re-parse the XML file in order to perform the XSLT. Obviously the extra rendering and parsing causes a large performance hit, and if the Saxonica implementation is not well-integrated with .NET, this will still be the case.

    (I’m sorry I don’t have time to study the .NET implementation of Saxonica’s product right now in order to be able to make more concrete statements about this.)

  2. Devon Young says:

    I’ve always been more of an XSLT user, so to me, I don’t see much of a reason for a standalone XQuery implementation. Using XSLT with XQuery, is likely what I’ll do when I first delve into XQuery (which I plan to do soon). Down the road, I might wish to use standalone XQuery…but not right now, or for at least 4-6 months (i’m quite sure). That’s my take.

  3. Gad D Lord says:

    I would love to have System.XQuery namespace working over a local xml file. It will save me the trouble to load XML files as objects with composition and learn *LINQ. Just cover the already ready XQuery 1.0 specification.

  4. Michael Rys says:

    Now that the W3C XQuery recommendation has been released, the XML Team, owners of MSXML and System.XML,…

  5. Bryan says:

    I would appreciate an Xquery implementation that took as the collection an variable that was an object composed of a file path and file extension of files to be queried. The implementation should go ahead and recursively build the collection for you.

    That would make .Net really interesting for me again.

  6. xmlhacker says:

    Similar to the way Saxon provides tightly coupled XSLT 2.0/XPath 2.0/XQuery 1.0 support, it seems to me that it would make a lot of sense to provide this same level of tightly integrated support natively.

    To me, anyway, probably the most important aspect of this is not whether or not MSFT should provide XQuery support for stand alone XML docs (I believe you should), and instead, proper documentation and evangelism efforts as to using the right tool for the job, implementing a "Patterns and Practices for XSLT 2.0, XPath 2.0, and XQuery 1.0" which showcases the proper usage of each technology using a tightly coupled approach to piping together queries and transforms effectively.

    Of course, LINQ would need to play into this as well, though obviously LINQ is much more C#/VB.NET-language centric, which seems to me is catered towards a completely different crowd of folks than those in whom would be looking at an XSLT 2.0/XPath 2.0/XQuery 1.0 as a potential solution.

    And lastly, if I could just throw the following snippet from a recent thread on XSL-List (this comes from a follow-up comment made by Andrew Houghton)

        > What I would like to see though, is Saxonica and Microsoft working

       > together to develop a plugable XSL infrastructure, like is available

       > in the Java world.  So when XSL 3.0 comes around, one can swap out

       > implementations without having to rewrite code.

    For what its worth, to see something similar to the above take place would be absolutely amazing, in my opinion (as well as, I assume, others.)

    Thanks for opening up the conversation!  *MUCH* appreciated!

  7. In a recent post to the MSFT XML Team Weblog, Dr. Michael Rys writes, Microsoft XML Team’s WebLog : Standalone XQuery Implementation in .NET? We very much wish to hear from our user community about their requirements that could be…

  8. gilstav says:

    I would appreciate an implementation of Xquery.

    It would be easier to allow an end user to compose an Xquey with an Xquery Builder than with Xslt (Template logic).Thank you.

  9. Kurt Cagle says:

    I believe that what makes the most sense here is an XQueryProcessor in the model of System.xml.xsl.xslTransform (e.g., System.xquery.xqueryTransform)- both have the potential to be precompiled, both may be applied to multiple XML entities (and quite possibly in the latter case on non-XML entities), both should be able to be run against streamed sax2 objects and DOM objects, and both may end up producing non-XML output. Is it worth it? Yes, absolutely – it completely abstracts the data input model.

    While you’re at it, take a real hard look at the eXist function extensions for XQuery ( – especially the transform:transform() method which invokes an XSLT transformation within an XQuery. The power of this one method is rather astounding, and it seems to provide the final key to making XQuery and XSLT complementary rather than competing technologies.

  10. Mike Champion says:

    > I would love to have System.XQuery namespace

    > working over a local xml file.

    LINQ to Objects gives you a powerful query language over filesystem objects  in .NET out of the box.  That illustrates why we are so enthusiastic about LINQ: It operates over the IEnumerable<T> view that is implicit in most data already, so those LINQ queries just work with no need to implement them for each class representing the data.  XQuery provides a data model that can represent data in filesystems, databases, XML documents, etc., but that requires an explicit mapping and implementation for each separate class of data.

    That said, we do understand that it doesn’t work outside .NET, and people need to write remoteable and interoperable queries.  I appreciate the suggestions, especially those about how to make our stack more consistent with Saxon and other 3rd parties.  Definite food for thought.

    What I’m not seeing, however, is clear explanations of WHY people want to use XQuery for scenarios that XSLT is already well suited.  

  11. Hal Hayes says:

    I, also, would like to see an XQuery implementation within the .Net Framework.

    If available, I would be more inclinded to use XQuery over XSLT, particularly in a more dynamic setting to extract data, and then to shape the output before consuming within an application.

    While the XQuery implementation in SQL Server 2005 is excellent, it is still only a subset of the standard, for example, the ability to query against multiple documents. And, I would prefer to see the implementation as native to the .Net Framework, rather than as a commercial add-on.

  12. Mike Champion says:

    If SQL Server supported the minimum conformance level of the W3C Recommendation, would you still see a need for XQuery in .NET?  

  13. D. Herr says:

    I’d also like to vote for XQuery support in .Net.

    To me, the primary benefit of XQuery over XSLT is the cleaner syntax.  It might seem like a minor thing, but the result is code that’s more readable, easier to debug, and cheaper to maintain.   I like the power of XSLT, but writing code in XML syntax is really tiring.

    LINQ is cool, but whenever I see Microsoft lagging in support for W3C standards I start looking around at other platforms.

    As mentioned earlier, an API that’s consistent with existing libraries (Saxon in this case) is always appreciated.

    Support for XQuery in Sql Server is great, but most of the data I’d like to process with XQuery comes from files or web services.

  14. Mike says:

    Our company started using the .NET framework because they had extensive support for XML standards.  At the time, this included XSLT 1.0 and XPATH.  I would like to see Microsoft continue to support W3C standards with regards to XML.  Our company does a lot of xml processing and adhering to standards is important to our business.

  15. Daniel says:

    It’s good thing to see in .NET framework. Already support XQuery in SQL Server 2005, it’s very useful. If .NET framework also can support XQuery, we can do much xml data processing on client side. That will be very good for developing.