Web API [Queryable] current support and tentative roadmap

  • Article

The recent preview release of OData support in Web API  is very exciting (see the new nuget package and codeplex project). For the most part it is compatible with the previous [Queryable] support because it supports the same OData query options. That said there has been a little confusion about how [Queryable] works, what it works with and what its limitations are, both temporary and long term. 

The rest of this post will outline what is currently supported, what limitations currently exist and which limitations are hopefully just temporary.

Current Support

Support for different ElementTypes

In the preview the [Queryable] attribute works with any IQueryable<> or IEnumerable<> data source (Entity Framework or otherwise), for which a model has been configured or can be inferred automatically.

Today this means that the element type (i.e. the T in IQueryable<T>) must be viewed as an EDM entity. This implies a few constraints:

  • All properties you wish to expose must be exposed as CLR properties on your class.
  • A key property (or properties) must be available
  • The type of all properties must be either:
    • a clr type that is mapped to an EDM primitive, i.e. System.String == Edm.String
    • Or clr type that is mapped to another type in your model, be that a ComplexType or an EntityType

NOTE: using IEnumerable<> is recommended only for small amounts of data, because the options are only applied after everything has been pulled into memory.

Null Propagation

This feature takes a little explaining, so please bear with me. Imagine you have an action that looks like this:

[Queryable]
public IQueryable<Product> Get()
{
…
}

Now imagine someone issues this request:

GET ~/Products?$filter=startswith(Category/Name,’A’)

You might think the [Queryable] attribute will translate the request to something like this:

Get().Where(p => p.Category.Name.StartsWith(“A"));

But that might be very bad…
If your Get() method body looks like this:

return _db.Products; // i.e. Entity Framework.

It will work just fine. But if your Get() method looks like this:

return products.AsQueryable();

It means the LINQ provider being used is LINQ to Objects. L2O evaluates the where predicate in memory simply by calling the predicate. Which could easily null ref if either p.Category or p.Category.Name are null.

The [Queryable] attribute handles this automatically by injecting null guards into the code for certain IQueryable Providers. If you dig into the code for ODataQueryOptions you’ll see this code:

…
string queryProviderAssemblyName = query.Provider.GetType().Assembly.GetName().Name;
switch (queryProviderAssemblyName)
{
case EntityFrameworkQueryProviderAssemblyName:
handleNullPropagation = false;
break;
case Linq2SqlQueryProviderAssemblyName:
handleNullPropagation = false;
break;
case Linq2ObjectsQueryProviderAssemblyName:
handleNullPropagation = true;
break;
default:
handleNullPropagation = true;
break;
}
return ApplyTo(query, handleNullPropagation);

As you can see for Entity Framework and LINQ to SQL we don’t inject null guards (because SQL takes care of null guards/propagation automatically), but for L2O and all other query providers we inject null guards and propagate nulls.

If you don’t like this behavior you can override it by dropping down and calling ODataQueryOptions.Filter.ApplyTo(..) directly.

Supported Query Options

In the preview the [Queryable] attribute supports only 4 of OData’s 8 built-in query options, namely $filter, $orderby, $skip and $top.

What about the 4 other query options? i.e. $select, $expand, $inlinecount and $skiptoken. Today you need to use ODataQueryOptions rather than [Queryable], hopefully that will change overtime.

Dropping down to ODataQueryOptions

The first thing to understand is that this code:

[Queryable]
public IQueryable<Product> Get()
{
return _db.Products;
}

Is roughly equivalent to:

public IEnumerable<Product> Get(ODataQueryOptions options)
{
// TODO: we should add an override of ApplyTo that avoid all these casts!
return options.ApplyTo(_db.Products as IQueryable) as IEnumerable<Product>;
}

Which in turn is roughly equivalent to:

public IEnumerable<Product> Get(ODataQueryOptions options)
{

    IQueryable results = _db.Products;
if (options.Filter != null)
results = options.Filter.ApplyTo(results);
if (options.OrderBy != null) // this is a slight over-simplification see this.
results = options.OrderBy.ApplyTo(results);
if (options.Skip != null)
results = options.Skip.ApplyTo(results);
if (options.Top != null)
results = options.Top.ApplyTo(results);

    return results;
}

This means you can easily pick and choose which options to support. For example if your service doesn’t support $orderby you can assert that ODataQueryOptions.OrderBy is null.

ODataQueryOptions.RawValues

Once you’ve dropped down to the  ODataQueryOptions you also get access to the RawValues property which gives you the raw string values of all 8 ODataQueryOptions… So in theory you can handle more query options.

ODataQueryOptions.Filter.QueryNode

The ApplyTo method assumes you have an IQueryable, but what if you backend has no IQueryable implementation?

Creating one from scratch is very hard, mainly because LINQ allows so much more than OData allows, and essentially obfuscates the intent of the query.
To avoid this complexity we provide ODataQueryOptions.Filter.QueryNode which is an AST that gives you a parsed metadata bound tree representing the $filter. The AST of course it tuned to allow only what OData supports, making it much simpler than a LINQ expression.

For example this test fragment illustrates the API:

var filter = new FilterQueryOption("Name eq 'MSFT'", context);
var node = filter.QueryNode;
Assert.Equal(QueryNodeKind.BinaryOperator, node.Expression.Kind);
var binaryNode = node.Expression as BinaryOperatorQueryNode;
Assert.Equal(BinaryOperatorKind.Equal, binaryNode.OperatorKind);
Assert.Equal(QueryNodeKind.Constant, binaryNode.Right.Kind);
Assert.Equal("MSFT", ((ConstantQueryNode)binaryNode.Right).Value);
Assert.Equal(QueryNodeKind.PropertyAccess, binaryNode.Left.Kind);
var propertyAccessNode = binaryNode.Left as PropertyAccessQueryNode;
Assert.Equal("Name", propertyAccessNode.Property.Name);

If you are interested in an example that converts one of these ASTs into another language take a look at the FilterBinder class. This class is used under the hood by ODataQueryOptions to convert the Filter AST into a LINQ Expression of the form Expression<Func<T,bool>> .

You could do something very similar to convert directly to SQL or whatever query language you need. Let me assure you doing this is MUCH easier than implementing IQueryable!

ODataQueryOptions.OrderBy.QueryNode

Likewise you can interrogate the ODataQueryOptions.OrderBy.Query for an AST representing the $orderby query option.

Possible Roadmap?

These are just ideas at this stage, really we want to hear what you want, that said, here is what we’ve been thinking about:

Support for $select and $expand

We hope to add support for both of these both as QueryNodes (like Filter and OrderBy), and natively by the [Queryable] attribute.

But first we need to work through some issues:

  • The OData Uri Parser (part of ODataContrib) currently doesn’t support $select / $expand, and we need that first.
  • Both $expand and $select essentially change the shape of the response. For example you are still returning IQueryable<T> from your action but:
    • Each T might have properties that are not loaded. How would the formatter know which properties are not loaded?
    • Each T might have relationships loaded, but simply touching an unloaded relationship might cause lazyloading, so the formatters can’t simply hit a relationship during serialization as this would perform terribly, they need to know what to try to format.
  • There is no guarantee that you can ‘expand’ an IEnumerable or for that matter an IQueryable, so we would need a way to tell [Queryable] which options it is free to try to handle automatically.

Support for $inlinecount and $skiptoken

Again we hope to add support to [Queryable] for both of these.

That said today you can implement both of these by returning ODataResult<> from your action today.
Implementing $inlinecount is pretty simple:

public ODataResult<Product> Get(ODataQueryOptions options)

{

    var results = (options.ApplyTo(_db.Products) as IQueryable<Product>);

    var count = results.Count;

    var limitedResults = results.Take(100).ToArray();

    return new ODataResult<Product>(results,null,count);

}

 

However implementing server driven paging (i.e. $skiptoken) is more involved and easy to get wrong. 
I’ll blog about how to do Server Driven Pages pretty soon.

Support for more Element Types.

We want to support both Complex Types (Complex Type are just like entities, except they don’t have a key and have no relationships) and primitive element types. For example both:

public IQueryable<string> Get(); – maps to say GET ~/Tags

and

public IQueryable<Address> Get(parentId); – maps to say GET ~/Person(6)/Addresses

where no key property has been configured or can be inferred for Address.

You might be asking yourself how do you query a collection of primitives using OData? Well in OData you use the $it implicit iteration variable like this:

GET ~/Tags?$filter=startswith($it,’A’)

Which gets all the Tags that start with ‘A’.

Virtual Properties and Open Types

Essentially virtual properties are things you want to expose as properties via your service that have no corresponding clr property. A good example might be where you to use methods to get and set a property value. This one is a little further out, but it is clearly useful.

Conclusion

As you can see [Queryable] is a work in progress that is layered above ODataQueryOptions, we are planning to improve both over time, and we have a number of ideas. But as always we’d love to hear what you think!