blambert/bugreport - ADO.NET Data Services Client URL encoding bug.

Today I was working with the ADO.NET Data Services Client, performing queries against Azure Table Storage, and I discovered that there is a bug in the URL encoding of some queries.  (Though I was performing queries against Azure Table Storage, this bug is in the ADO.NET Data Services Client; so it will manifest itself in any query, against any service.  I think.  Certainly it affects Azure Table Storage.)

I was able to write this up and get it to the ADO.NET Data Services Client team today.

I am writing it up here, in my blog, to get the information out there, share a workaround that works, but needs work.

The essence of the bug is this:  The ADO.NET Data Services Client does not properly URL encode certain values.

Steps to reproduce:

We’ll add two DataEntity values, one with a RowKey of “foo%bar” and the other with a RowKey of “foo_bar”:

 context.AddObject(EntitySetName, new DataEntity()
{
    PartitionKey = "SomeValue",
    RowKey = "foo%bar"
});
context.AddObject(EntitySetName, new DataEntity()
{
    PartitionKey = "SomeValue",
    RowKey = "foo_bar"
});
context.SaveChanges();

Next, we’ll query for these two values:

foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo%bar").Execute())
{
    Console.WriteLine("Query for foo%bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo_bar").Execute())
{
    Console.WriteLine("Query for foo_bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

And the output of this operation looks like this:

Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .

QUERY FAIL. We didn’t get the “foo%bar” entry back…

If we fiddler this, we see the following requests:

https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo%bar'

https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo_bar'

And there’s the issue!  The % in “foo%bar” in the first query was not URL encoded.  You can’t have a % in a URL.  It’s a reserved character.

If we change that first query to look like this:

string value = HttpUtility.UrlEncode("foo%bar");
foreach(DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == value).Execute())
{
    Console.WriteLine("Query for HttpUtility.UrlEncode(\"foo%bar\") returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

The fiddler traces looks like:

https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo%25bar'

https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo_bar'

And we can see that “foo%bar” is now properly URL encoded as “foo%25bar”.

The output of this operation looks like this:

Query for HttpUtility.UrlEncode("foo%bar") returned SomeValue-foo%bar
Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .

It works.

My testing indicates that of the reserved characters, the following errors will occur:

For ! result SUCCESS
For * result SUCCESS
For ' result SUCCESS
For ( result SUCCESS
For ) result SUCCESS
For ; result SUCCESS
For : result SUCCESS
For @ result SUCCESS
For & result EXCEPTION An error occurred while processing this request.
For = result SUCCESS
For + result FAIL
For $ result SUCCESS
For , result SUCCESS
For / result SUCCESS
For ? result SUCCESS
For % result FAIL
For # result EXCEPTION An error occurred while processing this request.
For [ result SUCCESS
For ] result SUCCESS

So, it appears that & + % and # are broken.

The quick and dirty workaround is to call HttpUtility.UrlEncode(…) on every input value being compared using Where’s with values that contains these characters.

Since % is not being encoded, we have a workaround, as every input value that is URL encoded before being passed into the ADO.NET Data Services Client will pass right through.

So your Where’s would look like:

Where(q => q.X == HttpUtility.UrlEncode(x) && q.Y == HttpUtility.UrlEncode(y))

The downside of this quick and dirty workaround is that when this bug gets fixed, queries with this workaround will break, because the %’s will be double URL encoded.  So a more sophisticated workaround will be needed…

Disclaimer: I don’t work on the ADO.NET Data Services team, I’m on another team; so I am not speaking authoritatively for Microsoft in this area. I am sharing my experiences as a developer who ran into a bug, and needed to understand it, report it, and work around it.

Brian