blambert/bugreport - ADO.NET Data Services Client URL encoding bug.
Today I was working with the ADO.NET Data Services Client, performing queries against Azure Table Storage, and I discovered that there is a bug in the URL encoding of some queries. (Though I was performing queries against Azure Table Storage, this bug is in the ADO.NET Data Services Client; so it will manifest itself in any query, against any service. I think. Certainly it affects Azure Table Storage.)
I was able to write this up and get it to the ADO.NET Data Services Client team today.
I am writing it up here, in my blog, to get the information out there, share a workaround that works, but needs work.
The essence of the bug is this: The ADO.NET Data Services Client does not properly URL encode certain values.
Steps to reproduce:
We’ll add two DataEntity values, one with a RowKey of “foo%bar” and the other with a RowKey of “foo_bar”:
context.AddObject(EntitySetName, new DataEntity()
{
PartitionKey = "SomeValue",
RowKey = "foo%bar"
});
context.AddObject(EntitySetName, new DataEntity()
{
PartitionKey = "SomeValue",
RowKey = "foo_bar"
});
context.SaveChanges();
Next, we’ll query for these two values:
foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo%bar").Execute())
{
Console.WriteLine("Query for foo%bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}
foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo_bar").Execute())
{
Console.WriteLine("Query for foo_bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}
And the output of this operation looks like this:
Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .
QUERY FAIL. We didn’t get the “foo%bar” entry back…
If we fiddler this, we see the following requests:
https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo%bar'
https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo_bar'
And there’s the issue! The % in “foo%bar” in the first query was not URL encoded. You can’t have a % in a URL. It’s a reserved character.
If we change that first query to look like this:
string value = HttpUtility.UrlEncode("foo%bar");
foreach(DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == value).Execute())
{
Console.WriteLine("Query for HttpUtility.UrlEncode(\"foo%bar\") returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}
The fiddler traces looks like:
https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo%25bar'
https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo_bar'
And we can see that “foo%bar” is now properly URL encoded as “foo%25bar”.
The output of this operation looks like this:
Query for HttpUtility.UrlEncode("foo%bar") returned SomeValue-foo%bar
Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .
It works.
My testing indicates that of the reserved characters, the following errors will occur:
For ! result SUCCESS
For * result SUCCESS
For ' result SUCCESS
For ( result SUCCESS
For ) result SUCCESS
For ; result SUCCESS
For : result SUCCESS
For @ result SUCCESS
For & result EXCEPTION An error occurred while processing this request.
For = result SUCCESS
For + result FAIL
For $ result SUCCESS
For , result SUCCESS
For / result SUCCESS
For ? result SUCCESS
For % result FAIL
For # result EXCEPTION An error occurred while processing this request.
For [ result SUCCESS
For ] result SUCCESS
So, it appears that & + % and # are broken.
The quick and dirty workaround is to call HttpUtility.UrlEncode(…) on every input value being compared using Where’s with values that contains these characters.
Since % is not being encoded, we have a workaround, as every input value that is URL encoded before being passed into the ADO.NET Data Services Client will pass right through.
So your Where’s would look like:
Where(q => q.X == HttpUtility.UrlEncode(x) && q.Y == HttpUtility.UrlEncode(y))
The downside of this quick and dirty workaround is that when this bug gets fixed, queries with this workaround will break, because the %’s will be double URL encoded. So a more sophisticated workaround will be needed…
Disclaimer: I don’t work on the ADO.NET Data Services team, I’m on another team; so I am not speaking authoritatively for Microsoft in this area. I am sharing my experiences as a developer who ran into a bug, and needed to understand it, report it, and work around it.
Brian