More about REST: File upload download service with ASP.NET Web API and Windows Phone background file transfer
Last week we discussed RESTful services, as well as how to create a REST service using WCF Web API. I'd like to remind you what's really important is the concepts, not how to implement the services using a particular technology. Shortly after that, WCF Web API was renamed to ASP.NET Web API, and formally entered beta.
In this post, we'll discuss how to upgrade from WCF Web API to ASP.NET Web API. We'll create a file uploading/downloading service. Before that, I'd like to give you some background about the motivation of this post.
The current released version of Story Creator allows you to encode pictures to videos. But the videos are mute. Windows Phone allows users to record sound using microphone, as do many existing PCs (thus future Windows 8 devices). So in the next version, we'd like to introduce some sound. That will make the application even cooler, right?
The sound recorded by Windows Phone microphone only contains raw PCM data. This is not very useful other than using XNA to play the sound. So I wrote a prototype (a Windows Phone program) to encode it to wav. Then I expanded the prototype with a REST service that allows the phone to upload the wav file to a server, where ultimately it will be encoded to mp4 using Media Foundation.
With the release of ASP.NET Web API beta, I think I'll change the original plan (write about OAuth this week), to continue the discussion of RESTful services. You'll also see how to use Windows Phone's background file transfer to upload files to your own REST services. However, due to time and effort limitations, this post will not cover how to use microphone and how to create wav files on Windows Phone (although you'll find it if you download the prototyp), or how to use Media Foundation to encode audio/videos (not included in the prototype yet, perhaps in the future).
You can download the prototype here. Once again, note this is just a prototype, not a sample. Use it as a reference only.
The service
Review: What is REST service
First let's recall what it means by a RESTful service. This concept is independent from which technology you use, be it WCF Web API, ASP.NET Web API, Java, or anything else. The most important things to remember are:
- REST is resource centric (while SOAP is operation centric).
- REST uses HTTP protocol. You define how clients interact with resources
using HTTP requests (such as URI and HTTP method).
In most cases, upgrade from WCF Web API to ASP.NET Web API is simple. Those two do not only share the same underlying concepts (REST), but also share a lot of code base. You can think ASP.NET Web API as a new version of WCF Web API, although it does introduce some break changes, and has more to do with ASP.NET MVC. You can find a lot of resources here. Today I'll specifically discuss one topic: How to build a file upload/download service. This is somewhat different from the tutorials you'll find on the above web site, and actually I encountered some difficulties when creating the prototype.
Using ASP.NET Web API
The first thing to do when using ASP.NET Web API is to download ASP.NET MVC 4 beta, which includes the new Web API. Of course you can also get it from NuGet.
To create a project that uses the new Web API, you can simply use Visual Studio's project template.
This template is ideal for those who want to use Web API together with ASP.NET MVC. It will create views and JavaScript files in addition to files necessary for a service. If you don't want to use MVC, you can remove the unused files, or create an empty ASP.NET application and manually create the service. This is the approach we'll take today.
If you've followed last week's post to create a REST service using WCF Web API, you need to make a few modifications. First remove any old Web API related assembly references. You have to add references to the new version: System.Web.Http.dll, System.Web.Http.Common, System.Web.Http.WebHost, and System.Net.Http.dll. Note many Microsoft.*** assemblies have been renamed to System.***. As for System.Net.Http.dll, make sure you reference the new version, even if the name remains the same. Once again, make sure you set Copy Local to true, if you plan to deploy the service to Windows Azure or somewhere else.
Like assmblies, many namespaces have been changed from Microsoft.*** to System.***. If you're not sure, simply remove all namespaces, and let Visual Studio automatically find which ones you need.
The next step is to modify Global.asax. Before, the code looks like this, where ServiceRoute is used to define base URI using ASP.NET URL Routing. This allows you to remove the .svc extension. Other parts of URI are defined in UriTemplate.
routes.Add(new ServiceRoute( "files", new HttpServiceHostFactory(),
typeof(FileUploadService)));
In ASP.NET Web API, URL Routing is used to define the complete URI, thus removes the need to use a separate UriTemplate.
public static void RegisterRoutes(RouteCollection routes)
{
routes.MapHttpRoute(
name: "DefaultApi",
routeTemplate: "{controller}/{filename}",
defaults: new { filename = RouteParameter.Optional }
);
}
The above code defines how a request is routed. If a request is sent to https://[server name]/files/myfile.wav, Web API will try to find a class named FilesController ({controller} is mapped to this class, and it's case insensitive). Then it will invoke a method which contains a parameter filename, and map myfile.wav to the value of the parameter. This parameter is optional.
So you must have a class FilesController. You cannot use names like FileUploadService. This is because now Web API relies on some of ASP.NET MVC's features (although you can still host the service in your own process without ASP.NET). However, this class does not have to be put under the Controllers folder. You're free to put it anywhere, such as under a services folder to make it more like a service instead of a controller. Anyway, this class must inherite ApiController (in WCF Web API, you didn't need to inherite anything).
For a REST service, HTTP method is as important as URI. In ASP.NET Web API, you no longer use WebGet/Invoke. Instead, you use HttpGet/AcceptVerbs. They're almost identical to the counter parts in old Web API. One improvement is if your service method begins with Get/Put/Post/Delete, you can omit those attributes completely. Web API will automatically invoke those methods based on the HTTP method. For example, when a POST request is made, the following method will automatically be invoked:
public HttpResponseMessage Post([FromUri]string filename)
You may also notice the FromUri attribute. This one is tricky. If you omit it, you'll encounter exceptions like:
No 'MediaTypeFormatter' is available to read an object of type 'String'
with the media type ''undefined''.
This has to do with ASP.NET MVC's model binding. By default, the last parameter in the method for a POST/PUT request is considered to be request body, and will deserialized to a model object. Since we don't have a model here, an exception is thrown. FromUri tells MVC this parameter is actually part of URI, so it won't try to deserialize it to a model object. This attribute is optional for requests that do not have request bodies, such as GET.
This one proved to be a trap for me, and web searches do not yield any useful results. Finally I asked the Web API team directly, where I got the answer (Thanks Web API team!). I'm lucky enough to have the oppotunity to talk with the team directly. As for you, if you encoutner errors that you're unable to find the answer yourself, post a question in the forum! Like all new products, some Web API team members will directly monitor that forum.
You can find more about URL Routing on
https://www.asp.net/web-api/overview/web-api-routing-and-actions/routing-in-aspnet-web-api.
Implement file uploading
Now let's implement the file uploading feature. There're a lot ways to handle the uploaded file. In this prototype, we simply save the file to a folder of the service application. Note in many cases this will not work. For example, in Windows Azure, by default you don't have write access to folders under the web role's directory. You have to store the file in local storage. In order for multiple instances to see the same file, you also need to upload the file to blob storage. However, in the prototype, let's not consider so many issues. Actually the prototype does not use Windows Azure at all.
Below is the code to handle the uploaded file:
public HttpResponseMessage Post([FromUri]string filename)
{
var task = this.Request.Content.ReadAsStreamAsync();
task.Wait();
Stream requestStream = task.Result;
try
{
Stream fileStream = File.Create(HttpContext.Current.Server.MapPath("~/" + filename));
requestStream.CopyTo(fileStream);
fileStream.Close();
requestStream.Close();
}
catch (IOException)
{
throw new HttpResponseException("A generic error occured. Please try again later.", HttpStatusCode.InternalServerError);
}
HttpResponseMessage response = new HttpResponseMessage();
response.StatusCode = HttpStatusCode.Created;
return response;
}
Unlike last week's post, this time we don't have a HttpRequestMessage parameter. The reason is you can now use this.Request to get information about request. Others have not changed much in the new Web API. For example, to obtain the request body, you use ReadAsStreamAsync (or another ReadAs*** method). Note in real world product, it is recommended to handle the request asynchronously. This is again not considered in our prototype, so we simply let the thread wait until the request body is read.
When creating the response, you still need to pay attention to status code. Usually for a POST request which creates a resource on the server, the response's status code is 201 Created. When an error occurs, however, you need to return an error status code. This can be done by throwing a HttpResponseException, the same as in previous Web API.
File downloading and Range header
While the audio recoding feature does not need file downloading, this may be required in the future. So I also implemented it in the prototye. Another reason is I want to test the Range header (not very relavent to Story Creator, but one day it may prove to be useful).
Range header is a standard HTTP header. Usually only GET requests will use it. If a GET request contains a Range header, it means the client wants to get partial resource instead of the complete resource. This can be very useful sometimes. For example, when downloading large files, it is expected to pause the download and resume sometime later. You can search for 14.35 on https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html, which exlains the Range header.
To give you some practical examples on how Range header is used in real world, consider Windows Azure blob storage. Certain blob service requests support the Range header. Refer to
https://msdn.microsoft.com/en-us/library/windowsazure/ee691967.aspx for more information. As another example, Windows Phone background file transfer may (or may not) use the Range header, in case the downloading is interrupted.
The format of Range header is usually:
bytes=10-20,30-40
It means the client wants the 10th to 20th bytes and the 30th to 40th bytes of the resource, instead of the complete resource.
Range header may also come in the following forms:
bytes=-100
bytes=300-
In the former case, the client wants to obtain from the beginning to the 100th byte of the resource. In the latter case, the client wants to obtain from the 300th bytes to the end of the resource.
As you can see, the client may request for more than one ranges. However in practice, usually there's only one range. This is in particular true for file downloading scenarios. It is quite rare to request for discrete data. So many services only support a single range. That's also what we'll do in the prototype today. If you want to support multiple ranges, you can use the prototype as a reference and implement additional logic.
The following code implements file downloading. It only takes the first range into account, and ignores the remaining.
public HttpResponseMessage Get([FromUri]string filename)
{
string path = HttpContext.Current.Server.MapPath("~/" + filename);
if (!File.Exists(path))
{
throw new HttpResponseException("The file does not exist.", HttpStatusCode.NotFound);
}
try
{
MemoryStream responseStream = new MemoryStream();
Stream fileStream = File.Open(path, FileMode.Open);
bool fullContent = true;
if (this.Request.Headers.Range != null)
{
fullContent = false;
// Currently we only support a single range.
RangeItemHeaderValue range = this.Request.Headers.Range.Ranges.First();
// From specified, so seek to the requested position.
if (range.From != null)
{
fileStream.Seek(range.From.Value, SeekOrigin.Begin);
// In this case, actually the complete file will be returned.
if (range.From == 0 && (range.To == null || range.To >= fileStream.Length))
{
fileStream.CopyTo(responseStream);
fullContent = true;
}
}
if (range.To != null)
{
// 10-20, return the range.
if (range.From != null)
{
long? rangeLength = range.To - range.From;
int length = (int)Math.Min(rangeLength.Value, fileStream.Length - range.From.Value);
byte[] buffer = new byte[length];
fileStream.Read(buffer, 0, length);
responseStream.Write(buffer, 0, length);
}
// -20, return the bytes from beginning to the specified value.
else
{
int length = (int)Math.Min(range.To.Value, fileStream.Length);
byte[] buffer = new byte[length];
fileStream.Read(buffer, 0, length);
responseStream.Write(buffer, 0, length);
}
}
// No Range.To
else
{
// 10-, return from the specified value to the end of file.
if (range.From != null)
{
if (range.From < fileStream.Length)
{
int length = (int)(fileStream.Length - range.From.Value);
byte[] buffer = new byte[length];
fileStream.Read(buffer, 0, length);
responseStream.Write(buffer, 0, length);
}
}
}
}
// No Range header. Return the complete file.
else
{
fileStream.CopyTo(responseStream);
}
fileStream.Close();
responseStream.Position = 0;
HttpResponseMessage response = new HttpResponseMessage();
response.StatusCode = fullContent ? HttpStatusCode.OK : HttpStatusCode.PartialContent;
response.Content = new StreamContent(responseStream);
return response;
}
catch (IOException)
{
throw new HttpResponseException("A generic error occured. Please try again later.", HttpStatusCode.InternalServerError);
}
}
Note when using Web API, you don't need to manually parse the Range header in the form of text. Web API automatically parses it for you, and gives you a From and a To property for each range. The type of From and To is Nullable<long>, as those properties can be null (think bytes=-100 and bytes=300-). Those special cases must be handled carefully.
Another special case to consider is where To is larger than the resource size. In this case, it is equivalent to To is null, where you need to return starting with From to the end of the resource.
If the complete resource is returned, usually status code is set to 200 OK. If only part of the resource is returned, usually status code is set to 206 PartialContent.
Test the service
To test a REST service, we need a client, just like testing other kinds of services. But often we don't need to write clients ourselves. For simple GET requests, a browser can serve as a client. For other HTTP methods, you can use Fiddler. Fiddler also helps to test some advanced GET requests, such as Range header.
I think most of you already know how to use Fiddler. So today I won't discuss it in detail. Below are some screenshots that will give you an idea how to test the Range header:
Note here we request the range 1424040-1500000. But actually the resource size is 1424044. 1500000 is out of range, so only 4 bytes are returned.
You need to test many different use cases, to make sure your logic is correct. It is also a good idea to write a custom test client with all test cases written beforehand. This is useful in case you change service implementation. If you use Fiddler, you have to manually go through all test cases again. But with some pre-written test cases, you can do automatic tests. However, unit test is beyond the scope of today's post.
Windows Phone Background File Transfer
While we're at it, let's also briefly discuss Windows Phone background file transfer. This is also part of the prototype. It uploads a wav file to the service. One goal of the next Story Creator is to show case some of Windows Phone Mango's new features. Background file transfer is one of them.
You can find a lot of resources about background file transfer. Today my focus will be pointing out some points that you may potentially miss.
When to use background file transfer
Usually when a file is small, there's no need to use background file transfer, as using a background agent do introduce some overhead. You will use it when you need to transfer big files (but it cannot be too big, as usually using phone network costs users). Background file transfer supports some nice features, like automatic pause/resume downloading as the OS thinks it's needed (requires service to support Range header), allows user to cancel a request, and so on.
In the prototype, we simply decide to use background file transfer to upload all files larger than 1MB, and use HttpWebRequest directly when the file size is smaller. Note this is rather an objective choice. You need to do some test and maybe some statictics to find the optimal border between background file transfer and HttpWebRequest in your case.
if (wavStream.Length < 1048576)
{
this.UploadDirectly(wavStream);
}
else
{
this.UploadInBackground(wavStream);
}
Using background file transfer
To use background file transfer, refer to the document on https://msdn.microsoft.com/en-us/library/hh202959(v=vs.92).aspx. Pay special attention to the following:
The maximum allowed file upload size for uploading is 5MB (5242880 bytes).
The maximum allowed file download size is 20MB or 100MB (depending on whether wifi is available). Our prototype limits the audio recording to 5242000 bytes.
It's less than 5242880 because we may need additional data. For example, additional 44 bytes are required to write wav header.
if (this._stream.Length + offset > 5242000)
{
this._microphone.Stop();
MessageBox.Show("The recording has been stopped as it is too long.");
return;
}
In order to use background file transfer, the file must be put in isolated storage, under /shared/transfers folder. So our prototype saves the wav file to that folder if it needs to use background file transfer. But if it uses
HttpWebRequest directly, it transfers the file directly from memory (thus a bit faster as no I/O is needed).
In addition, for a single application, at maximum you can create 5 background file transfer requests in parallel. If the limit is reached, you can either notify the user to wait for previous files to be transferred, or manually queue the files. Our prototype, of course, takes the simpler approach to notify the user to wait.
The following code checks if there're already 5 background file transfer requests and notifies the user to wait if needed. If less than 5 are found, a BackgroundTransferRequest is created to upload the file. The prototype simply hardcodes the file name to be test.wav. Of course in real world applications, you need to get the file name from user input.
private void UploadInBackground(Stream wavStream)
{
// Check if there're already 5 requests.
if (BackgroundTransferService.Requests.Count() >= 5)
{
MessageBox.Show("Please wait until other records have been uploaded.");
return;
}
// Store the file in isolated storage.
var iso = IsolatedStorageFile.GetUserStoreForApplication();
if (!iso.DirectoryExists("/shared/transfers"))
{
iso.CreateDirectory("/shared/transfers");
}
using (var fileStream = iso.CreateFile("/shared/transfers/test.wav"))
{
wavStream.CopyTo(fileStream);
}
// Transfer the file.
try
{
BackgroundTransferRequest request = new BackgroundTransferRequest(new Uri("https://localhost:4349/files/test.wav"));
request.Method = "POST";
request.UploadLocation = new Uri("shared/transfers/test.wav", UriKind.Relative);
request.TransferPreferences = TransferPreferences.AllowCellularAndBattery;
request.TransferStatusChanged += new EventHandler<BackgroundTransferEventArgs>(Request_TransferStatusChanged);
BackgroundTransferService.Add(request);
}
catch
{
MessageBox.Show("Unable to upload the file at the moment. Please try again later.");
}
}
Here we set TransferPreferences to AllowCellularAndBattery, so the file can be transferred even if no wifi is available (thus the user has to use phone's celluar network) and battery is low. In real world, please be very careful when setting the value. 5MB usually will not cost the user too much. But if you need to transfer larger files, consider to disallow cellular network. Battery is usually less of a concern in case of file transfer, but it is still recommended to do some test and gather practical data.
Once a file transfer is completed, the background file transfer request will not be removed automatically. You need to manually remove it. You may need to do that quite often, so a global static method (such as a method in the App class) will help.
internal static void OnBackgroundTransferStatusChanged(BackgroundTransferRequest request)
{
if (request.TransferStatus == TransferStatus.Completed)
{
BackgroundTransferService.Remove(request);
if (request.StatusCode == 201)
{
MessageBox.Show("Upload completed.");
}
else
{
MessageBox.Show("An error occured during uploading. Please try again later.");
}
}
}
Here we check if the file transfer has completed, and remove the request if it has. We also check the response status code of the request, and display either succeed or failed information to user.
You invoke this method in the event handler of TransferStatusChanged. Note you do not only need to handle this event when the request is created, but also need to handle it in case of application launch and tomestone. After all, as the name suggests, the request can be executed in background. However, if your application is not killed (not tomestoned), you don't need to handle the events again, as your application still remains in memory.
Below is the code handling application lifecycle events and check for background file transfer:
// Code to execute when the application is launching (eg, from Start)
// This code will not execute when the application is reactivated
private void Application_Launching(object sender, LaunchingEventArgs e)
{
this.HandleBackgroundTransfer();
}
// Code to execute when the application is activated (brought to foreground)
// This code will not execute when the application is first launched
private void Application_Activated(object sender, ActivatedEventArgs e)
{
if (!e.IsApplicationInstancePreserved)
{
this.HandleBackgroundTransfer();
}
}
private void HandleBackgroundTransfer()
{
foreach (var request in BackgroundTransferService.Requests)
{
if (request.TransferStatus == TransferStatus.Completed)
{
BackgroundTransferService.Remove(request);
}
else
{
request.TransferStatusChanged += new EventHandler<BackgroundTransferEventArgs>(Request_TransferStatusChanged);
}
}
}
Finally, in a real world application, you need to provide some UI to allow users to monitor background file transfer requests, and cancel them if necessary. A progess indicator will also be nice. However, those features are really out of scope for a prototype.
Upload files directly
Just to make the post complete, I'll also list the code that uses HttpWebRequest to upload the file directly to the service (same code you can find all over the web).
private void UploadDirectly(Stream wavStream)
{
string serviceUri = "https://localhost:4349/files/test.wav";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serviceUri);
request.Method = "POST";
request.BeginGetRequestStream(result =>
{
Stream requestStream = request.EndGetRequestStream(result);
wavStream.CopyTo(requestStream);
requestStream.Close();
request.BeginGetResponse(result2 =>
{
try
{
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result2);
if (response.StatusCode == HttpStatusCode.Created)
{
this.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show("Upload completed.");
});
}
else
{
this.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show("An error occured during uploading. Please try again later.");
});
}
}
catch
{
this.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show("An error occured during uploading. Please try again later.");
});
}
wavStream.Close();
}, null);
}, null);
}
The only thing worth noting here is Dispatcher.BeginInvoke. HttpWebRequest returns the response on a background thread (and actually I already start the wav encoding and file uploading using a background worker to avoid blocking UI thread), thus you have to delegate all UI manipulations to UI thread.Finally, if you want to test the protypte on emulator, make sure your PC has a microphone. If you want to test it on a real phone, you cannot use localhost any more. You need to use your PC's name (assume your phone can connect to PC using wifi), or host the service on internet such as in Windows Azure.
Conclusion
I didn't intend to write such a long post. But it seems there're just so many topics in development that makes me difficult to stop. Even a single theme (RESTful services) involves a lot of topics. Imagine how many themes you will use in a real world application. Software development is fun. Do you agree? You can combine all those technologies together to create your own product.
While this post discusses how to use ASP.NET Web API to build file upload/download services, the underlying concept (such as the Range header) can be ported to other technologies as well, even on non-Microsoft platforms. And the service can be accessed from any client that supports HTTP.
If I have time next week, let's continue the topic by introducing OAuth into the discussion, to see how to protect REST services. And in the future, if you're interested, I can talk about wav encoding on Windows Phone, and more advanced encodings on Windows Server using Media Foundation. However, time is really short for me...