[This post originally appeared in http://blogs.msdn.com/b/azuremobile/archive/2014/04/07/deep-dive-on-the-offline-support-in-the-managed-client-sdk.aspx. I’ll be writing Azure Mobile Service related posts in the Azure Mobile blog primarily, and cross-posting them here for people who have feeds linked to this blog only]
Last week we announced a new feature in the Azure Mobile Service SDK (managed code only for now): support for offline handling of data. While before all table operations required an active internet connection, with this new support the application can store table operations in a local data store, and later when connected push the changes to the mobile service (and also pull changes from the service into the local table). The local data store is defined by an interface, so you can use whatever implementation you want, but we also released a new NuGet package with a SQLite-based implementation of the store to help you get started quickly.
New table types: IMobileServiceSyncTable and IMobileServiceSyncTable<T>
One thing which we decided to do when implementing the offline support is that we wanted the developers to be aware of what kind of data would be offline and which data would be online. We thought about implementing the support in a transparent way and data would be synchronized to the server whenever a connection could be established, but almost always the “magic” that we would implement would be wrong. Instead, we released an alpha version of the SDK with offline capabilities (meaning that when managing the NuGet packages you’ll need to select the “Include prerelease” option in the combo box on top of the package names), and based on the feedback we get from all of you we’ll decide the direction to go next (which can be implementing a full “auto-sync” framework).
In practice, that means that to use the offline feature, you’ll need to use a different kind of table. There are two new methods in the MobileServiceClient class:
GetSyncTable(string tableName) and
GetSyncTable<T>(), which return instances of the
IMobileServiceSyncTable<T> interfaces, respectively. They’re in most ways very similar to the IMobileServiceTable and IMobileServiceTable<T> interfaces, exposing similar methods. The biggest difference is that for the local (sync) tables, there are no overloads for the CRUD operations with additional query string parameters (there’s no HTTP request going on the operation on those tables). Other differences include the fact that sync tables only work with entities with string ids (those with integer ids are not supported locally) and the shape of the response for insert and update operations (on regular tables, the responses can be any JSON value, including arrays and primitives; for sync tables they need to be objects).
One more thing which I think is important – a few people have asked me why the interface was named sync table – why not name it local table, which is : the interface name tells us something about its behavior. We’re not talking simply about a local table. It’s an object which can synchronize the state of the remote table (on Azure) with the table in the local store. We’ll get to more details on the synchronization later.
Let’s get coding. To start, let’s try to insert an item into a sync table to see what happens…
When we run that, it doesn’t work. Instead, we get the following exception:
System.InvalidOperationException: SyncContext is not yet initialized.
The problem is that the SDK has no idea about where to store the local data. Before using any of the sync operations, we first need to initialize the synchronization context of the client so that those operations can start working.
The synchronization context
In addition to the two new methods to get sync table instances from the client, there’s a new property,
SyncContext, which needs to be initialized with an instance of the actual local store where the data will be saved. Before you can use any of the local operations, the sync context in the client needs to be initialized with an
IMobileServiceLocalStore object. That means that you can define whatever mechanism to store the local data, but the large majority of developers doesn’t need to go into that level of details, so we’ve also released an implementation of that interface based on a SQLite database. To access that store implementation (in the class
MobileServiceSQLiteStore) you’ll need a new NuGet package, Azure Mobile Services SQLiteStore. On to fixing the code above. When we instantiate the store, we need to define the tables which will be used to store data locally. There are two ways to define the tables that the SDK will use: you can either pass a JObject instance containing the properties which will be stored to the
DefineTable method, or even easier, you can also use the
DefineTable<T> method where you just pass the type which will be used as the generic parameter, and the SDK will deal with it.
Now, if you run the code above it will work – a
TodoItem table will be created in the local store and an item will be added to it.
The synchronization context is mainly responsible for, well, synchronizing the data between the local database (represented by the local store) and the remote database (accessed via the Azure Mobile Service). This synchronization is done via an explicit push / pull mechanism which must be invoked by the developer – at this point we don’t have any “auto-sync” framework which will handle those calls automatically, but this feature may be implemented in a future version of the SDK.
So back to the synchronization context. Once we start calling operations on the local tables, those operations start getting queued up by the sync context. Those operations become “pending” and are persisted locally, so that even if the application is closed and reopened, the list of pending operations is returned. You can check the number of operations which are pending by looking at the
PendingOperations property in the synchronization context. As more operations are executed in the local tables, the queue will grow, until there is a synchronization event (which we’ll talk more in the next section). Let’s look at that property a little closer, by expanding the example above and performing additional operations in the local table.
As I mentioned before, the sync tables have basically the same API as “regular” (remote) tables, so queries, updates and inserts (as well as deletes, not shown above) look just like methods on regular tables. As we run the code below (assuming that the local store was empty) we’ll get an output similar to the one shown below. It looks expected – when we insert the first item the pending operation count goes to one; when we insert another item the count is incremented once more; when we read from the local table the count is not incremented – read operations are not synchronized. But there’s one interesting aspect which happens when we update one of the items which we had just inserted: the number of operations in the queue is not changed. What the current implementation of the synchronization context does is to “merge” pending operations for the same item, so that during synchronization only one operation (in this case, insert) is sent to the server (where the value of the item is the most recent one).
Defined table in the store
Initialized the sync context
Pending operations in the sync context queue: 0
Inserted into local store: 9e61196d-55df-4869-8b30-4a4a6eb792f2
Pending operations in the sync context queue: 1
Inserted another item into the local store: c349b30d-d603-48db-8509-b2fd170f4499
Pending operations in the sync context queue: 2
Things to do Buy bread, Buy milk
Pending operations in the sync context queue: 2
Updated item: c349b30d-d603-48db-8509-b2fd170f4499
Pending operations in the sync context queue: 2
I talked about synchronization operations without introducing them. Let’s look at them now.
Push / pull / purge
There are three basic operations which can trigger synchronization. The simplest of all is the
PushAsync method in the synchronization context. Once that method is called, the changes which were performed in the local table are sent over to the server. In the example below, there will be one more item in the server once the call to
PushAsync is completed (or maybe more, if there were other insert operations pending in the synchronization queue).
Push is executed on the whole context, not on specific tables. It’s implemented this way to support relationships between entities on the client side. For example, if you have an “Order” and an “OrderItem” table, you can insert an item in the first table, and having the id of that entity, insert the children items with the appropriate foreign key. And when the operations are sent to the server, they will be sent in order so that any FK relationships in the database will be satisfied.
The other operation which triggers a synchronization is a call to
PullAsync on the local table. That call can either pull all items from the remote table, or just pull a subset of the items. Only pulling some items from the table is often advisable, as stuffing everything from the (potentially large) remote database table into the (restricted by the device memory) local table may have some bad performance implications. You can pass an OData-formatted query to select which items to pull from the server, or you can also use the (more friendly) Linq expressions to determine the query of items to be pulled.
One important thing to notice regarding pull operations – if there are items in the pending synchronization queue, those items are first pushed over to the server, then the pull operation takes place. That prevents a scenario where an update is done to a local item, but a pull operation would overwrite the changes locally and potentially leave the data in an inconsistent state. That’s one the first synchronization rule: a pull triggers a push. In the example below, the insert operation for the “Buy milk” item will first be pushed to the server, then the items will be pulled into the local table.
Another operation which triggers a synchronization event is a call to
PurgeAsync on the local table. Often we want to clear the local cache to update the data which the application doesn’t need anymore. For example, the canonical TODO app, we only display the items in the client which are not complete. In this case, there’s no need to store locally any items which have already been complete. A call to purge such items can be done as shown below.
Notice that, just like in the pull case, a call to purge will first send any pending operations to the server (the second synchronization rule: purge also triggers a push). This way, if we had marked an item as complete locally, we want to make sure that this information is in the server before we remove the item from the local store.
Handling conflict errors
Until now we’ve looked at many synchronization scenarios where everything works fine. There are cases, however, where errors happen. If there are multiple sources changing a single entity (such as a row in the database), you may have conflicts when a second update is attempted (since the version of the item will have change – for more information see this document on optimistic concurrency implementation on the server). In this case, a push operation would fail. Take the code below: the item is updated in the remote table, but when we try to push the update to the local item, since the version of the items will not match, the push operation will fail, and a
MobileServicePushFailedException will be thrown. The exception has a list of all errors which happened for the individual elements from the synchronization queue.
There are scenarios where you want to catch and deal with the synchronization conflicts in the client. You can control all the synchronization operations by implementing the
IMobileServiceSyncHandler interface and passing it when initializing the context. For example, this is an implementation of a sync handler which traces all the operations which are happening.
And we can use this synchronization handler by passing it to the overload of
InitializeAsync in the sync context, as shown below:
This context implementation doesn’t do much, but we can catch the exception which is thrown by the client when the server returns a Precondition Failed (HTTP status code 412) and retry the call again after updating the version on the client.
And this is how we can resolve conflicts on the client. This sample shows another conflict handling policy (letting the user choose which version to keep), but the structure is similar to the one above. And a final note about resolving synchronization conflicts: to use the optimistic concurrency feature (which prevents two clients from overriding modifications in the same row), you’ll need to define a version column in the class used in the client.
Some additional information about this release which I think is interesting. Unlike on remote tables, local tables can store arbitrary types, including complex ones (for example, a “Person” class can have an “Address” property), and when stored in the local table the complex property will be stored as a JSON-ified version of its value. You won’t be able to query on those types (for example, list all people whose “Address.City” property is “Springfield”), but it can be stored and retrieved without any extra code.
When writing a sync handler (like the one used in the previous section to resolve conflicts) you can also abort the whole push operation if you find an error for which you don’t want to continue. If this is the case, you can call the
AbortPush method on the
IMobileServiceTableOperation instance passed to the
One more thing: the implementation of the local store uses the SQLite database, which is a x86/arm only binary. The project cannot be configured to “AnyCPU”, it needs to be configured to a specific architecture.
In this release we introduced offline capabilities in the .NET SDK for Mobile Services. We released it as an alpha NuGet package so you can try and give us feedback on what works, what doesn’t and what we can improve. Please let us know in the comments, in our forum or via twitter @AzureMobile.