Using IndexedDB in your Windows 8 Game
As I was pulling together my previous post on notifications in an HTML 5 game on Windows 8, I couldn’t help but think there must be a better way to handle the storage of the local leaderboard. If you’re not familiar with that sample, it’s a simple game where you see how many times you can touch a bouncing ball before it hits the boundaries of the display ten times.
What’s Wrong with Using ApplicationData for the Leaderboard?
The sample implements a local leaderboard that leverages the roaming data store in Windows 8 application data storage, as you can see from this snippet of code in the Scores.setScores method:
var roamingSettings = Windows.Storage.ApplicationData.current.roamingSettings;
roamingSettings.values["scores"] = JSON.stringify(scoresToSet);
Here scoresToSet is an array of JavaScript objects that each contains a player, score, and skill level, like
{ player:"Jim", score: 32, skill: 1 }
Since roaming storage is synced to the cloud, the current implementation brings the leaderboard to whatever other devices the user has installed the game on, and that may or may not be a feature you’d want to expose. After all the game may be harder to play on different form factors, touch versus mouse, etc., so does consolidating the leaderboard into one across those disparate devices make sense? If not, it would be a trivial matter to switch to local application data storage, simply by referencing Windows.Storage.ApplicationData.current.localSettings instead of roamingSettings.
That’s not the part that bothered me though; it was this code:
for (var i = scores.length - 1; i >= 0; i--) {
if (score.score > scores[i].score) {
if (i < scores.length - 1) {
scores[i + 1] = scores[i];
scores[i] = score;
newScoreRank = i + 1;
}
if (i === 0) {
scores[i] = score;
newScoreRank = 1;
}
} else if (score.score === scores[i].score && i < scores.length - 1) {
scores[i + 1] = score;
newScoreRank = i + 2;
break;
}
}
Granted, this is just a sample application, but look at the amount of code needed to manipulate the leaderboard and insert the new score at the right place, so it can later be serialized en masse into roaming storage. On top of that, there’s limited capability to show other interesting views of the game play, like a history of all the games Joe completed, or a leaderboard that’s filtered by skill level.
The case for IndexedDB
To get the flexibility desired, we need to store more than just the snapshot of the leaderboard, and we need some capability to do at least rudimentary ad hoc queries on the data. That’s the kind of thing that SQL Server would be good for! Of course, in the context of Windows Store applications, much less the JavaScript variety, SQL Server and relational database servers aren’t a supported option, unless hosted as a service.
The good news is that JavaScript applications built in Windows 8 can leverage the Indexed Database API specification of the W3C, or IndexedDB for short. IndexedDB is a non-relational object store of JavaScript objects providing transactional semantics via the following constructs:
| database | a set of object stores; each application (or more specifically, origin) can contain multiple databases |
| object store | a collection of key-value pairs within the database; analogous to a table in a relational database |
| key | a value of type float, date, string, or array that uniquely identifies an object with a given object store, much like the primary key columns in a relational database table. The key also imposes an ascending sort order on the associated objects in the store. |
| value | the JavaScript object associated with a given key. Like NoSQL document databases such as MongoDB and CouchBase, values can be complex objects with no schematization requirements across the object store. |
| key path | a string that defines how to extract the key from a value, typically a period-separated concatenation of the attributes that define the path through the object to the key property |
| index | a specialized storage construct that supports searching objects in the store by attribute values |
| transaction | an atomic and durable operation against the database, in one of three modes: readonly, readwrite, or versionchange. |
| request | a read or a write operation, analogous to a SQL statement (although IndexedDB doesn’t use SQL). |
| key range | a single or range of key values for which to retrieve values from an object store |
| cursor | an iterator over the results of a request which can be traversed in one of four modes or directions: next, nextunique, prev, or prevunique. |
If you've worked with relational database management systems before, these concepts should be familiar, but be aware there are no relational semantics here – that means no joins! If you've been tinkering with some of the key-value NoSQL offerings, that won’t be a shock and you'll be right at home.
By the way, if you’re looking for relational database-like functionality for your C# or VB Windows Store app, take a look at SQLite (and in particular the excellent blog posts by Tim Heuer). With SQLite you essentially embed a transactional SQL engine right into your application’s process!
There’s a port of SQLite to JavaScript as well, but I’ve not explored its viability for Windows Store applications. Lastly, for those of you familiar with Web SQL,note that the W3C has ceased work on that specification.
Re-implementing the Local Leaderboard
Defining the Schema
The leaderboard in the current HTML 5 sample contains objects with a consistent schema of three properties (or attributes):
player: the name of the player
score: the score of the game
skill: the skill level of the associated game (from 0 to 2, Basic to Advanced)
One challenge of this object structure is that uniqueness is not necessarily guaranteed. I could score the same on two games played at the same level. In this case, I’d be compelled to add a timestamp that indicates when the game was completed, and that’s something I’d probably want to show on the leaderboard anyway - the longer someone at the top the more compelling it is to try to beat them!
A timestamp (in UTC) will work fine as a key in this case, since it will be unique on that device. But you could also opt to use a key generator which simply creates a monotonically increasing sequence of values (up to 2^53) whenever a key is needed.
Now let's revisit the core IndexedDB constructs and put some concrete names against them that will be used in the actual code:
| database | leaderboard |
| object store | games |
| key | gamedate |
| value | an object with the following prototype { gamedate: <date>, player: <string>; score: <number>, skill: <number> } |
| key path | gamedate |
| index | three indexes: player, score, and skill+score |
To upgrade the leaderboard to use IndexedDB, we simply need to replace the implementation of the methods that currently access roaming storage, and that code is localized to three methods located in scores.js
- getScores returns a JavaScript array of the top ten scores as retrieved from roaming storage. This method will be modified to return scores from the IndexedDB, perhaps with some filter criteria allowing a richer set of options
- setScores write the snapshot of the leaderboard, serialized as JSON, back to roaming storage. The updated implementation won’t need this method.
- newScore adds a score from a newly completed game into the appropriate position of the JavaScript leaderboard array and then writes the entire array back to roaming storage (via setScores). In the updated implementation, this work is reduced to a simple insert operation (or more precisely write request) on IndexedDB.
Creating a Database
We need to start by creating a database if one doesn’t exist (and it won’t the first time the application is run on a Windows 8 device). A good spot for doing this in the HTML 5 game’s application framework is the initialize method of the Game module (in game.js):
1: initialize: function (state) {
2:
3: if (GameManager.gameId === null) {
4: this.stateHelper = state;
5: this.state = state.internal;
6: this.settings = state.external;
7: }
8:
9: var db = window.indexedDB.open("leaderboard", 1);
10: db.onupgradeneeded = function (e) {
11: GameManager.leaderboard = e.target.result;
12:
13: scoresTable = GameManager.leaderboard.createObjectStore(
"scores", { keyPath: "gamedate" });
14: scoresTable.createIndex("playerIdx", "player", { unique: false });
15: scoresTable.createIndex("scoreIdx", "score", { unique: false });
16: scoresTable.createIndex("skillIdx", "skill", { unique: false });
17:
18: txn = e.target.transaction;
19: txn.onerror = function () { console.log("Schema definition failed"); };
20: txn.oncomplete = function () { console.log("Schema definition worked"); };
21: };
22: db.onerror = function () { console.log("Database creation failed"); };
23: db.onsuccess = function (e) {
24: GameManager.leaderboard = e.target.result;
25: };
26: },
| Lines 1 - 7 | original implementation of initialize is retained |
| Line 9 | open an existing database called leaderboard, or create it, if it doesn’t already exist. When created, a database has a version designation that’s an empty value, so the second parameter here (1) is interpreted as a request to upgrade the database version. Only through upgrade requests can you modify the schema of the database. |
| Line 10 | the onupgradeneeded callback is invoked whenever there is a database version upgrade request, as occurs when the database is opened with a version number that does not yet exist. |
| Line 11 | the database object is stored as a new attribute of the GameManager singleton instance. |
| Line 13 | the scores object store is created with a key path of gamedate, the unique key of the store. |
| Lines 14 - 16 | three different indexes are created referring to different key paths (to values that have yet to be added to the object store). |
| Lines 18 - 20 | callbacks are associated with the transaction under which the schema changes are being made. This transaction is operating in versionchange mode , which is the only context under which the database schema can be modified. After the changes have been made, the oncomplete callback is invoked; it this implementation it merely pushes a message to the JavaScript console. |
| Line 22 | the error callback would be invoked if the database open request fails. |
| Line 23-25 | a success callback is invoked when the open succeeds (and a version upgrade is not required). Here the database context is retained for convenience as an attribute of GameManager. |
Adding New Scores to the Database
The database is empty at this point, and records to be added whenever a player completes a game and a new score is logged. That happens in the newScore method of the Scores module (scores.js). Where the code had pulled the last known leaderboard from roaming storage and manually inserted the new score (if it appeared in the top ten), now the new score record is simply inserted into the database:
1: newScore: function (score) {
2:
3: var txn = GameManager.leaderboard.transaction(["scores"], "readwrite");
4:
5: scoreTable = txn.objectStore("scores");
6: insRequest = scoreTable.add(
7: {
8: player: score.player,
9: score: score.score,
10: skill: score.skill,
11: gamedate: new Date()
12: });
13: },
| Line 3 | a new transaction is created involving the scores object store and of mode readwrite. |
| Line 5 | a handle to the stores object store is obtained. |
| Line 6-12 | a new object containing information on the score of the game just completed is added to scores. The key for this value is the gamedate attribute; recall that was the key path defined for the scores object store when the schema was first created. What’s perhaps not obvious is that the add method is asynchronous. Upon successful completion, the onsuccess callback of the request (insRequest) will fire. Likewise, onerror will execute if there were a problem inserting the new key-value pair. For our leaderboard scenario, there is no specific action that needs to be taken for the success case; the record has been inserted. As for the error path, production code would likely include some error handling in a callback, but to keep things focused, I’ve those details to the reader. |
Retrieving the Scores from the Database
In the original implementation, the code flow to populate the leaderboard went something like this:
The scoresPage UI contains a ListView that is bound to the JavaScript array (the ViewModel) returned by getItems in scoresPage.js. getItems in turn invokes getScores to pull the last snapshot of the leaderboard from the roaming settings, essentially grabbing the Model and creating the ViewModel. All of this happens in a synchronous fashion. In the schematic above, the invocation chain is captured by the blue solid arrows and the data that is returned is represented by the red dashed arrows.
To adapt this workflow for IndexedDB, conceptually the only change is to have getScores access IndexedDB versus roaming storage. It’s not quite that easy though: data access for IndexedDB all occurs asynchronously, and the existing code is expecting getScores to block until it has all the data. So, instead of returning a JavaScript array to getItems, we pass callback functions forward. When the query for IndexedDB had completed, its callback fires and in turn spawns the execution of the forwarded callbacks, which set up the ViewModel and trigger the binding. The flow now looks more like:
As you might expect there is a bit of a ripple effect in the code as both getItems and getScores need to participate in the asynchronous invocation chain by accepting callbacks to execute. getScores (implementation below) now accepts a callback function (passed to it by getItems) that is invoked when the database retrieval is complete. That callback function (shown a bit later) takes the data and creates the ViewModel, and when that completes, it invokes another callback that sets up the ListView binding.
The IndexedDB code though is confined to the getScores method, and there are three primary constructs in play:
- a transaction, created in Line 5 below and with its mode set to readonly, since it’s only reading data. When all the requests on the transaction are complete, the oncomplete callback fires, which in turn populates the ViewModel via the populateViewModel function forwarded from getItems. Not shown are implementations for onabort and onerror, which should be include for handling errors and unexpected failures.
- a request (here a query), created in Line 10, and leveraging one of the indexes defined on the stores schema. Since this query is to return a leaderboard, the scored should be sorted in descending order. In a SQL scenario, you’d add an ORDER BY clause; here, the index always sorts objects in ascending order, so just traverse the index space backwards – hence, the prev argument for openCursor.
- a cursor, which is the result of the query and traversed in Lines 14-18. Each value (“row”) from the cursor is pushed onto a JavaScript array, which will ultimately be passed into the populateViewModel callback. The continue method here advances the cursor in the required direction. The request object is reused; therefore, onsuccess continues to be called until the query results have been exhausted and cursor becomes undefined.
1: getScores: function (populateViewModel) {
2:
3: var scoresFromDb = [];
4:
5: var txn = GameManager.leaderboard.transaction(["scores"], "readonly");
6: txn.oncomplete = function () {
7: populateViewModel(scoresFromDb);
8: };
9:
10: var query = txn.objectStore("scores")
11: .index("scoreIdx")
12: .openCursor(null, "prev");
13: query.onsuccess = function (e) {
14: var cursor = e.target.result;
15: if (cursor) {
16: scoresFromDb.push(cursor.value);
17: cursor.continue();
18: }
19: };
20: },
For completeness sake, the revision to getItems follows.Instead of getScores returning a list of scores, an anonymous function that populates the ViewModel is passed in (Lines 6 – 24), so that when the scores are retrieved from the database, that function can be invoked to set the values in the ViewModel.
1: function getItems(updateBindings) {
2: // TODO: Update desired background styling values of the score table here
3: var colors = ["rgba(209, 211, 212, 1)", "rgba(147, 149, 152, 1)", "rgba(65, 64, 66, 1)"];
4:
5: var scores = GameManager.scoreHelper.getScores(
6: function (scores) {
7: var items = [];
8: var groupId = 0;
9:
10: // TODO: Update to match your score structure
11: for (var i = 0; i < scores.length; i++) {
12: items.push({
13: group: pageData.groups[groupId],
14: key: "item" + i,
15: rank: i + 1,
16: player: scores[i].player,
17: score: scores[i].score,
18: skill: GameManager.scoreHelper.getSkillText(scores[i].skill),
19: backgroundColor: colors[i % colors.length],
20: });
21: }
22:
23: updateBindings(items);
24: });
25: }
getItems (below) similarly accepts a function as an argument, and that function is how the newly populated ViewModel (items) is made accessible to the page initialization code that sets up the UI binding. There is another invocation of getItems in this same JavaScript module that must be similarly modified, but I’m leaving the details out since it’s tangential to the discussion of IndexedDB.
1: function ready(element, options) {
2:
3: // Set up ListView
4: WinJS.UI.processAll(element)
5: .done(function () {
6: if (list) {
7: var listView = element.querySelector(".collectionList").winControl;
8: pageData.groups = getGroups();
9: getItems(function (items) {
10: pageData.items = items;
11: list = new WinJS.Binding.List(pageData.items);
12: groupedItems = list.createGrouped(groupKeySelector, groupDataSelector);
13: listView.forceLayout();
14: });
15: }
16: });
17: }
Retrieving the Scores for One Skill Level from the Database
Now that the revamped leaderboard has feature parity with the original implementation, let’s add some new functionality. The current game has three levels, Basic, Intermediate, and Advanced, so it might be nice to have an option to display leaderboards for each of those levels individually. Let’s assume that there is some UI element, a three-valued slider for instance, that is introduced allowing the user to pick the skill level reflected on the leaderboard. The value obtained from that UI element could be made accessible to the getScores function, perhaps as an argument.
The operative part of getScores is reproduced below; this is the same version we discussed above that retrieves all the scores in descending order.
var query = txn.objectStore("scores")
.index("scoreIdx")
.openCursor(null, "prev");
query.onsuccess = function (e) {
var cursor = e.target.result;
if (cursor) {
scoresFromDb.push(cursor.value);
cursor.continue();
}
};
Recall that when the database was created, we set up three different indexes, including one on skill level, so one approach would be to use that index and confine the elements retrieved – the key range – to the skill level that was selected. That query would look something like the following, with the changes highlighted:
var query = txn.objectStore("scores")
.index("skillIdx")
.openCursor(IDBKeyRange.only(skillLevel), "prev");
query.onsuccess = function (e) {
var cursor = e.target.result;
if (cursor) {
scoresFromDb.push(cursor.value);
cursor.continue();
}
scoresFromDb.sort(function (a, b) {
return b.score - a.score;
});
};
Since we want only the score records associated with the level selected (available here as skillLevel), it makes sense to use the index defined on that attribute of the score objects. Furthermore, we need only a subset of the range that index spans. In this case, it’s a single value, so we can use the only key range. You can also specify other key ranges bounded (or not) on either the high or the low ends of the range of values that the index spans. With only a single key value being returned, the cursor mode isn’t really relevant, so that optional argument (with the value prev) could be removed from the call to openCursor.
You might be surprised to see a call to sort the array! Since IndexedDB does not support compound indexes, you can’t easily do what you might do in SQL with a ORDER BY clause specifying two columns. By going with skillIdx, we got the benefit of the cursor returning only those scores for the desired game skill level, but there is no guaranteed ordering of them. The leaderboard needs to display them in descending order, so that’s why the call to the built-in JavaScript sort method is necessary.
An alternative would be to stay with the original index (scoreIdx), which does sort by score in descending fashion, but only add to scoresFromDb those values that have the correct skill value. A simple if condition (if (cursor.value.skill == skillLevel)) prior to pushing the value to scoresFromDb would suffice.
Quotas and Other Details
Hopefully, the sample scenario here has provided some insight into how you might leverage IndexedDB in your own applications, games or otherwise. Keep in mind that it is just one of several storage options for JavaScript Windows Store applications, and each has its own unique characteristics and constraints. For IndexedDB in particular note:
- IndexedDB is implemented via an Extensible Storage Engine (ESE) database which uses ISAM storage semantics and runs in a separate process, communicating with your application via RPC.
- The databases managed by the application can only be removed if the application exposes that functionality or if you uninstall the application from your machine.
- Each Windows Store application has a quota of 250MB of IndexedDB storage that it can access; furthermore, IndexedDB across all Windows Store applications combined is limited to
- 375MB if the hard drive of the device is less than 30GB; otherwise,
- 4% of the disk size or 20GB, whichever is smaller
- Note too there is no API to determine how much space remains, so you need to code defensively by handling the quotaExceededError exception.