Change tracking Libraries in Universal Windows Apps

  • Article

December 2016: This content is out of date for Windows Anniversary Update and later

I've been working to make change tracking easier, and it shipped in the Windows Anniversary Update. There is an MSDN Magazine article and new blog post with all the details. I'm going to leave this post as a historical reference how how hard it used to be.

Many developers have been looking for a way to bring the change tracking power of Win32 apps to universal apps. Whether it be for uploading user's files to a cloud backup or for preparing the in app experience ahead of time, apps need to know about changes to the users' files.

In Windows 10 it is easier than ever to track changes to a user's files and keep up-to-date even when your app isn't running.

Why Change Tracking?

An app could want to track changes to a user's files for a number of reasons, but the most common ones are:

  • Being notified when a picture is taken to do some post processing
  • Being notified when a file is modified to sync the changes to the cloud
  • Being notified when media is downloaded so it is ready to play in your app right away

All of these scenarios and more can be accomplished in Windows 10 with ease. And we are so confidant of this that we built the OneDrive, Photos, and Xbox Music applications using the very same APIs and techniques that I will outline below.

How we make it easier in Windows 10

There are two new things that will make change tracking easier than ever for you in Windows 10. First of all, the indexer and Windows.Storage.Search are enabled on all device families. This means developers are going to have access to the same powerful searching engine that powers Cortana on everywhere.

Secondly the new StorageLibraryContentChangedTrigger will allow apps to be activated in the background whenever something changes in a library. By leveraging this your app will be notified whenever there is a change in a library, even if it isn't running.

Sample Walkthrough

We are going to walk through a quick example of how an application can change track. In this case we are only going to be looking for files that are new or modified. To do this, there are 3 steps:

  1. Snapshot the current state of the file system
  2. Register the background task to be run when there is a change
  3. Track changes in the file system since the last snapshot

The sample will only be adding the files into an in app database, but your app can do a number of different things with the files that is finds.

This is the same method that is used by OneDrive and the Photos on Windows 10, although I've simplified the code somewhat to factor out their business logic.

Manifest Changes

The sample app is also going to need the picturesLibrary capability added to the manifest so let's do that first. You can either use the designer in VS, or if you love editing XML, add the line:

     <uap:CapabilityName="picturesLibrary"/>

You'll note that the namespace has changed for UWAs, but the capability has the same name as in Windows and Windows Phone 8.1.

First Run Grovel

The very first time your app runs, it is going to have to find the current state of the file system. In a perfect world, the file system would be completely empty the first time your app runs, but the real world is rarely that nice.

What we're going to do is first find all the files in the library that we are interested in. We will also have to track the time of the snapshot since step 3 will rely on this information.

First let's grab the current time, so that we know when the snapshot was taken

 lastSearchTime = DateTimeOffset.UtcNow;
//Store it in the application settings so the background task can read it
//We are not checking if there is an existing value here because it is 
//going to trample the DB entirely
var appSettings = ApplicationData.Current.LocalSettings;
appSettings.Values["LastSearchTime"] = lastSearchTime;

This is done first so that if there is a change to the file system while the rest of the code is running, we will pick it up in the change tracking later.

Next, I create a query for all files in the pictures library that my app is able to handle.

 StorageFolder photos = KnownFolders.PicturesLibrary;
//supportedExtentions is a list of file extensions my app supports
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//We are going to use indexer only properties for the query later            
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);

Now that all the files are included in the query there is an optional step. You could attach an event handler to the ContentChanged event on the resultSet object. This isn't strictly necessary in this case because we've already saved the time, so the future change tracking will include changes that happen while this query is open.

Next I'm going to enumerate the files and add them to an in app database for tracking.

 uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;
for (; files.Count != 0; 
  files = await resultSet.GetFilesAsync(currentIndex,stepSize), currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
      addOrUpdateDatabase(file);
  }
}

There is a very important note here: NEVER CALL GetFilesAsync() ON A DIRECTORY YOU DON'T CONTROL!! At this point the app doesn't know if there are 10 or 10 million files in the pictures library. If the user has a lot of files in the directory, creating all those StorageFile objects is going to cause the system to run out of memory.

The full function to copy and paste is here:

 lastSearchTime = DateTimeOffset.UtcNow;
//Store it in the application settings so the background task can read it
//We are not checking if there is an existing value here because it is 
//going to trample the DB entirely
var appSettings = ApplicationData.Current.LocalSettings;
appSettings.Values["LastSearchTime"] = lastSearchTime;
StorageFolder photos = KnownFolders.PicturesLibrary;
//supportedExtentions is a list of file extensions my app supports
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//We are going to use indexer only properties for the query later            
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);

uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;
for (; files.Count != 0; 
  files = await resultSet.GetFilesAsync(currentIndex,stepSize), currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
      addOrUpdateDatabase(file);
  }
}

 

Now we have all the files currently on the disk tracked in the app database, let's set up the app to be trigged when something changes.

Registering for StorageLibraryChangedTrigger

By registering for the StorageLibraryChangedTrigger, your app will be activated in the background whenever something changes in the library you are interested in. The registration is the same as for other background tasks, so we are going to walk through it really quickly.

If you want more details on how background tasks are registered, you can see overview or the quickstart article.

Manifest Registration

Your app will need to register for the general background task type using either the designer or entering the raw XML.

 <Extensions>              
  <ExtensionCategory="windows.backgroundTasks"
  EntryPoint="StorageLibraryChangedBackgroundTask.StartupTasks">
      <BackgroundTasks>
        <TaskType="general"/>
      </BackgroundTasks>
    </Extension>
</Extensions>

The entry point is going to change depending on the name of your background task.

Registering the Task in Code

Again the code registration is very similar to the registrations for the other task types, so we are going to focus on the unique parts.

First you are going to want to make sure that your task hasn't been already registered

 foreach (var cur inBackgroundTaskRegistration.AllTasks)
{
  if (cur.Value.Name == backgroundTaskName)
  {
    //already registered the task
    return;
  }                
}

Next your application is going to have to get access to the libraries that it wants to change track and create the appropriate trigger.

 var library = awaitStorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures);
var trigger = StorageLibraryContentChangedTrigger.Create(library);

Remember to make sure that you have added the correct library capabilities in the manifest before requesting access.

Note, change tracking is only available in the libraries, but your application can track as many libraries as it would like. In this case I'm only going to track the pictures library because my application is only able to handle images. This includes pictures that are coming from the camera on devices that have one.

Finally I'll request access for the background task run and register it.

 var access = awaitBackgroundExecutionManager.RequestAccessAsync();
if (access == BackgroundAccessStatus.Denied)
{
  //If you hit this, make sure the manifest registration is correct
 return;
}
string taskEntryPoint = typeof(StorageLibraryChangedBackgroundTask.StartupTasks).ToString();
var builder = newBackgroundTaskBuilder();
builder.Name = backgroundTaskName;
builder.TaskEntryPoint = taskEntryPoint;
builder.SetTrigger(trigger);
BackgroundTaskRegistration task = builder.Register();

And that is it. The background task will now be activated anytime that something changes in the pictures library.

Tracking Incremental Changes

Now that your app is going to be notified when something is changed we need to be able to find out what file changed. In this case we are only going to track additions and modifications, while ignoring deletions.

The tracking is done from a background task so that it can be run when the app isn't in the foreground. The background tasks can be triggered when the app is in the foreground though, so be aware of potential concurrency issues when you are writing your own app.

The first thing that we are going to do when the app is triggered is to hold the deferral (since there will be async operations run in the task) and retrieve the last search time from the application settings.

 var appSettings = ApplicationData.Current.LocalSettings;
if (!appSettings.Values.ContainsKey("LastSearchTime"))
{
 //We haven't done a first run crawl, so lets just bail out
  return;
}
DateTimeOffset lastSearchTime = (DateTimeOffset)appSettings.Values["LastSearchTime"];

In this case, I'm electing not to do a first crawl in the background since it is going to be too resource intensive for a background task.

Next we are going to create the query that will find the items which have been changed since the last time the app was able to crawl. To do the filtering we will leverage the System.Search.GatherTime property from the Windows Property system. This property is an indexer only property that tracks the last time that the file was modified, created, or moved.

To use this property there are two important preconditions. First of all, the query must be set to use the indexer with the IndexerOption.OnlyUseIndexer property. Without the indexer this query will not work because System.Search.GatherTime isn't a property of the file, it is a calculated property stored by the indexer.

The other important note is that dates in the indexer are stored using Zulu time format. Thus, the ApplicationSearchFilter will have to be created with the lastSearchTimeConverted to Zulu time.

 StorageFolder photos = KnownFolders.PicturesLibrary; 
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//This is important because we are going to use indexer only properties for the query
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
//Files that have been modified or created since the last search time. The date time 
//format being used is Zulu
string timeFilter = "System.Search.GatherTime:>=" + 
    lastSearchTime.ToString("yyyy\\-MM\\-dd\\THH\\:mm\\:ss\\Z");           

option.ApplicationSearchFilter += timeFilter;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);
lastSearchTime = DateTimeOffset.UtcNow;

Once we have the query in hand, we can reuse the same code from the first run grovel to check for new or modified files.

We are ignoring the case where files may be deleted and not show up in the query, since the app is only interested in new files. If your app is interested in files that have been deleted, it will have to compare a full query of the location with the results in its database.

 uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;


for (; files.Count != 0; files = await resultSet.GetFilesAsync(currentIndex, stepSize),
                            currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
    //Note we are assuming that the database will handle any concurrency issues
      addOrUpdateDatabase(file);
  }
}

And on the way out, release the deferral and store the last time that we completed a successful search for the next time that the background task is triggered.

 appSettings.Values["LastSearchTime"] = DateTimeOffset.UtcNow;
deferal.Complete();

The full function is below for those who are looking for something to copy and paste:

 public async void Run(IBackgroundTaskInstance taskInstance)
{
  BackgroundTaskDeferral deferal = taskInstance.GetDeferral();

  var appSettings = ApplicationData.Current.LocalSettings;
  if (!appSettings.Values.ContainsKey("LastSearchTime"))
  {
    //We haven't done a first run crawl, so lets just bail out
    return;
  }
  DateTimeOffset lastSearchTime = (DateTimeOffset)appSettings.Values["LastSearchTime"];


  StorageFolder photos = KnownFolders.PicturesLibrary;
  QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, 
                                          supportedExtentions);
  //This is important because we are going to use indexer only properties for the query
  //later
  option.IndexerOption = IndexerOption.OnlyUseIndexer;
  option.FolderDepth = FolderDepth.Shallow;
  //Set the filter to things that have changed since the lastSearchTime. Note that the 
  //time must be formatted as Zulu time as shown here.
  string timeFilter = "System.Search.GatherTime:>=" + 
                    lastSearchTime.ToString("yyyy\\-MM\\-dd\\THH\\:mm\\:ss\\Z");
  option.ApplicationSearchFilter += timeFilter;


  StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);
  lastSearchTime = DateTimeOffset.UtcNow;
            

  uint currentIndex = 0;
  const uint stepSize = 10;
  IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex,
      stepSize),
  currentIndex += stepSize;
  
  for (; files.Count != 0; files = 
      await resultSet.GetFilesAsync(currentIndex, stepSize),
      currentIndex += stepSize)
  {
    foreach (StorageFile file in files)
    {
    //Note we are assuming that the database will handle any concurrency issues
      addOrUpdateDatabase(file);
    }
  }
  //Update the last time that we searched for files
  //Edit 2/9/16 fixed a race condition, see comments for details
  appSettings.Values["LastSearchTime"] = lastSearchTime;
  deferal.Complete();
}

Conclusions

Using the methods above an app is able to leverage the same change tracking technology that is used by first party experiences. It allows your app to be up to date with the latest changes on the device so your users can jump right into being delighted as soon as your app is opened.

 

Edit: (10/26)

We have discovered an issue where sometimes app registrations for the notification are getting lost if the app or machine were not shut down properly. As a workaround, please check the registration for the background task every time your app starts up and re-register as needed.