Change Tracking: More Betterness

In a couple previous posts on this blog we have covered how a UWP can track changes that are happening in a user's library. Although there were notifications and some apps could enable the scenarios they are looking for, the code required was still complicated. We heard a lot of feedback from developers that it could be a made simpler and took it to heart. So for the Windows 10 Anniversary Update I'm happy to announce the StorageLibraryChangeTracker.

The StorageLibraryChangeTracker is a new feature that is going to allow apps to easily track files and folders as users move them around the system. Using the StorageLibraryChangeTracker, an app can now easily track:

• File operations including add, delete, modify
• Folder operations such as renames and deletes
• Items moving through the system to differentiate between a move within a library, a move outside a library and a delete.

Since the change tracker tracks folder level operations and file deletions, it greatly reduces the effort for an app to track changes in a library.

In this blog post we are going to briefly walk through the programing model for working with the change tracker, a brief outline of some sample code, and the different types of file operations that are tracked.

Using the Change Tracker

The change tracker is implemented on system as a circular buffer storing the last N file system operations. Apps are able to read the changes off of the buffer and then process them into their own experiences. Once the app is finished with the changes it marks the changes as processed and will never see them again

Enabling the change tracker

The first thing that the app needs to do is to tell the system that it is interested in change tracking a given library. It does this by calling Enable() on the change tracker for the library of interest

StorageLibrary videosLib = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos); StorageLibraryChangeTracker videoTracker = videosLib.ChangeTracker; videoTracker.Enable();

A few important notes:

• Make sure your app has permission to the correct library in the manifest before creating the StorageLibrary object
• Enable is thread safe, will not reset your pointer, and can be called as many times as you'd like (more on this later)

Waiting for changes

Once the change tracker is initialized it will begin to record all of the operations that occur within a library, even while the app isn’t running. If you'd like your app's background task to be activated when changes happen, please see the StorageLibraryChangedTrigger or my blog post.

Reading the Changes

The app can then poll for changes from the change tracker, and receive a list of the changes since the last time the app checked. The code to get a list of changes is below, but we will cover how to understand those changes in another section.

StorageLibrary videosLibrary = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos); videosLibrary.ChangeTracker.Enable(); StorageLibraryChangeReader videoChangeReader = videosLibrary.ChangeTracker.GetChangeReader(); IReadOnlyList changeSet = await changeReader.ReadBatchAsync();

The app is now responsible for processing the changes into its own experience or database as needed.

Pay special attention to the second call to enable here, this is to defend against a race condition if the user adds another folder to the library. Without the extra call to Enable(), the code will work in most scenarios, but if the user is changing their library folders it will fail with the exception ecSearchFolderScopeViolation (0x80070490).

Accepting the changes

Once your app is done processing the changes that have been delivered to it, it should tell the system to never show it those same changes again. This is done by calling:

await changeReader.AcceptChangesAsync();
Note:

• If changes have happened between calling ReadBatchAsync and AcceptChangesAsync, the pointer will be only be advanced to the most recent change the app has seen. Those other changes will still be available the next time it calls ReadBatchAsync
• Not accepting the changes will cause the system to return the same set of changes the next time the app calls ReadBatchAsync.

Important Things to Remember

When using the change tracker, there are a few things that you should keep in mind to make sure that everything is working correctly.

Buffer Overruns

Although we try to reserve enough space in the change tracker to hold all the operations happening on the system until your app can read them, it is very easy to imagine a scenario where the app doesn't read the changes before the circular buffer overwrites itself. Especially if the user is restoring data from a backup or syncing a large collection of pictures from their camera phone.

In this case a special error will be returned from ReadBatchAsync, StorageLibraryChangeType.ChangeTrackingLost. If your app receives this error code, it means a couple things:

1. The buffer has overwritten itself since the last time you looked at it. The best course of action is to recrawl the library, since any information from the tracker will be incomplete
2. The change tracker will not return any more changes until you call Reset(). Once the app calls reset, the pointer will be moved to the most recent change and tracking will resume normally.

It should be rare to get these cases, but in scenarios where the user is moving a large number of files around on their disk we don't want the change tracker to balloon and take up too much storage. This should allow apps to react to massive file system operations while not damaging the customer experience in Windows.

Changes to a StorageLibrary

StorageLibraries are an interesting concept in UWPs. They exist as a virtual group of root folders that may contain other folders. To reconcile this with a file system change tracker, we made the following choices:

1. Any changes to descendent of the root library folders will be represented in the change tracker
   • The root library folders can be found using StorageLibrary.Folders
2. Adding or removing root folders from a StorageLibrary (through RequestAddFolderAsync and RequestRemoveFolderAsync) will not create an entry in the change tracker
   • These changes can be tracked through the DefinitionChanged event on StorageLibrary or by enumerating the root folders in the library using StorageLibrary.Folders.
   • This will also mean that if a folder with content already in it is added to the library, there will not be a change notification or change tracker entries generated. Any subsequent changes to the descendants of that folder will generate notifications and change tracker entries.

Calling Enable()

It’s mentioned above, but just to make sure that it is clear: Apps should call ChangeTracker.Enable() as soon as they start tracking the file system and before every enumeration of the changes. This will ensure that the change tracker is not going to miss changes to the folders included in the library.

Putting It all Together

Here is all the code that is used to register for the changes from the video library and start pulling the changes from the change tracker.

 private async void EnableChangeTracker()
{
    StorageLibrary videosLib = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
    StorageLibraryChangeTracker videoTracker = videosLib.ChangeTracker;
    videoTracker.Enable();
}

private async void GetChanges()
{
    StorageLibrary videosLibrary = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
    videosLibrary.ChangeTracker.Enable();
    StorageLibraryChangeReader videoChangeReader = videosLibrary.ChangeTracker.GetChangeReader();
    IReadOnlyList changeSet = await changeReader.ReadBatchAsync();


    //Below this line is for the blog post. Above the line is for the magazine
    foreach (StorageLibraryChange change in changeSet)
    {
                    
        if (change.ChangeType == StorageLibraryChangeType.ChangeTrackingLost)
        {
            //We are in trouble. Nothing else is going to be valid. 
            log("Reseting the change tracker");
            videosLibrary.ChangeTracker.Reset();
            return;
        }                    
        if (change.IsOfType(StorageItemTypes.Folder))
        {
            await HandleFileChange(change);
        }
        else if (change.IsOfType(StorageItemTypes.File))
        {
            await HandleFolderChange(change);
        }
        else if (change.IsOfType(StorageItemTypes.None))
        {
            if (change.ChangeType == StorageLibraryChangeType.Deleted)
            {
                RemoveItemFromDB(change.Path);
            }
        }
    }
    await changeReader.AcceptChangesAsync();
}

Sorting the changes

The file system can be a complicated beast, and there are many different ways that users or programs can change how it will appear to your app. To help make it easier to understand what is going on, the change tracker provides two pieces of information to your app - the type of item that was changed, and the change that occurred.

The following code shows an example of how the changes might sorted in an app that is going to be maintaining a cache of the state of the file system. (Yes, the indexer team does write indexers for their indexer code samples - you really don't want us writing anything else)

 private async Task HandleFileChange(StorageLibraryChange change)
{
    //Temp variable used for instantiating StorageFiles for sorting if needed later
    //We are only going to create the StorageFile if needed to save the overhead
    StorageFile newFile = null;
    switch (change.ChangeType)
    {
        case StorageLibraryChangeType.Created:
        case StorageLibraryChangeType.MovedIntoLibrary:
            if (IsPathInAppDB(change.Path))
            {
                //Should never hit this, but check to make sure our sync isn't busted
                RemoveItemFromDB(change.Path);
            }

            //Need to filter the item to make sure that it is going to be something we are interested in
            newFile = (StorageFile)(await change.GetStorageItemAsync());
            if (newFile.ContentType.Contains("audio") || newFile.FileType == ".mp4")
            {
                AddItemToDB(newFile);
            }
            break;

        case StorageLibraryChangeType.Deleted:
        case StorageLibraryChangeType.MovedOutOfLibrary:
            if (IsPathInAppDB(change.Path))
            {
                //If we are tracking the file then lets remove it from the database
                RemoveItemFromDB(change.Path);
            }
        break;

        case StorageLibraryChangeType.MovedOrRenamed:
            if (IsPathInAppDB(change.PreviousPath))
            {
                MoveExistingItemInDB(change.PreviousPath, change.Path);
            }
            else
            {
                //Need to filter the item to make sure that it is going to be something we are interested in
                newFile = (StorageFile)(await change.GetStorageItemAsync());
                if (newFile.ContentType.Contains("audio") || newFile.FileType == ".mp4")
                {
                    AddItemToDB(newFile);
                }
            }
        break;

    case StorageLibraryChangeType.EncryptionChanged:
    case StorageLibraryChangeType.IndexingStatusChanged:
    case StorageLibraryChangeType.ContentsReplaced:
        if (IsPathInAppDB(change.Path))
        {
            UpdateExistingItemInDB(change.Path);
        }
    break;

    default:
    return;
    }
}

Now let's walk through what is going on in the code and each of the possible states of a StorageLibraryChange.

Types of Items in a StorageLibraryChange

The types of items that can be changed are by calling StorageLibraryChangeType.IsOfType(StorageItemTypes). There are 3 options that are possible here:

Your app should be able to handle all 3 types of items being returned, or at least fail gracefully if a type is returned that it isn’t expecting.

Types of Changes in a StorageLibraryChange

There are 10 different values of StorageLibraryChangeType that can be returned to your app. Each change type gives enough information about a change to the files system for your app to know what item is being changed and to find the new location of it on disk.

MovedOutOfLibrary/MovedOutOfLibrary

These are going to be returned as best as possible by the platform. In cases where we cannot determine if the destination or origin is in a library, it will return MovedOrRenamed.

The theory is that it would be great to have all the information about if the file was moved out of the library right away, but sometimes it isn't possible to calculate that. In those cases, your app will still be able to see the change and react accordingly.

Coalescing the change notifications

It is possible that between two subsequent reads of the change tracker by an application, there may be a number of operations to a single file. If possible the change tracker will attempt to reduce the load on the application by simplifying the operations. For example, an add then move operation will be combined to a single add operation in the file’s final location. In cases where it isn't possible to simplify the operations they will be presented separately, for example if there was a move then a modify operation.

Finishing Up

And that's it. Your app can now track all the changes that are happening in a given library and find out exactly where the files are going. I'm looking forward to seeing all the new apps that are built using the change tracker and the exciting experiences they bring to users.

Happy tracking