Last week, we posted about an issue that might cause your Live Mesh Folders to either appear to have missing files, or have unexpectedly large numbers of conflicts. We mentioned that we were testing a fix and we’d update you as soon as possible…and then it got quiet here for a bit. The good news is that we’ve made a lot of progress since then, so here’s an update.
The root cause of the issue (clients not synching properly after encountering too many tombstones in a row) is fixed. We actually put this fix into production early last week, and then watched to make sure tombstones were no longer causing any issues. It’s been a week now, and we’re feeling good that we have this particular tombstone issue fixed. We grudgingly admit that we are not perfect, and there might be other bugs left in our Tech Preview that cause incorrect conflicts or synch behavior – keep reporting issues that you see, and we’ll keep fixing them.
Of course we also want to help users who previously hit this bug and have folders that are in a confused state. If you were hitting the issue where folders were only partially synchronizing, you should have already seen this fixed – the first time your client software contacted the server with the fix in production, the server should have returned the complete file list.
If you hit the issue where a folder had a large number of conflicts, we have good news for you here as well. This particular tombstoning issue affects folders in a very distinct way, since it ends up creating conflicting items whose names and file contents are identical. Conveniently, this also means that it’s fairly easy to automate clean up of such a folder. Since we can tell that there are no real conflicts (that is, the files are in fact identical as determined by hashing file contents), we can remove any conflicts and just keep one copy of each file. The net result is that we’ve written a tool which can identify folders affected by this particular bug, and then automatically clean them up. The tool runs entirely in our datacenter, it never actually accesses file contents (we create/store the hash on upload), and it never exposes any of the filename data outside the datacenter. We can ask the tool to give us back a list of LiveIDs of users who have affected folders, and then we can tell the tool to go fix all the folders that a given LiveID owns (if you create a folder, Live Mesh treats you as the owner.)
This tool should fix many of the conflicts created by this tombstone issue, but not all of them. If your folders have been out of synch, and you’ve been making changes to files on multiple PCs, well then you have actually generated a real conflict (two different copies of the file exist), and it will take your attention to decide which copy to keep.
Here’s the announcement about the tool that we’re posting to the forums with details on how to get your affected folders cleaned up:
Last week we posted a sticky in our forums containing the details of a recently discovered issue which has resulted in some Live Mesh folders being only partially synchronized. To better serve our customers, we have analyzed our storage service to identify folders that might have been impacted. If our analysis shows that you have a folder that might have been affected by this issue, resulting in either a large number of conflicts associated with your folders or fewer than expected files, you will be receiving an e-mail from us within the next three days asking whether you would like us to perform an automated clean up tool on our servers that will resolve spurious conflicts and retrieve missing files. Further details about this process will be included in that e-mail. Note that while this tool can greatly assist those with large numbers of affected folders, we cannot guarantee that all files and folders will be restored to their original state.
We will also soon be posting steps for performing a manual recovery to help those of you who have a small number of affected folders.
Finally, please remember that Live Mesh is currently a Tech Preview and that you should always backup any data stored in Live Mesh.
UPDATE: We have posted instructions for how to manually clean up affected folders here.