Occasionally Connected Systems Sync Framework


Recently I’ve been giving some presentations on our upcoming sync framework.  This new sync framework is designed to fill the void between the simplicity of RDA (Remote Data Access), full featured, but complex, Merge Replication, and raw components of DataSet.  This new sync framework is designed to work over services, 2 tier and work with any back end system. 


Because a picture is worth a thousand words, and a video is worth a book, I’ve posted my deck and a screen cast to demonstrates the features we’re working on.


OCS Power Point Deck
Screen Cast of the new Sync Framework demonstrating the architecture and details of download scenarios


Steve

Comments (45)

  1. As you may be aware, after all I have given some presentations about it, I like SQL Everywhere and its

  2. mauricedb says:

    Hi Steve,

    Nice job 🙂

    Let me know when you start sending out CTP bits. I would very much like to give them a try.

    Maurice

  3. Steve Lasker has posted a good screencast that walks through how new sync framework will work. …

  4. Troy says:

    Will OCS Sync Framework support column-level change detection as an option, or does it deal with rows only?

  5. Steve.Lasker says:

    Hi Troy,

    Unfortunately, no, not in the first release.  There are a few features we just didn’t have the time to complete in the short Orcas cycle.  However, just as we’re shipping SSEv frequently, we’ll be doing the same for the sync runtime.  

    To respond to several other questions that I’ve received in emails:

    The server side components will work with any relational database that has an ADO.net provider

    Because the sync model is highly componentized, you can sync with arbitrary services on any platform.  We’ll ship a web service proxy and service, but it will be trivial to add a WCF service and proxy.  Likewise, we have samples that sync via RSS.

    We hope to have a CTP available later this year.  It will be part of the Orcas Beta, but our sync framework is based on FX 2,0, so while we want developers to adopt 3.0/5, it’s not a requirement to use the sync framework.

    Steve

  6. Troy says:

    So it sounds like this feature "column level change detection" is on the roadmap!  Cool!

    I currently use SQL Server 2005 merge replication with SQL Express subscribers for remote offices (and it works great!).  I’ve been considering using SQL Server merge replication as a future solution for SSEv for mobile users.  But using the OCS sync framework is intriguing me.  It appears as if much of the functionality is based on the same code (if only in concept) from Access and/or SQL Server merge replication.  It seems to require you to get your hands a little dirtier setting it all up, but then it allows much greater control and flexibility.  On the one hand, I like the "black box" of merge replication, but on the other hand, I like having more control (even with the learning curve).

    In your OCS Power Point Deck, you have a slide called "Synchronization Options" that compares merge replication and OCS Sync.  I’d love a more detailed explanation of the differences betweens merge replication and OCS sync (even within a SQL Server only environment).

  7. You mentioned that it can be used to sync services.  Do you have any way of doing change detection of the data coming from these web services or is it up to the developer to build this into the web services?  

  8. Steve has a great post (with Video and slides) for building offline apps… Worth a look! My guess is lots

  9. Steve.Lasker says:

    Troy,

    You’re trade-off example of OCS and merge are accurate.  We view Merge as our Replication Product that has lots of features and lots of “we’ll take care of it” level of functionality.  While it’s very feature rich, including the ability to put business rules inline, it does run as a black box with lots of cubbies to place things in.  In the end, Merge Replication will replicate your database and will keep things absolutely accurate.  To keep that level of accuracy, we don’t expose as many knobs.

    OCS is much more a componentized model, exposing just about everything for you to have ultimate power to synchronize data, not replicate a database.  This gives you a lot more power, but also means you can easily confuse the system and wind up with strange results.  The sync designer and sync command builder will do a great job to keep the safety on for you.  

    OCS/Merge comparison.  I’m sure Michael, our documentation guru, will have a bunch of info comparing Merge with OCS.  These runtimes are being built by the same team, so we’re getting the depth of experience incorporated into the OCS sync runtime.

    Replication Person

    I like to say we don’t have pixie dust in our sync runtime.  The sort of stuff you sprinkle on the black box to make it work.  This means we can’t synchronize arbitrary web services that don’t expose some level of change tracking information.  However, we do have a SnapShot mode where the sync runtime can take the results of a web service and continually stuff them in the local database.  Lets say you have a States service.  The list of states aren’t that big, and don’t change that often.  Rather then ask the developer who owns the service to add functionality, you could simply call the States service and just keep replacing the local values.  This would still be better then calling the service for every combo box, or caching the data in memory.  Each time the app starts up, or once a week, SnapShot the list of states, and use the local persistence as your cache.

    Steve

  10. Karl says:

    Steve,

    where and how does conflict resolution come into play here?

    Conflicts cannot be avoided in disconnected scenarios with fully bidirectional sync:

    – client A makes some changes while offline

    – client B syncs conflicting changes (w.r.t. A’s local changes) to server

    – client A back online, tries to sync

    – conflict has to be resolved

    As I understand it, the server sync agent would have to detect stale writes / conflicts (on a per row basis by means of a revision number, for example) and report them back to the client (both the client’s values and current server values) for resolution.

    Karl

  11. Donta says:

    Very interested in the testing the framework. How do I go about getting the bits? I have a project that could benefit from this type of synchronization. Also, will this work on the compact framework? I have another project looming for a PPC device.

    Thanks, Donta

  12. Steve.Lasker says:

    Hi Karl,

    Yes, these are the scenarios we’ll support.  We’ll support resolving the conflicts on the server or the client.  Yes, you’ll have both the client and the server row available to determine how you want to handle the conflict.

  13. Steve.Lasker says:

    Hi Donta,

    We’re hoping to have CTP ready bits available in the November/December timeframe.  I’ve been keeping a collection of the emails people have been sending me, so please use the Contact link above to send me the request.  When we’re ready, I’ll be sending emails to those that have asked and we’ll get started.

    For Compact Framework, as much as I hate to say it, we won’t have CF support in the first release.  The Compact Framework team just had too much to get done to take on the work.  Because we will be shipping the OCS and SSEv runtimes out of band from the .NET FX and .NET CF frameworks we have a lot more flexibility.  We hope to ship the .NET CF versions shortly after Orcas.

    We are doing the engineering work to make sure everything is designed to work on .NET CF.  Meaning, we’re testing the overloads and making sure the types we use are available on .NET CF, but the "taxes" we pay to fully support a platform are just beyond the scope of what we could finish in the Orcas timeframe for devices.

    Steve

  14. WillSmith says:

    Does this OCS framework work with a client using a locally installed SQL Express DB?

    Thanks.

    Will

  15. Steve.Lasker says:

    Hi Will,

    I’m assuming you’re asking about Express being installed locally.

    Not initially.  Our initial focus is for SSEv as we’re focused on the breadth of client apps and we see SSEv having a much broader adoption curve.  In order to abstract the change tracking details on the client, we require the local store to have knowledge for change tracking.  SSEv will ship a new release on, or before Orcas.  When the next release of SQL Server ships, code named Katmai, SQL Express will be supported and another release of SSEv will be available with additional functionality.

    Steve

  16. gorshing says:

    I am curious what application you used to create the screencast?

  17. Steve.Lasker says:

    I’ve been using Camtasia: http://www.techsmith.com/camtasia.asp  It’s a great product.

    Steve

  18. SQL Server Everywhere gets a new name

  19. jsh_ec says:

    Steve,

    Will SQLce still work ‘out of the box’ with SQL2000? I have never got SQL2005 working reliably. It always fails with IPC issues which require an OS reinstall. The current CTP works well with SQL2000.

    I would be very interested in looking at the new conflict resolution in 3.5. Please could you count me in on the CTP.

  20. Steve Lasker has posted a great whitepaper that describes the differences between SQL/e and SSCE. …

  21. Matthew Phillips says:

    Hi,

    Following your presentations on the forthcoming sync services at TechEd:Developers EMEA, I’m very much looking forward to taking advantage of it.

    I’d be grateful if you could include us in the CTP/early programs

    thanks

    Matthew Phillips

  22. On Wednesday morning Steve Lasker delivered his second public webcast (the first…

  23. billb59 says:

    Steve… very nice… do you have the source you used for the demo posted anywhere? Keep up the awesome work… bill

  24. Derek says:

    Steve, do you have an update on when the CTP bits are going to be available? Thanks…derek

  25. Tras semanas de cacharreo con la beta 2 del Visual Studio "Orcas" decidí sacar un pequeño seminario

  26. Troy says:

    Hi,

    When syncing from Sql2005 (full or express) How does the sync framework handle data types that do not exist in the compact framework, such as Xml, varchar(max), nvarchar(somevaluegreaterthan4000) etc ?

  27. Steve.Lasker says:

    Hi Troy,

    We use the same mapping as Merge Replication.

    XML and VarChar(Max) become nText data types.  

    With Sync Services you have a bit more control.  If you pre-create the table, and the data "fits" we’ll continue to insert/update the data.  For instance, if you sync an int into a nVarChar column on SQLce, it will continue to work.  Of course if you put Six in the local table, it will fail going up.  We don’t do mapping from english to decimal <g>

    Steve

  28. Cette semaine, à Barcelone, se tient le plus grand événement européen des développeurs sur les technologies

  29. Cette semaine, à Barcelone, se tient le plus grand événement européen des développeurs sur les technologies

  30. Reka Kovacs says:

    hello Steve,

    As far as I seen on the web, I miss some codes/samples/detailed description regarding 3 topics in Sync Framework:

    1.mixing the synchronization – like synchronizing db to file system or vice versa

    2. Implementation of custom providers

    3. Sample code using the rss synchronization.

    You’ve mentioned above having some samples using the RSS synchronization. Can you share this code?

    Thank you in advance!

    Reka K.

  31. EQ says:

    Is it possible to use Sync to temporarily store Client-side text data from a Web Application that goes offline?  Google Gears solves this problem, and while its seems to me that Sync supports Client/Server sync and offline storage, it doesnt seem that this is the case for web applications.  Is this right?

  32. Balaji says:

    Hi,

      I would like to develop the sample web application by using Sync Framework. Is there is any sample application available. And can you please guide me about how to develop the application.

  33. Steve.Lasker says:

    Balaji,

    When you say "web application", are you referring to server side, or client side?

    Steve

  34. swineetha says:

    Hi,

    I would like to develop a Web application(Not a windows).I would like to enable the application to be available in offline mode.My exact requirement scenario would be,

    I have a web application with a web server.

    I need to sense the network ,whether it is offline or online.

    Then in online mode i can enter data directly to the server(this needs to be in sync with my local cached data).

    When i am unable to connect to the server,i should be able to access the application and save the data locally and sync them when i am connected back to server.

    I hope this would be definitely possible with MS Sync.

    But on browsing through various forums i got some sample applications which are windows application.

    But mine would be a web application.

    I sit possible with MS Sync and if yes is there any sample application available?Please redirect me in correct direction .

    Thanks in advance.

  35. swineetha says:

    Missed to mention some thing,

    I got a sample console application for bidirectional sync using MSF.I tried with that and the console application worked fine.I tried to convert it into a web application.

    i.e in my local machine i had .sdf file created through MS SQL Server CE and a sql server 2005 as the remote server.Data sync was perfect through that console application.But when i tried to execute my WEB application i got an exception message stating "SQL Server compact edition is not intended for ASP.Net application development".

    I was using SQL server Compact edition 3.1 with my VS2005.I sthe mistake with thw version or I can not use SQL Server CE with a web application and only with a windows application.

    Please guide me in this.

  36. Balaji says:

    Hi Steve,

      Thanks for your reply. I need the web application which is working both online and offline(When network is not available). My web application has asp 2.0 in the front end and sql server 2005 in the back end. Please guide me how MS synch sense the network and synch features.

  37. swineetha says:

    I have a clarification regarding MSF.Is MSF targetted only for desktop application?I have a requieremnt for making my web application to work offline.So,that user can access the site uninterrupted,inspite of the network connection.On browsing for MSF and with the sample applications it is mentioned as MSF can take data offline and synchronize any kind of data when connected to the server.But,is it possible to render a web page in offline mode?(

    So that when the user is offline data will be captured in the local DB and when he is online data will be captured in the server and in online mode he can sync the data changes in local DB to the server)

    If yes please redirect me to the right place where i could get information on this.

  38. Steve.Lasker says:

    Hi guys,

    Sorry for the delayed response.  I’ve been out of town visiting customers.  I hear you saying “web application” and want to “work offline”.  However, what it appears you’re asking for is for the database to be offline, but the web server is still available.  I don’t think that’s what you mean to ask, but your saying you’re getting an error running Compact under ASP.net.  In this case, you’re actually attempting to run SQL Server Compact on the Web Server, so you’re making an assumption the “browser client” can connect.  

    So, you can enable SQL Server Compact under ASP.net, which I discuss here.  My question is really what is the “client” to your “web application”?  Is the user running in the browser?  Or, is the user running WinForms or WPF on the client, and communicating to the database through a “web server” running “services”?

    For the WinForm/WPF client, you can absolutely run Compact there.  For running code within the local browser, we don’t yet have anything to announce here.  You could of course run Compact locally, and host it in an ActiveX control.  

    Steve

  39. Tras semanas de cacharreo con la beta 2 del Visual Studio &quot;Orcas&quot; decidí sacar un pequeño seminario

  40. Avi says:

    I am not been able to download "OCS Power Point Deck", seems it is share with window authentication on.

    How do i go about it ?

    -Avi

  41. I am implementing sync services across WCF, which so far seems to be working well. However, I need to be able to allow the client to resolve conflicts. From what I can see, you would want to subscribe to the ApplyChangeFailed event. That would be fine, but it only exists under the ServerSyncProvider which, unless I am missing something, cannot be accessed client side. I know you can subscribe to this event on the server, from within the WCF application in my case. But handling this server side only limits you as the user would not be able to decide how the conflict should be handled.

    My question is what is the recommended way of handling conflicts client side in an n-tier environment and is it possible to handle the ApplyChangeFailed event client side(clientProvider_applyChangeFailed is firing only at INSERT-INSERT conflicts) ?

  42. Steve.Lasker says:

    Hi,

    All the events you’re looking for exist on the SqlCeClientSyncProvider.  http://msdn.microsoft.com/en-us/library/microsoft.synchronization.data.sqlserverce.sqlceclientsyncprovider.aspx

    You can get this from the SyncAgent.LocalProvider.

    Hope this helps,

    Steve

  43. Haytham says:

    Di Steve,

    Your blog is really helpful and informative, it’s been helping me in the current project i am working on, but i have a question.

    We are developing a web-application that needs to do the following:

    1-local web application on the client deployed on IIS that works on a compact/express SQl database

    2-occasionally this database sync with the main server database (2way)

    The question is:

    1-is it better off using SSIS?

    2-or i can use SyncFramework? though it is a web application and i heard that sdf is not accessed in the iis appdomain.

  44. Hi Steve,

    Can you kindly make a tutorial on how to deploy N-Tier VB applications on different machines.

    This application should be based on VB 2008 and WCF.

    I am struck with the deployment of presentation Tier. It works on same machine, but gives error after installing it on different machine. May be Endpoint Address Issue. but I tried a lot, but couldn't succeed.

    Thanks,

    Akbar