How user profile synchronization works in SharePoint 2010


While we have a bunch of documents (and blogs) on provisioning the user profile synchronization service, configuring it and running it, we don’t address the larger conceptual question of “what’s going on under the hood” when synchronizing data from a directory source into SharePoint. This blog entry is meant to address exactly that question.


But before we dive, let’s first look at how we think of the User in the User Profile Service, and how that leads us to solving this seemingly simple but fundamental need for rich user data in the profile store.


The “User” in the User Profile Service
In a typical business system, a user is determined by the identity presented to the system. Once this identity is confirmed, business system follows a certain set of rules and policies to give appropriate access. But there is rich information about this user’s activity in the system that the corporation overall can benefit from and the user can benefit from. In the User Profile Service, we think of the user at the center of it all as a “person” in the profile store who has a face -through the Profile, My Site, Newsfeed; has a web service that can be crawled – for example for Social search; a store backing the person up and; relationships and a networks in the enterprise.


Need for the User Profile Synchronization Service
With that mindset, that is “person” at the center, as the cornerstone of the User Profile Service architecture, the question becomes, how does one get started in accumulating interesting information about a user in the user profile store. That is where the User Profile Synchronization service comes in. User Profile Synchronization service facilitates bringing user and group information from directory sources, such as Active Directory. If there is additional interesting data in a business system, say employee ID or hire date, a Business Data Connectivity model can be used to augment the user records with this data. SharePoint 2010 utilizes the synchronization engine from Forefront Identity Manager (FIM) 2010. With SharePoint 2010, data can be written back to the directory source (AD or LDAP), if appropriate permissions are present. A popular scenario for this capability is to write back pictures uploaded in SharePoint back to say, Active Directory, which than can be used by Microsoft Outlook.


Inner workings of the User Profile Synchronization Service
User Profile Synchronization service undergoes a series of steps to stage data from the directory source, process it and bring it over to the user profile store.


Before understanding how data flows from the directory source to/from user profile store, let’s look at two key storage areas for the synchronization service:


1.      Connector Space (CS) is a staging area that contains representations of the designated objects from a connected data source and the attributes. Synchronization Service uses the connector space to determine what has changed in the connected data source and to stage incoming changes. Synchronization Service also uses the connector space to stage outgoing changes for export to the connected data source. Synchronization Service maintains a distinct connector space as a staging area for each management agent (connection in SharePoint 2010).


2.      Metaverse (MV) is a storage area that contains the aggregated identity information from multiple connected data sources, providing a single global, integrated view of all combined objects.


 


Now let’s consider a few examples to illustrate the data flow in common scenarios.



Scenario 1 – Full Sync: if Active Directory was the directory source and a full AD synchronization was being performed to bring users, groups and group membership information into the user profile store. 


 


Step 1: Import from Active Directory into Active Directory CS
In this step, data is brought into the Connector Space from the directory source.


Step 2: Import from User Profile Store into the SharePoint CS
Similar to step 1, in this step, data is brought into the Connector Space from the User Profile Store. Note that this step is conditional skipped for the very first full sync but is performed for any subsequent full syncs.


Step 3: Synchronization of data with the MV
This step moves and reconciles the data and its references into the MV. If there was existing data present in the MV, or if there were additional sources of data, for example, multiple forests, this is where reconciling of data and references will occur. This is also the step that transfers data to the SharePoint Connector Space to gets it ready for export to user profile store. 


Step 4: Export of data to user profile store
Once data is available in the SharePoint CS, it can now be moved to the user profile store. Note that some of this data requires further processing, for example, an employee’s manager is a reference to an incoming record, which might or might not have arrived before the employee. So SharePoint stages this data for further post-processing. Similar post-processing is required for groups and memberships.

Export is the most interesting and critical step from a SharePoint perspective, as this is the step that brings the data into the profile store. Before this step is reached, while much work is being done, there is no data flowing into the user profile store.


Step 5: Import and confirmation that the data has been received
Once the data is added to the profile store, a file is created to confirm its receipt back to the FIM synchronization engine. This file is also used to send data back, if there is additional data being sent back to AD, such as a picture.


Step 6: Synchronization of the data with MV
This step is similar in outcome as the previous synchronization Step 2. Now the data is being sent back to the MV from user profile store, including any data needed to be written back to AD. During this step, along with reconciling the data in the MV, any data needing to go back to AD is also prepared and marked ready for export in AD CS.


Step 7: Export to AD
Similar to Step 3, this step carries data back from AD CS to AD. This step will be a no-op if there is nothing being written back to AD.


Step 8: Import and confirmation that the data has been received
Similar to Step 5, this step closes the loop on data sent to AD.


 Scenario 2 – Full Sync with BCS data: if Active Directory was the directory source and along with AD data, additional data from business systems, such as SAP, was being brought in.


Along with bringing in the identity data for the user from a directory source, often additional interesting data might reside in business systems. For example, employee ID might exist in another system, which if brought into the user’s profile, can be used for interesting scenarios, for example combined with the Audience functionality in SharePoint.



While a directory source, such as AD or LDAP, is required to create users (rows) in profile store, additional data (columns) can be brought in using Business Connectivity Service BCS (previously Business Data Catalog) models. Documentation on how to create these models or configure them for synchronization is elsewhere and is beyond the scope of this document. What we describe here is how the data flow works in such scenarios.



For bringing in the data through a BCS model, in addition to the steps outlined in Scenario 1 above, following steps are performed, after Step 8.


Step 9: Import from BCS
BCS Data is joined, using the attribute selected for BCS connection creation, and then the property values for mapped properties are brought into the BCS Connector Space


Step 10: Sync BCS data to MV
BCS Data is synchronized into the MV as well as setup for export from SP CS to the Profile Store. Notice the one-way sync arrow, as the BCS data is only brought into the profile store. We do not write the data back to the respective BCS business system.


Step 11: Export of data to user profile store
Similar to step 4, BCS data is written into the user profile store.


Step 12: Import and confirmation of data receipt
Same as step 5, we close the loop with the SharePoint CS that the data has been received.



Scenario 3 – Incremental Sync: if Active Directory was the directory source, a full synchronization was completed earlier, and an incremental synchronization is being run.


The steps performed for incremental sync are exactly the same as full sync, except wherever a sync or import step were performed, it’s now replaced by Delta sync or Delta import, respectively. Only the changes flow through, not everything. Note that regardless of incremental or full synchronization, BCS synchronization will bring in all user data from the BCS source system.



 Few final thoughts
As discussed here, user profile synchronization is a key element of populating the ‘person’ in your profile store. It’s critical to understand and plan what information resides in the profile store, where it comes from, how it gets there and how it’s maintained. Behind each of these key questions is the complex process of bringing data to-from directory sources such as AD and business systems such as SAP. Our hope is that this discussion helped you gain a better understanding of how it all works and how the data flows, so you can manage and maintain the profile store more effectively.


 

Comments (26)

  1. C Bear says:

    Thanks for the post. This is really helpful.

    A couple of questions:

    1. From your description it sounds like if you want to bring in data via BCS to the user profile store, you must also use AD or another directory for the initial user creation. Is this correct or can you bring in all the data via the BCS?

    2. I was under the impression that the BCS had 2 way sync capability (supposed to be one of the advantages over the BDC) but you state that sync from the BCS data is one way. Can you please clarify?

    Thanks!

  2. SPSocial says:

    Hi Caedmon,

    1. You can’t create new users (rows) through BCS, you can only augment existing user’s properties (columns).

    2. Yes, BCS does have two-way capability, to read and write. But that’s the BCS service/models, independently of sync. While the sync engine and BCS both are capable of doing so, the 2010 BCS based sync is scoped to read-only and will not write back the data to the business sytem through BCS.

    – Tanujb

  3. snymanpaul says:

    Hi,

    I have one question I’m sure many others have too i.e. how does the above AD example differ if you choose to use a generic LDAP store like OpenLDAP?

    Thanks for the great info.

    Regards,

    Paul

  4. Robert Greig says:

    Hi – very useful article.

    Is it possible to write a custom connector? I have an internal user master that is not AD (and is not LDAP at all) that I need to use to synchronize with the Sharepoint user profile store. I'm just looking for some pointers to docs or is this something that is just Not Possible?

  5. Frank van Rijt says:

    If you have already FIM 2010 deployed within your organisation, can you leverage that for Sharepoint 2010 provisioning or does Sharepoint 2010 needs it''s own FIM instance?

  6. Please see the following technet.microsoft.com/…/ff959234.aspx

    Configure profile synchronization using a Lightweight Directory Interchange Format (LDIF) file (SharePoint Server 2010)

    Published: September 16, 2010

    This article describes how to use a Lightweight Directory Interchange Format (LDIF) file to synchronize user and group profile information between Microsoft SharePoint Server 2010 and a Lightweight Directory Access Protocol (LDAP) provider not directly supported by Microsoft SharePoint Server 2010. For a list of directly supported LDAP providers, see Identify directory services and business systems.

  7. Simo Filali says:

    I have a different issue : on my user profile sync service i have 2 connections ( from AD, and From BCS) , i am using BCS to augment the users properties , on my BCS i am using  a WCF that i created which invoking a method getemployee(string unique id)  //my unique id is caming from my user profile so i can get the rest of my properties to sync , my issue is that my unique id is in this format (domainuserid) thats how its stored on Active directory ,so the WCF method is throwing an error because of the special caractere("") , when i use WCFclientTest and i invoke the method using a string in the following (domain\uniqueId) it works , the prob is that i was thinking to run a timer job  to manipulate the unique id after the Sync is done , but the as i said i have 2 connections when the sync is done  the bcs sync will be done too , i want to stop the sync after AD sync is done , run the timer job then continue the sync with BCS. any idea?? thanks.

  8. Dorene says:

    Hello, I am not sure if my issue is related to the above coments, but any help would be greatly appreciated.  As the Admin of our SharePoint site I deleted some files that were listed under the "List".  The delete was successfully from my view, but all other users on the Site are still seeing the files that were deleted.  Please, please help! Is this a sync issue?

  9. Dorene says:

    I wanted to make a change…I was changing the content site and the content site was a list…

    Thank you

  10. Michael Brockman says:

    Excellent post; very informative. I have one question: Do you know if it's possible to use a BCS connection by itself just to update existing profiles from an external system without having an AD or other type of primary connection? I've set up an ECT which returns data from the external system into an external list without problem. Profile synchronization completes all steps in FIM successfully. But no profiles ever get updated. And I missing something here?

    Thanks.

  11. Neha says:

    Wonderful article. Good to know about profile import and what happens in the background. But I have a query, I read this article http://www.ekhichdi.com/…/How-to-give-Replicating-Directory-Changes-permission-270.html

    Now I do not have permission on the Active directory and the AD team is not going to make the changes. I am testing SP 2010 profile import on my test machine. Is there a way I can test import apart from importing from AD? Maybe for 10-20 users?

    Regards

  12. Todd Fleenor says:

    We have full installation of FIM 2010 synchronization service in our organization.

    How does this information for SharePoint differ from the sychronization service in FIM 2010?

    Can we use FIM 2010 for SharePoint instead of using what is being described here?

  13. venpal says:

    I have a question

    Does it have any relation Sharepoint FAST

    I have unistalled, reinstalled and configured FAST since then i noticed Membership details are not updating

    i became member of few more sites but these sites are not listed in Membership section of mysite

    what could be the possible reason

    user profile sync, crawl of search everything is working fine

  14. Cathy says:

    We are using AccountName to join to an external BCS to pull in more profile data.  the connection is setup, and a couple of mapings created, but not sure if it is synching.  Here is my question:  AccountName is of Type=People.  Will it successfully join to a string field of format "domainusername"?

    Same question for the Manager field in Sharepoint 2010?  When I try to map a BCS connection field to Manager property, there are no fields in the drop down.  So I assume no fields of compatible data type.  I have tried the "adusername" format, and the Distinguished Name format "CN=xx,OU=xx,DN=xx,DN=xx,DN=xx" but neither shows up in the drop down when I try to map.

    Appreciate your help

  15. ehabzag says:

    I am doing a SharePoint user profile sync from AD, the process is working fine, from "Synchronization service manager" 2373 profiles were added.

    but only 9 were added to SharePoint.

    i checked AD and found that the 9 accounts have type "USER" and others have "inetOrgPerson". How can i add these marked as "inetOrgPerson" to SharePoint user profile??

    please advice

  16. Andrew says:

    So we can't use BCS to create the profiles?

    User Profiles can only be sync from LDAP source?

    Actually I have a CSV file containing user info extract from AD and I need to enable them on the user profile sync.

    I imported the file into a table in a database and created BCS, but some how after I do a sync no profiles can be created.

    and it is not allowed to connect to the AD to do the user profile sync

    Any thoughts? please help

  17. Anuj says:

    If you are updating user profiles using SharePoint 2010 APIs then please remember following –

    1. Profile update takes lot of time

    2. More the number of properties in each profile, more the time it will take to update

    3. Update only the properties which are changing

    4. Try to load configuration values before running the main logic (of updating the properties)

    5. Reason it takes more time to update profile/properties is internal to SP 2010

  18. RichCool says:

    SharePoint 2010 user profile sync not importing new AD security groups since the server upgraded from the 2007 to 2010.

  19. New SAP BCS Connector says:

    Here you find a how-to tutorial about the new and easy SAP BCS Connector tool from Theobald Software:

    http://www.parago.de/…/how-to-integrate-sap-business-data-into-sharepoint-2010-using-bcs-services-and-bcs-connector-from-theobald-software

  20. Venkat says:

    Hi All,

    We've a mixed mode authentication site and we have AD and LDAP users. How to set a filter to exclude AD users and include only LDAP users in UPS.

    Thanks in advance.

  21. Anshuman Dash says:

    Hi! First of all a very informative article. Great work!

    Secondly, I had the following question, if anyone could help me with,

    1. I have a BCS connection to a custom source.

    2. It is mapped with an attribute, say Employee Id

    3. The Profile Property import works fine for all the custom Properties mapped to the BCS Source.

    But, the problem arises when I change the connection string of the source. I do not get any of the Properties populated.

    The only difference between the old and the new source is that, the Value of Employee Id is different. I have updated the Employee Id value for the users in SharePoint. But still I don't get the required values from the Data Source.

    Can anyone throw some light on this problem? Is there a caching happening anywhere?

  22. Jim says:

    Thanks for explaining this complex process. I've got a sync with BCS augmentation working, but it takes FOREVER! It took 20 hours to sync 30,000 users and nearly all of the time is spent during the DS_FULLIMPORT phase of the BCS import. We added an index to the table and that took it down to 15 hours but that is still brutal.

    It seems to process each user one at a time, running a SELECT * FROM table WHERE User_ID = [the user id]. Each select takes only milliseconds to return but when profiling it seems to take 2.5 seconds per user. That time seems to be spent by SharePoint doing something after the record is returned. Why the heck does it select each user one at a time? Why doesn't it pull in all the data from the table or view and then join the data to the AD data in the sync database or in memory? This seems very inefficient.

  23. Jim says:

    I finally fixed it. There's a bug in the UPS sync code (as of the June 2013 CU) which tries to log whether the BDC server name is an intranet server or internet server name, for each user.

    The problem is that if you use an Oracle data source, the server name isn't a valid DNS host name and you get an exception for every user getting synced!

    On the server hosting BCS, I added a fake entry in the hosts file to the Oracle TNS name pointing to 127.0.0.1 and the sync time went from 15 hours to 15 minutes!

  24. Faisal Masood says:

    Thanks for the UPS explanation.

    Faisal Masood

    http://www.FaisalMasood.com

  25. Naresh says:

    Hi, i was wondering what is the basic difference between full sync v's incremental sync? in our organization we have just noticed that the updates from AD are not reflected in SharePoint directory search for 1 week but new profiles does reflect the next day. i checked the schedule and noticed that we have only incremental import scheduled once every night.

    do we need full import to see the modifications of existing user profiles? searched several places and most of them say incremental is sufficient and full import is just a one time deal?

    any clarifications is greatly appriciated

Skip to main content