Support-Info:(SYNC SERVICE): stopped-server: msidmCompositeType or Asynchronous Processing Scenario

  • Article

Overview

The stopped-server status is a common status that is seen in the FIM Synchronization Service Engine when executing an import and/or export run on a given management agent.  The following information has been compiled to provide guidance on how to begin troubleshooting a stopped-server status and possibily resolve a stopped-server status.

The stopped-server status is a WMI Return Code (https://msdn.microsoft.com/en-us/library/windows/desktop/ms699322(v=vs.100).aspx) from the RunStatus Property.

Additionally, this blog will cover at-least one specific scenario that had a stopped-server status.

Stopped-Server Checklist

Checklist Item Description
SQL Server Instance Check
  • Check to see if the FIM Synchronization Service Database and the FIM Service Database are running on the same SQL Server Instance.
  • Check to see if the FIM Synchronization Service database is on a shared SQL Server that has other databases as well.
SQL Server
  • Has the SQL Server Service been stopped/started recently?
  • You can use the System Event Log to determine this information
  • Check the disk space that is housing the MDF and LDF files for the FIM Service.
    • Issues have been seen where the drive housing the MDF and/or the LDF file is running low or out of disk space.  You may check the Forefront Identity Manager Event Log and/or the Application Event log.
Determine which Management Agent is returning the Stopped-Server
  • It is important to identify which Management Agent is throwing the error, as it can guide your troubleshooting steps.
Determine which Run Profile is returning the stopped-server
  • If you are running a combined single-step run profile (IE: Full Import and Full Synchronization) then split them and run them individually.
  • If it is failing on an Export, then try a Full Import (Stage Only) followed by a Full Synchronization followed by an Export again.
Refresh Credentials
  • Refresh the credentials on the management agent
Review the Run History (Operations Tab)
  • Is there a lot of Exports that have been run back to back with no imports?  If so, then run a Full Import (Stage Only) followed by a Full Synchronization.
Application Event Log
  • It is very possible that the stopped-server status could place more detailed information into the application event log.  You may find things like a BAIL error message.
Storechk.Exe
  • You will need Microsoft Product Support to obtain this tool.
  • It is build specific.
  • Storechk.Exe analyzes the backend FIM Synchronization Service Database and produces a report.
FIM Service Management Agent
  • FIM Request Processing (Search Requests)
  • FIM Service Tracing
  • FIM Management Agent Tracing
  • ETL/ETW Tracing

NOTES:

  • If the FIM Synchronization Service does not hear back from the FIM Service for 10 minutes, it throws a stopped-server.
  • You can utilize configuration file changes to help the issue.

 

Stopped-Server Scenario: msidmCompositeType or Asynchronous Processing Scenario

Problem Statement

FIM 2010 R2 – Receiving a stopped-server when running an export on the FIM Service Management Agent. The export will sit there for a few minutes and then return a stopped-server status and stop the export.  Actions Taken

  1. Application Event Log: In any stopped-server issue, we should always start with the Application Event Log, as the stopped-server may throw some valuable information into the Application Event Log.  In review of the Application Event Log the following line was found in the BAIL error message. “The operation failed because the object cannot be found”.
Application Event Log
Source:        FIMSynchronizationServiceEvent ID:      6301Description:The server encountered an unexpectederror in the synchronization engine:"BAIL: MMS(6588):d:\bt\9394412\private\source\miis\server\sqlstore\csobj.cpp(15136):0x80230405 (The operation failed because the object cannot be found)BAIL: MMS(6588):d:\bt\9394412\private\source\miis\server\sqlstore\csobj.cpp(14776):0x80230405 (The operation failed because the object cannot be found)

 

2. Request History:  Based on the fact that we are running an Export on the FIM Service Management Agent, and that we are seeing a message indicating that the object cannot be found the next stop is the Request History.  You can review Request History via the following steps:

    1. Open Internet Explorer and navigate to the FIM Portal

    2. Click Search Requests on the menu on the left

    3. To search requests for today, simply click the magnifying glass

      I. If you need to review prior days, you may have to execute an advanced search

d. If you see a request for msidmCompositeType then the issue could very well be asynchronous processing.

In FIM 2010 R2 Asynchronous Processing was enabled by default.

NOTE: Asynchronous Processing is where we process a batch of records at one time.  

3. Review Asynchronous Processing Configuration: Since we have an idea that the problem is asynchronous processing, let’s review the current asynchronous processing configuration.

    1. Open a Windows Explorer window and navigate to where you have the FIM Synchronization Service application installed
    2. Double Click on the Bin Folder
    3. Locate the MIISERVER.EXE.CONFIG file
    4. Make a backup copy of the MIISERVER.EXE.CONFIG File
    5. Edit the MIISERVER.EXE.CONFIG file
    6. Search for the section resourceSynchronizationClient
      i.     By default it will look like <resourceSynchronizationClient />
      ii.    There is a commented section with the specified property names that can be utilized here.  I have put them in the below table.
Property Name Default Value Description
asynchronous True This property controlswhether FIM MA uses the older synchronous channel or the new asynchronouschannel when communicating with FIM Service.
Aggregate True  This property controls whether FIM MA cansend data to FIM Service in batches.
aggregationThreshold 1000  This property controls the number ofattributes per aggregated batch.
delayUpdateAcknowledgements False This property controlswhether the FIM Management Agent sends acknowledgements to FIMSynchronization Service immediately upon Request completion, or withholdsthem until the end for Update operations. Setting this to true may reduce theoverall time to export data, but the FIM Synchronization Service UIstatistics are not updated as frequently.
exportRequestsInProcessMaximum 50 This property controls themaximum number of export requests thatcan be in process.

 

Resolution

To resolve this scenario it is recommended to go through the following steps:

  1. Disable Asynchronous Processing (Throttling) *NOTE: More information on next slide
  2. Run the Export on the FIM Service Management Agent followed by the Confirming Delta Import (Stage Only) then the Delta Synchronization.
  3. Once you have cleared the Pending Exports, you can then re-enable Asynchronous Processing.  However the recommendation is not to use the default value of 1000.  Start at a low number, like 100 and let it run for a while to ensure no stopped-server.  Increment from there.
Default:: Asynchronous Processing isenabled with a 1000 record threshold <resourceSynchronizationClient/> 
Disable Asynchronous Processing <resourceSynchronizationClientasynchronous=“false” aggregate=“false”/>
Asynchronous Processing on with athreshold of 300 objects <resourceSynchronizationClientasynchronous=“true” aggregate=“true” aggregationThreshold=“300” />

 

Additional Resources