CRM 4: System-wide Duplicate Detection

Here I am continuing the duplicate detection story of CRM 4.0 with a new aspect of this feature. Organizations face a common problem where in spite of several policies and guidelines in place users end up entering duplicate data in MSCRM system via data import or data generated during many internal operations or simply knowingly creating duplicate records. Creating duplicate records at any point of time might be crucial but this ends in a lot of duplicate data in system which can affect sales, marketing and auditing of the organization. System-Wide Duplicate detection feature in Titan solves this problem and empower users to find duplicates in a part or all the records of an entity in the MSCRM system.

System-wide duplicate detection is an asynchronous type job and it detects all the duplicates in the system for the given records. System-wide duplicate detection can be run on selected records of any entity or records under any specific user query or all the records of that entity. I will cover each of these scenarios one by one.

System-wide duplicate detection wizard can be launched from Duplicate Detection option under Tools menu in MSCRM or Workplace > Duplicate Detection grid New button. First page of this wizard is welcome page and informs user about the capabilities of this wizard.


Proceed to Select Records page to select the entity on whose records you are going to run the duplicate detection. All the entities that have one or more duplicate rules published in MSCRM system will be listed under Look for dropdown on this page. This interface is standard Advance find interface and you can either select any existing view to run the duplicate detection on all the records of that view or can create a custom query also. You can also see the records identified according to the selected query by clicking Preview Records button.


Now Proceed to Select Options page for giving name to your job and selecting the time at which you want to run this job. This duplicate detection job can be run recursively also by checking “Run this job after every” check box and providing number of days in the days drop down box. This feature is very useful in scenarios where duplicates are creating regularly in MSCRM system and you want to keep track of them after every few days or months. You can also choose to send an email to you and others on completion of this job.


Now go next to Start Duplicate detection Job page and submit the job. Your job will be submitted for processing at the time provided by you in the wizard.

Duplicate Detection can also be launched from any entity grid for which one or more duplicate rules are published. While running duplicate detection on entity grid you can choose either to run duplicate detection for manually selected records or for all records under that view. For running duplicate detection for selected records first select one or multiple records on grid and go to grid view of desired entity and launch duplicate detection dialog from More Actions > Detect Duplicates > For Selected Records. This will launch Check for Duplicates dialog which is same as Select Options page of System Wide duplicate detection wizard. Just fill your options and your job will be submitted for processing.



For running duplicate detection for all records in any view select More Actions > Detect Duplicates > For All Records on All Pages.

You can see your duplicate detection jobs under Workplace > Duplicate Detection grid or Settings > Data Management > Duplicate Detection Jobs grid. Here you can check the status of your jobs and see the duplicates found in any duplicate detection job. Open the system form by double clicking and go to View Duplicates under Details section on left hand side.


This form gives user the power of editing, merging and deleting the duplicate records which is essential for data cleansing.

Some noticeable points about System wide Duplicate detection are

a) Duplicate Detection runs for all the published rules for any entity and there is no concept of choosing duplicate rules while doing duplicate detection.

b) Duplicate detection wizard can be run on only one entity at a time. Though you can run multiple jobs for different entities.

c) There should be duplicate rules present as published for the entity you want to run duplicate detection.

d) User level privileges apply while showing duplicate records to users.

This story will continue and in next post I will discuss security model of duplicate detection.

Sudhakar Singh

Comments (35)

  1. dmbrandon says:


    Very helpful disucssion on dup detection.  Question – is dup detection run automatically when Outlook is synchrnized with CRM? For example,a new contact is entered offline, when returning online is the contact checked for dup detection?

  2. Welcome to the next in the series of posts on Duplicate Detection in Titan. In case you missed the action

  3. Sudhakar says:

    Hi dmbrandon,

    Duplicate detection runs automatically when Outlook goes online from offline if settings says so. By default it is disabled. To enable it in outlook go to CRM > Options menu > Synchronization tab and select Do not create duplicates option and now when going from offline to online outlook will throw Data Synchronization error dialog if any duplicate is found and will ask you to choose appropriate actions like Resolve duplicate or Ignore duplicate.

  4. Mohit says:

    I am imporeting data via Data migration manager and not through Bulk import wizard , can dupicate detection work while inserting data thro DMM tool ?

  5. Sudhakar says:

    No, Duplicate detection doesn’t work with DMM. This is as per design so that user dont get a lot of errors during migration. To clean the data after migration you can anytime run the bulk duplicate detection wizard.

  6. Alex F says:

    Excellent article!

    Quick question – In my environment, then when selecting ‘send email to me when job is finished’, the email message has a link using an internal name of the crm server and NOT the fully qualified name that is used to access the system.

    The fully qualified name with https is displayed OK when sending ‘short cuts’ from within the crm system.

    Any ideas on where I may be able to fix this?

    Thanks a bunch!

  7. Sharad says:

    This is currently a limitation where the hyperlink works fine for on premise deployments. This will be fixed in next version.

    In single organization environment, you might give a try to replacing ‘@servername’ with  fully qualified name in the following email templates:

    1. Bulk Deletion Task Completed With Failure Template

    2. Bulk Deletion Task Completion Template

    3. Bulk Deletion Task Failed Template

    I should warn you that in multi organization environment there might not be a workaround which works for all.

  8. Sonal says:

    Hi Alex,

    You can update the URL in the e-mail template being used for duplicate detection completion.

    For this navigate to

    Settings->Templates->E-mail Templates and open

    "Duplicate Detection Task Completion Template".Here you can update the link to reflect the fully qualified name.Once done all e-mails specific to this template will be created with updated link.

  9. Bernhard says:

    The detection works very fine!

    How is it possible to delete the detected records?



  10. Sudhakar says:

    Hi Bernhard,

    In the View Duplicates dialog itself, on lower duplicate records grid toolbar there are options for deleting/deactivating/editing/merging the duplicate record.

  11. This blog post will cover custom filter-based mass record deletion for a given entity System. Note: If

  12. This blog post will cover custom filter-based mass record deletion for a given entity System. In this

  13. This blog post will cover custom filter-based mass record deletion for a given entity System. In this

  14. Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

  15. Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

  16. Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

  17. stacy says:

    I started a duplicate detection rule and now I cannot stop it.  I get an error message "The duplicate detection rule is currently being used and cannot be updated or deleted. Please try again later."  

    I cannot delete, unpublish, edit or anything else.  I created this for use with a custom entity and cannot delete the entity either.

  18. Frank Lee says:

    There may be running rule(s) that are in "wait" mode.  Drill down on the duplicate detecton rule via system jobs and find out which ones are pending/wait and cancel those.  You can then delete, unpublish, edit, etc.

    Frank Lee, Microsoft Dynamics CRM MVP

  19. Chris Kiec says:

    I can not remove a rule on Contacts.  This started 7-21.  It has no rule in wait or pending mode.  Please help

  20. Peter McKenzie-Bridle says:

    2 years ago someone added some products, quotes, orders and invoices to our production CRM system when testing this feature.  We now want to use the integrated ordering processes but want to remove the old test data – because they are related records (i.e. a product is contained in an order which was cancelled) I’m unable to remove the data through the GUI.  What other methods can I use to remove this redundant data?

  21. Kevin Roy says:

    Hi Im trying to remove duplicate Account names is there any way to do this, every time i select account it then gives me equals not equals etc and wants a value, thing is if i put in a value I would be as well going through manually, all I want it that any accounts with the same name are highlighted so I can remove them is this possible ?

  22. reicoskyr says:

    I too cannot unpublish a system rule either and there is nothing in a wait status either.  

  23. Debbie Grale says:

    I also cannot unpublish a rule, there is nothing in wait status. (there is nothing about duplicate detection in system jobs)  

    I’ve deleted the results of the last duplicate detection job I ran, with no luck. Have logged out of CRM and logged back in and still unable to unpublish.

  24. RC@ says:

    Duplicate Detection runs for all the published rules for any entity and there is no concept of choosing duplicate rules while doing duplicate detection.

    What does the above line mean?

    Actually I created a workflow on Email creation and has nothing to do with the duplicate detection rule created for lead entity. But there seems to be an entry of a system job of this duplicate detedtion rule under that workflow created on the email entity!!

    Any suggestions?? Please help!



  25. P Uday Kumar says:

    I was also unable to unpublish/delete duplicate detection rule. I tried searching for the duplicate-detection rule in waiting state from Settings>>System jobs but i couldn’t find the job in waiting state. Thanks to "Advance Find" which helped me to find the duplicate-detection job in waiting state. I used following query: Look For: System Jobs

    System Job Type: Equals : Bulk Duplicate Detection

    Status Reason : Equals : Waiting

    This query displayed a job that was in waiting state. After cancelling the job I was able to unpublish the rule.

  26. Rafiq says:

    System Admin cannot publish the the rule. Publish button not seen

  27. Bhaskar says:

    Duplicate detection rules have been unpublished and re-published and each time they are stuck in ‘Publishing’ status.

    Tried creating new rules after the re-deploy and get the same results when publishing these.

    Any ideas about how I can get the rules to publish?

  28. Sam says:


    does anyone know how to set a system wide dedupe rule that ignores deactivated accounts?  


  29. SJM says:

    Duplicate detection has been failing since Feb. In the System Job Failure with Errors view, I get a Failed Status with an error code -2,147,220,969.

    The Text is Unhanded Exception: System.Web.Services.Protocols.SoapException: Server was unable to process request. Detail:<detail><error><code>0x80040217</Code><descrition>systemuser With Id = cfc99cba-ca58-dd11-bc42-001ec9365ec7 Does Not Exists</description><type>Platform</type></error></details> atSystem.Web.Services.Protocols.SoapHttpClientProtocol.RealResponce(SoapClentMessage message, WebResponce responce, Stream responceStream, Boolean asyncCall) at System.WebServices.Protocols,SoapHttpClientProtocol.Invo

    In the users window I can double click on each user and see the assisgned UserID. Non of my users ahve this ID. It must be an internal ID.

    Little advice?

  30. Fred Kaemmer says:

    The deduplication tools in CRM 4.0 are fine but the deduplication rules are that basic that they are not a helpful feature of a state of the art CRM

  31. Yiannis Patrikiadis says:

    I have the same problem with the publishing status of the duplicate rules. Does anyone find a solution. How can I resolve this issue?

  32. Is there a way to change the number of rows that display in the upper portion of the screen when viewing a duplicate detection job?  I can only see 12 rows at maximum.  Also, would be nice if there was an Export to Excel option for the list.

  33. Pino says:

    Nice tool to delete duplicate files from the drive but i use a tool to delete duplicates called duplicate finder 2009

    Find Here :

  34. peterfequet says:

    I have a question concerning duplicate contacts.  I want to send an email to various contacts (for example contact #1, contact #2, etc.) for different entities.  It is possible that a same person be contact #1 for one entity and contact #2 for another entity, but I only want this person to get the Email once.  Is there anyway I can get the system to send the email only once to this person.  The system will recognize it only if he appears twie for the same entity.  Any help is greatly appreciated!!!!

Skip to main content