Sharepoint 2016 – MysiteCleanup Job functionality changes


Here is a Quick look at what's new  with the  Mysite Cleanup job in Sharepoint On-Prem  2016 &  how it impacts the  functionality of  profile deletion & cleanup .

How this worked in Sharepoint 2010 / Sharepoint 2013 ?

The user profile in the profile store can be marked for deletion (bdeleted=1) when the actual user is deleted or it meets a User filter in FIM (eg:UserAccountControl bitonequal 2). When the mysite clean up job is executed, such profiles are immediately deleted and corresponding mysites are kept for 14 days for data recovery.

More Info : https://blogs.msdn.microsoft.com/kaevans/2012/06/25/inside-the-sharepoint-2010-my-site-cleanup-timer-job/

Note : While AD import is configured, you will need to execute a set of commands to get the disabled / deleted users marked for deletion

Ref: https://blogs.msdn.microsoft.com/spses/2015/03/04/sharepoint-2013-active-directory-import-and-known-behaviors/ and https://blogs.msdn.microsoft.com/spses/2014/04/13/sharepoint-2013-adimport-is-not-cleaning-up-user-profiles-in-sharepoint-whose-ad-accounts-are-disabled/

Whats new in Sharepoint 2016 ?

The SharePoint 2016 works similar to Sharepoint 2013 , however the profiles marked for deletion (bdeleted=1) will not be immediately deleted, such profiles will be preserved for 30 days.

This is how a profile look like when its active.

upa1_a

 

 

 

 

When the user is marked for deletion , the bdeleted and NTName values are updated. The NTName column value will be appended with the UserID.

upa1_b

 

 

 

 

To calculate the 30 days retention , we have introduced a new table (upa.userprofilescheduledforremoval) and the value will be populated when the user is marked for deletion. When the mysite Clean up job runs , it queries this table and picks up the profiles that are more than 30 days and process them.

upa2_a

 

 

 

 

Note : It is not recommended to Query any Sharepoint databases manually or to make changes other than ones described at http://support.microsoft.com/kb/841057 .

I see the Profiles in my farm are never  Deleted , Even after 30 days & Successful Run of Mysite Cleanup Job ?

We  at Microsoft are already aware of this issue  &  working  to  get this addressed in one our upcoming updates .  Such profiles  can be removed manually, using Central Admin  or   PowerShell script as needed .  We will update this post once we have more info on the fix .

 

Update [Aug 9, 2017] : The above mentioned issue with MysiteCleanup job on Sharepoint 2016 has been resolved in Aug 2017 CU.

Download Sharepoint 2016 updates : https://technet.microsoft.com/en-us/library/mt715807(v=office.16).aspx#BKMK_2016

Please test , plan and review , back up the data and then install the CU on production farms.

 

POST By : Manjesh Menon [MSFT]


Comments (12)

  1. MP says:

    Thanks for the update . We have large no of disabled User profiles not getting deleted even after Running Mysite cleanup job in our SharePoint 2016 environment running MIM . Do you have PowerShell script somewhere to delete those till Microsoft releases the fix?

    1. Spses says:

      Hi MP,
      Below is the sample script that can be used to remove the profiles only. You may need to modify it to remove the Mysites if you wish to do so. You may need to prepare an input CSV file (Profiles_tobe_deleted.csv) copying the result of SQL query towards the profile DB “select ntname from upa.userprofile_full where bdeleted=1 (nolock)”. Its pretty important to have proper back up of the databases before you proceed.

      Disclaimer: Sample Code is provided for the purpose of illustration only and is not intended to be used in a production environment. THIS SAMPLE CODE AND ANY RELATED INFORMATION ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE. We grant You a nonexclusive, royalty-free right to use and modify the Sample Code and to reproduce and distribute the object code form of the Sample Code, provided that. You agree: (i) to not use Our name, logo, or trademarks to market Your software product in which the Sample Code is embedded; (ii) to include a valid copyright notice on Your software product in which the Sample Code is embedded; and (iii) to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use or distribution of the Sample Code.

      # Adding PS Snapin SharePoint
      $snapin = Get-PSSnapin | Where-Object {$_.Name -eq ‘Microsoft.SharePoint.Powershell’}
      if ($snapin -eq $null)
      {
      Write-Host “Loading SharePoint Powershell Snapin” -ForegroundColor Green
      Add-PSSnapin “Microsoft.SharePoint.Powershell”
      }

      # Importing the set of users.
      $fileName = “C:\Profiles_tobe_deleted.csv”;
      $inputFile = “”;

      try
      {
      $inputFile = Import-CSV $fileName -Delimiter ‘,’ -Header NTName
      }
      catch
      {
      Write-Error “Could not open file $fileName”
      exit;
      }

      # Delcaring the Sharepoint Variables.
      $site = new-object Microsoft.SharePoint.SPSite(“http://sps16app”);
      $ServiceContext = [Microsoft.SharePoint.SPServiceContext]::GetContext($site);
      $pm = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($ServiceContext)

      # Declaring and creating the log files. Each time the script is executed, a new file will be created with the current time in the filename.
      $dateTime=Get-Date -format “dd-MMM-yyyy HH-mm-ss”
      $UPLogFile=”UserProfiles_Remove_bdeleted”+”_”+ $dateTime + “.log”

      $inputFile | Foreach-Object($_){

      $User=$_.ntname

      try
      {
      $profile = $pm.GetUserProfile($User)
      $DisplayName = $profile.DisplayName
      Write-host “Current User:” $DisplayName
      $messageDisplayname = “Current User:” + $DisplayName
      Add-Content -Path $UPLogFile -Value $messageDisplayname

      $AccountName = $profile[[Microsoft.Office.Server.UserProfiles.PropertyConstants]::AccountName].Value
      $id=$profile.ID
      Write-host “ID for the user :” $DisplayName “is ” $id
      $messageid = “ID for the user :” + $DisplayName + “is ” + $id
      Add-Content -Path $UPLogFile -Value $messageid

      Write-host “Removing the Profile..”
      $messageremove = “Removing the Profile..”
      Add-Content -Path $UPLogFile -Value $messageremove

      try
      {
      $pm.RemoveUserProfile($id)
      Write-host “Successfully Removed the Profile” $AccountName
      $messagesuccess = “Successfully Removed the Profile” + $AccountName
      Add-Content -Path $UPLogFile -Value $messagesuccess
      Add-Content -Path $UPLogFile -Value ” ”
      }
      catch
      {
      Write-host “Failed to remove the profile ” $AccountName
      $messagefail = “Failed to remove the profile ” + $AccountName
      Add-Content -Path $UPLogFile -Value $messagefail
      Add-Content -Path $UPLogFile -Value ” ”
      }
      }
      catch
      {
      Write-host “Exception when handling the user ” $User
      $messageexcp = “Exception when handling the user ” + $User
      Add-Content -Path $UPLogFile -Value $messageexcp
      Add-Content -Path $UPLogFile -Value ” ”

      }
      }

      1. MP says:

        Thanks for the script . I installed feature Pack2 in our Test environment and it seems to have fixed Mysitecleanup job issue . I also changed MIM filter ( to bit on equals 546) so it can properly sync disabled accounts in AD. I can now see mysite deletion emails being sent to Managers . The problem is if we run this in production it will send hundreds of emails to managers for employees that left the company long time back . Is there a way to change the timer job to send email to specific account spadmin@abc.com rather than the manager ?

        1. Spses says:

          Hello MP,
          Unfortunately there is no way we can control this – we can completely disable the outgoing emails so that the managers will NOT be updated about the deletion, however there will be a reminder email 3 days prior to the deletion as well. Keeping the outgoing email settings for longer days may impact the “alert” features too. Hence you can modify the script in such a way that , once the profile is removed, delete the corresponding mysite as well. In the same script , you may add the following lines right before $pm.RemoveUserProfile($id)
          $acctprofile = $profileManager.ResolveProfile($AccountName)
          $acctprofile.PersonalSite.Delete()
          Further checks for the personal site existence before deleting will help the Script run better.

  2. Hello Manjesh,
    Do the entries in the userprofilescheduledforremoval table get cleared when the user profile is re-imported? Does it prevent the deletion of mysites in such cases unlike in previous SharePoint versions when once marked for deletion, the mysite will remain queued for deletion and will be deleted no matter what.

    Regards,
    Aneesh

    1. Hi Aneesh,
      As of today, our tests shows that the entry in the userprofilescheduledforremoval table will be removed or the value for the schedulestate will be updated once the user is imported back, This will prevent the mysite being deleted. We will further test a couple of scenarios once the above mentioned issue is completely addressed.

  3. Umr says:

    I am testing August CU for this issue and waiting for 30 days to complete and get rid of all accounts marked with “deleted” in name and moved in to “upa.userprofilescheduledforremoval”. Is there any way to get rid of them before 30 days ?

    Another question is , What about the disabled accounts? I notice if you disable an account and even thou you have filter applied to exclude disabled account, it still does not get rid of those accounts in database and you can see disabled accounts in “UserProfile_Full” with older data in “LastUserUpdate” column. It seems like it is not syncing disabled account any more due to filter but not removing those users from the database. Do you know what is right behavior for disabled accounts?

    1. Umr says:

      Just an update on the my question above , after I remove the filter query , save connection , import profiles, reenter the same filter to exclude disabled and executing MySiteCleanup job , i can see the same behavior as deleted users. Now question is do i really have to wait for 30 days for these to be gone or any safe way to remove them for further testing ?

      1. Spses says:

        Yes Umr. As of now (Aug 2017) its by design that the property is NOT manageable and its hard coded to 30 days. However you may use PowerShell / OM codes to remove such profiles before the wait period if you wish so. Please note that a profile that is marked as Bdeleted=1 will not be crawled, will not be picked up for Org chart etc.

        1. Umr says:

          Thanks for your reply . I tried -PurgeNonImportedObjects , hoping it will remove users from “UserProfilesScheduledForRemoval” table but that did not do any thing. Is there any other PS command you are referring to in your response above or same command ?
          Thanks again.

          1. Spses says:

            Hello Umr,
            The -purgeNonimportedObjects command will NOT remove any item from any table (be it SP2010 /2013 / 2016). It marks the users as bdeleted=1(in case for 2010 and 2013). In 2016, this command should mark the users as bdeleted=1 (userprofile_full table) and create an entry in the “UserProfilesScheduledForRemoval” table. A successful execution of the “MysiteCleanup Job” should remove such entries from userprofile_full table (in case of 2010 / 2013) AND from “UserProfilesScheduledForRemoval” table (in case of 2016)

Skip to main content