Debugging the FIM 2010 SSPR "PWUnrecoverableError" error

Last night, I had a particularly devilish time trying to figure out why when users were attempting to reset their passwords, they were receiving the following error:

"An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)"

Nothing like an error list this to make a FIM guy *sigh*

Checking the Password Reset server's event log revealed the following errors:

Error logged in the FIM SSPR Server's event log:
Message: An error has occurred. Please try again, and if the problem persists, contact your help desk or system administrator. (Error 3000)
Source:
Attributes:
Details: System.InvalidProgramException: Error while performing the password reset operation: PWUnrecoverableError
   at Microsoft.IdentityManagement.CredentialManagement.Portal.Reset.AttemptToResetPassword()
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
CorrelationId:
RequestId:
ErrorCode: 3000

Knowing that the PWUnrecoverableError means that the source of the error must have originated on the FIM Service box, I checked that server's event log to discover the following error logged:

Error logged in the FIM Portal/Service Server's event log:
mscorlib: System.Runtime.InteropServices.COMException (0x80070721): A security package specific error occurred. (Exception from HRESULT: 0x80070721)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.ResourceManagement.PasswordReset.ResetPassword.ResetPasswordHelper(String domainName, String userName, String newPasswordText)

I ran through the usual check-list of things to verify:

  1.     The FIM Service is a member of the FIMSyncPasswordReset and FIMSyncBrowse grops on the Synchronization Server
  2.     The password management on the AD management agent is enabled
  3.     The FIM service account has the appropriate WMI permissions on the sync server
  4.     Enabled DCOM for the FIM service account
  5.     Firewall settings are cool
  6.     The user in question is in the the Password Reset Users Set
  7.     The user in question's MV object has a domain and accountName values

It was only after exhausting all other possibilities that I dug deeper to find that, if you're FIM Service and Synchronization services reside on separate boxes, you must ensure that all of the boxes impersonation levels are set to Identify. My issue was that while my sync server was set to this, my portal/service servers were set to Delegate. Once I brought them all in line, my happiness was restored.
 
For sake of posterity, here are the steps you should take to verify that your sync and portal/service server have the correct impersonation levels.

  1.     Start --> Run --> DCOMCNFG.exe
  2.     Component Services --> Computers --> My Computer, right-click Properties
  3.     Select the Default Properties tab
  4.     Verify that the Default Impersonation Level is set to Identify (if it is not, fix it).
  5.     Click OK to close the dialog.

Anyway, hope this helps anyone down the line that needs it.