NoMAS and Other Solutions to 9548 and 9551 Events (Exchange Server 2003)

I thought I had posted this information several years ago, but a customer recently contacted me seeking it and I realized it never left the publishing stage. Please note that this post is more specific to 5.5 to 2003 migrations, but may still be relevant to many folks out there.

After installing Exchange 2003 in your environment, you may see the following types of events in the event log of your Exchange Server 2003 machine.

Event Type: Warning

Event Source: MSExchangeIS

Event Category: General

Event ID: 9548

Date: 6/14/2005

Time: 8:57:29 AM

User: N/A

Computer: [2003 server name]

Description:

Disabled user /O=[LEGACY DN PATH]/CN=dZazzo (this could be any valid path to a user account) does not have a master account SID. Please use Active Directory MMC to set an active account as this user's master account.

Event Type: Warning

Event Source: MSExchangeIS Public Store

Event Category: General

Event ID: 9551

Date: 4/19/2005

Time: 11:00:17 AM

User: N/A

Computer: [Exchange 2003 Server]

Description:

An error occurred while upgrading the ACL on folder [Public Folders]/[Complete Path] located on database "First Storage Group\Public Folder Store (Server Name)".

The Information Store was unable to convert the security for /O=[LEGACYDN]/cn=DZazzo into a Windows 2000 Security Identifier.

It is possible that this is caused by latency in the Active Directory Service, if so, wait until the user record is replicated to the Active Directory and attempt to access the folder (it will be upgraded in place). If the specified object does NOT get replicated to the Active Directory, use the Microsoft Exchange System Manager or the Exchange Client to update the ACL on the folder manually.

The access rights in the ACE for this DN were 0x4fb.

It is important to have a full understanding of the reasons why these errors might be generated and know that the agency responsible for these errors may be remote to yours. The following resources can be helpful in diagnosing the problem (excerpts below):

XADM: Addressing Problems That Are Created When You Enable ADC-Generated Accounts: This article provides information about disabled accounts that the Active Directory Connector (ADC) creates and describes how to enable those disabled accounts. The information store expects that disabled accounts will have a msExchMasterAccountSID attribute and that enabled user accounts will not have a msExchMasterAccountSID attribute.

How to troubleshoot public folder performance issues that are related to ACL conversions in Exchange 2000 and in Exchange 2003: This article describes how Access Control List (ACL) entries can affect Public Folder performance on a computer that is running Exchange 200x.

You cannot move or log on to an Exchange resource mailbox: This article has a general explanation of 9548 Event IDs and introduces the msExchMasterAccountSID attribute and how to eliminate 9548 events manually.

Additional Information about the MasterAccountSID

MsExchMasterAccountSid is generally used on disabled Windows 2000/2003 user accounts with Exchange mailboxes to store the SID of an NT4 account in a trusted domain. This allows the NT4 account to access the Exchange 2000/2003 mailbox. The Exchange Information Store logic looks at this attribute when a disabled user account is encountered, and uses this SID as the active SID, instead of the objectSid of the user, which is used if the account is enabled. The active SID (either objectSID or msExchMasterAccountSID) is used to apply permissions within the information store, for example delegate permissions or public folder permissions.

If the user account is disabled, and no msExchMasterAccountSid is set, the store cannot determine which SID to add to the ACL. In this case, the user is not added to the ACL, and 9548 event gets logged. This can also cause a 9551 event to be logged, if the store is in the process of upgrading a DN-style ACL to a NT-style ACL.

Over the long-term, the accumulation of 9548 events can warn administrators of potential performance problems on the Exchange store, related to the enumeration of ghosted objects (disabled users). If an ACE is a disabled account's objectsid, store expects the disabled account's mexchuseraccountcontrol to be 2, and msexchmasteraccountsid to be populated with another SID. If MAS isn't populated, the store won't know who the primary associated user is in order to grant/deny access to this folder later; hence the 9548 gets logged.

The performance degradation will occur, regardless of whether the account accessing the folder is in a valid state or not. So long as the folder has any Access Control Entries (ACEs) that are in bad states (disabled without MAS, or DN-to-SID convert failed, which is the 9552 errors associated with converting Distribution Groups to Security groups), the store takes a performance hit. In the case of zombie permissions, users can also be denied access to the folder resource, even if they are ACL'd correctly. For that reason, I always recommend using the DS/IS consistency adjuster to remove unknown permissions from any folders before introducing E2k/E2k3 into a 5.5 site.

The “Associated External Account” (which can be found on the Mailbox Rights button from the Exchange Advanced tab of user properties in ADU&C) and msExchMasterAccountSid should always be in sync. These requirements are not enforced by either AD or Exchange, but if they are not followed, either 9548 events will be logged, or access to resources may be unexpectedly denied, or both.

It is also possible (and the default behavior by the ADC when creating mail-enabled disabled user accounts) that the built-in security principle SELF can be set as the msExchMasterAccountSID. This is perfectly valid, and tells the store to use the objectSid of the disabled account as the active SID when performing security operations. This can also be used for scenarios where regular user accounts will be disabled for a period of time, and will possibly be enabled for use later, or deleted.

Resolution

At MCS, we recommend that if you disable your users as part of the standard employee exit process, you should grant the SELF built-in account full mailbox access and associated external account to the mailbox. This will prevent problems with permissions and the proliferation of 9548 and 9551 event IDs in the application logs.

If a customer requires changes to hundreds of objects, they can also use the NoMAS.exe tool from Microsoft, which provides for bulk manipulation of these objects. The NoMAS tool has the following requirements:

1. NoMAS must be run in a domain with an Exchange 2003 representation.
2. NoMAS cannot traverse domain boundaries
3. NoMAS requires that the administrator running the utility be a member of the domain Admins group of the local domain.
4. NoMAS requires Exchange Administrator permissions or greater on the mailbox stores of those objects being changed (this is typically accomplished with an account that has Exchange Administrator delegations at the Administrative Group level.

NoMAS is not supported by Microsoft. Although PSS may assist with troubleshooting NoMAS issues, hotfixes and escalations for the tool will not be provided.