OAB Generation and the 9339 event id

The 9338 and 9339 event ID’s that are logged in the application log indicate that there was a problem with the data that is returned from the Active Directory during an Offline Address List generation. The reason why there are two possible events that can be logged, are due to the fact that we might / might not find a Display Name for the object that we failed on.

Knowing that OABGen.dll is a MAPI application, we know that OABGen.dll will need to make calls to the Active Directory via the NSPI interface for any data needed. Once the call has made its way to the Active Directory, it is the responsibility of the Active Directory to serve this data request back to OABGen.dll.

For those of you that are not very familiar with MAPI, here is a quick overview on how MAPI applications store/manipulate data. Before OABGen.dll can query the Active Directory for its data, OABGen.dll will have to create a MAPI table that is comprised of columns. Each column will be represented by a MAPI property (Example – PR_DISPLAY_NAME). Each row consists of a subset of properties. Now one thing to keep in mind is that a MAPI table is an abstract data type that represents a set of objects that all have common characteristics. Each object returned from the Active Directory will occupy a row in the table, and each column will contain the property data found. I do not want to go very deep in to MAPI because this is outside the scope of this blog, but I want you to have an understanding on how this works.

Below is a visual representation of a 1 row and 3 columns that would represent this object in the Active Directory:

Now that we have covered the basics of how MAPI sees data, we can move on to how the OAB Generation process obtains this data.

During the generation an Offline Address List, OABGen.dll will make MAPI function calls to QueryRows() to retrieve all the objects from the /Global Address List. For more information on the MAPI QueryRows() function call, please refer to this MSDN article. Each time OABGen.dll calls QueryRows(), it will pass in a set number of rows to be retrieved per call. OABGen.dll receives these query results in the form of pages.

Below is example of a QueryRows() call to Active Directory. Here we are requesting 5 rows per function call.

Now here is where the 9339 comes in to play. If one of the entries in the page has a bad property, the entire page retrieval will fail. In the event of a failure OABGen.dll will log the last entry it finds in the *last* page of results that were successfully retrieved. From the example above you can see that User 4 has a bad property in the Active Directory, so this page retrieval will fail.

Below is a list of events that you will see in the Windows Event Viewer Application Log.

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9339
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
Active Directory (domain controller) returned error 8004010e while generating the offline address list for '\Global Address List'. The last recipient returned by the Active Directory was 'Dave Goldman'. This offline address list will not be generated.
- OAB Address Book

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9330
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
OALGen encountered error 8004010e(internal ID 5000f5f) accessing Active Directory (domain controller) for '\Global Address List'.
- Address Book

Event Type : Error
Event Source : MSExchangeSA
Event Category : OAL Generator
Event ID : 9126
Date : 4/27/2005
Time : 10:07:24 AM
User : N/A
Computer : OABGenSrv
Description:
OALGen encountered error 8004010e while calculating the offline address list for address list '\Global Address List'. This offline address list will not be available for client download. 

Now that I have the events from the application log, I want to look closer to see what the error code being returned means. If you look at the three above listed event id’s you will see that there is a common error return ‘8004010e. By using The Microsoft Exchange Server Error Code Look-up I can see what the error code means.

C:\WINXP\system32>err 8004010e
# for hex 8004010e/ decimal -2147221234
ecInsufficientResrc ec.h
MAPI_E_NOT_ENOUGH_RESOURCES mapicode.h
# 2 matches found for "8004010e"

The error code MAPI_E_NOT_ENOUGH_RESOURCES is a default error code the NSPI interface returns whenever it catches an error while retrieving data from the Active Directory. 

Invalid Active Directory Attributes
Mangled, null and incorrect attribute values can cause problems for the OAB Generation process, and stop an Offline Address List from being generated. By default directory attributes that are not populated will have a <Not Set> for its value. If you look at a property and you see that it is blank or null, this indicates that something is wrong with the property. In addition to the null or blank, values can also contain incorrect values.

As of right now there are only 4 known attributes that will cause a problem for the OAB Generation process if they have bad, null or incorrect values. This doesn’t mean that there couldn’t be more, but through most of the cases that I have reviewed and triaged, these are the 4 that always come up:

1 & 2. The HomeMDB and HomeMTA attributes - Which signify which Exchange server holds the mailbox. The homeMDB contains the complete DN of the mailbox store for the mailbox and the homeMTA contains the complete DN of the Message Transfer Agent (MTA) on that server.

3. The Manager attribute - Contains the distinguished name of the user who is the user's manager. The manager's user object contains a directReports property that contains references to all user objects that have their manager properties set to this distinguished name.

4. The Secretary attribute - Contains the distinguished name of the secretary for an account.

These values must point to a valid and correct object. For example, if a mailbox was moved from server one to server two and there was an active directory replication latency or problem, we might end up with an incorrect value for the homeMDB and homeMTA.

Another example is where the manager of a group of users leaves the company and their account is removed but the changes never replicate around.

How to fix this

This is where things start to get tricky. Right now without calling in a support call to PSS, you will have to try to do this manually. I am currently working on a solution for OABInteg that will allow you to scan for invalid entries and I hope to have this completed soon.

Looking above you can see that there are 4 attributes that have been the top offenders. You can run LDIFDE export commands to scan through your active directory and look for objects that have invalid entries. Below I am listing a KB Article that talks about using the LDIFDE utility. KB Article ID: 237677 Using LDIFDE to Import and Export Directory Objects to Active Directory - https://support.microsoft.com/?id=237677

Keep in mind that when you are looking at these attributes mentioned above, you need to make sure that they do not contain and null, invalid or mangled values. If they do contain values and are not supposed too, make sure they are set to <Not Set>. If they are supposed to contain values make sure that they point to valid objects in the active directory and are not set to null, or have a blank space.

One thing that will help you with your search is to look at the 9339 event and look at the last recipient returned. From here you can use the MAPI Editor (known as MFCMapi) to expose the Global Address List. You can download MFCMapi here: https://www.microsoft.com/downloads/details.aspx?familyid=55FDFFD7-1878-4637-9808-1E21ABB3AE37&displaylang=en

Steps for exposing the Global Address List

1. Run MFCMapi.

2. Click 'OK' on the main screen.

3. Select from the menu 'Session'.

4. Select 'Logon and Display Store Table'.

5. Select the profile that you want to use.

6. Select from the menu 'Address Book'.

7. Select 'View AB Hierarchy'.

8. When the dialog box appears, expand the Root Container.

9. Double click the 'Global Address List'

Now that you have the Global Address List exposed, you can search for the last recipient displayed in the 9339 event and start your search down from there. The next 50 results are the ones that you will want to look at, as this will be the next batch of results returned.

I will post more once I have added this functionality to OABInteg.

Dave