HOWTO: Install and Use CustomAuth on IIS 6

It never ceases to amaze me how easily users will download and install arbitrary binaries from arbitrary sources as administrator… but the security concerns aside, I am going to address the popular question of how exactly to configure CustomAuth and how exactly does it work…


Can someone attempt to install the CustomAuth.dll from, and see if they can get it to work?  The article linking to the zip file is here:

I’ve tried simply taking the dll in the zip file and installing it as an ISAPI filter and wild card script map but I never get prompted.

If someone could successfully install it and then guide me through the steps, it’d be very much appreciated.



Umm… I am not going to download and install a binary from an arbitrary location that you posted, and you should not, either. Always get your binaries from a trusted, original source. In this case, CustomAuth is a sample ISAPI produced by Microsoft and can be obtained in the following official locations:

  1. IIS Resource Kit Tools – CustomAuth.dll pre-compiled and CustomAuth.ini
  2. IIS Platform SDK – CustomAuth source code is located at “%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\CustomAuth”. I am going to assume that you can compile it Visual C++ 2005 Express Edition. It is really no different than the standard ISAPI instructions I posted in this blog entry, except you have to also:

    1. Add an additional “include” directory of “%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\inc”
    2. Compile “%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\src” into a Library named IsapiTools.lib
    3. Add the directory containng IsapiTools.lib file as an additional “library” directory
    4. Add IsapiTools.lib, crypt32.lib, and advapi32.lib as additional Link dependencies 

I did all of my experiments with the copy from IIS Resource Kit Tools. If you see a different behavior with your version, I suggest you abandon it and use the official one from Microsoft.

Configuring CustomAuth

Regarding your question on how to get CustomAuth to work – I really suggest reading and following the instructions contained within CustomAuth.ini – because if you do not follow instructions, you will not get prompted and it just looks like it is not working. Users of URLScan should be familiar with this approach; incidentally, the same author wrote both ISAPIs.

For the benefits of future readers, I am going to reproduce the setup instructions from CustomAuth.ini below and describe why each step is necessary, then show one semi-automatic way to configure CustomAuth.

; Sample CustomAuth.ini
; For CustomAuth.dll to function correctly, the following steps
; must be taken:
; – It must be running on at least IIS 6.0
; – CustomAuth.dll must be installed as a filter
; – CustomAuth.dll must be installed as a wildcard script map
; – In the wildcard script map dialog, “verify file exists” must
; be unchecked.
; – CustomAuth.dll must be allowed in the IIS MMC Web Service
; Extensions folder
; – CustomAuth.ini must exist in the same directory as CustomAuth.dll
; – If not using the built-in logon and logoff pages, then the
; specified pages must have ACLs set such that the Internet Guest
; Account must have access to them
; – For any pages that require successful logon, set an ACL that
; denies the anonymous user account read access.

Since CustomAuth is customized authentication solution modifying IIS’s anonymous authentication behavior, proper configuration is more than just a checkbox, even though using it is exactly the same as other Authentication protocols. The reasons each step is neccesary is as follows:

  • IIS 6 is required because the wildcard application mapping portion requires HSE_REQ_EXEC_URL to change the NT user token used to access the requested resource. It completes the CustomAuth protocol which uses a web page to gather NT username/password.instead of the plain user login dialog.
  • Installation as ISAPI Filter is important because it traps the 401 on denial by ACL of the anonymous user on the protected resource and initiates the CustomAuth protocol to send the login webpage which gathers NT username/password to the client.
  • Installation as wildcard application mapping is necessary because it allows CustomAuth to automatically apply to all resource extensions in the protected URL namespace.
  • “Verify file exists” must be disabled because some of the internal request traffic used by CustomAuth to complete its protocol (such as receiving the login webpage which gathers the NT username/password) uses non-existent URLs (from IIS’s perspective), and we do not want IIS to prematurely interfere.
  • CustomAuth.dll must be enabled in Web Service Extensions because it has an ISAPI Extension component configured as wildcard application mapping which must function…
  • CustomAuth.ini must be located with CustomAuth.dll because that is how the ISAPI is written to look for its configuration.
  • If you customize the login/logout web page, then it must be accessible to the Anonymous user or else when CustomAuth redirects the browser to those web pages, the browser gets rejected because it is not authenticated nor is anonymous access allowed. You need to avoid a catch-22 when protecting URLs with content that is also a URL…
  • To determine which resource requires authentication via CustomAuth, you MUST disable access to it by the Anonymous user. This triggers IIS to send the 401 denied by ACL (which gets captured by the ISAPI Filter portion and kicks off CustomAuth protocol).

Sample Automation to Install CustomAuth

Here is a little batch script snippet that automates configuration according to the rules in CustomAuth.ini. You have to put in the appropriate values for the various scripts up front – such as the website to install CustomAuth on, location of the CustomAuth ISAPI DLL, location to the script tools, and the sample URL to test CustomAuth with. No, the script is not idempotent; if you run it >1 times the result may not be correct, and you may lose custom ISAPI filter configuration, but the point is to illustrate one way to configure CustomAuth.

  • ChgList.vbs can be found here (Manipulates LIST properties like an array)

  • FiltTool.js can be found here (Manipulates ISAPI  Filters, global or site)

  • documentation to built-in IIS sample administration script tools can be found here

[04/25/2006 – modified to use FiltTool.js to manipulate filters]

SET CMD_ADSUTIL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\adsutil.vbs
SET CMD_CHGLIST=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\chglist.vbs
SET CMD_FILTTOOL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\filttool.js
SET FILE_PROTECT=%SYSTEMDRIVE%\Inetpub\wwwroot\pagerror.gif

%CMD_FILTTOOL% -action:add -site:%WIBSITE% -name:CustomAuth -dll:%FILE_ISAPI%
%CMD_CHGLIST% %WEBSITE%/root/scriptmaps first *,%FILE_ISAPI%,1 /insert /commit
%CMD_IISEXT% /addfile %FILE_ISAPI% 1 CustomAuth 1 “MS CustomAuth Sample”
%CMD_ADSUTIL% set %WEBSITE%/root/AuthFlags 1
CACLS %FILE_PROTECT% /P Administrators:F


Th test is based on trying to access /pagerror.gif, which will trigger CustomAuth login and display the built-in login dialogbox. If it does not display for you because https:// is not enabled, then you need to configure UseSSLForFormSubmission=0 (or you can enable SSL by using SelfSSL from the IIS Resource Kit Tools…).


Based on your description, I suspect that you either did not  force that all URLs under CustomAuth protection to be Anonymous Authentication only, or you failed to deny Anonymous user read ACL to the protected resource. In either case, the ISAPI-related configuration all looks good and CustomAuth.dll loads into memory as verified by “TASKLIST /m CustomAuth.dll”, but without removing anonymous user access to protected resources to trigger 401 as well as force Anonymous authentication by clients, you will not see any prompts for credentials.


Comments (24)

  1. Vlad says:

    David, are the other options to rewrite or simulate what CustomAuth does in .NET? One example would be for SharePoint Single Sign On solution. Since SPS has their own ISAPI filter to manage url, what customers are doing is using CustAuth to intercept calls before SPS. Any other approaches besides writing it in C++?

    Thanks a lot.

  2. David.Wang says:

    Vlad – You cannot do what CustomAuth does with .NET and IIS6.

    CustomAuth simply provides you a way to use a URL-based Form to retrieve username/password which is converted to a NT User Token on the server and assigned to IIS for use on a given request.

    .NET has no way to assign a NT User Token to IIS. You may be able to do this with HttpModules for purely managed resources, but there is no way to provide CustomAuth behavior using managed code.

    The ability to fully extend server behavior using both native and managed code is something we are doing in IIS7.

    This seems like something interesting to clarify with examples. I am soliciting for questions/analysis of SharePoint/IIS customization:


  3. Richard says:

    I am using CustomAuth.dll as ISAPI filter for form authentication in SharePoint Portal server. It works fine. But after a iisreset somehow the authentication doesn’t work anymore. I get the an error: "You cannot view this area, either because the area no longer exists or because you do not have the rights to view this area".  

    Do you have any suggestion?

    Thank you in advance

  4. Han Chaal says:

    Greetings Richard,

    The reason you got the error was because you must turn off Anonymous Access BEFORE resetting IIS. This bug is inherent with IIS6 and no, there is no patch for it to the best of my knowledge.

    The only way around this is to goto IIS and disable Anonymous Access for all the sites, then reset IIS, and then re-enable Anonymous Access. Some users do not encounter this problem for some reason which is why this bug is not readily apparant.

  5. David.Wang says:

    Richard – do not use IISRESET on IIS6. You should never need to use it on IIS6, and it does bad things to IIS6 including possibility of corrupting configuration. Please describe WHY you are using IISRESET, and whether you are running in IIS5 Compatibility Mode or IIS6 Worker Process Isolation Mode

    Han – Can you give definitive steps to reproduce the issue? Without definitive repro steps, I can tell you that it will *never* be patched. In other words, since software is a creative process, without user feedback, issues will never be addressed and you simply have to live within your own silence.

    I suspect that something simpler explains what you are seeing, but it cannot be diagnosed unless it can be readily reproduced.

    Right now, one thing I am considering is if Integrated authentication is in use across a worker process recycle because that can lead to loss of authentication/rights to a website without the client noticing. But that is a known condition.


  6. Richard says:

    Thank you, Han and David.

    You are right. I Do not need resset IIS.  The only reason I use iisreset is that it can reproduce the problem. In my case, after I configure the CustomAuth,  it works fine. However,  I always got same error message the next day. It means I have to reset IIS every day to get CustomAuth to work.

    By the way, IIS is running in IIS6 Worker Process Isolation Mode ("Run WWW service in IIS 5.0 isolation mode" is unchecked), and ASP.Net is 1.1.4322.

    Thank you again,


  7. Han Chaal says:

    Greetings David Wang,

    The reason I reset IIS was to reload the web.config file for that site as I had made changes to the SafeControl Assemblies for my custom web parts. To the best of my knowledge that is the only way to reload the web.config file?

    But I agree that resetting IIS should only be done as a last-resort. If anyone has a better way please let me know.

    Best Regards

    Harnam S Chaal

  8. Akif says:

    Hi  David Wang

    I m using a wild card ISAPI extension with SPS. It is working properly but when I reset IIS, it generates an exception in its CHttpServer::GetExtensionVersion method and I have to first remove it, then start SPS and then again register it and it works fine then.

    So is there any way to avapod this entire cycle again and again or it is the prob at my side.

    Regards and Thanx,

    Syed Akif kamal

  9. David.Wang says:

    Akif – the problem has to be with your ISAPI Extension DLL since IIS is simply loading/invoking the ISAPI and the ISAPI is responsible for handling whatever environment it is placed in. Your workaround illustrates that your ISAPI has some problem with the environment; you have to figure out what it is.

    My guess is that SPS uses one version of the .Net Framework in the process while your ISAPI DLL triggers code requiring another .Net Framework version to load… which fails due to the well-known CLR Versioning issue.

    You will have to debug GetExtensionVersion of your ISAPI DLL to figure out what is going on. This is easy to do using:

     "sx e ld <module name>"

    with the debuggers in the the freely available Microsoft Debugging Toolkit.


  10. Akif says:

    Thanx for ur reply.

    Actually I m calling managed Assemblies of SPS from the ISAPI. So if these assemblies are not loaded before hand, my ISAPI fails to load them. Is there any workaround, a proper one 🙂

    Regards and thanx

    Syed Akif Kamal

  11. Rasmus Foged says:

    Just wanted to share that you can use .NET to make what customauth does. But this does not apply to Sharepoint – only  .NET/classical ASP solutions.

    You will need the activets library and a custom HttpHandler,  map all request to your handler and write the token so IIS interpets the handle.


  12. David.Wang says:

    Rasmus – I do not think what you are describing is comparable to what CustomAuth does.

    CustomAuth calls LogonUser with the username/password to get a NT User Token, and then calls HSE_REQ_EXEC_URL with the NT User Token to change the Impersonation Token IIS uses to execute the remainder of the request.

    The only ways to modify the IIS Impersonation Token are through authentication protocol, ISAPI Filter subscribing to either SF_NOTIFY_PREPROC_HEADERS, SF_NOTIFY_AUTHENTICATION, or SF_NOTIFY_AUTH_COMPLETE event, or ISAPI Extension calling HSE_REQ_EXEC_URL. No .Net library can call any of these functions.

    I can believe that your component changes the thread token that remains in effect for the remainder of the .Net request, but it is in no way similar to the what CustomAuth does. If it was real, then it would also apply to Sharepoint (like CustomAuth) because Sharepoint is merely a .NET solution.


  13. Mark says:

    David – You last post seems to seems to contradict another post you made awhile back on another topic. (I’m certain I’m just not quite understanding something)…

    your last post replying to Rasmus Foged on Monday, March 13, 2006 5:00 AM, you say "No .Net library can call any of these functions."  referring to ‘HSE_REQ_EXEC_URL’ along with a number of other functions.

    But how then does the DefaultHttpHandler in IIS6 (with ASP.Net 2.0) call ‘HSE_REQ_EXEC_URL’ as you mention in your other thread post at:

    The specific post I’m referring to was made on:

    Monday, October 24, 2005 9:41 PM

    I know it’s a little off topic, Sorry,


  14. David.Wang says:

    Mark – What I mean is that:

    1. Microsoft does not provide any .Net library nor managed code to directly call those native code functions.

    2. Microsoft does provide limited and wrapped access to those native code functions from managed code (for example, DefaultHttpHandler is indirect way for User to control pszUrl of HSE_REQ_EXEC_URL – ASP.Net ISAPI DLL is mapping between User’s Managed Code modules and ISAPI, so it translates the String returned by DefaultHttpHandler into pszUrl on its internal HSE_REQ_EXEC_URL call. But, User Managed Code module STILL has no way to set any other HSE_REQ_EXEC_URL properties like request header, entity body, user token…)

    3. It’s just code, so User can always write native/managed interop to allow Managed code access to those native code functions.

    Most users cannot do #3, so it is easier for me to say "No .Net library can call any of these functions." than "Microsoft does not provide any .Net library to call those functions". Because if I do, the next User question would be "since you did not say it is impossible, how do I do that with managed code"… and that would be opening a Pandora’s Box of questions and design details beyond most people’s comfort level.

    In other words, if you are not satisfied with #2, you will either have to wait for us to change #1 (in IIS7), or you will need to figure out how to do #3 yourself.


  15. Mark says:

    Great explanation, thanks for clearing that up for me.


  16. Ramon says:

    Hi David,

           i’m trying to use CustomAuth with my SPS and I’m having some problems. After I have readed your topic I have achieved to see the built-in logon page when requesting a page. My problem is when I push de "Log On" Button, then browser is redirected to "Logon" and a 404 Error is displayed. This seems like CustomAuth.dll is not intercepting this Page. Can you help me?

    Thanks in advanced

  17. Ramon Espuga says:


        I’m here another time. Finally seems that I have solved the 404 error. I think the problem was with IISRESET bug or with Exclude Path.

        The problem now is that I cannot enter into Sharepoint, I’m pushing the submit button and the Logon.htm page displays another time. I don’t know if CustomAuth.dll is validating me or not, is there any way to debug the process or to see where is the problem?

  18. Ramon Espuga says:

    Hi David,

            finally CuastomAuth is working in my development environment, but not in my real server. But this is not the question this time. Now I’m trying to make a redirection back to the user’s requested page before the authentication and I’m not sure how to make it. Can you help me?

  19. Tom says:

    I have never been able to get this option to work any assitance would be helpful.

    I installed the ISAPI Filter and set Mapping for the IIS extensions and webservices extensions.

    I set the appropriate settings and paths in the .ini file.

    Yet when I go to the portal site I still get the default login prompt from IIS and it also apears to block me from other items even after I have logged in. so I get 2 prompts and the second no longer works.

  20. David.Wang says:

    Tom – Since we are talking about security and it is completely custom, every single detail has to be perfect or else it won’t work. Please re-read the blog entry to understand and follow all details because that is all you need to get CustomAuth to work with default IIS.

    For example, CustomAuth *requires* ONLY Anonymous authentication to be enabled, any custom portal applications *must* work with ONLY anonymous authentication enabled, but all protected resources must DENY the anonymous user access.

    You may need to do additional steps to get other applications on IIS to work with it, but since the application is arbitrary, the steps are also arbitrary.


  21. Johnny says:

    Windows Authentication & CustomAuth

    I am struggeling to find a solution to my fairly basic problem. On the global intranet site I am developing the authentication is done using Windows Authentication and Active Directory groups. Our company has a desk sharing policy and in some cases multiple users share 1 PCs. My website would then require all these users to constantly be logging in and out. I looked at the CustomAuth Extension in Detail, to me this looks like all users would then have to type in username and password.

    Ideally I would like to have a start page which has 2 links:

    1. Login with current Windows User

    2. Login as a different User

    When 1. is pressed the current Windows credentials would be used without any manual typing. When 2. is pressed a page (or pop-up) appears where the credentials can be entered manually.

    This solution would solve many of my issues:

    – if a user with auth is sitting at the PC of a user without auth, then the credentials can be entered of the user who has access

    – my app has 2 types of user: normal & admin….hence if  an admin user is sitting at the PC of a non admin user, then the admin can enter the required credentials

    I have looked all over and find many people looking to solve the same issue as me, but with no solution around. There are some ‘dirty’ work arounds, like starting IE using the ‘RunAs’ command, changing the IE settings to prompt  for all sites, and so forth.

    Is there a way to achieve my goal of offering my users maximum comfort & flexibility with IIS6.0?

    Will there be an easy way to achieve this in IIS 7.0? Is there an official date for release yet?

    Many thanks, I hope this blog is still being used.



  22. Tim Mansell says:

    Hi David,

    I will most likely need to use this functionality within IIS7, can you confirm whether CustomAuth will work with IIS7.



  23. David.Wang says:

    Johnny – your requirements actually have nothing to do with IIS. You want the browser to perform such logins patterns.

    IIS will accept the login no matter how the user clicked the button.


  24. junli999 says:

    my customAuth only works within a virtual directory. any link to a different virtual directory will redirect to the login page. what's wrong with it?