Using an HTTP Module to assist in adjusting the value of aspnet:MaxHttpCollectionKeys imposed by MS11-100


Internet applications using ASP.Net recently had an update (MS11-100) pushed out of band back in December that set a limit of 1000 items to be accepted by a web form.  While this is not the only limit imposed, it is the one that some applications are hitting.  If you exceed this value, an exception is thrown that looks like the one below:

System.Web.HttpException:
The URL-encoded form data is not valid. —> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at System.Web.HttpValueCollection.ThrowIfMaxHttpCollectionKeysExceeded()
   at System.Web.HttpValueCollection.FillFromEncodedBytes(Byte[] bytes, Encoding encoding)
   at System.Web.HttpRequest.FillInFormCollection()
   — End of inner exception stack trace —
   at System.Web.HttpRequest.FillInFormCollection()
   at System.Web.HttpRequest.get_Form()

The key here is ThrowIfMaxHttpCollectionKeysExceeded.  If that is in your stack trace, you know that you have exceeded the value the patch imposes.

To keep this exception from being thrown, you need to change the value of aspnet:MaxHttpCollectionKeys in the web.config of your application to a value that is as high as the highest allowed count of keys in your hash table. 

To help you figure out what value you should use in production, I have created a HTTP Module that counts the number of keys in that collection for each request and stores the high water mark in the event log once the module is unloaded.  I have included a sample that shows how this works.

For you to test this in your application, you need to build the MS11-1000Helper.DLL and place it in your web application’s Bin directory.  Once you have done this, you need to add the following to your web.config file to enable the HTTP Module:

<system.web>

    <httpModules>

      <add type=MS11_100_Helper.CounterModule, MS11-100-Helper name=CounterModule />

    </httpModules>

</system.web>

 

Additionally, you need to add the following to your appSettings:

<appSettings>

    <!–

      This is a super high number to use as a starting point in TESTING your application. 

      You should never have a number like this in your PRODUCTION web.confg.

      Instead, you should use this HTTP module (MS11-100-Helper.dll) to help you determine

      the appropriate number to use for your setting in PRODUCTION.

    –>

    <add key=aspnet:MaxHttpCollectionKeys value=9999 />

</appSettings>

 

Then you should TEST your application.  If you are dynamically generating Forms items in your application, you need to test with the highest amount of generated UI as you expect to ever see while your application is being run. 

After you have tested the above scenario, you need to close the browser and terminate the web server (stop the development server or issue an IISRESET).  Once you have done this, the high water mark should be written to the event log “Applications and Services Logs/CounterSourceLog” and you should have an Information entry that looks like this:

Largest Value Seen = X

Where X is some number. 

You can then use this number to help you set the value of aspnet:MaxHttpCollectionKeys in production. 

I would personally set the number to be a little higher than the value here, but that depends on the application.  If you don’t have a lot of dynamically generated forms controls, then the number is probably pretty close to what you can use for your deployed application.  You might want to add just a few more to the value just in case you have a last minute tweak that puts you over the limit.

If you have pages that can dynamically generate a ton of controls without limit (indicating a bad design), then this isn’t going to help you much.

Just be sure to remove the module and 9999 limit from your production code!!! 

There are a couple of additional caveats with this as far as being the end all to this issue. 

First, realize that MS11-100 is a workaround to help limit a DOS attack.  The final fix will be to randomize the hash tables used in the ASP.Net will be in a change that comes later.  Once that change is available, this workaround may not be needed.

Second, this helper only counts the keys in the Form collection.  If you are running into an issue where you are seeing an exception thrown for Json, then this code will not help you.

Third, if the high water mark is less than 1000, I wouldn’t recommend changing the default app setting of aspnet:MMaxHttpCollectionKeys.  A DOS attacker will probably be hitting numbers much higher than this to cause the issue to occur, which is why we choose the number to begin with.

Fourth, anytime you make an application change, you should reuse this tool in TEST to help you figure out the value of the setting in production.

Special thanks to Wyn Lewis-Bevan for the idea on developing this helper!

MS11-100-Helper.zip

Comments (32)

  1. Sam L says:

    So 1000 form fields is below any type of DoS threashold.

    What it too large a value?

    Is 3000 form fields "safe"?

  2. The value limits the impact a DoS attack would have on your website, it does not eliminate it.  Too large of a value would be some arbitraty number like X thousand that you just picked out of the air.  That is why I created this HTTP module; to give people a better idea of what value they should place in their configuration.

    Proper load testing can give you some idea on the amount of CPU generated from high value settings and is dependant on your hardware and farm configurations.

  3. Sam L says:

    Paul, thanks for the reply. I'm not picking a value out of the air – I know the value.

    I'm trying to decide whether to write code or edit the web.config 🙂

    Here is why I am asking the question.

    MS says 1000 form fields is the default limit which implies it is "safe". But you've given us a way to increase that limit — how do we know what is "safe" to increase that limit too?

    In other words,  when do we move from "safe" to "unsafe" in respect to values for aspnet:MaxHttpCollectionKey?

    btw – maybe MS is fixing the hashing function and we won't have to worry about this "work around"?

  4. Sam,  Safe and unsafe can only be determined by testing the application.  1000 was choosen because a high percentage of internet facing sites (those that would be subject to this attack) have values below this limit and the limit could be imposed by the update.  Some sites have interfaces that require a value greater than this for their applications to run correctly.  This tool helps those sites determine what the correct number should be for their site.

    Prior to the patch, there were no limits and a specifically crafted response with many keys in it could cause the CPU to spike to 100% for long periods of time causing a DoS.  There are no implied or intended terms for safe or unsafe.  We will be working on fixing the hashing funciton in the future.

  5. Mahesh Bhatt says:

    Thanks a lot. It works fine 🙂

  6. Kevin says:

    Paul, you may want to point out what web.config files need to be changed for this to work in a CRM environment.  We found that changing the config in …ReportManager on the application server and in …CRMWeb on the front end server did the trick.

    The real pain here wasn't having to add that key to the web.config, but rather it was determining what was going on in the first place.  The error messages were pretty useless and we ended up having to use log files and Fiddler to get us pointed in the right direction.  It was a good week of our time to finally determine and resolve the true problem.

  7. Thanks Kevin.  You're correct in that this can happen for applications like CRM as well.  Any ASP.Net application that takes this update may have this exception and the module can be used to assist in changing the setting on those systems as well.

  8. Yun Lian says:

    Hello Paul,

    Is the key " aspnet:MaxHttpCollectionKeys" suitable for application run on .Net 1.1?

    Thanks

  9. Hi Yun,

    The article states that MS11-100 fixes .Net 1.1 SP1, so that setting should be available, but I have not tested it on that version.  See technet.microsoft.com/…/MS11-100

  10. SuKe says:

    But how to build the MS11-1000Helper.DLL

  11. SuKe – using the source code above, you can build the DLL using Visual C# Express 2010 available for free from here: http://www.microsoft.com/…/visual-csharp-express

  12. SuKe says:

    Hi,Paul,Thank you very much for your reply.But I hve two problems: first,It always has a problem about config file when I built the source code above with VS2005;second ,where is the 'Applications and Services Logs/CounterSourceLog'?So can you give me a suggest what i should do next

  13. Hi Suke.  The source code was built with VS 2010, I have no idea what will occur when you try and use VS 2005 to build this solution as I did not target that platform.  Please us the compiler from the link above.  The 'Applications and Services Logs/CounterSourceLog' is a available in the Event Viewer.  Expand the 'Applications and Services' folder under the Event Viewer and select the CounterSourceLog to see your results.  This was tested under the latest releases of Windows Server 2008 R2 and Windows 7.  You can always modify the source code to change the behavior to write to a text file if you want, you do not need to log to the event log.

  14. ND says:

    in IIS 7 when this value at server level…it doesn't propagate it to individual applications, even though IIS 7 claims on individual site/app that the setting is inherited.

    Any ideas?

  15. ND – I haven't tried this, but my guess is that you will need to add the MS11-1000Helper.DLL to the GAC instead of placing it in your web application’s Bin directory.

  16. ND says:

    Hi Paul, really appreciate a response. I'm sorry I wasn't asking about the helper.

    Basically we want to increase this setting for all of about 25 sites on this particular IIS7 server.

    However, setting in IIS App Settings doesn't seem to help individual sites. We end up having to set it in the individual web.config's of individual sites.

    Things is we have more servers with more individual sites. Do you know if using MaxHttpCollectionKeys setting at server level for some reason doesn't propagate down to sites?

    I'm asking here because I can't find any answers anywhere else.

    Thanks

  17. Hi ND,  Sorry, I misunderstood your original question.  In looking at the documentation, your setting should work, however you have found otherwise, sorry about that.  Since this is a problem relating to a security update, you can contact support for assistance.

    Customers in the U.S. and Canada can receive technical support from Security Support or 1-866-PCSAFETY.  There is no charge for support calls that are associated with security updates.

    International customers can receive support from their local Microsoft subsidiaries. Again, there is no charge for support that is associated with security updates.

  18. ND says:

    Hi Paul,

    turns out that you can't do it for all sites on a server, only individual web.configs.

    forums.iis.net/…/2020036.aspx

  19. Hi ND – that is what I suspected as well.  Thanks for the pointer.

  20. Kamran Shahid says:

    It's working fine with Windows 7 or windows 2008 server.But we are getting error in windows XP

    Security Exception

    Description: The application attempted to perform an operation not allowed by the security policy.  To grant this application the required permission please contact your system administrator or change the application's trust level in the configuration file.

    Exception Details: System.Security.SecurityException: The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security.

    Source Error:

    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

    Stack Trace:

    [SecurityException: The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security.]

      System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly, Boolean wantToCreate) +664

  21. It looks like it is failing trying to write to the event log.  Why don't you change the code and have it write to a text file?  That would give you the same result.  You can get the compiler for free from here – http://www.microsoft.com/…/visual-csharp-express

    I crated this code to work on Windows 7 and Windows 2008.  If you continue to have problems, then my recommendation is that you upgrade from Windows XP.

  22. JG says:

    Hi Paul

    You said –

    First, realize that MS11-100 is a workaround to help limit a DOS attack.  The final fix will be to randomize the hash tables used in the ASP.Net will be in a change that comes later.  Once that change is available, this workaround may not be needed.

    Do you know what the status is on the final fix ?

    Thanks

  23. Hi JG.  

    I'm not aware of the final fix being available at this time.  I would suspect this to be available on newer versions of the framework, but I have no insight into when.  Scott Guthrie references that this will likely to be fixed in the future, but does not give a timetable.  weblogs.asp.net/…/asp-net-security-update-shipping-thursday-dec-29th.aspx

  24. Jay says:

    Hi Paul,

    I am getting "Server Error in '/' Application" error if i add this DLL to my web application bin folder and add the code lines you mentioned in web.config. My application is VS 2008 application and we are just trying to figure out the best httpcollectionkey value for our application.

    Do i need to make any change in the MS11-100-Helper app before i add it to my application bin?

  25. Jay,

    You will probably need to rebuild the DLL for your target platform; I did not test this on anything other than VS 2010, sorry.  

    Paul

  26. Craig Nicholas says:

    Hi Paul,

    Thanks for the post. The appSettings change works fine for reports accessed via Report Manager but when I attempt to view the report via a Web Browser control in my Win Forms application, the error occurs as if it has not processed this setting.

    Where should this configuration setting be applied in this case?

    Thanks

    Craig

  27. Craig Nicholas says:

    I found the answer. The setting needed to be added to the ReportServer web.config too.

  28. Thanks for the update Craig!

  29. The Brazilian says:

    I'd like to know the limit, if it parameter has one.

    Please, various blogs sites etc explain that "you have to redesign your appl" bla bla bla.  Thanks a lot, but, I only want to know the limit OR if it is possible to put <add key="aspnet:MaxHttpCollectionKeys" value="-1" /> for unlimited.

    Paul, you are the guy!!

  30. The Brazilian – I don't see a limit listed.  The value is an Int32 type, so you should be able to set it to the MaxValue for that type.  The downside of this is that Increasing this value above the default setting increases your susceptibility of your server to the Denial of Service vulnerability.

  31. cheong00 says:

    Hello.

    Ping for status update. Is it safe to lift the limitation now (i.e.: have the DOS issues been fixed)? Thanks.