For users with large groups (100,120+), any webpage with integrated authentication enabled and hosted on IIS 6 did not come up and returns 401.1.
Data type: REG_DWORD
The default value for the above key is 12,000 on Win 2k3 systems but in this case we had found that the token size was never more than 7000 (figured with network traces), so it did not make sense to increase this on all the Windows Based systems in the forest.
So increasing the reg MaxTokenSize may not always help and something on the webserver needs to be tweaked to support this too.
We have this article http://support.microsoft.com/kb/820129 which discusses two reg keys MaxFieldLength and MaxRequestBytes. We increased the values of MaxFieldLength and MaxRequestBytes on the IIS server from the default of 16384 to 32768 (may vary according to ticket size) and this fixed the issue.
The full path of the registry keys is HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters. The keys have to be added as DWORD’s. Their description says
MaxFieldLength – Sets an upper limit for each header. See MaxRequestBytes. This limit translates to approximately 32k characters for a URL. Default Value – 16384, Range 64 – 65534 (64k – 2) bytes
MaxRequestBytes –Determines the upper limit for the total size of the Request line and the headers. Its default setting is 16KB. If this value is lower than MaxFieldLength, the MaxFieldLength value is adjusted.
Default Value – 16384, Range 256 – 16777216 (16MB) bytes
So, to summarize, we may need to tweak above settings for webserver as well if Kerberos authentication does not work for an account which is a part of large number of groups along with focussing at the MaxTokenSize key, since we need IIS too, to support large sized headers.