HOWTO: Retrieve Request Headers using ISAPI, ASP, and ASP.Net


Developers frequently confuse Request Headers, Response Headers, and Server Variables as well as the appropriate syntax to retrieve/manipulate each of them, depending on the API (ISAPI, ASP, and ASP.Net). I am going to clarify all of this right now. 🙂


Question:


So I’ve managed (thanks to your samples) to create a filter that adds a header. I call it SM_USER. When I use ASP to enumerate all the headers I can see it as HTTP_SM_USER in the ALL_HTTP and ALL_RAW header (with a value set).


Unfortunately though when I list out the HTTP_SM_USER variable there is no value for it.


I am setting it in the onPreProcessHeaders with a call sort of like this


fRet = pPPH->SetHeader(pfc, szMyHeader, szMyHeaderValue);


The question is: What’s happening here? Why can I see it in the ALL but not when I list out the particular header?


Cheers,


Answer:


Request Headers, Response Headers, and Server Variables represent three totally different logical concepts available to server-side applications (though not all APIs expose these logical concepts), and you manipulate them in different ways.


The main syntactic sugar which exists for legacy compat reasons with CGI is that Server Variables ALSO allow retrieval of certain Request Headers when using the special HTTP_ prefix.


The following is how I treat the logical concepts…


Request Headers


Request Headers represent the header name/value pairs that the HTTP client, such as a web browser, sent to the server. A typical request looks like the following, and request headers are highlighted:

GET / HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n

\r\n

Response Headers


Response Headers represent the header name/value pairs that the HTTP server, such as a web server, sends to the client. A typical response looks like the following, and response headers are highlighted:

HTTP/1.1 200 OK\r\n
Content-Type: text/html\r\n
Server: Microsoft-IIS/6.0\r\n
Date: Thu, 20 Apr 2006 09:00:00 GMT\r\n
My-Response-Header: Dash\r\n
My_Response_Header: Underscore\r\n
Content-Length: 11\r\n

\r\n
Hello World

Server Variables


Server Variables represent runtime state available to the server-side application for request processing. The list of variables and syntax available on IIS is described here. They include state like:



  • SCRIPT_NAME (client-requested URL, such as “/”)
  • AUTH_TYPE (type of authentication performed)
  • etc

As stated earlier, it also supports syntactic sugar to access Request Headers via the HTTP_ prefix, in accordance to the CGI 1.1 Specification.


The CGI Specification for Accessing Request Headers


The CGI 1.1 Specification defines the HTTP_ prefix to retrieve Request Headers. Dashes in Header Name transform into Underscores with HTTP_ prefix prepended and make them available as Server Variables (in the CGI context, available as Environment Variables).


In other words, the HTTP_ACCEPT_ENCODING Server Variable name retrieves the Accept-Encoding: Request Header.


The astute reader should notice a quirk between the HTTP and CGI specifications – in particular, how does one retrieve the Accept_Encoding: Request Header in CGI???


Remember, HTTP Request Header names are message-headers, which by definition are (I have collected all the relevant BNF definitions for reference):

       message-header = field-name “:” [ field-value ]
field-name = token
field-value = *( field-content | LWS )
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>

token = 1*<any CHAR except CTLs or separators>
separators = “(” | “)” | “<” | “>” | “@”
| “,” | “;” | “:” | “\” | <“>
| “/” | “[” | “]” | “?” | “=”
| “{” | “}” | SP | HT

OCTET = <any 8-bit sequence of data>
CHAR = <any US-ASCII character (octets 0 – 127)>
CTL = <any US-ASCII control character
(octets 0 – 31) and DEL (127)>
SP = <US-ASCII SP, space (32)>
HT = <US-ASCII HT, horizontal-tab (9)>

LWS = [CRLF] 1*( SP | HT )
TEXT = <any OCTET except CTLs,
but including LWS>


According to HTTP specifications, header names can include any CHAR except CTLs or separators. In particular, both “-” (dash) and “_” (underscore) are valid characters in header name.


The problem should be clear now. The CGI 1.1 Specification defines a substitution of “-” in Request Header to “_” in Server Varible names, but what is the substitution of “_” in Request Headers in Server Variable??? It cannot be “_” in Server Variable names, and it is conveniently undefined…


Oops! Did we just spot a flaw in a sacred specification underlying major Internet Applications? You bet. Oh no, heavens forbid that there is a flaw in a publicly implemented specification! The sky is falling!


And you can also bet that it is not the only one in existence… so the next time you think that a peculiar software behavior is due to a software bug and not a specification bug, think again… because believe it or not, even the W3C recommendations are not flawless. Remember, to err is human, and humans write those documents and software.


When troubleshooting, I always recommend to think and evaluate the situation instead of assuming one or another is magically sacred. For example, developers tend to assume that their code is perfect and that the flaws must be from the system.


The Solution in IIS 6.0


Yes, we decided to do something unspecified in IIS 6.0 to resolve this situation. Otherwise, how can you possibly retrieve the SM_USER request header within ASP?


We introduced the HEADER_ prefix for Server Variable names which uses the name as-is to retrieve request headers. In other words:



  • ServerVariable( “HEADER_SM_USER” ) retrieves SM_USER: Request Header
  • ServerVariable( “HEADER_SM-USER” ) retrieves SM-USER: Request Header

It is a neat solution to the problem. Yes, it is unspecified, but I think users are better off with than without it.


Table of Operations


Whew… soapbox aside, the following table categorizes the available syntax to retrieve the three logical concepts of Request Header, Response Header, and Server Variable. Without parsing, of course…




























API / Concept Request Header Response Header Server Variable
ISAPI Filter PreprocHeaders:
GetHeader(“Header_As-is:”)

AuthComplete:
GetHeader(“Header_As-is:”)

Any event after PreprocHeaders:
GetServerVariable:
HTTP_ prefix
HEADER_ prefix
SendResponse:
GetHeader(“Header_As-is:”)

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ISAPI Extension

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ASP

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix

ASP.Net Request.Headers(“Header_As-is”)

GetServerVariable:
HTTP_ prefix
HEADER_ prefix
Not Possible

GetServerVariable:
HTTP_ prefix
HEADER_ prefix


The key take-aways:



  • ISAPI Filters can retrieve everything as-is, assuming you are in the right filter event. You just need to remember that header names MUST have the “:” appended, while GetServerVariable() calls using HTTP_ or HEADER_ prefix do NOT have “:”
  • GetServerVariable() calls using HTTP_ or HEADER_ prefix work everywhere (after PreprocHeaders event for ISAPI Filters) to retrieve request headers, especially when using HEADER_ prefix
  • ASP.Net offers the Request.Headers collection to give you parsed access to the Request Headers
  • Notice that ASP and ASP.Net behavior mirror that of ISAPI Extension… because they are implemented as ISAPI Extension DLLs on IIS

ASP Example:


For example, using WFetch, try making the following raw request:

GET /GetServerVariable.asp HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n

\r\n

And have the following source code for /GetServerVariable.asp

<%
headerDash = “My-Request-Header”
headerUnder = “My_Request_Header”
HTTP_headerDash = “HTTP_” & headerDash
HTTP_headerUnder = “HTTP_” & headerUnder
HEADER_headerDash = “HEADER_” & headerDash
HEADER_headerUnder = “HEADER_” & headerUnder

Response.Write( HTTP_headerDash & ” = ” & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & ” = ” & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & ” = ” & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & ” = ” & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
%>


You get the following output:

HTTP_My-Request-Header = Dash\r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = Dash\r\n
HEADER_My_Request_Header = Underscore\r\n

Notice that HTTP_ syntax only retrieves the header name with dashes, while HEADER_ syntax retrieves both headers with dashes and underscores individually.


ASP.Net Example


For example, using WFetch, try making the following raw request:

GET /GetServerVariable.aspx HTTP/1.1\r\n
Host: localhost\r\n
Accept: */*\r\n
My-Request-Header: Dash\r\n
My_Request_Header: Underscore\r\n

\r\n

Have the following source code for /GetServerVariable.aspx

<%
dim headerDash,headerUnder, HTTP_headerDash, HTTP_headerUnder, HEADER_headerDash, HEADER_headerUnder

headerDash = “My-Request-Header”
headerUnder = “My_Request_Header”
HTTP_headerDash = “HTTP_” & headerDash
HTTP_headerUnder = “HTTP_” & headerUnder
HEADER_headerDash = “HEADER_” & headerDash
HEADER_headerUnder = “HEADER_” & headerUnder

Response.Write( HTTP_headerDash & ” = ” & Request.ServerVariables( HTTP_headerDash ) & VbCrLf )
Response.Write( HTTP_headerUnder & ” = ” & Request.ServerVariables( HTTP_headerUnder ) & VbCrLf )
Response.Write( HEADER_headerDash & ” = ” & Request.ServerVariables( HEADER_headerDash ) & VbCrLf )
Response.Write( HEADER_headerUnder & ” = ” & Request.ServerVariables( HEADER_headerUnder ) & VbCrLf )
Response.Write( “Header(” & headerDash & “) = ” & Request.Headers( headerDash ) & VbCrLf )
Response.Write( “Header(” & headerUnder & “) = ” & Request.Headers( headerUnder ) & VbCrLf )
%>


You get the following output:

HTTP_My-Request-Header = \r\n
HTTP_My_Request_Header = Dash\r\n
HEADER_My-Request-Header = \r\n
HEADER_My_Request_Header = \r\n
Header(My-Request-Header) = Dash\r\n
Header(My_Request_Header) = Underscore\r\n

With ASP.Net, the Headers collection is the only way to get all the header values. HTTP_ prefix only retrieves the header name with dashes, as expected.


Conclusion


Whew… a very long winded entry. 🙂


But… I wanted to be thorough on the subject of Request Headers, Response Headers, and Server Variables when it comes to ISAPI, ASP, and ASP.Net. So, now you have everything…


//David

Comments (32)

  1. Jeff Parker says:

    Well I don’t know what to say. Here I thought I knew a heck of a lot about the response, request and server variables. I mean I have been doing web dev for about 13 years now. I have read the specs, never noticed the flaw, I just kind of accepted the specs as they have been around for years.

    I love it and hate it when do this to us. Love it cause I really learn something new and it makes me think. Hate it because now I will be spending the rest of the day churning over code in my head I have written for several years and processing this info and writing my own samples to play with my new found knowledge.

    However good post. I learned a heck of a lot now I just need to digest it all.

  2. Phylyp says:

    <trivia>

    The HTTP_SM_USER header is used by Netegrity SiteMinder, for holding a single sign-on user ID.

    </trivia>

    In a previous project, we ran into problems where the – to _ replacement was causing a bit of a pain, in our ASP code that needed to parse the user’s SSO.

    However, our ever-friendly SiteMinder team changed the header to HTTP_SMUSER, thus alleviating our woes.

  3. David.Wang says:

    Jeff – hehe. My job is done, then. 😉

    I actually started writing more code snippets in ISAPI, ASP, and ASP.Net before I decided to tone it down and just focus on the basics. I’ll leave those details to another time.

    //David

  4. David.Wang says:

    Phylyp – well… now you know why all this is going on and how to account for it in various web framework technologies that run on IIS. 🙂

    I know you all think we are crazy sometimes, but we are not that random. 😉

    //David

  5. … I don’t see the ambiguity.

    OK, Some-Header: and Some_Header: both map to the same environment variable.  Fine.  That just means the header-to-environment mapping isn’t one-to-one.  We knew that anyway… environment variables aren’t case-sensitive, for example.

    So as far as CGI is concerned, the following are all the same:

    Some-Header: a

    Some_Header: a

    sOmE-HeAdEr: a

    Isn’t this only a problem when you have two headers, both important to your app, that differ only in case or "- vs. _"?

  6. David.Wang says:

    Maurits – The classic problem here is when you are an ISAPI, like ASP or ASP.Net, and want to retrieve a Request Header with "_" (underscore) using Server Variable… because you cannot.

    For example, SM_USER used by SiteMinder. You will have to parse the ALL_RAW Server Variable to get it in ASP. On IIS6, you can finally use ServerVariable( "HEADER_SM_USER" ) and get it declaratively, without parsing.

    //David

  7. This much is clear:

    * On IIS, HTTP_SM_USER does not catch the value of SM_User:

    You imply that * is a weakness in the CGI spec. In particular, you claim that there is no CGI means to access the value of SM_User: via server/environment variables.

    I disagree.  I think there is a CGI means, and HTTP_SM_USER is precisely it.

    Therefore I consider * a bug in IIS.

    See this post:

    http://channel9.msdn.com/ShowPost.aspx?PostID=184522

    And this repro/analysis:

    http://www.geocities.com/mvaneerde/sm_user.txt

    Note at the end of the analysis that there is at least one third-party CGI ISAPI that correctly pulls SM_User: from HTTP_SM_USER.

  8. David.Wang says:

    Maurits – Sorry, but you still have not provided any proof that this is an issue with IIS.

    You should know by now that just because it is a KB does not mean it is a bug in a Microsoft product… the Microsoft KB system is quite broken, in my opinion – if you only knew how it really works… you’d be surprised that useful information comes from it at all. It’s part of the reason that I blog – because I want the KB to go away.

    All it means is that someone called in due to some confusion, wanted to do something but couldn’t, and someone in PSS wrote a KB article explaining it. The person writing the KB article usually NEVER consults the product team on whether it is right or wrong – they usually just assume the product is wrong and move on. They are paid to create that KB, not whether it is correct. This is why for most interesting keywords, you see DOZENS of KB articles that run the entire gamut of clarity, conciseness, or correctness.

    Yes, a major case of left hand not knowing what the right hand is doing.

    Anyhow, back to the discussion at hand. I can vent about the MS KB system at another time when they really piss me off. 🙂

    The argument is very clear in my mind: show me:
    1. a CGI 1.1 EXE (not ISAPI) that can
    2. correctly and individually retrieve header values for BOTH SM_USER and SM-USER headers
    3. from the same request
    4. using JUST the HTTP_ prefix syntax

    and I will file a bug for IIS7 to get it fixed.

    ISAPI can work around the HTTP_ specification flaw by parsing ALL_RAW or ALL_HTTP server variables for headers. That is not interesting to me – the specification flaw is what we are discussing.

    Bottom line: the CGI specification is flawed as soon as it gave a mapping that is not bijective. Period. End of discussion.

    BTW: None of the “solutions” offered at geocities.com will work for a request with both SM_USER and SM-USER request headers. They work if EITHER SM_USER or SM-USER are given. And since we are nitpicking over specifications, it is the mere possibility, not practicality (I can’t see applications intentionally using SM_USER and SM-USER as different things) that matters. Meaning if one gives code snippets, it most likely missed the boat.

    //David

  9. > show me:

    > 1. a CGI 1.1 EXE (not ISAPI) that can

    > 2. correctly and individually retrieve header values for BOTH SM_USER and SM-USER headers

    > 3. from the same request

    > 4. using JUST the HTTP_ prefix syntax

    You have very clearly illustrated the non-one-to-one-ness of the CGI mapping.

    This is NOT the bug:

    — REQUEST —

    SM_User: foo

    SM-User: bar

    — CGI ENVIRONMENT —

    HTTP_SM_USER: ? // cannot be both foo and bar — THIS IS NOT A BUG IN IIS but a FUNDAMENTAL result of the CGI mapping

    IIS is free to do what it likes here.  CGI does not specify what to do if two headers are redundant.

    This, on the other hand, IS the bug:

    — REQUEST —

    SM_User: foo

    — CGI ENVIRONMENT —

    HTTP_SM_USER: foo // well, the expected result is "foo"; IIS gives "". THIS IS A BUG IN IIS.

    Other CGI implementations do not have this bug, and CAN sniff the SM_User header through HTTP_SM_USER — provided that it is not masked by a SM-User header.

    The bug, in particular, lies in ISAPI’s GetServerVariable(…)

    This blog doesn’t allow indentation, so I’ve posted pseudocode here:

    http://www.geocities.com/mvaneerde/sm_user2.txt

  10. There’s a bug in the KB article, too.  It says "Retrieve and parse the HTTP_ALL header variable" instead of "Retrieve and parse the ALL_HTTP header variable" — I’ll report that through the appropriate method.

  11. David.Wang says:

    Maurits – I think we are merely arguing whether the glass is half empty or half full.

    One can argue that the CGI specification says that HTTP_SM_USER should retrieve SM_USER or SM-USER, whichever one exists, and if both exist, the behavior is unspecified.

    Or HTTP_SM_USER only retrieves SM-User, there is no way to retrieve SM_USER, and there is no unspecified behavior.

    Yes, the specification did not say “you cannot retrieve headers containing ‘_'”, so you may be inclined to say that IIS interpretation is too narrow. However, it also did not say “you cannot retrieve headers containing CTRL characters”, either… so it is a matter of whether one wants unspecified or underspecified behavior.

    Neither option is more “correct” than other, so until the CGI specification clarifies the confusion, there is no way for one to say “this is a bug in X”. This is the problem with an underspecified specification – things get too open for interpretation. 😉

    //David

  12. FWIW, here’s the CGI/1.1 RFC:

    http://www.ietf.org/rfc/rfc3875.txt



    4.1.18.  Protocol-Specific Meta-Variables



    The HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "_" and has "HTTP_" prepended to give the meta-variable name

  13. David.Wang says:

    Maurits – Good information…

    But, this is not a specification… it’s more like an "oh, we did it this way; we think it should be codified".

    And it still does not deal with what happens with retrieving header values for requests containing both SM_USER and SM-USER headers. tsk tsk.

    And it is dated after all versions of IIS under discussion, so kinda moot point except for maybe IIS7, but it’s not an endorsed specification, so it is hard to rally behind it

    //David

  14. … and both of the authors work for Apache 😉

  15. There’s also this Internet-Draft from 1999, by IBM and E*Trade…

    http://cgi-spec.golux.com/draft-coar-cgi-v11-03-clean.html#6.1.5

    Each HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "_", and has "HTTP_" prepended to form the metavariable name.



    If multiple header fields with the same field-name are received then the server MUST rewrite them as though they had been received as a single header field having the same semantics before being represented in a metavariable.

    </quote>

    That last sentence is wonderfully vague 🙂

    This draft hasn’t had a great deal of work done on it recently, though 🙁

  16. David.Wang says:

    Maurits – hehe… yeah. It’s like patting yourself on the back and writing the spec *after* the implementation. 😉

    The vague sentence is dealing with what happens when the browser sends multiple SM_USER headers on the same request (i.e. HTTP spec says to merge their values into one value, comma-delimited)… which is also wonderful information to have in a specification… but sigh, it still ignores the SM_USER and SM-USER ambiguity.

    I wonder if people just copy/paste the original sentences of the spec and then just augment/embelish the parts they are interested in…

    //David

  17. David Wang says:

    I finally have enough blog entries about various portions of IIS6 request processing that I can stitch…

  18. gaetano giunta says:

    Sorry, but I think that the spec being vague and broken is a very poor excuse in this case.

    As has been stated before, the spec misses the case of two different http headers that will result in the same converted value.

    It has no ambiguity whatsoever oh the handling of ‘SM_USER’: zero, nil, zilch. Anybody doing his best for implementing it, should treat it as valid. And even if the spec had been twice as shady and broken, the main goal for everybody is interoperability, isn’t it?

  19. David.Wang says:

    gaetano – Good point, but I hope you agree that vague/broken spec is the very bane against interoperability.

    As a general rule, when you have communication between two parties, miscommunication and misinterpretations are bound to happen, and having vague/broken specifications (which basically set the foundation of common understanding) does NOT help.

    We can argue about relative correctness of miscommunication all day long, but at the end of the day, I think it is the root vagueness that matters and NOT the various instance-manifestations of that vagueness. If you solve the root problem (mapping of headers to server variables), the instance-problems naturally solve themselves. Those are sound design principles.

    Now, I caution you against mistaking miscommunication for "not doing his/her best for implementing it". Who says that we are not doing our best?

    I mean, you should agree that EVEN IF everyone is doing their best-effort for supporting interoperability, any vagueness in the specification that causes miscommunication destroys interoperability… and you somehow want to imply that someone did NOT do their best effort? Give me a break. That logic is down-right arrogant and insulting.

    My statement is that mistakes/ambiguity in specifications happen more than you think, and they introduce mis-communication which can affect interoperability while implying NOTHING about the effort put forth by the implementors.

    //David

  20. Jim Thorstad says:

    David, I’ve found your efforts to explain the situation to be very useful. Our company has struggled to support customers doing single sign-on between our product and Netegrity SiteMinder because of the fact that the header containing the SiteMinder Id (by default, without customizations) is either sm_user, or sm-user and we see them (I think correctly) as different.

    Thus our customers get frustrated and I had to put a debug page into our product just to show our customers that its not *our* fault they spelled the header wrong.

    I’ve also done some research to educate myself why Tomcat shows headers in one case on WebSphere on Unix shows them in another case (only to later find in the spec that case is not relevant). Your thread here was helpful  as you explain why the ASP sample code I downloaded that uses HTTP_ might be contributing to our confusion.

  21. Arun Chaudhary says:

    Hi,

         I am very new in ISAPI but i have my very urgent requiremnet that i have to set a cutsom varibale and then again need to fetch it on next page or any other page like

    suppose first i wll create a custom server valriable like "username" and on in other page i will just check that it have any value or not

    like

    if request.servervariable("http_username")<>"" then

    some processing

    end if

    as im new so please if you explain me step by step how i can create ISAPI as also i dont know much more abour ISAPI please give me full step that how i can craete it and then again how i can access it in my asp page.

    Thanks :

    Arun Chaudhary

  22. David.Wang says:

    Arun – I suggest using Session Variables to accomplish what you want.

    Using ISAPI to set custom variables will not do what you describe because such variables do not "carry over" between pages.

    I suggest searching the web for basic documents on how web technologies work.

    //David

  23. Ravichandran says:

    I am trying to retrieve an header value and empty it before it is written to the IIS log. Based on the code example given I am able to access the header. But how to modify it. Primarily we are having issue with Site mind assertions being logged in the IIS and we do not want to have it logged.

  24. Vir says:

    How to Retreive Request Headers using VB.net.can any one knows the synatx for that.if yes please provide with a sample code for that

  25. naveen says:

    How to forward the request headers when we want redirect the request.

  26. Frankel Lipshitz says:

    Duuuuuuuude….  this was really helpful!  Thanks, man!

  27. Gopi says:

    This is an excellent post, I am surprised to see the IIS behavior and understood now. Parsed all_http to solve my problem. Thanks a lot David.  Thanks!

  28. Paul says:

    Great information David!  Quick question for you.  I see that you can set a value for a header in an ISAPI filter using "SetHeader".  How can you do this using an ISPI Extension?  I’ve got an existing extension that we’re using to block SQL injection.  We want to modify it to prevent XSS using header variables like the HTTP_ACCEPT_LANGUAGE header.

  29. Paul says:

    Great information David!  Quick question for you.  I see that you can set a value for a header in an ISAPI filter using "SetHeader".  How can you do this using an ISPI Extension?  I’ve got an existing extension that we’re using to block SQL injection.  We want to modify it to prevent XSS using header variables like the HTTP_ACCEPT_LANGUAGE header.

  30. Nimesh says:

    My iis7.5 is give me permision denied error..

Skip to main content