Why does turning on reliable sessions sometimes cause the client to take much longer to fail an authentication check?
As part of doing an authenticated operation, the client must send a request to the server that contains information about the client's identity. This is just an abstract description of what actually goes on. The identity information and the request may be spread out across one or more messages or protocols. The authentication check on the server determines whether the information provided by the client is sufficient to allow the client to perform the operation. If the authentication check passes, the server continues with the operation as normal. If the authentication check fails, the server fails the client request by sending back an error message.
An error message may be formatted as a transport protocol error message, such as an HTTP status code, or some other mechanism for communicating errors. The details of how errors are communicated vary from application to application. The important point to note is that an authentication failure is communicated via an error indication that looks different than the expected application message.
Without a reliable session, any errors or exceptions encountered while processing a client request will cause the request to fail. If the client encounters anything but the expected application message, it knows that's an error and can report the failure immediately.
With a reliable session, the client is supposed to ignore some kinds of errors and exceptions while letting others through. The client is supposed to ignore a recoverable error and use the reliable protocol to retransmit messages or otherwise recover from the error. The client is not supposed to ignore an unrecoverable error because nothing the reliable session can do will make an unrecoverable error better. Of course, there's no clear marking indicating which errors are recoverable and which are not. The server has no idea what kinds of things the client might be able to recover from.
Thus, distinguishing a recoverable error from an unrecoverable error is a heuristic process performed by the reliable session. The reliable session has to make a guess about the intent of the error. Since an authentication failure is a type of error, the reliable session will report the error quickly if it correctly guesses that the error is unrecoverable and will take a while to report the error if it incorrectly guesses that the error is recoverable and spends some time trying to recover from it. The variability of details for how errors are communicated means that different application configurations may result in different guesses being made for the same underlying condition.