Anaylyze HttpWebRequest Hangs with a VB DebugDiag script`

Here is a VB Version of the JScript DebugDiag script for analyzing WebRequest hangs using DebugDiag  It has several improvements over the previous version.  First you will notice many similarities to the previous script.  Some minor changes were added to translate from Jscript to VBscript.  One big change is the .loadby command.  This will load the sos.dll from the same path the mscorwks is loaded.  You would normally run this script on the machine that you took the dump from.  This would ensure that the sos version matches the mscorwks version in the dump.  Another change is to seach not only for GetRequest but for SubmitRequest as well.  Finally I am dumping out more information like when there request is fine and the ConnectionGroupName.

Problem! All available connections used up.
HttpRequest URI:
ServicePoint – ConnectionLimit:12 CurrentConnections:14
ConnectionGroupName: MyConnectionPool

I thought it would be equally important to dump when there were no issues:

 No Hang Problems found
HttpRequest URI:http://Mydev/output.asmx/test?&Name=test2
ServicePoint – ConnectionLimit:10 CurrentConnections:1

Finally, fomatting the URI in a more readable format was fairly simple.  I used the -nofields switch when dumping the string and parsed out the String: text.

function getUriString(aObjAddr)
Dim aRetStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_Uri”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)” )

aFoundIdx = InStr(1,aCmdOutput,”String:”,1)


if (aFoundIdx <> 0) then
  aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if
getUriString = aRetStr

end function


As with ANY sample code, I leave it to you to debug and test!  I also left some “TODO” sections for you to fill in yourself.  I hope you find it useful.  An industrious person could embed this into the existing DebugDiag scripts to included it in the HangAnalysis portion of the scripts and really have a powerful tool for troubleshooting WebRequest issue.


 Full script below (Copy Code):

@ Language = vbscript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from…%><%@ TopLevelScript = True %><%

Option Explicit
Dim g_Progress
Dim g_Debugger

Dim g_HttpObjectOffsets
Dim g_SvcPointObjectOffsets
Dim g_UriObjectOffsets
Dim g_StringObjectOffsets

Dim g_MaxConnectionErrorThreads ‘ array of thread id’s that have exceed max connection.
Dim g_MaxConnectionWarningThreads ‘ array of thread id’s that have 50% max connection.


‘ this kicks off everything
sub main()

Dim a_DataFile
Dim i
Dim a_NumberOfDumps

Manager.Write “<b>System.Net sample analysis script</b><BR>”

Manager.Progress.SetOverallRange 0,100
Manager.Progress.OverallPosition = 10
a_NumberOfDumps = Manager.DataFiles.Count

‘ for each data file specified…

for i=0 to a_NumberOfDumps-1

‘ get the dump
a_DataFile = Manager.DataFiles(i)
Manager.Progress.OverallStatus=”Starting analysis of: ” & a_DataFile

‘ get an instance of a debugger
set g_Debugger = Manager.GetDebugger(a_DataFile)

Manager.Write “<h2>Analysis of: ” & a_DataFile & “</h2>”

if g_Debugger.IsCrashDump then
  Manager.ReportInformation “DebugDiag determined that this dump file (” & a_DataFile & “) is a <font color=’Red’><b> crash </b></font>” &_
“dump and this script does not analyze crashes (only hangs). Please run analysis using a crash Script.”

end if

‘ try to load the managed debugging extension

‘ initialize some object information for performance

‘ walk all threads but report only specific managed ones

‘ todo
end if

‘ done with the debugger, so close it
Manager.CloseDebugger a_DataFile
Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1)

end sub

sub initGlobals()

set g_HttpObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_HttpObjectOffsets, Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”,”_ConnectionGroupName”),”system.dll!System.Net.HttpWebRequest”

set g_SvcPointObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_SvcPointObjectOffsets, Array(“m_ConnectionLimit”, “m_CurrentConnections”),”system.dll!System.Net.ServicePoint”

set g_UriObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_UriObjectOffsets, Array(“m_String”), “system.dll!System.Uri”

set g_StringObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_StringObjectOffsets, Array( “m_firstChar”), “mscorlib.dll!System.String”

g_MaxConnectionErrorThreads = “”
g_MaxConnectionWarningThreads = “”

end sub


‘ this starts the analysis…

sub walkManagedThreads()

Dim aCmdOutput
Dim aStackOutput
Dim aThreadInfo
Dim aDebugThread
Dim aNumThreads
Dim aAddr

Dim i
Dim aFoundIdx
Dim bFoundFirstThread
bFoundFirstThread =false

set aThreadInfo = g_Debugger.ThreadInfo
aNumThreads = aThreadInfo.Count

Manager.Progress.SetCurrentRange 0,aNumThreads
Manager.Progress.CurrentStatus=”Finding threads waiting on ‘System.Net.HttpWebRequest.GetResponse()’ or ‘System.Net.HttpWebRequest.SubmitRequest()'”

for i=0 to aNumThreads-1

Manager.Progress.CurrentPosition = i

set aDebugThread = aThreadInfo(i)

if Not( aDebugThread is nothing ) then

aStackOutput = g_Debugger.Execute(“~” & aDebugThread.ThreadID & “e!CLRStack”)
aFoundIdx = InStr(aStackOutput,”System.Net.HttpWebRequest.GetResponse”)

if (aFoundIdx = 0)then
  aFoundIdx = InStr(aStackOutput,”System.Net.HttpWebRequest.SubmitRequest”)
end if

‘ highlight the call???
‘is the tread processing GetResponse?

if (aFoundIdx <> 0) then
  if Not(bFoundFirstThread) then

Manager.Write “<br><h2>Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'</h2><br>”

  end if

Manager.Write “<A NAME=’ThreadAnchor” & aDebugThread.ThreadID & “‘>”

‘ dump managed objects on the stack
aCmdOutput = g_Debugger.Execute(“~” & aDebugThread.ThreadID & “e!dso”)

‘ Collect the HttpWebRequest objects on the stack.
aFoundIdx = InStr(aCmdOutput,”System.Net.HttpWebRequest”)

while aFoundIdx <> 0

‘ if we found an object of interest on the stack,
‘ sav the address fo the object and the thread ID.

aAddr = Mid(aCmdOutput,aFoundIdx-9, 8)

checkObj aAddr,aDebugThread.ThreadID

‘build error/warning strings here instead.
‘display total number of managed threads?

‘ look for another one…
aFoundIdx = 0 ‘aCmdOutput.indexOf(“System.Net.HttpWebRequest”, aFoundIdx + 25)


Manager.Write regexReplace(aStackOutput, “\n”, “<br>” )
end if
end if


end sub


function regexReplace(aSrcStr, aPattern, aReplacement )

dim aRetStr
dim aRegEx
set aRegEx = New RegExp

aRetStr = aRegEx.Replace( aSrcStr,aReplacement)
regexReplace = aRetStr

end function


sub reportErrors()

Dim i
Dim threadList
if Len(g_MaxConnectionErrorThreads) then

Manager.ReportError “The number connections exceeds the number of available connections for these threads:” & g_MaxConnectionErrorThreads,”Increase the number of connections available per this article: <a TARGET=_blank href=’http:’’>Managing Connections</a>”

end if

end sub

”””””’/ Managed code helpers ””””””””’/

‘ use .loadby to find where mscorwks is loaded and then load SOS from the same place
function LoadSOS()

Dim abSuccess
Dim begPos
Dim endPos
Dim CmdOutput
Dim Module

abSuccess = false
CmdOutput = g_Debugger.Execute(“.loadby sos mscorwks”)
‘todo, check CmdOutput to ensure SOS loaded!

LoadSOS = true

end function


‘ given an Array, populate a dictionary object with offets for these array items to be used later.
sub ManagedFieldOffsets( dictObj ,aArray, aObject)

Dim aStr
Dim i
Dim aCmdOutput
‘ get the Class information
aCmdOutput = g_Debugger.Execute(“!DumpClass ” + getEEClass(aObject))

for each aStr in aArray
  dictObj.Add aStr, findOffset(aCmdOutput,aStr)

End Sub

‘ simple helper functions

function getManagedLongVal( aObjAddr, aOffset )
  getManagedLongVal = g_Debugger.Execute(“dt long ” & aObjAddr & “+” & aOffset)
end function


function getManagedStringVal( aObjAddr, aOffset )

Dim retStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(” & aObjAddr & “+” & aOffset & “)”)

aFoundIdx = InStr(aCmdOutput,”String:”)

if (aFoundIdx <> 0) then
retStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if

getManagedStringVal = retStr

end function

””””’ Helper Function – findOffset
‘ given a string, extract the field offset specified
‘ this is really an excercise it string parsing
function findOffset(aCmdOutput, aField)

Dim retStr

Dim aFoundIdx
Dim aBOL
Dim sArray

‘ find the field
aFoundIdx = InStr(1,aCmdOutput,aField,1)

if aFoundIdx <> 0 then

‘ look backwards from there for the offset.
‘ find the previous line
aBOL = InStrRev(aCmdOutput,Chr(10),aFoundIdx,1)+1

‘need to strip all but one space from the string and make an array of the column values
retStr=normalizeWhitespace(Mid(aCmdOutput,aBOL, aFoundIdx-aBOL))
sArray = Split(retStr)
‘ 7 columns total, offset is third column (0 based array)
retStr = sArray(2)

  ‘TODO Manager.Write “Did not find Field: ” & aField & “<<<<<<<<<<<<<<”
end if

findOffset= retStr

end function

””””’ Helper Function – getEEClass
‘ given a string, extract the EEClass address
‘ theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM
function getEEClass( theClassName )

Dim retStr
Dim aCmdOutput
Dim aFoundIdx
Dim aEOL

aCmdOutput = g_Debugger.Execute(“!Name2EE ” & theClassName)

aFoundIdx = InStr(1,aCmdOutput,”EEClass:”,1)

if (aFoundIdx <> 0) then
  aEOL = InStr(aFoundIdx,aCmdOutput,”Name:”,1)
  retStr = Mid(aCmdOutput, aFoundIdx + Len(“EEClass:”) , aEOL – (aFoundIdx + Len(“EEClass:”)))
end if

getEEClass= retStr

end function

‘ utility functions

‘ utility function removing extra whitespace and replacing it with a single space
‘ for example ” some text here” will be converted to ” some text here”

function normalizeWhitespace(stringToTrim)

dim aRegEx
set aRegEx = New RegExp
normalizeWhitespace = aRegEx.Replace( stringToTrim,” “)

end function


‘ calculate and save offsets for the Object fields we need
”””””””” Non-generic managed object helpers….

‘ Given an address of an object get the service point object address
function getSvcPointFromReq(aHttpObjAddr)

Dim aRetStr
Dim aCmdOutput

aCmdOutput = g_Debugger.Execute(“dd ” & aHttpObjAddr & “+” & g_HttpObjectOffsets.Item(“_ServicePoint”) & ” L1″)

aRetStr = Mid(aCmdOutput, Len(aCmdOutput) – 8,8)
getSvcPointFromReq = Trim(aRetStr)

end function

function getUriString(aObjAddr)
Dim aRetStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_Uri”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)” )

aFoundIdx = InStr(1,aCmdOutput,”String:”,1)
if (aFoundIdx <> 0) then
  aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if

getUriString = aRetStr

end function


function getConnectionGroupName(aObjAddr)

getConnectionGroupName=g_Debugger.Execute(“du poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_ConnectionGroupName”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)+” & g_StringObjectOffsets.Item(“m_firstChar”)).substr(9)

end function


‘ obj is an address we can do !do on to get information from
sub checkObj(aObjAddr, threadID)

‘get service point and see if it is OK.
Dim bSvcPointErrorOrWarning
bSvcPointErrorOrWarning = false

Dim maxConn
Dim currConn
Dim connGroup
Dim aCmdOutput
Dim i
Dim aFoundIdx

aCmdOutput = getSvcPointFromReq(aObjAddr)

if( (aCmdOutput<>”00000000″) And (aCmdOutput<>”????????”)) then

maxConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item(“m_ConnectionLimit”))
currConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item(“m_CurrentConnections”))
connGroup = getManagedStringVal( aObjAddr, g_HttpObjectOffsets.Item(“_ConnectionGroupName”))

if(currConn>=maxConn) then

‘ report out info and error
g_MaxConnectionErrorThreads = g_MaxConnectionErrorThreads & “<a href=’#ThreadAnchor” & threadID & “‘>” & threadID & “</a>, “

Manager.Write “<br> <font color=’red’><b>Problem! All available connections used up.</b><br> HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”


if(((currConn)*2)>=maxConn) then

‘ report out info and error
g_MaxConnectionWarningThreads = g_MaxConnectionWarningThreads & “<a href=’#ThreadAnchor” & threadID + “‘>” & threadID & “</a>, “

Manager.Write “<br> <font color=’orange’><b>Warning, half of the availabe connections are being used</b><br> HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”


‘ no problems found
Manager.Write “<br> <font color=’green’> <b>No Hang Problems found</b><br>HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”

end if
end if
end if

End Sub

Comments (0)