Anaylyze HttpWebRequest Hangs with a VB DebugDiag script`



Here is a VB Version of the JScript DebugDiag script for analyzing WebRequest hangs using DebugDiag http://blogs.msdn.com/jpsanders/archive/2008/10/24/analyzing-httpwebrequest-connection-saturation-in-net.aspx.  It has several improvements over the previous version.  First you will notice many similarities to the previous script.  Some minor changes were added to translate from Jscript to VBscript.  One big change is the .loadby command.  This will load the sos.dll from the same path the mscorwks is loaded.  You would normally run this script on the machine that you took the dump from.  This would ensure that the sos version matches the mscorwks version in the dump.  Another change is to seach not only for GetRequest but for SubmitRequest as well.  Finally I am dumping out more information like when there request is fine and the ConnectionGroupName.


Problem! All available connections used up.
HttpRequest URI:http://10.10.10.10/Service/Service.asmx
ServicePoint – ConnectionLimit:12 CurrentConnections:14
ConnectionGroupName: MyConnectionPool


I thought it would be equally important to dump when there were no issues:


 No Hang Problems found
HttpRequest URI:http://Mydev/output.asmx/test?&Name=test2
ServicePoint – ConnectionLimit:10 CurrentConnections:1
ConnectionGroupName:


Finally, fomatting the URI in a more readable format was fairly simple.  I used the -nofields switch when dumping the string and parsed out the String: text.


function getUriString(aObjAddr)
Dim aRetStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_Uri”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)” )


aFoundIdx = InStr(1,aCmdOutput,”String:”,1)


 


if (aFoundIdx <> 0) then
  aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if
getUriString = aRetStr


end function


 


As with ANY sample code, I leave it to you to debug and test!  I also left some “TODO” sections for you to fill in yourself.  I hope you find it useful.  An industrious person could embed this into the existing DebugDiag scripts to included it in the HangAnalysis portion of the scripts and really have a powerful tool for troubleshooting WebRequest issue.


 


 Full script below (Copy Code):




<%
@ Language = vbscript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from…%><%@ TopLevelScript = True %><%


Option Explicit
Dim g_Progress
Dim g_Debugger



Dim g_HttpObjectOffsets
Dim g_SvcPointObjectOffsets
Dim g_UriObjectOffsets
Dim g_StringObjectOffsets


Dim g_MaxConnectionErrorThreads ‘ array of thread id’s that have exceed max connection.
Dim g_MaxConnectionWarningThreads ‘ array of thread id’s that have 50% max connection.


main()



‘ this kicks off everything
sub main()


Dim a_DataFile
Dim i
Dim a_NumberOfDumps

Manager.Write “<b>System.Net sample analysis script</b><BR>”


Manager.Progress.SetOverallRange 0,100
Manager.Progress.OverallPosition = 10
Manager.Progress.OverallStatus=”Initializing”
a_NumberOfDumps = Manager.DataFiles.Count


‘ for each data file specified…


for i=0 to a_NumberOfDumps-1


‘ get the dump
a_DataFile = Manager.DataFiles(i)
Manager.Progress.OverallStatus=”Starting analysis of: ” & a_DataFile


‘ get an instance of a debugger
set g_Debugger = Manager.GetDebugger(a_DataFile)



Manager.Write “<h2>Analysis of: ” & a_DataFile & “</h2>”


if g_Debugger.IsCrashDump then
  Manager.ReportInformation “DebugDiag determined that this dump file (” & a_DataFile & “) is a <font color=’Red’><b> crash </b></font>” &_
“dump and this script does not analyze crashes (only hangs). Please run analysis using a crash Script.”


end if


‘ try to load the managed debugging extension
if(LoadSOS)then


‘ initialize some object information for performance
initGlobals


‘ walk all threads but report only specific managed ones
walkManagedThreads
reportErrors


else
‘ todo
end if


‘ done with the debugger, so close it
Manager.CloseDebugger a_DataFile
Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1)


next
end sub


sub initGlobals()


set g_HttpObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_HttpObjectOffsets, Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”,”_ConnectionGroupName”),”system.dll!System.Net.HttpWebRequest”


set g_SvcPointObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_SvcPointObjectOffsets, Array(“m_ConnectionLimit”, “m_CurrentConnections”),”system.dll!System.Net.ServicePoint”


set g_UriObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_UriObjectOffsets, Array(“m_String”), “system.dll!System.Uri”


set g_StringObjectOffsets = CreateObject(“Scripting.Dictionary”)
ManagedFieldOffsets g_StringObjectOffsets, Array( “m_firstChar”), “mscorlib.dll!System.String”


g_MaxConnectionErrorThreads = “”
g_MaxConnectionWarningThreads = “”

end sub


 


‘ this starts the analysis…


sub walkManagedThreads()


Dim aCmdOutput
Dim aStackOutput
Dim aThreadInfo
Dim aDebugThread
Dim aNumThreads
Dim aAddr


Dim i
Dim aFoundIdx
Dim bFoundFirstThread
bFoundFirstThread =false


set aThreadInfo = g_Debugger.ThreadInfo
aNumThreads = aThreadInfo.Count


Manager.Progress.SetCurrentRange 0,aNumThreads
Manager.Progress.CurrentStatus=”Finding threads waiting on ‘System.Net.HttpWebRequest.GetResponse()’ or ‘System.Net.HttpWebRequest.SubmitRequest()'”

for i=0 to aNumThreads-1

Manager.Progress.CurrentPosition = i


set aDebugThread = aThreadInfo(i)


if Not( aDebugThread is nothing ) then


aStackOutput = g_Debugger.Execute(“~” & aDebugThread.ThreadID & “e!CLRStack”)
aFoundIdx = InStr(aStackOutput,”System.Net.HttpWebRequest.GetResponse”)

if (aFoundIdx = 0)then
  aFoundIdx = InStr(aStackOutput,”System.Net.HttpWebRequest.SubmitRequest”)
end if


‘ highlight the call???
‘is the tread processing GetResponse?


if (aFoundIdx <> 0) then
  if Not(bFoundFirstThread) then

    bFoundFirstThread=true
Manager.Write “<br><h2>Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'</h2><br>”


  end if


Manager.Write “<A NAME=’ThreadAnchor” & aDebugThread.ThreadID & “‘>”

‘ dump managed objects on the stack
aCmdOutput = g_Debugger.Execute(“~” & aDebugThread.ThreadID & “e!dso”)


‘ Collect the HttpWebRequest objects on the stack.
aFoundIdx = InStr(aCmdOutput,”System.Net.HttpWebRequest”)

while aFoundIdx <> 0

‘ if we found an object of interest on the stack,
‘ sav the address fo the object and the thread ID.

aAddr = Mid(aCmdOutput,aFoundIdx-9, 8)

checkObj aAddr,aDebugThread.ThreadID

‘build error/warning strings here instead.
‘display total number of managed threads?

‘ look for another one…
aFoundIdx = 0 ‘aCmdOutput.indexOf(“System.Net.HttpWebRequest”, aFoundIdx + 25)


wend

Manager.Write regexReplace(aStackOutput, “\n”, “<br>” )
end if
end if


next


end sub


 


function regexReplace(aSrcStr, aPattern, aReplacement )


dim aRetStr
dim aRegEx
set aRegEx = New RegExp
aRegEx.Pattern=aPattern
aRegEx.Global=True


aRetStr = aRegEx.Replace( aSrcStr,aReplacement)
regexReplace = aRetStr


end function


 


sub reportErrors()


Dim i
Dim threadList
if Len(g_MaxConnectionErrorThreads) then

Manager.ReportError “The number connections exceeds the number of available connections for these threads:” & g_MaxConnectionErrorThreads,”Increase the number of connections available per this article: <a TARGET=_blank href=’http:’msdn.microsoft.com/en-us/library/7af54za5.aspx’>Managing Connections</a>”


end if


end sub


”””””’/ Managed code helpers ””””””””’/


‘ use .loadby to find where mscorwks is loaded and then load SOS from the same place
function LoadSOS()


Dim abSuccess
Dim begPos
Dim endPos
Dim CmdOutput
Dim Module


abSuccess = false
CmdOutput = g_Debugger.Execute(“.loadby sos mscorwks”)
‘todo, check CmdOutput to ensure SOS loaded!


LoadSOS = true


end function


 


‘ given an Array, populate a dictionary object with offets for these array items to be used later.
sub ManagedFieldOffsets( dictObj ,aArray, aObject)


Dim aStr
Dim i
Dim aCmdOutput
‘ get the Class information
aCmdOutput = g_Debugger.Execute(“!DumpClass ” + getEEClass(aObject))


for each aStr in aArray
  dictObj.Add aStr, findOffset(aCmdOutput,aStr)
next


End Sub


‘ simple helper functions


function getManagedLongVal( aObjAddr, aOffset )
  getManagedLongVal = g_Debugger.Execute(“dt long ” & aObjAddr & “+” & aOffset)
end function


 


function getManagedStringVal( aObjAddr, aOffset )


Dim retStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(” & aObjAddr & “+” & aOffset & “)”)

aFoundIdx = InStr(aCmdOutput,”String:”)

if (aFoundIdx <> 0) then
retStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if


getManagedStringVal = retStr

end function


””””’ Helper Function – findOffset
‘ given a string, extract the field offset specified
‘ this is really an excercise it string parsing
function findOffset(aCmdOutput, aField)


Dim retStr
retStr=””

Dim aFoundIdx
Dim aBOL
Dim sArray

‘ find the field
aFoundIdx = InStr(1,aCmdOutput,aField,1)

if aFoundIdx <> 0 then

‘ look backwards from there for the offset.
‘ find the previous line
aBOL = InStrRev(aCmdOutput,Chr(10),aFoundIdx,1)+1

‘need to strip all but one space from the string and make an array of the column values
retStr=normalizeWhitespace(Mid(aCmdOutput,aBOL, aFoundIdx-aBOL))
sArray = Split(retStr)
retStr=””
‘ 7 columns total, offset is third column (0 based array)
retStr = sArray(2)

else
  ‘TODO Manager.Write “Did not find Field: ” & aField & “<<<<<<<<<<<<<<”
end if

findOffset= retStr


end function


””””’ Helper Function – getEEClass
‘ given a string, extract the EEClass address
‘ theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM
function getEEClass( theClassName )


Dim retStr
retStr=””
Dim aCmdOutput
Dim aFoundIdx
Dim aEOL

aCmdOutput = g_Debugger.Execute(“!Name2EE ” & theClassName)

aFoundIdx = InStr(1,aCmdOutput,”EEClass:”,1)

if (aFoundIdx <> 0) then
  aEOL = InStr(aFoundIdx,aCmdOutput,”Name:”,1)
  retStr = Mid(aCmdOutput, aFoundIdx + Len(“EEClass:”) , aEOL – (aFoundIdx + Len(“EEClass:”)))
end if

getEEClass= retStr


end function



‘ utility functions

‘ utility function removing extra whitespace and replacing it with a single space
‘ for example ” some text here” will be converted to ” some text here”


function normalizeWhitespace(stringToTrim)


dim aRegEx
set aRegEx = New RegExp
aRegEx.Pattern=”\s+”
aRegEx.Global=True
normalizeWhitespace = aRegEx.Replace( stringToTrim,” “)

end function


 


‘ calculate and save offsets for the Object fields we need
”””””””” Non-generic managed object helpers….



‘ Given an address of an object get the service point object address
function getSvcPointFromReq(aHttpObjAddr)


Dim aRetStr
aRetStr=””
Dim aCmdOutput

aCmdOutput = g_Debugger.Execute(“dd ” & aHttpObjAddr & “+” & g_HttpObjectOffsets.Item(“_ServicePoint”) & ” L1″)


aRetStr = Mid(aCmdOutput, Len(aCmdOutput) – 8,8)
getSvcPointFromReq = Trim(aRetStr)


end function


function getUriString(aObjAddr)
Dim aRetStr
Dim aCmdOutput
Dim aFoundIdx
aCmdOutput = g_Debugger.Execute(“!do -nofields poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_Uri”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)” )


aFoundIdx = InStr(1,aCmdOutput,”String:”,1)
if (aFoundIdx <> 0) then
  aRetStr=trim( Mid(aCmdOutput, aFoundIdx + Len(“String:”)))
end if


getUriString = aRetStr


end function


 


function getConnectionGroupName(aObjAddr)


getConnectionGroupName=g_Debugger.Execute(“du poi(poi(” & aObjAddr & “+” & g_HttpObjectOffsets.Item(“_ConnectionGroupName”) & “)+” & g_UriObjectOffsets.Item(“m_String”) & “)+” & g_StringObjectOffsets.Item(“m_firstChar”)).substr(9)


end function


 


‘ obj is an address we can do !do on to get information from
sub checkObj(aObjAddr, threadID)


‘get service point and see if it is OK.
Dim bSvcPointErrorOrWarning
bSvcPointErrorOrWarning = false


Dim maxConn
Dim currConn
Dim connGroup
Dim aCmdOutput
Dim i
Dim aFoundIdx

aCmdOutput = getSvcPointFromReq(aObjAddr)


if( (aCmdOutput<>”00000000″) And (aCmdOutput<>”????????”)) then


maxConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item(“m_ConnectionLimit”))
currConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.Item(“m_CurrentConnections”))
connGroup = getManagedStringVal( aObjAddr, g_HttpObjectOffsets.Item(“_ConnectionGroupName”))

if(currConn>=maxConn) then

‘ report out info and error
g_MaxConnectionErrorThreads = g_MaxConnectionErrorThreads & “<a href=’#ThreadAnchor” & threadID & “‘>” & threadID & “</a>, “


Manager.Write “<br> <font color=’red’><b>Problem! All available connections used up.</b><br> HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”


else


if(((currConn)*2)>=maxConn) then


‘ report out info and error
g_MaxConnectionWarningThreads = g_MaxConnectionWarningThreads & “<a href=’#ThreadAnchor” & threadID + “‘>” & threadID & “</a>, “


Manager.Write “<br> <font color=’orange’><b>Warning, half of the availabe connections are being used</b><br> HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”


else


‘ no problems found
Manager.Write “<br> <font color=’green’> <b>No Hang Problems found</b><br>HttpRequest URI:” & getUriString(aObjAddr) & “<br>ServicePoint – ConnectionLimit:” & maxConn & ” CurrentConnections:” & currConn &”<br>ConnectionGroupName: ” & connGroup & “</font><br><br>”


end if
end if
end if


End Sub
%>


Comments (0)