Kudu is the engine behind GIT deployments in Azure Web Sites. Using the feature provided by Kudu, we can capture hang dump for worker process which is very helpful for analysis performance issues. Please note this is still an experimental feature.
In this article, I will show you how to get the dump step by step. I use a slow web site hosted at http://slowsite.azurewebsites.net/ as a sample.
Upload the tool
1. Download ProcDump tool from Microsoft site, and extract the tool from the zip package.
ProcDump
http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
2. Logon to the site FTP and upload the tool. For example, I created a folder named dump and put the tool into the dump folder.
Open the Kudu Console
1. Open IE and browser to https://slowsite.scm.azurewebsites.net
2. And you will see a credential prompt, please give your deployment user and password.
a. You can find the deployment user on the portal. For my site it is wzhaoFtpUser(not case sensitive)
b. Key in the user name and password, like this.
3. Now, you should see the Kudu site, like this
4. Click the “Launch” to open the Kudu Diagnostics Console.
Capture the Dump
1. Run “powershell.exe -Command Get-Process” to list the process (es). Here is the sample output. You should able see php-cgi.exe or node.exe if you are running PHP or node.js application. The process ID for w3wp.exe is 28456.
2. Go to the folder where ProcDump was uploaded to. For this sample, it is “site\wwwroot\dump”
3. Run “ProcDump -accepteula -ma 28456” to create the dump. You will see the dump was created.
4. Run “dir” command to show the dump file.
Download the dump and analyze
1. Logon to the site FTP and download the dump file to local.
2. Open the dump in debugger, and you can see Default.aspx was busying on something.
0:000> !runaway
User Mode Time
Thread Time
34:6f54 0 days 0:00:09.171
7:7698 0 days 0:00:00.734
0:000> ~34kc
mscorlib_ni!System.DateTime.GetDatePart(Int32)
mscorlib_ni!System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc(System.DateTime, Boolean ByRef)
mscorlib_ni!System.DateTime.get_Now()
App_Web_t31wtklw!ASP.default_aspx.__Render__control1(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)
System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)
System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)
System_Web_ni!System.Web.UI.Page.Render(System.Web.UI.HtmlTextWriter)
System_Web_ni!System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)
System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
System_Web_ni!System.Web.UI.Page.ProcessRequest()
System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
App_Web_t31wtklw!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
Remind
1. 1. Resource Quotation
Generate and download the dump uses the site’s resource quotation, for example, disk storage, outband network, etc. By default, Azure Web Sites runs the site using a 32bit worker process on 64 bit machine. The process can use up to 4 GB memory. It is very easy to exceed the disk storage limitation (1GB).
I use a FREE site for example, after I download the dump file, I saw this on my site dashboard. The result is my site was suspended.
In order to reduce the resource usage, you can consider a mini dump which is much smaller. Run command “Procdump -accepteula 8636” will generate a mini dump. Or you can use the KUDU restapi to create a mini dump directly.
https://github.com/projectkudu/kudu/wiki/Process-list-and-minidump
2. Multiple Instance Issue
If you are running site with multiple instance, this method only generates dump file for the instance where the request was routing to. It can’t generate dumps for all instance.
Reference:
Kudu
https://github.com/projectkudu/kudu/wiki/Process-list-and-minidump
Dump Type
http://msdn.microsoft.com/en-us/library/windows/desktop/ms680519.aspx
ProcDump
http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
See you next time,
Wei Zhao from APGC DSI Team
I am seeing the below error. May I know what kind of perms is needed to run procdump in console?
[10:46:35] Dump 1 error: Error writing dump file: 0x80070005
Error 0x80070005 (-2147024891): Access is denied.