Using Kudu to capture hang dumps for Windows Azure Web Sites

Kudu is the engine behind GIT deployments in Azure Web Sites. Using the feature provided by Kudu, we can capture hang dump for worker process which is very helpful for analysis performance issues. Please note this is still an experimental feature.

In this article, I will show you how to get the dump step by step. I use a slow web site hosted at http://slowsite.azurewebsites.net/ as a sample.

Upload the tool

1. Download ProcDump tool from Microsoft site, and extract the tool from the zip package.

ProcDump

http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

2. Logon to the site FTP and upload the tool. For example, I created a folder named dump and put the tool into the dump folder.

Open the Kudu Console

1. Open IE and browser to https://slowsite.scm.azurewebsites.net

2. And you will see a credential prompt, please give your deployment user and password.

a. You can find the deployment user on the portal. For my site it is wzhaoFtpUser(not case sensitive)

b. Key in the user name and password, like this.

3. Now, you should see the Kudu site, like this

4. Click the “Launch” to open the Kudu Diagnostics Console.

Capture the Dump

1. Run “powershell.exe -Command Get-Process” to list the process (es). Here is the sample output. You should able see php-cgi.exe or node.exe if you are running PHP or node.js application. The process ID for w3wp.exe is 28456.

2. Go to the folder where ProcDump was uploaded to. For this sample, it is “site\wwwroot\dump”

3. Run “ProcDump -accepteula -ma 28456” to create the dump. You will see the dump was created.

4. Run “dir” command to show the dump file.

Download the dump and analyze

1. Logon to the site FTP and download the dump file to local.

2. Open the dump in debugger, and you can see Default.aspx was busying on something.

0:000> !runaway

User Mode Time

Thread Time

34:6f54 0 days 0:00:09.171

7:7698 0 days 0:00:00.734

0:000> ~34kc

mscorlib_ni!System.DateTime.GetDatePart(Int32)

mscorlib_ni!System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc(System.DateTime, Boolean ByRef)

mscorlib_ni!System.DateTime.get_Now()

App_Web_t31wtklw!ASP.default_aspx.__Render__control1(System.Web.UI.HtmlTextWriter, System.Web.UI.Control)

System_Web_ni!System.Web.UI.Control.RenderChildrenInternal(System.Web.UI.HtmlTextWriter, System.Collections.ICollection)

System_Web_ni!System.Web.UI.Control.RenderChildren(System.Web.UI.HtmlTextWriter)

System_Web_ni!System.Web.UI.Page.Render(System.Web.UI.HtmlTextWriter)

System_Web_ni!System.Web.UI.Control.RenderControlInternal(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)

System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter, System.Web.UI.Adapters.ControlAdapter)

System_Web_ni!System.Web.UI.Control.RenderControl(System.Web.UI.HtmlTextWriter)

System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)

System_Web_ni!System.Web.UI.Page.ProcessRequest()

System_Web_ni!System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)

System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)

App_Web_t31wtklw!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)

Remind

1. 1. Resource Quotation

Generate and download the dump uses the site’s resource quotation, for example, disk storage, outband network, etc. By default, Azure Web Sites runs the site using a 32bit worker process on 64 bit machine. The process can use up to 4 GB memory. It is very easy to exceed the disk storage limitation (1GB).

I use a FREE site for example, after I download the dump file, I saw this on my site dashboard. The result is my site was suspended.

In order to reduce the resource usage, you can consider a mini dump which is much smaller. Run command “Procdump -accepteula 8636” will generate a mini dump. Or you can use the KUDU restapi to create a mini dump directly.

https://github.com/projectkudu/kudu/wiki/Process-list-and-minidump

2. Multiple Instance Issue

If you are running site with multiple instance, this method only generates dump file for the instance where the request was routing to. It can’t generate dumps for all instance.

Reference:

Kudu

https://github.com/projectkudu/kudu/wiki/Process-list-and-minidump

Dump Type

http://msdn.microsoft.com/en-us/library/windows/desktop/ms680519.aspx

ProcDump

http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

See you next time,

Wei Zhao from APGC DSI Team