.NET Debugging Demos Lab5: Crash – Review


 


Since I already posted a challenge for this lab earlier I didn’t want to wait too long with publishing the review… 


Previous demos and setup instructions


If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.

Information and setup instructions
Lab 1: Hang
Lab 1: Hang – review
Lab 2: Crash
Lab 2: Crash – review
Lab 3: Memory
Lab 3: Memory – review
Lab 4: High CPU hang
Lab 4: High CPU hang – review

Reproducing the issue:

1. Recycle IIS (IISReset)

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx 

3. Type some message and click the Send button 

Note: If just-in-time debugging is enabled a message box might pop up because the w3wp.exe process crashed.  If this comes up, please ignore it.


Q: What do you see in the browser?


A: A “Internet Explrer cannot display the webpage” page indicating that something is wrong on the server.


Explore the eventlogs


1. Open the eventlogs for system an application


Q: What events do you notice relating to the crash? (note: you may see different events on different operating systems)


A:
Application

Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 2008-03-04
Time: 12:57:33
User: N/A
Computer: MYMACHINE
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 4626467c, debug? 0, fault address 0x00022366.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


System

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
Date: 2008-03-04
Time: 12:57:19
User: N/A
Computer: MYMACHINE
Description:
A process serving application pool ‘DefaultAppPool’ terminated unexpectedly. The process id was ‘5764’. The process exit code was ‘0x800703e9’.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 2008-03-04
Time: 12:57:48
User: N/A
Computer: MYMACHINE
Description:
A process serving application pool ‘DefaultAppPool’ suffered a fatal communication error with the World Wide Web Publishing Service. The process id was ‘5764’. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80               m..


 


Q: What does exit code ‘0x800703e9’ mean?


A: Searching on the internet or looking it up in Error Lookup we find that 0x800703E9 means “Recursion too deep; the stack overflowed.”


Q: Based on the eventlogs, what caused the crash?


A: A stackoverflow exception in the application.  The most likely cause for this is infinite recursion, i.e. function a calls function b calls function a calls function b etc. until the memory allocated for the stack is exhausted.


Reproduce the issue again and gather a memory dump:

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash


adplus -crash -pn w3wp.exe -quiet


4. Enter a message on the page and click send


Q: What files get created in the dump folder (i.e. the folder named crash_mode current date and time under the debuggers directory)?


A:


2008-03-04 14:28 4 415 ADPlus_report.txt
2008-03-04 14:28 <DIR> CDBScripts
2008-03-04 14:28 151 874 536 PID-2548__W3WP.EXE__1st_chance_Process_Shut_Down__full_06f0_2008-03-04_14-28-17-145_09f4.dmp
2008-03-04 14:28 111 648 PID-2548__W3WP.EXE__Date_03-04-2008__Time_14-28-0606.log
2008-03-04 14:28 13 156 Process_List.txt

Open the dump to figure out what it is doing:

1. Open the process_shut_down dump (crash_mode with todays date and time in the debuggers directory) in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the stacks:

Note: In a dump file the active thread is the thread that caused the debugger to dump the process, i.e. a process exit exception or other exception. 

One of a few things can cause the process to shut down.


  • An unhandled exception (2nd chance exception), in most cases if this is causing the shutdown, you will also get a 2nd chance exception dump, and this will be the one you want to look at to figure out what caused the crash

  • An external shutdown, i.e. an iisreset, a preemptive recycle (based on recycling options) or someone killing the process from task manager.  If this is the case, the active thread in the process shutdown dump is usually the main thread and you will see a stack similar to the following
    0:000> kL
    ChildEBP RetAddr
    0014fbfc 7d503faf ntdll!ZwTerminateProcess+0x12
    0014fc38 7d503f5a kernel32!_ExitProcess+0x4b
    0014fc4c 79fd9e8f kernel32!ExitProcess+0x14
    0014fe74 79f7479c mscorwks!SafeExitProcess+0x157
    0014ff10 79004fab mscorwks!HandleExitProcessHelper+0x27
    0014ff10 79004fab mscoree!CorExitProcess+0x46
    0014ff20 77bcaddb mscoree!CorExitProcess+0x46
    0014ff2c 77bcaefb msvcrt!__crtExitProcess+0x29
    0014ff5c 77bcaf52 msvcrt!doexit+0x81
    0014ff70 01001a3c msvcrt!exit+0x11
    0014ffc0 7d4e7d2a w3wp!wmainCRTStartup+0x144
    0014fff0 00000000 kernel32!BaseProcessStart+0x28

    If this is the case you should look at the eventlog as it usually contains the recycling reason, and if you are troubleshooting a real crash you should consider disabling recyling while troubleshooting the real crash so that you won’t get these type of “red herring” dumps.

  • Something in the process called process exit.  This is usually caused by some type of exception considered fatal, i.e. heap corruption, stack overflow, out of memory exceptions or fatal execution engine exceptions.  In this case the active thread in the process shutdown stack will look (and probably is) completely unrelated to the crash. In this case you will want to look at the log to see what happened just prior to the crash in order to figure out how to proceed and gather new dumps if necessary.

1. Run kb to see which of the above categories this crash/process exit falls into


Q: Which category did it fall into and why?



A: In my particular dump I got this stack for the last remaining executing thread.  Since this is not the main thread (didn’t start with w3wp!wmainCRTStartup) this means that the process died and shut down the threads.  The fact that we didn’t get a 2nd chance dump means we didnt die from an unhandled exception, and the fact that it wasn’t the main thread means that it isn’t a normal recycle. 

0:000> kL
ChildEBP RetAddr
0290fe54 7d4d8c82 ntdll!NtWaitForSingleObject+0x15
0290fec4 79e789c6 kernel32!WaitForSingleObjectEx+0xac
0290ff08 79e7898f mscorwks!CLREventWaitHelper+0x2f
0290ff58 79e78944 mscorwks!CLREvent::WaitEx+0x117
0290ff6c 79efa706 mscorwks!CLREvent::Wait+0x17
0290ff98 79fc8583 mscorwks!SVR::gc_heap::gc_thread_function+0x30
0290ffb8 7d4dfe21 mscorwks!SVR::gc_heap::gc_thread_stub+0x92
0290ffec 00000000 kernel32!BaseThreadStart+0x34
 

Examine the log file to find out what happened just prior to the crash

Q: What events occurred just prior to the crash? 

A: Right before the process exit event I see the following exception (Unknown exception – code e053534f).  Since I don’t have the symbol path set up in the environment variables I can’t really trust the stack, but what I can see is that we have some .net code (the mscorlib_ni frames) followed by some calls into mscorwks and eventually the kernel32!RaiseException that raises the e053534f exception.  Since this exception code is not a known exception code adplus reports it as unknown exception.  

0:025> g
Tue Mar 4 14:28:16.551 2008 (GMT+1): (9f4.30c): Unknown exception – code e053534f (first chance)

— 1st chance UnknownException exception —-
—————————————————————

Occurrence happened at:
Debug session time: Tue Mar 4 15:20:44.687 2008 (GMT+1)
System Uptime: 19 days 12:10:42.146
Process Uptime: 0 days 0:00:17.546
Kernel time: 0 days 0:00:00.578
User time: 0 days 0:00:00.640

Faulting stack below —
# ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0ff3aecc 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x4e
01 0ff3aee8 79fff01a 0024a900 6ea810e3 0024a900 mscorwks!GetCompileInfo+0x1d896
02 0ff3b06c 79f067d3 06f26a08 6ea81197 06fd9788 mscorwks!NGenCreateNGenWorker+0x213a7
03 0ff3b118 7948d92a 793826f1 06fd98c4 79e82084 mscorwks!GetCLRFunction+0x2371f
04 0ff3b214 793983c8 00000000 06fd975c 00000000 mscorlib_ni+0x3cd92a
05 0ff3b254 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
06 0ff3b278 79398550 0ff3b2d8 06fd9470 06f2657c mscorlib_ni+0x2d84e3
07 0ff3b28c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d8550
08 0ff3b2c4 793983c8 06fd8d90 06fd8fe4 06f2657c mscorlib_ni+0x3d959c
09 0ff3b33c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d83c8
0a 00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c


We can rebuild the native piece of the stack by using ln (list near) on the mscorwks methods in the process exit dump.
0:000> ln mscorwks!GetCLRFunction+0x2371f
(79f06728) mscorwks!GetResourceFromDefault+0xb6 | (79f0681a) mscorwks!GetResourceStringFromManaged
0:000> ln mscorwks!NGenCreateNGenWorker+0x213a7
(79f0681a) mscorwks!GetResourceStringFromManaged+0x44 | (79f069e5) mscorwks!NativeCompareInfo::CompareStringW
0:000> ln mscorwks!GetCompileInfo+0x1d896
(7a0ac375) mscorwks!ReportStackOverflow+0x61 | (7a0ac3dc) mscorwks!ChangeThreadToSOTolerant
Substituting the unresolved method names (method names with bad symbols) we get this…  
 # ChildEBP RetAddr  Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0ff3aecc 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x4e
01 0ff3aee8 79fff01a 0024a900 6ea810e3 0024a900 mscorwks!ReportStackOverflow+0x61
02 0ff3b06c 79f067d3 06f26a08 6ea81197 06fd9788 mscorwks!GetResourceStringFromManaged+0x44
03 0ff3b118 7948d92a 793826f1 06fd98c4 79e82084 mscorwks!GetResourceFromDefault+0xb6
04 0ff3b214 793983c8 00000000 06fd975c 00000000 mscorlib_ni+0x3cd92a
05 0ff3b254 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
06 0ff3b278 79398550 0ff3b2d8 06fd9470 06f2657c mscorlib_ni+0x2d84e3
07 0ff3b28c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d8550
08 0ff3b2c4 793983c8 06fd8d90 06fd8fe4 06f2657c mscorlib_ni+0x3d959c
09 0ff3b33c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d83c8
0a 00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c

So we can see that it was a stackoverflow exception that brought the process down.  If we wanted to we could also resolve the top of the .net stack by running !ip2md 790c0000+0x3cd92a for example where 790c0000 is the base address for mscorlib.dll

0:000> !ip2md 790c0000+0x3d959c
MethodDesc: 7925bc40
Method Name: System.IO.StreamWriter..ctor(System.String)
Class: 7910381c
MethodTable: 7910389c
mdToken: 06003696
Module: 790c2000
IsJitted: yes
m_CodeOrIL: 79499580

But I will leave that as an excercise for you since we only have a short piece of the stack so we don’t get deep enough to find the recursion.  Note here that the recursion doesn’t happen in the streamwriter constructor.  This is just the last method call when we are running out of stack space as you’ll see later 

Configure adplus to gather full dumps on the exception in the log file 

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Copy the attached adplus configuration file to your debuggers directory  (Note: this configuration file causes adplus to take a full dump on the unknown exception)

4. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash


adplus -pn w3wp.exe -c Unknown.cfg


4. Enter a message on the page and click send


Open the dump to figure out what it is doing:

1. Open the 1st_chance_unknown_exception dump in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the active stack:


1. Run kb 2000 and !clrstack to see what the active thread is doing (i.e. the thread that caused the stackoverflow)

0:014> kb
ChildEBP RetAddr Args to Child
0250ae9c 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x53
0250aeb8 79fff01a 001a67c0 c29e1ac8 001a67c0 mscorwks!ReportStackOverflow+0x61
0250b03c 79f067d3 02f367d4 c29e1a1c 0309dfe0 mscorwks!GetResourceStringFromManaged+0x44
0250b0e8 7948d92a 793826f1 0309e11c 79e82084 mscorwks!GetResourceFromDefault+0xb6
0250b1e4 793983c8 00000000 0309dfb4 00000000 mscorlib_ni+0x3cd92a
0250b224 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
0250b248 79398550 0250b2a8 0309dcc8 02f35fd0 mscorlib_ni+0x2d84e3
0250b25c 7949959c 00000400 02f3640c 00000000 mscorlib_ni+0x2d8550
0250b294 793983c8 0309d5e8 0309d83c 02f35fd0 mscorlib_ni+0x3d959c
0250b30c 7949959c 00000400 02f3640c 00000000 mscorlib_ni+0x2d83c8
00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c

0:014> !clrstack
OS Thread Id: 0x177c (14)
ESP EIP
0250b090 7d4e2366 [HelperMethodFrame_2OBJ: 0250b090] System.Environment.GetResourceFromDefault(System.String)
0250b0f0 7948d92a System.IO.__Error.WinIOError(Int32, System.String)
0250b11c 79395557 System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean)
0250b214 793983c8 System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions)
0250b240 793984e3 System.IO.StreamWriter.CreateFile(System.String, Boolean)
0250b250 79398550 System.IO.StreamWriter..ctor(System.String, Boolean, System.Text.Encoding, Int32)
0250b270 7949959c System.IO.StreamWriter..ctor(System.String)
0250b27c 0fe90b52 Utility.WriteToLog(System.String, System.String)
0250b2b0 0fe90b0b ExceptionHandler.LogException(System.Exception)

0252f0c4 0fe90bd4 Utility.WriteToLog(System.String, System.String)
0252f170 0fe90b0b ExceptionHandler.LogException(System.Exception)
0252f174 0fe90bd4 Utility.WriteToLog(System.String, System.String)
0252f220 0fe90b0b ExceptionHandler.LogException(System.Exception)
0252f224 0fe90a81 BuggyMail.SendEmail(System.String, System.String)
0252f2c8 0fe90a28 CompanyInformation.btnContact_Click(System.Object, System.EventArgs)
0252f2d8 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
0252f2ec 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
0252f300 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
0252f304 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
0252f30c 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
0252f31c 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0252f518 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0252f550 6614d80f System.Web.UI.Page.ProcessRequest()
0252f588 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
0252f590 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0252f5a4 0fe90205 ASP.companyinformation_aspx.ProcessRequest(System.Web.HttpContext)
0252f5a8 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0252f5dc 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0252f61c 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
0252f66c 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
0252f688 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
0252f6bc 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
0252f6c8 65fcfa6d System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
0252f8d8 79f047fd [ContextTransitionFrame: 0252f8d8]
0252f90c 79f047fd [GCFrame: 0252f90c]
0252fa68 79f047fd [ComMethodFrame: 0252fa68]


 


Q: Can you tell what the issue is from here?


A:  From !clrstack it appears that we are processing companyinformation.aspx and in the page_load method it calls into BuggyMail.SendEmail.  From there we go into some type of recursion between ExceptionHandler.LogException(System.Exception) and Utility.WriteToLog(System.String, System.String), and eventually we run out of stackspace and throw the stackoverflow exception that kills the process…  To resolve the issue we need to figure out a way to stop the recursive loop between LogException and WriteToLog.  You may be tempted to believe that it is the System.IO.__Error.WinIOError or the GetResourceFromDefault methods that cause the stackoverflow but in reality those methods are just the straws that broke the camels back… They are called in each


Verify your assumption in the code and modify the code to resolve the issue 

1. Open ExceptionHandler.cs and Utility.cs and device a plan for how to fix the problem

    public static void LogException(Exception ex)
{
Utility.WriteToLog(ex.Message, “c:\\log.txt”);
}

public static void WriteToLog(string message, string fileName)
{
try
{
using (StreamWriter sw = new StreamWriter(fileName))
{
//Log the event with date and time.
sw.WriteLine(“————————–“);
sw.WriteLine(DateTime.Now.ToLongTimeString());
sw.WriteLine(“——————-“);
sw.WriteLine(message);
}
}
catch (Exception ex)
{
ExceptionHandler.LogException(ex);
}
}


A: From this and the stack we can backtrack what happened… We got an exception in BuggyMail.SendMail and asked to log the Exception which calls WriteToLog, unfortunately we also got an exception in WriteToLog when actually writing to the log (for those of you interested, you can find these exceptions with !dso and the reason we get an exception in WriteToLog is because we (the w3wp.exe process) don’t have access to write to c:\log.txt.  The solution shold be to have some other exceptionhandling mechanism in WriteToLog.

Have fun,

Tess

unknown.cfg

Comments (26)

  1. Sharepoint No VSE WSS for VS 2008 Until Summer [Via: Tariq ] WPF Podder v2 has been released! [Via:…

  2. I’m facing an issue with a StackOverflow problem, I get the dump, but when I load it it says "unable to verify the checksum for System.Data.ni.dll" or something similar, no matter what I do, I’ve tried updating to .NET 2 SP1, regenerating the native images, I’ve made sure it reads the correct files (filemon), I’ve ran windbg directly from the server, etc

    The weird thing is that this only happens on the server though, we actually have reproduced the issue in two servers, but we cannot get the same error on the developer machines

    these are server 2003 with service pack 2 that have .NET 2, 3 and 3.5 installed

    when I run the windbg commands everything points to an issue in System.Data.dll, but since the symbols for that file are not loaded I don’t get a lot of information, I get a bunch of calls to system.data.ni.dll –

    how can I get the right symbols file?

    any other clues you might have?

    thanks for all these tutorials Tess, I really appreciate all your work

  3. Tess says:

    There are no symbols for the native images (.ni) and you dont really need them.  You will get the calls on the stack if you use !clrstack.

    Tess

  4. this is the output running kb and !clrstack

    0:026> kb 2000

    ChildEBP RetAddr  Args to Child              

    072d3664 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x114f8b

    072d40c0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d4b1c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d5578 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d5fd4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d6a30 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d748c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d7ee8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d8944 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d93a0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d9dfc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072da858 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072db2b4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dbd10 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dc76c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dd1c8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ddc24 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072de680 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072df0dc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dfb38 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e0594 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e0ff0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e1a4c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e24a8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e2f04 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e3960 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e43bc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e4e18 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e5874 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e62d0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e6d2c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e7788 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e81e4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e8c40 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e969c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ea0f8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eab54 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eb5b0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ec00c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eca68 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ed4c4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072edf20 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ee97c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ef3d8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072efe34 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f0890 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f12ec 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f1d48 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f27a4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f3200 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f3c5c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f46b8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f5114 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f5b70 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f65cc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f7028 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f7a84 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f84e0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f8f3c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f9998 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fa3f4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fae50 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fb8ac 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fc308 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fcd64 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fd7c0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fe21c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fec78 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ff6d4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07300130 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07300b8c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073015e8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07302044 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07302aa0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073034fc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07303f58 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073049b4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07305410 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07305e6c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073068c8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07307324 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07307d80 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073087dc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07309238 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07309c94 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730a6f0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730b14c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730bba8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730c604 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730d060 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730dabc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730e518 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730ef84 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730efd0 65218285 00000000 186ed6b4 000000bc System_Data_ni!_load_config_used+0x1140e0

    0730f030 651feb12 00000092 14307e20 1c6b0de4 System_Data_ni!_load_config_used+0xa6bb5

    0730f164 79e880df 01bf1f0e 0730f198 84adef90 System_Data_ni!_load_config_used+0x8d442

    0730f124 081684e7 0730f198 062b43e0 00000001 mscorwks!PreStubWorker+0x139

    *** WARNING: Unable to verify checksum for System.Web.RegularExpressions.ni.dll

    WARNING: Frame IP not in any known module. Following frames may be wrong.

    0730f1b0 66f12980 1c0579b0 18312720 6628efd2 0x81684e7

    00000000 00000000 00000000 00000000 00000000 System_Web_RegularExpressions_ni+0x12980

    0:026> !clrstack

    OS Thread Id: 0x1740 (26)

    ESP       EIP    

    072d2c3c 6528665b System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])

    that’s all BCL, where do I go from there?

  5. the windows event I get is this:

    EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 system.data, P5 2.0.0.0, P6 471ebf27, P7 1840, P8 0, P9 system.stackoverflowexception, P10 NIL.

    and this:

    A process serving application pool ‘ProcessName’ suffered a fatal communication error with the World Wide Web Publishing Service. The process id was ‘5664’. The data field contains the error number.

    8007006d

  6. Tess says:

    Eber, the stackoverflow you are getting with the System.Data.BinaryNode.EvalBinaryOp is a known issue that occurrs when you have an extremely long filter i.e. condtionx and conditiony and conditionz etc.

    basically for each and or or it goes into a separate eval of the left site and eval of the right site recursively, so that is why you get a stackoverflow.  Run !dso to see the condition and look to see if it is meant to be like that or if you have some logic issues that cause the filter exception to be set like that inadvertedly.  I don’t have access to my tools at the moment but when I get back into the office I can write a post describing why it happens and more details about it.

  7. When you use very complex Rowfilters or expressions on datasets or datatables you may end up getting

  8. thanks Tess, the weird thing is that it only happens on the server, not on any of the development machines

    admittedly we use some complex expressions

    I’ll check the new article, thanks a lot

  9. H Debugger says:

    I cannot run 0:025> g, Why?

    Why should you create the full dump?

    Thanks

  10. H Debugger says:

    I did not follow the stpes "Examine the log file to find out what happened just prior to the crash

    Q: What events occurred just prior to the crash?  

    "

    Could you explain it with more details?

  11. Tess says:

    H Debugger,

    0:025> g

    is something you would find in the log file created by adplus…

    as a sidenote: the 0:025> meant that it was on thread 25,  the g means go…   in a dump you can’t go anymore so you can’t run g at the prompt unfortunately…

    ____________________________

    regarding "I did not follow the stpes "Examine the log file to find out what happened just prior to the crash

    Q: What events occurred just prior to the crash?  

    "  

    During the time you are attached in crash mode, the debugger will keep spitting out debugging info to the log file based on your adplus configuration.  This means that at the time of the crash you can look in the log file and have a look at what debugging info it had spit out just prior to the crash, in order to see what events led up to the crash…

  12. Nizar says:

    I think there could be an infinite loop like:

    private void function_name(Leave_Application_Table lat)

       {

          function_name();

       }

  13. Tony Wu says:

    ————————————————————————————

    unfortunately we also got an exception in WriteToLog when actually writing to the log (for those of you interested, you can find these exceptions with !dso and the reason we get an exception in WriteToLog is because we (the w3wp.exe process) don’t have access to write to c:log.txt.

    —————————————————————————————-

    Unfortunately,I can not reproduce the bug.Because the ASPNET account has access to write to the c:logtxt.:((

  14. Tess says:

    you can just switch the path to a folder that the ASPNET account doesn’t have access to.  Typically it shouldn’t have access to c: though…

  15. It might be an idea to change path in the lab to something like @"ct:log.txt".

    Exception will be: System.NotSupportedException

    Message: The given path’s format is not supported.

    In my case I’m using Vista Home for labs (just being a nerd and Vista dummy) and setup the BuggyBits to run under Local System. So had to look into the walkthrough to see what actually is expected to be wrong. Missed the thrill :).

  16. 飞天舞者 says:

    Debug入门之旅-StackoverFlow exception的调试

  17. Loek says:

    You rock 🙂 I was ready to start pounding my head into a brick wall over that ‘unknown exception’ 🙂

  18. JP says:

    Hi Tess,  

    You referred me over here from your "ASP.Net 2.0 Crash Case Study" blog entry.  

    I'm trying to execute Adplus with the referenced Unknown.cfg, but I'm getting the following Adplus error:

    "Errors reading configuration file: unknown.cfg

    Invalid root tag: expected <ADPlus Version='2'>

    !!!ERROR – ASPlus failed to run"

    So, I updated the root tag to be <ADPlus Version="2">, but that resulted in the following:

    "Errors reading configuration file: unknown.cfg

    Invalid tag in the Exceptions section of a configuration string: Config

    !!! ERROR – ADPlus failed to run"

    I'm running ADPlus version 6.1.0.0 with the following command "adplus -pn w3wp.exe -c unknown.cfg"

    Is this config file no longer valid?

    Thanks again,

    JP

  19. Tess says:

    I think the newest adplus requires a different config file, but try installing debug diag, create a crash rule and set up an exception for 0xe053534f

  20. JP says:

    Hi Tess,

    I set up the crash rule for exception 0xe053534f but the issue came and went without creating a dump.  I also tried creating a generic crash rule for w3wp.exe with no exception specified but got the same result….  no dump.

    Any other suggestions?  

    Your help on this is greatly appreciated!  With the occurrence happening 10+ times a day, this is causing both us and our customers quite a bit of grief.  Can you recommend (or do you personally offer) any professional (i.e. for pay) services in this area?  I'm interesting in learning how to troubleshoot these issue, but right now solving them is priority number 1.  I'm willing to try additional steps, but time is of the essence.  I'm sure you understand.

    Thoughts?

    Thanks so much,

    JP

  21. Tess says:

    JP, if it didnt get caught with that rule, then it is likely not a stackoverflow.  if you see a "real crash" i.e. a stopped unexpectedly in the eventviewer, or simply that the process exits, a dump should be generated with debug diag… can you post the eventlog entry of the crash? (you can remove any identifying names if you want)

  22. JP says:

    Hi Tess,  

    Looks like my last post didn't go through….

    Here's the contents of the event log:

    Date: 3/28/2011       Source: W3SVC

    Time: 4:06:48 PM    Category: None

    Type: Warning         EventID: 1011

    User:  N/A

    Description:

    A process serving application pool 'XXXXXXX' suffered a fatal communication eror with the World Wide Web Publishing Service.  The process id was '3338'  The ata field contains the error number.

    000: 6d 00 07 80

    0000: 8007006d

    This error is then followed by one or more:

    A process serving application pool 'XXXXXX' terminated unexpectedly.  The process id was '840'.  The process exit code was '0x800703e9'

    Not sure how I missed this before, but I assume my next step is to run debugdiag for exception 800703e9 ("Recursion too deep; the stack overflowed.").  I set this up to try to catch the exception.

    But in the meantime, I went back to review an older dump that i took and hit another dead end….

    In a previous dump, i received the following files:

    FULLDUMP_FirstChance_epr_Process_Shut_Down_w3wp.exe__0878_2011-03-15_17-33-35-486_047c.dmp

    followed by 30 or so

    MINIDUMP_FirstChance_av_AccessViolation_w3wp.exe__0878_2011-03-15_14-16-47-051_047c.dmp

    So i opened the FullDump in WinDbg and got:

    Loading Dump File [C:adplus20110315_141615_Crash_ModeFULLDUMP_FirstChance_epr_Process_Shut_Down_w3wp.exe__0878_2011-03-15_17-33-35-486_047c.dmp]

    User Mini Dump File with Full Memory: Only application data is available

    Comment: 'FirstChance_epr_Process_Shut_Down'

    Symbol search path is: *** Invalid ***

    ****************************************************************************

    * Symbol loading may be unreliable without a symbol search path.           *

    * Use .symfix to have the debugger choose a symbol path.                   *

    * After setting your symbol path, use .reload to refresh symbol locations. *

    ****************************************************************************

    Executable search path is:

    Windows Server 2003 Version 3790 (Service Pack 2) UP Free x86 compatible

    Product: Server, suite: Enterprise TerminalServer DataCenter SingleUserTS

    Machine Name:

    Debug session time: Tue Mar 15 17:33:36.000 2011 (UTC – 4:00)

    System Uptime: 5 days 17:31:05.963

    Process Uptime: 0 days 3:28:12.000

    ……………………………………………………….

    ……………………………………………………….

    ……………………………………………………….

    ……………………………………………………….

    ……………………………………………………….

    …………………………………………

    Loading unloaded module list

    …………..

    eax=000000c0 ebx=5a3e142c ecx=00000000 edx=00000000 esi=000001e4 edi=00000000

    eip=7c82847c esp=01a8fa88 ebp=01a8faf8 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll –

    ntdll!KiFastSystemCallRet:

    7c82847c c3              ret

    i then entered:

    0:000> ~

    .  0  Id: 47c.f10 Suspend: -1 Teb: 7ffaf000 Unfrozen

    Sweet – got a dot.  That's good right?

    I then entered:

    0:000> kb 2000

    ChildEBP RetAddr  Args to Child              

    WARNING: Stack unwind information not available. Following frames may be wrong.

    01a8faf8 77e61c8d 000001e4 ffffffff 00000000 ntdll!KiFastSystemCallRet

    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for w3core.dll –

    01a8fb0c 5a3b81ef 000001e4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12

    01a8ffb8 77e6482f 00000000 00000000 00000000 w3core!W3_FILE_INFO_CACHE::GetFileCache+0xd86c

    01a8ffec 00000000 5a3b80c9 00000000 00000000 kernel32!GetModuleHandleA+0xdf

    And here I am again?  Not really sure where to go next.

    Referencing some other entries of yours, I came up with the following:

    0:000> .load C:windowsmicrosoft.netframeworkv2.0.50727sos.dll

    0:000> !threads

    *********************************************************************

    * Symbols can not be loaded because symbol path is not initialized. *

    *                                                                   *

    * The Symbol Path can be set by:                                    *

    *   using the _NT_SYMBOL_PATH environment variable.                 *

    *   using the -y <symbol_path> argument when starting the debugger. *

    *   using .sympath and .sympath+                                    *

    *********************************************************************

    PDB symbol for mscorwks.dll not loaded

    ThreadCount: 27

    UnstartedThread: 0

    BackgroundThread: 27

    PendingThread: 0

    DeadThread: 0

    Hosted Runtime: no

                                         PreEmptive   GC Alloc           Lock

          ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception

    XXXX    1  294 000e36a0   1808220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    2  1b8 000f6010      b220 Enabled  00000000:00000000 000e8930     0 Ukn (Finalizer)

    XXXX    3  bb0 00113610    80a220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Completion Port)

    XXXX    4  214 00116d88      1220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX    5  a88 05d40ae0   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    6  80c 05d75c38   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    7  c28 05d709d8   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    8  5c0 05e1b008   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    a  780 05e1ec70       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX    d  dd0 0bf59e78   180b220 Enabled  00000000:00000000 109b4460     1 Ukn (Threadpool Worker)

    XXXX    e  3dc 0bf4d4b0   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    f  490 0bf5a248   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX   10  3c4 0bf92900   880a220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Completion Port)

    XXXX   12  454 0bf80408       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   14  e74 0bf9b1d8   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX   16  124 0c00d270       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   17  d38 0c022738       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   18  d7c 0c080cb0       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   13  460 0c0a31c0       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX    c  de4 10989598       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   15 1700 0bf8b078   200b220 Enabled  00000000:00000000 109b4460     0 Ukn

    XXXX    9  cf4 109891c8   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX    b 179c 0c0a3590   180b220 Enabled  00000000:00000000 000e8930     0 Ukn (Threadpool Worker)

    XXXX   19 1350 0bfdca88   180b220 Disabled 00000000:00000000 109b4460     1 Ukn (Threadpool Worker) System.StackOverflowException (026c106c)

    XXXX   11 1740 0c068d68       220 Enabled  00000000:00000000 000e8930     0 Ukn

    XXXX   1a 1144 05e1cdb0   200b220 Enabled  00000000:00000000 109b4460     1 Ukn

    XXXX   1b 1094 108ca1b0       220 Enabled  00000000:00000000 000e8930     0 Ukn

    0:000> !do 026c106c

    Name: System.StackOverflowException

    MethodTable: 04930dd8

    EEClass: 0475262c

    Size: 72(0x48) bytes

    (C:WINDOWSassemblyGAC_32mscorlib2.0.0.0__b77a5c561934e089mscorlib.dll)

    Fields:

         MT    Field   Offset                 Type VT     Attr    Value Name

    04930b24  40000b5        4        System.String  0 instance 00000000 _className

    0492ff98  40000b6        8 …ection.MethodBase  0 instance 00000000 _exceptionMethod

    04930b24  40000b7        c        System.String  0 instance 00000000 _exceptionMethodString

    04930b24  40000b8       10        System.String  0 instance 00000000 _message

    0492a480  40000b9       14 …tions.IDictionary  0 instance 00000000 _data

    04930cb8  40000ba       18     System.Exception  0 instance 00000000 _innerException

    04930b24  40000bb       1c        System.String  0 instance 00000000 _helpURL

    04930740  40000bc       20        System.Object  0 instance 00000000 _stackTrace

    04930b24  40000bd       24        System.String  0 instance 00000000 _stackTraceString

    04930b24  40000be       28        System.String  0 instance 00000000 _remoteStackTraceString

    04932d70  40000bf       34         System.Int32  1 instance        0 _remoteStackIndex

    04930740  40000c0       2c        System.Object  0 instance 00000000 _dynamicMethods

    04932d70  40000c1       38         System.Int32  1 instance -2147023895 _HResult

    04930b24  40000c2       30        System.String  0 instance 00000000 _source

    049333ec  40000c3       3c        System.IntPtr  1 instance        0 _xptrs

    04932d70  40000c4       40         System.Int32  1 instance -532459699 _xcode

    Am I going in the right direction?  Where to next?

    Thanks again for all your help,

    JP

  23. Tess says:

    JP, that is definitely a stackoverflow so it should get caught with a debug diag 0xe053534f rule, you can also try to just set up a rule and in the exceptions select SOV and ask it to dump on that exception.

    If you fail and need help you are welcome to open up a support incident with Microsoft Support at http://support.microsoft.com

  24. Ajeeth says:

    Hi JP,

    Please try the following config for new ADPlus, it works for me

    <ADPlus Version='2'>

       <Settings>

           <RunMode> CRASH </RunMode>

       </Settings>

       <!– defining and configuring exceptions –>

       <Exceptions>

           <!– First we redefine all exceptions –>

           <AllExceptions>

               <Actions1>Log</Actions1>

               <Actions2>MiniDump;Log;EventLog</Actions2>

           </AllExceptions>

           <NewException Code="0xe053534f " Name="Unknown_Exception"/>

           <!– Configuring the custom exception –>

           <Exception Code="0xe053534f ">

               <Actions1>FullDump;Log;EventLog</Actions1>

               <Actions2>FullDump;Log;EventLog</Actions2>

           </Exception>

       </Exceptions>

    </ADPlus>