Troubleshooting guide - part 2
Below is the second half of a well executed document from guestRx: Bulent Elmaci. Bulent has worked with Windows Mobile debugging for a long time and backs up his writing with a lot of experience. It is the first of a series of articles he has written to help our OEM's become better debugging developers. Enjoy!
TOOLS
Using correct tools to collect relevant and sufficient data, and to analyze them not only makes the life a little easier for the engineers, it also helps making sure correct information can be gathered and correct conclusions can be reached, that would otherwise be missed if things were done only manually. It also shortens the investigation time and thus leaves more time to better assessing and implementing resolution alternatives.
Below, some of the important tools are discussed briefly that are available to Windows Mobile partners, with the goal of presenting where they are applicable/usable, how they can be located and available references for detailed usage.
CELog
CELog is an event-tracking tool that can log both a set of predefined kernel events (related to processes, threads, synchronization, memory, etc.) as well as application defined ones. It collects the data in a user-configurable location that can later be viewed using various tools. The data captured can be divided into “zones” to be able to filter out events that are relevant to the problem. More details can be found in the following pages:
o https://blogs.msdn.com/ce_base/archive/2005/11/30/a-tour-of-windows-ce-performance-tools.aspx
o https://blogs.msdn.com/ce_base/archive/2006/01/23/516586.aspx
Perfman
Perfman is an on-device tool that helps collecting CELog data on a Windows Mobile powered device without needing any connection to the device. It provides a simple user interface to make it easier to start and control the logging on the device. More details can be found in the following pages:
- OEM documentation topic “Tuning the Platform for Optimum Performance”
o https://blogs.msdn.com/ce_base/archive/2005/11/30/a-tour-of-windows-ce-performance-tools.aspx
Application Logs
Even though this is obvious, it can easily be missed (if it is not being overused). For issues that originate from applications or components for which you have access to the sources, application level logging/tracing can be very powerful to help narrow down the problem area.
Platform Builder Debugging
If a consistent repro of the issue is available, you have a good understanding of the possible components that the issue might be related to, and you have access to a KITL enabled device, live debugging using Platform Builder would be the best way to go. This is true even if you don’t have the correct symbols. Following pages provide good tricks that can aid in debugging using Platform Builder:
- Resolving symbols manually on Windows CE
- Windows CE Virtual Memory Layout for Debugging
- Platform Builder Debug Symbols
- Virtual Memory and Thread Stacks
Netlog
Netlog is an on-device network packet sniffer (a.k.a. network monitor) that can be used to log network activity on the device. The output of Netlog can be viewed with popular desktop viewer tools like Microsoft Network Monitor, or Wireshark/Ethereal. More details can be found in the following pages:
- MSDN Documentation
o https://msdn.microsoft.com/en-us/library/ms883125.aspx
o https://msdn.microsoft.com/en-us/library/aa926774.aspx
o https://msdn.microsoft.com/en-us/library/aa922639.aspx
o https://msdn.microsoft.com/en-us/library/ms886701.aspx
- Windows CE Networking Team Blog
o https://blogs.msdn.com/cenet/archive/2004/10/11/getting-network-captures-on-windows-ce.aspx
RIL Proxy Logs
RIL Proxy is a component that is part of the Radio Interface Layer (RIL) architecture. Besides its main function, it also serves the purpose of being able to log radio events that originate either from the radio or from the operating system. RIL Proxy logs can be very useful especially for troubleshooting radio or connectivity related issues. More details can be found in the following pages:
- Windows CE Networking Team Blog
o https://blogs.msdn.com/cenet/archive/2005/09/27/474650.aspx
o https://blogs.msdn.com/hopperx/archive/2007/07/26/using-the-radio-interface-layer.aspx
RapiConfig
RapiConfig is a desktop configuration tool, part of the Windows Mobile 6 SDK, which allows applying a provisioning XML on a device/emulator connected using ActiveSync. It can be used both to make quick configuration changes (e.g. change Connection Manager settings, make registry changes, etc.), and to capture current configuration on the device (e.g. read Connection Manager settings, read a registry value, etc.).
Usage of RapiConfig is controlled by security settings on Windows Mobile which means that any “RapiConfig request” could be rejected if RAPI API usage is disabled (look for policy 4097 in the “OEM Documentation”). In such a case security configuration of Windows Mobile device must be changed before using RapiConfig tool.
More details can be found in the following pages:
- MSDN Documentation
o https://msdn.microsoft.com/en-us/library/aa454232.aspx
o https://msdn.microsoft.com/en-us/library/ms889520.aspx
o https://msdn.microsoft.com/en-us/library/aa924367.aspx
o https://msdn.microsoft.com/en-us/library/ms889589.aspx
o https://msdn.microsoft.com/en-us/library/30dtsstx.aspx
o https://msdn.microsoft.com/en-us/library/bb384149.aspx
- MSDN Blogs
o https://blogs.msdn.com/marcpe/archive/2005/01/18/355158.aspx
- OEM Documentation
o “Configuration Service Providers”
MakeCab
This tool, which runs on Windows XP/Vista and which is also part of Windows Mobile SDK, can generate a CPF file (which is special CAB file with only settings but no files) from a provisioning XML. The CPF file can be put on the device and run to make the actions (read or change some configuration on the device) defined by the provisioning XML it contains.
Installation of a CAB/CPF file is controlled by security settings on Windows Mobile which means that CAB/CPF file installation could be rejected (look for policy 4101 in the “OEM Documentation”). In this case the file(s) have to be signed with a trusted certificate that is already installed on the Window Mobile device (provided by OEM or Mobile Operator).
More details can be found in the following pages:
- OEM Documentation
o “How to Create a .cpf File”
o “Cab Provisioning Format (CPF) File”
o “Packaging the XML File for Delivery”
- MSDN Documentation
o https://msdn.microsoft.com/en-us/library/ms889557.aspx
o https://msdn.microsoft.com/en-us/library/aa455993.aspx
Remote Tools
Platform Builder comes with various remote tools that target specific troubleshooting/debugging scenarios. OEM documentation includes comprehensive documentation about each of these tools. Some of these topics in the documentation are listed below:
- “Remote Tool Connectivity”
- “Remote Tools for Debugging”
- “Tools for Performance Tuning”
- “Remote Tools for Information Management”
CEDebugX
CEDebugX is an extension to Platform Builder that helps collecting system information from a device, to better understand the state of the device at a given time. More details can be found in the following pages:
- MSDN Documentation
o https://msdn.microsoft.com/en-us/library/bb509784.aspx
- MSDN Channel 9
o https://channel9.msdn.com/posts/mikehall/Using-CEDebugX-with-Windows-Embedded-CE-60-SP1/
Watson/Kernel Dumps
This mechanism (a.k.a. “Windows Error Reporting” or “Watson”) takes a snapshot of the system memory and system state upon a system exception (e.g. memory access violation, division by 0, etc.). It is by default installed in all Windows Mobile devices and can be turned on or off using “Error Reporting” settings UI or alternatively via using some registry keys.
When “Windows Error Reporting” is enabled, you should take care that as soon as the Windows Mobile device is connected to a PC via ActiveSync, it will by default try to remove the dump file from the device and to upload it to “Windows Error Reporting” server. This behavior can be turned off using registry keys. If you want to get the dump file, you should either disable the upload mechanism or copy the file before connecting the device to ActiveSync. More details can be found in the following pages:
- OEM Documentation
o “Error Reporting Overview”
o “Using CELog and Watson to Debug a Driver Exception in a Retail Device Without KITL”
o “Generating A Dump From A Handled Exception”
o “Exception Mode Error Reporting”
- MSDN Documentation
o https://msdn.microsoft.com/en-us/library/bb905581.aspx
o https://msdn.microsoft.com/en-us/library/aa935972.aspx
o https://msdn.microsoft.com/en-us/library/aa935583.aspx
o https://msdn.microsoft.com/en-us/library/aa935778.aspx
- MSDN Blogs
Post Mortem Debugging
In addition to Just in Time Debugging capabilities, Platform Builder also provides post-mortem debugging capabilities via an extension called “Post Mortem Debugger”. It is used to debug Watson dump files using Platform Builder, as if what is being debugged is a live device under debugger. More details can be found in the following pages:
- OEM Documentation
o “Post-Mortem Debugging”
o “Types of Crash Dump Files”
o “Using CELog and Watson to Debug a Driver Exception in a Retail Device Without KITL”
o “Generating A Dump From A Watched Process”
o “Capturing a Dump File on a Standalone Device”
o “Capturing a Dump File While Debugging”