The principle you should follow is the one used by Dr House in his movies: explore all the paths in order to discard a possible conclusion. It’s an attitude more than a process. And after some years you start dealing this way with everything… So, when facing a problem with an application you're writing for Windows Mobile, ask yourself the following questions.
1- Make sure the problem is not OEM-specific. Remember that, as I’ve written here, you should contact that OEM's Technical Support in order to troubleshoot the issue, as only the OEM knows in detail what they included in their OS based on the Windows CE\Mobile platform. So, having been able to reproduce it on different devices from different makers, or even better on the Device Emulator, is one of the first steps. If you don’t know precisely how the interaction between Microsoft and the OEMs work, you may for example read my answer #4 on my friend Chris’ blog here.
2- Before wasting huge amount of time on whatever type of troubleshooting, make sure you’re using the state-of-the-art technologies. For example, if you have a weird problem with a managed application running against v2.0, even if SP2, then you can check if v3.5 included a fix precisely to address the issue you’re experiencing. You don’t need to recompile via VS2008, as I’ve discussed about this here. Or, another example is SQL Compact 3.5: SP1 was recently released – if your issue is somehow related to SSCE3.5 then you should make sure if it was addressed in 3.5 SP1. Finally, when talking about state-of-the-art, I’m also referring to possible ROM Upgrades and Updates published by OEMs so that you can verify if the issue was raised by a older version of a OS' or OEM's component.
3- A problem is not something random. It’s random until we discover its pattern, and when we do it, we have a “deterministic problem”. So, a hint is: invest some time on trying to understand if the problem doesn’t happen under certain circumstances. In really very rare cases I had to do with real random problems, due to random failures of the CPU… but this was fixed by the CPU-manufacturer some years ago.
4- Remember that the goal of this phase is to exclude possible problem’s causes… so what about 3RD party libraries? I’m talking about the controls and libraries that the application depends on, and which you don’t own the source of. Many times you simply have to make sure that you’re using them correctly (Is this library the right version for this device? Was the control meant to work for NETCF v3.5? …).
5- Once you exclude all the possible external problems, it’s time to dig into it. But… we can’t work on the whole solutions, plenty of projects! You need to separate the “offending” portion. How do you know that? A possible approach, if the problem is reproduced on the Device Emulator, is to instrument your code, for example by adding calls to the OutputDebugString function in critical parts of your code and configure Device Emulator to bring up a serial debug output console. To do so go to the "File", "Configure..." menu item, switch to the "Peripherals" property sheet and check the "Create text console window for serial port 1" option. To make this option permanent, you can use Visual Studio menu Tools\Options\Devices, select the emulator and then you’ll have the same dialog.
6- Last but not least, EQATEC recently came out with a gift to all Mobile Programmers: a new tool called EQATEC Tracer that may be tremendously useful when troubleshooting.