KITL Troubleshooting Tips



Posted by: Upender Sandadi
 
(reposting since text formatting got messed up first time)


This is the second entry in the KITL series. In the first entry we learnt aboust basic KITL packet layout. In this we will look at some of the common things we have run across in debugging KITL issues on devices in-house. This covers mostly KITL over ethernet. For KITL over serial and KITL over USB, I would recommend getting a trace and analyzing the KITL traffic using the packet layout information.


Tracing tools:
Some of the tools which can be used to capture KITL and debug traffic are:



  • You can use netmon.exe (from Microsoft) or Ethereal 

  • You can use a null modem cable to capture all the serial output traffic from the device and use Hyperterm or Teraterm to capture output from a serial port on the desktop.

Connectivity Issues:


Image downloads to the device but KITL doesn’t connect to the desktop



  • Make sure KITL is initialized on the device: Unfortunately this is very early in the boot phase. So you would have to use custom debug messages in OEMInit() to debug further. One possible problem could be a conflict of an IRQ with another device on the target. For CEPCs check the autoexec.bat to make sure IRQ is set to 0 or to a non-conflicting value. 

  • Make sure the device is not connecting to a wrong Platform Builder session on the host: This mostly happens if you have multiple PB sessions and one session is already connected to a device you are trying to establish connection with. You can detect this in the serial output where KITL usually prints the IP address of the host it is connecting to.

  • Make sure you are using compatible network card: Some images require you to have right set of network card in the device for KITL to establish a connection. For the most part if you use NE2000 compatible network card, this issue should be resolved.

  • Some times KITL interrupt initialization can fail and image will boot in polling mode. This is something one should be aware of if there seem to be any perf issues or issues booting up. Serial output from the device will indicate if the image is running in polling mode or interrupt mode. One can also break into the debugger and look at {,,nk.exe}Kitl.Interrupt (in Windows CE 6.0 this is referenced as {,,kitl.dll}Kitl.Interrupt) value to see if the KITL interrupt is initialized or not. If it is not initialized, then that would be set to 0xFF.

  • Make sure that the device name generated by the bootloader matches the device name generated by KITL inside the download image – Platform Builder looks for the same device name to identify which device to connect with.

      Best tools to debug in these scenarios are:



  • Serial output from the device. If you are creating your own images then enable all KITL debug zones and one should see all the KITL activity on device serial output. If you see KitlSend messages on serial output and yet the KITL packet doesn’t show up in the netmon trace, then most likely the problem is with the network card.

  • Network capture (using Ethereal or netmon) of all traffic to/from the device.

  • Custom debug messages in the OAL and KITL startup code.

Image downloads but KITL hangs after printing a message which says “Loading nk.exe …”.



  • One would typically see this if the image has debugger built in but the settings on desktop for debugger are not properly set. Make sure KdStub is selected as the debugger option in the Platform Builder target connectivity options for your device.


Image downloads but KITL hangs after printing a message which says “Loading relsfd.dll …”.



  • If the image has debugger, then try and break in the debugger to see why KITL relfsd calls are hung. This could be for several reasons:


    • Platform Builder version is not compatible with the OS version.

    • Some high priority thread is spinning leading to starvation for the KITL threads.

    • KITL Interrupts are turned off inadvertently leading to connectivity failure.

  • If the image doesn’t have debugger, then get the KITL trace and analyze the KITL packets to see why the desktop and device are not in sync.

This is definitely not an exhaustive list. Please add to this list if you run into a KITL issue with your custom devices and how you solved it.
 

Comments (8)

  1. Helge says:

    Upender, you mention to use Ethereal for monitoring. Is there a dissector plugin for KITL?

    Regards, Helge

  2. ce_base says:

    I don’t know of any KITL filters for Ethereal. We have KITL filters for Microsoft tool netmon which we can definitely release to wider audience if interested.

    You can read about netmon in general from here:

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/netmon/netmon/about_network_monitor_2_1.asp

    Upender

  3. Naveen says:

    I am very fresher to KITL. Can I get basic concepts of KITL.

  4. ce_base says:

    To summarize at a very high level, KITL is a software connectivity layer between a windows CE device and a desktop. KITL relies on underlying transport stacks such as Serial, USB, Ethernet to send and receive data between a CE device and a desktop. Functionality provided by KITL on top of these network stacks is packet and stream management. KITL is the only software solution which lets you debug at the kernel level of a CE device remotely from a desktop.

    There are currently two articles on KITL on this site (this and the KITL packet and transport overview). Please feel free to browse through those and send us any questions you have.

    Upender

  5. Richard R. Lee says:

    An additional cause for: Image downloads to the device but KITL doesn’t connect to the desktop

    The BOOT_ARGS struct may be used to pass important KITL parameters from the boot loader to the kernel, through an agreed upon address in physical memory. If this memory is corrupted, or the boot loader and kernel are using different addresses, download will proceed, but the host will not conenct.

    Another KITL related problem I’ve seen occurs when some device or software filter imposes a maximum transmission unit (MTU) of less than 1500 bytes. This may split the raw UDP packets. KITL does not have any mechanism to reassemble the packets, simply rejecting what it views as partial and there for invalid messages from the host. With a packet sniffer like Ethereal, you will see two packets from the host, and then a nack from the target when this happens.

    Most host/target debugger traffic is much smaller and is not a problem. the download of NK.BIN uses TFTP, with a 512 byte standard payload. However, RelFSD will use 1500 packets copying files to and from the host, and will effectively hang the system on the load of the first file.

    One significant culprit is Cisco Systems VPN client, which by default sets the MTU to 1300 bytes. This happens even when the VPN is not engaged. Fortunately, Cisco does provide a utility for setting a ‘custom’ packet MTU of up to 1500 bytes.

    Dean Ramsier found this problem independently, and  the ‘elegent’ solution using  the Cisco utility (I wrote a packet splicing KITL filter initially).

  6. Mukesh Modi says:

    Hi,

    I found that in my platform builder, when I tried to connect through USB kitl, its connecting properly but in other platform builder its not connecting

    Only message coming waitting for connect

    successfully disconnected then some message display and then hang.

    Kindly tell me reason.

    Or mail me to mukesh.modi@rediffmail.com

  7. ce_base says:

    I am assuming you are talking about USB-Serial KITL connection.

    USB Serial KITL has dependencies on Microsoft Active Sync software on the host system. Make sure Active Sync is installed and working properly on the host machine. You can test this by syncing between any Windows Mobile device and host machine over USB.

    Also disable any other application from using USB ports. For example by default Active Sync will have USB port checked in its connection settings. Uncheck that. The reason being Platform Builder will try to open the USB port for exclusive access and will fail the call if that port is already in use.

    Other than these two, there shouldn’t be any other dependencies or settings for USB-Serial KITL to work. If none of these help, then you can enable logging on the host machine for USB transport and get more information on why it is failing. Here is some information on how to enable USB logging on the host side:

    Key: HKLMSoftwareMicrosoftUSBSERKITL

    Value Name: DebugZones

    Value Type: DWORD

    Value Data: Can be an OR combination of one of the following values.

    #define TRANZONE_INIT       0x0001

    #define TRANZONE_WARN       0x0002

    #define TRANZONE_ERR        0x0004

    #define TRANZONE_READ       0x0010

    #define TRANZONE_WRITE      0x0020

    #define TRANZONE_RDRTHR     0x0040

    #define TRANZONE_FUNCTION   0x1000

    #define TRANZONE_DUMPFRAME  0x2000

    You might want to enable all zones initially to get maximum data and then you can sort out the results to fine tune the zones.

    Log should be written to %TEMP%usbasertran.log. Remeber to remove this key once you are done with the investigation. Otherwise you will end up wasting disk space on the host machine with huge USB logs.

    I hope this information helps.

    -Upender

  8. ce_base says:

    Thanks Richard for that info.

    I did not know that there are systems out there with MTU less than a typical IP packet size. That is good information to know.

    -Upender