Getting Started with Unmanaged Tablet Development

Background on Pen-computing for Tablet, Inking, and Recognition

Taking input from a pen device and transforming that data into other meaningful data is a common and potentially arduous task for pen-computing developers but is relatively easy on Microsoft’s pen-computing platform, the Tablet PC. In the case of Tablet PC, this process is simplified because Microsoft has designed the pen-computing system and has implemented the design so that you don’t have to. The following example will give a quick introduction to the architecture for ink capture and recognition for Tablet PC and will then give an example of how to use the Tablet PC SDK to recognize handwriting as text.

In the Tablet PC world, input from the pen to the PC is called “ink”. Ink is made up of packets which contain all sorts of nifty information such as the x position, y position, and state of the pen. These packets combine to form strokes which are meaningful collections of related packets. The stroke items are aggregated into collections that are called strokes. These stroke collections all share the same attributes such as width, color, and transparency values.

The following diagram illustrates the high level design of pen input (ink) data for Tablet PCs:

Just like reading any other input stream, these strokes are captured by OS hooks and then after capture can be rendered or recognized. As you may have guessed, Microsoft has implemented interfaces, classes, and managed objects for representing, rendering, and recognizing each of the aforementioned data types. Microsoft distributes these objects to developers in the Tablet PC SDK in Windows XP and ships them with the Vista platform SDK.

Ink recognition, the process of transforming pen input into text, is built into the tablet SDK and is done on the Tablet PC by capturing ink and then passing this ink to an object that can recognize the ink. Ink is captured using the InkCollector or a derived object such as InkOverlay. After the InkCollector or InkOverlay has captured packets, which are formed into strokes, the strokes can be retrieved. By passing the strokes to a recognizer (the InkAnalyzer in this example) and running the analyze method on the captured strokes, the ink data can be translated to recognized content. Note that the RecognizerContext would be sufficient for recognizing the ink input and that the InkAnalyzer may even be considered overkill for cases such as this. More information on Ink Analysis (with the Divider and RecognizerContext) can be found in the MSDN article Ink Analysis with the Tablet PC SDK; CoDe Magazine has a great article on Ink Recognition and Ink Analysis that explains how the InkAnalyzer is essentially a superset of the divider and recognizer.

The following diagram illustrates the ink recognition process in this example:

From a high level perspective, the process for ink capture and analysis on the Tablet PC is rather straightforward. In managed code, where object creation is consistent and simple, it is easy to develop managed applications for Tablet. In unmanaged code, seasoned COM developers will easily be able to develop Tablet PC applications. To the uninitiated developer, creating COM applications can be rather daunting and can prevent them from electing to develop Tablet applications in an unmanaged environment. To address this scenario, I am breaking down the MFC/COM tasks as well as the high level Tablet APIs for the reader.

Creating a Simple Reco Application

The recognition application that will be developed will be a very simple dialog that contains an area for writing on and a button that when pressed will trigger the recognizer. The following figure illustrates this as a mock-up:

Next, you will create a new project in Microsoft Visual studio that creates a dialog with a UI similar to the recognizer mock-up. If you already know how to do this, you can skip the following instructions but make sure that you create a placeholder groupbox that has the id IDC_PLACEHOLDER. The following instructions will walk you through the steps I took to create a bare-bones MFC application and dialog.

On the first screen, select Visual C++, MFC, and then MFC application. Change the project name to Simple Recognizer, or something else if you prefer another name.

Click OK to come to the second screen.

The second screen will give you a quick low-down on project settings.

Click Next > you do not need to change anything, the default settings should be fine. 

On the Application type page, you will need to set the application type to Dialog.

Click Next > to see the User Interface features page.

Uncheck every checkbox on the User Interface Features page.

Click Next > and you will be sent to the advanced page.

You shouldn’t need to change any of the advanced features on this page.

You have set everything that needs to be set for the project. Click Next > to send you to another summary page, click Next > again, and click Finish on the last page and your project will be created.

Under the resources tab for your project, you will find the .rc file that contains UI information for your dialog application. Double click on this to see how the application will look when it runs. The following figure shows how the application might look:

This application has all these items that you don’t want. Delete the “TODO: Place dialog controls here.“ text, the “OK” button, and the “Cancel” button by simply selecting the objects with your mouse and pressing delete on your keyboard.

Now, drag a Button (our recognizer trigger), a GroupBox (for a placeholder for the reco area), and some static text onto your new dialog. The following image illustrates this:

Note the caption for the button has been changed to “Recognize”. Select the button and change the caption property from the properties dialog to enable this. The following image illustrates this.

The static text caption has been changed in the same manner. I have also changed the ID for the group box because that box (and its related HWnd) will be overridden with the ink control. The following image shows that I am changing the ID for this control to IDC_PLACEHOLDER.

Since you’ve been mucking around with the look and feel of your application, you should perform a quick build to make sure it looks right. Hit F5 and your Visual Studio should prompt you to build before running. You should have a relatively boring application at this point that looks like the following:

Press Alt+F4 to exit, as we elected to not create any sort of icons that will close this application. Granted, our application is not identical to the planned application, but it’s good enough! You’re now ready to get the code working.

Getting the Code Working

You now have the MFC skeleton for your application and it’s time to hook everything together in code.

Add the following includes to SimpleRecognizerDlg.h:

#include <msinkaut.h>

#include <iacom.h>

Add the following includes to SimpleRecognizerDlg.cpp:

#include <msinkaut_i.c>

#include <iacom_i.c>

You will first add a few protected declarations and a destructor (for CoUninitialize) to the SimpleRecognizerDlg class by adding the following code to the header.

// The destructor, which is used for CoUninitialize

~CSimpleRecognizerDlg();

 

// an inkCollector control

CComPtr<IInkCollector> inkCollector;

// InkPlaceholder()

// Replaces the groupbox placeholder with an inkCollector control

void InkPlaceholder();

// AddStrokesToInkAnalyzer

// Adds the strokes contained in a IInkStrokes collection to an InkAnalyzer

void AddStrokesToInkAnalyzer(IInkAnalyzer* inkAnalyzer, IInkStrokes* strokes);

// AddStrokeToInkAnalyzer

// Adds a single IInkStrokeDisp stroke to an InkAnalyzer

IContextNode* AddStrokeToInkAnalyzer(IInkAnalyzer* inkAnalyzer, IInkStrokeDisp* stroke);

After you have added the function and ink collector declarations, it’s time to implement the functions in SimpleRecognizerDlg.cpp.

 

First, add CoInitialize to the constructor and CoUninitialize to the destructor. These functions are used to set up the application for COM. The following code example shows how this is done.

 

CSimpleRecognizerDlg::CSimpleRecognizerDlg(CWnd* pParent /*=NULL*/)

    : CDialog(CSimpleRecognizerDlg::IDD, pParent)

{

    m_hIcon = AfxGetApp()->LoadIcon(IDR_MAINFRAME);

    ::CoInitialize(NULL);

}

CSimpleRecognizerDlg::~CSimpleRecognizerDlg()

{

    ::CoUninitialize();

}

Now create the InkPlaceholder function which will be called upon application initialization to destroy the placeholder groupbox and will then create the ink collector. This is a little funny as the COM code is rather tricky. The first thing that happens is the control is retrieved using the IDC_* tag that identifies it, IDC_PLACEHOLDER in this case. After that control is retrieved, the information about its position and window settings are stored. We then destroy the old control and create a new one with the same settings. This new window is used to host the InkCollector thatis CoCreated into the variable, inkCollector.

 

The following code example shows how this is done:

 

void CSimpleRecognizerDlg::InkPlaceholder(){

      if(inkCollector == NULL){

            try{

                  // swipe the group control with our new control

                 

                  CWnd* pTargetCtrl = this->GetDlgItem(IDC_PLACEHOLDER);

                  DWORD style = pTargetCtrl->GetStyle();

                  DWORD styleEx = pTargetCtrl->GetExStyle();

                 

                  // use the control to figure out how large is the initial on-screen area for the graphics window

                 

                  CRect rect;

                  pTargetCtrl->GetWindowRect(&rect);

                  pTargetCtrl->DestroyWindow();// I <3 destroy!

                  pTargetCtrl->m_hWnd = NULL;

                  // convert the screen rectangle to the dialog's client coordinates

                  this->ScreenToClient(&rect);

                  BOOL success = pTargetCtrl->CreateEx(

                        styleEx,

                        NULL, //CWnd default

                        NULL, //has no name

      style|WS_CHILD|WS_CLIPSIBLINGS|WS_CLIPCHILDREN|WS_VISIBLE,

                        rect, // Dimension and location of the place-holder

                        this, //this is the parent

                        IDC_PLACEHOLDER); //Use the ID of the just destroyed static

                  BringWindowToTop();

                 

// coinitialize must be called before cocreate

CoInitialize(NULL);

 

                  HRESULT hr = S_OK;

                  if (hr != 0)::MessageBox(NULL,L"1",L"",NULL);

                  // check HR

                  hr = inkCollector.CoCreateInstance(CLSID_InkCollector);

            if (hr != 0)::MessageBox(NULL,L"3",L"",NULL);

                  // check HR

                  hr = inkCollector->put_hWnd((long)pTargetCtrl->m_hWnd);

                  if (hr != 0)::MessageBox(NULL,L"2",L"",NULL);

                  hr = this->inkCollector->put_Enabled(VARIANT_TRUE);

                  if (hr != 0)::MessageBox(NULL,L"4",L"",NULL);

                  // check HR

            }catch(CString &message){

                  ::MessageBoxEx(NULL, message, L"Error", NULL, NULL);

            }

      }

}

 

After you have created the InkPlaceholder function, add a call to it to the OnDialogInit function in the Simple RecognizerDlg.cpp file.

 

// TODO: Add extra initialization here

InkPlaceholder(); // replace your placeholder

 

If you’re into checking your incremental progress, you can now build your application and will see that you have this fun ink control now instead of the boring groupbox:

 

Clicking around in the new area will enable you to capture ink with the accuracy and hilarity of MSPaint. Now that your application is capturing ink, it’s time to create the code to send the strokes to the analyzer object which can translate the content from strokes to text.

 

The next function, AddStrokeToInkAnalyzer, will add a stroke of data to the analyzer. This is not as easy as one would hope because the data that must be passed to the AddStroke method will need to first be parsed from the individual strokes. AddStrokeToInkAnalyzer retrieves the packet data, packet descriptions, class IDs, and GUID counts from the stoke using a VARIANT structure to safely store the data and description content. The content is looked over and additional data is retrieved from it. After the necessary information is gathered, this information is passed to the AddStroke method for the ink analyzer.

 

// AddStrokeToInkAnalyzer

// Adds a single IInkStrokeDisp stroke to an InkAnalyzer

IContextNode* CSimpleRecognizerDlg::AddStrokeToInkAnalyzer(IInkAnalyzer* inkAnalyzer, IInkStrokeDisp* stroke)

{

      VARIANT varPacketData;

      HRESULT hr = stroke->GetPacketData(

        ISC_FirstElement,

        ISC_AllElements,

    &varPacketData);

      // check HR

    long id;

    hr = stroke->get_ID(&id);

      // check HR

    VARIANT varPacketDesc;

    hr = stroke->get_PacketDescription(&varPacketDesc);

      // check HR

    LONG * plPacketData = NULL;

    BSTR * pbstrPacketDesc = NULL;

    ::SafeArrayAccessData(varPacketData.parray,(void **)&plPacketData);

    ::SafeArrayAccessData(varPacketDesc.parray, (void **)&pbstrPacketDesc);

   

    ULONG guidCount = varPacketDesc.parray->rgsabound[0].cElements;

    ULONG packetDataCount = varPacketData.parray->rgsabound[0].cElements;

    GUID *pPacketDescGuids = new GUID[guidCount];

    for( ULONG ul = 0; ul < guidCount; ul++ )

    {

        ::CLSIDFromString(

            pbstrPacketDesc[ul],

            &pPacketDescGuids[ul]);

    }

    IContextNode* node = NULL;

    hr = inkAnalyzer->AddStroke(

        id,

        packetDataCount,

        plPacketData,

        guidCount,

        pPacketDescGuids,

        &node);

    // check HR

    if(pPacketDescGuids != 0)

    {

        delete [] pPacketDescGuids;

    }

    ::SafeArrayUnaccessData(varPacketDesc.parray);

    ::SafeArrayDestroy(varPacketDesc.parray);

    ::SafeArrayUnaccessData(varPacketData.parray);

    ::SafeArrayDestroy(varPacketData.parray);

    return node;

}

After you can add the ink analysis data for one stroke to the ink analyzer, you can add the data for many strokes.

 

This final function, AddStrokesToInkAnalyzer, will use the AddStrokeToInkAnalyzer, function to add all of the strokes in the strokes collection to the analyzer.

// AddStrokesToInkAnalyzer

// Adds the strokes contained in a IInkStrokes collection to an InkAnalyzer

void CSimpleRecognizerDlg::AddStrokesToInkAnalyzer(IInkAnalyzer* inkAnalyzer, IInkStrokes* strokes)

{

    long count = 0;

    strokes->get_Count(&count);

    // Add the strokes to the InkAnalyzer

    for(int index = 0; index < count; index++)

    {

        CComPtr<IInkStrokeDisp> stroke;

        strokes->Item(index, &stroke);

        CComPtr<IContextNode> unclassifiedNode;

        unclassifiedNode.Attach(AddStrokeToInkAnalyzer(inkAnalyzer, stroke));

    }

}

 

Now that you are capturing ink and have functions to analyze the strokes, you are ready to add the final hook into your code that will recognize the ink when you click the Recognize button. Return to the dialog editor (click on Resource View… Dialog… IDD_SIMPLERECOGNIZER_DIALOG) and then double click on the recognize button to add a handler.

 

The following code is an example of how this can be done using the and the InkAnalyzer interface and the GetRecognizedString function:

 

try   {

            CComPtr<IInkAnalyzer> inkAnalyzer;

            inkAnalyzer.CoCreateInstance(CLSID_InkAnalyzer);

            CComPtr<IInkDisp> ink;

            this->inkCollector->get_Ink(&ink);

            CComPtr<IInkStrokes> strokes;

            HRESULT hr = ink->get_Strokes(&strokes);

            // check hr

            // Add the strokes to the InkAnalyzer (See InkAnalysisHelper.cpp)

            AddStrokesToInkAnalyzer(inkAnalyzer, strokes);

           

            // Perform synchronous analysis

            CComPtr<IAnalysisStatus> status;

            hr = inkAnalyzer->Analyze(&status);

            // check hr

            // Confirm that analysis was successful

            VARIANT_BOOL wasSuccessful = VARIANT_FALSE;

           

            // Get the recognized string from analysis

            CComBSTR recoString;

            hr = inkAnalyzer->GetRecognizedString(&recoString);

            // check hr

            // Display the reco result

            ::MessageBoxEx(NULL, recoString, L"Recognized string from analysis", NULL, NULL);

      } catch(CString &message){

            ::MessageBoxEx(NULL, message, L"Error", NULL, NULL);

      }

 

Now that you have added the handler for your code, you can test recognizing ink. Press F5 to run your application, write some text into the reco area, and click the Recognize button. You should get a popup message with the recognized text.

You can download my complete project solution for Visual Studio 2005/2008 here.