How to retrieve text under the cursor (mouse pointer)


Microsoft Active Accessibilty is the technology that exposes information about objects on the screen to accessibility aids such as screen readers. But that doesn't mean that only screen readers can use it.

Here's a program that illustrates the use of Active Accessibility at the most rudimentary level: Reading text. There's much more to Active Accessibility than this. You can navigate the objects on the screen, read various properties, even invoke commands on them, all programmatically.

Start with our scratch program and change these two functions:

BOOL
OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
{
  SetTimer(hwnd, 1, 1000, RecalcText);
  return TRUE;
}

void
PaintContent(HWND hwnd, PAINTSTRUCT *pps)
{
  if (g_pszText) {
      RECT rc;
      GetClientRect(hwnd, &rc);
      DrawText(pps->hdc, g_pszText, lstrlen(g_pszText),
               &rc, DT_NOPREFIX | DT_WORDBREAK);
  }
}

Of course, the fun part is the function RecalcText, which retrieves the text from beneath the cursor:

#include <oleacc.h>

POINT g_pt;
LPTSTR g_pszText;

void CALLBACK RecalcText(HWND hwnd, UINT, UINT_PTR, DWORD)
{
  POINT pt;
  if (GetCursorPos(&pt) &&
    (pt.x != g_pt.x || pt.y != g_pt.y)) {
    g_pt = pt;
    IAccessible *pacc;
    VARIANT vtChild;
    if (SUCCEEDED(AccessibleObjectFromPoint(pt, &pacc, &vtChild))) {
      BSTR bsName = NULL;
      BSTR bsValue = NULL;
      pacc->get_accName(vtChild, &bsName);
      pacc->get_accValue(vtChild, &bsValue);
      LPTSTR pszResult;
      DWORD_PTR args[2] = { (DWORD_PTR)(bsName ? bsName : L""),
                            (DWORD_PTR)(bsValue ? bsValue : L"") };
      if (FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER |
                        FORMAT_MESSAGE_FROM_STRING |
                        FORMAT_MESSAGE_ARGUMENT_ARRAY,
                        TEXT("Name: %1!ws!\r\n\r\nValue: %2!ws!"),
                        0, 0, (LPTSTR)&pszResult, 0, (va_list*)args)) {
        LocalFree(g_pszText);
        g_pszText = pszResult;
        InvalidateRect(hwnd, NULL, TRUE);
      }

      SysFreeString(bsName);
      SysFreeString(bsValue);
      VariantClear(&vtChild);
      pacc->Release();
    }
  }
}

Let's take a look at this function. We start by grabbing the cursor position and seeing if it changed since the last time we checked. If so, then we ask AccessibleObjectFromPoint to identify the object at those coordinates and give us an IAccessible pointer plus a child identifier. These two pieces of information together represent the object under the cursor.

Now it's a simple matter of asking for the name (get_accName) and value (get_accValue) of the object and format it nicely.

Note that we handled the NULL case of the BSTR in accordance with Eric's Complete Guide to BSTR Semantics.

For more information about accessibility, check out Sara Ford's WebLog, in particular the bit titled What is Assistive Technology Compatibility.

Comments (23)
  1. Anonymous says:

    That’s pretty cool. Two questions:

    1: How heavy is that parsing and traversal?

    2: could you defer the calculation until needed and just cache the current mouse position?

  2. Anonymous says:
    1. It’s up to the implementation of IAccessible.

      2. Certainly. This was just a Q&D program.
  3. Anonymous says:

    (About to look clueless — hey, I’m a PM!)

    Aren’t you freeing an invalid g_pszText the first time round?

    And wouldn’t you want to VariantInit(&vtChild) to begin with?

    (OK, back to making coffee for the devs…)

  4. Anonymous says:
    1. LocalFree is documented to do nothing when passed NULL.

      2. The pvarChild parameter to AccessibleObjectFromPoint is an [out] parameter, so you don’t need to initialize the value on the way in.

  5. Anonymous says:

    But, g_pszText is never initialized to NULL though, therefore the first call to LocalFree is called on a random pointer.

  6. Anonymous says:

    Global pointer variables are initialized to NULL by default.

  7. Anonymous says:

    I did as the instruction says and I got the following error.

    (‘accessibility’ is the name of the project.)

    accessibility error LNK2019: unresolved external symbol _AccessibleObjectFromPoint@16 referenced in function "void stdcall RecalcText(struct HWND *,unsigned int,unsigned int,unsigned long)" (?RecalcText@@YGXPAUHWND__@@IIK@Z)

    I use VC7.1. Did anyone have this error?

  8. Anonymous says:

    The documentation for AccessibleObjectFromPoint says

    Library: Use Oleacc.lib.

  9. Anonymous says:

    Thanks! I was able to compile and link.

  10. Anonymous says:

    Hi, this is me again.

    I followed the example by creating a Win32 project. Because Win32 applications do not look cool these days, I am wondering if this can be done from a managed code. If there is a tip to do it, can you please let me know?

    Your help is very much appreciated. Thank you.

  11. Anonymous says:

    "not actually a .NET blog"

  12. Anonymous says:

    I managed to find a way to call the function from a C# program using pinvoke. It took me so long, but I am glad that it can be done.

    So, here is my problem. :-). It looks like your code cannot display non-ASCII characters. Do you have a plan to make the code globalized using Unicode? If not, can you suggest me how I can do it?

    Thanks!

  13. Anonymous says:

    Um, you need to remember to #define UNICODE and then select an appropriate font into the DC. I had hoped it wasn’t necessary to mention that. I left it out because that wasn’t the point of the article. My articles aren’t complete solutions; they are highlights of specific technology.

  14. Anonymous says:

    This program grabs the whole sentence under the cursor. Is there any way that it could just get the individual word under the cursor?

  15. Anonymous says:

    I don’t know. The point here was to get people to notice IAccessible. I leave you to explore it more on your own.

  16. Anonymous says:

    How about in visual basic 6.0?

    I’m currently developing code to press a button on a particular program window.

    If you have any advice on this, that would be great.

    pls email: s4008460@student.uq.edu.au

    thnks

  17. Anonymous says:

    I’m assuming my readers are smart enough to translate programs from one language to another. In fact, IAccessible itself is dispatch-based so it is even easier to use from a scripting language than from C++.

  18. Anonymous says:

    Commenting on this entry has been closed.

  19. Anonymous says:

    You’ll probably use it too.

  20. Anonymous says:

    Because not everybody uses the same class library.

Comments are closed.