Reading the word under the cursor via UI Automation


A while back, I showed how to use UI Automation to extract the text under the mouse cursor. Let's refine the program so it can extract the word under the cursor.

The ability to extract subunits of text is provided by the Text­Pattern class. You can ask for the text around a specific point by asking Range­From­Point to create a zero-length range directly under the cursor, and then adjust the range to grab the text you want.

Start with the first Little Program and make these changes:

using System;
using System.Windows;
using System.Windows.Forms;
using System.Windows.Automation;
using System.Windows.Automation.Text;

class Program
{
 static Point MousePos {
  get { var pos = Control.MousePosition;
        return new Point(pos.X, pos.Y); }
 }

 public static void Main()
 {
  for (;;) {
   AutomationElement e = AutomationElement.FromPoint(MousePos);
   if (e != null) {
    object o;
    if (e.TryGetCurrentPattern(TextPattern.Pattern, out o)) {
     var textPattern = (TextPattern)o;
     var range = textPattern.RangeFromPoint(MousePos);
     range.ExpandToEnclosingUnit(TextUnit.Word);
     Console.WriteLine(range.GetText(-1));
    }
   }
   System.Threading.Thread.Sleep(1000);
  }
 }
}

We grab the element under the mouse and see if it supports Text­Pattern. If so, we ask the text pattern for a Text­Pattern­Range that corresponds to the position of the cursor. It's where the insertion point would go if you clicked there. Since the insertion point always goes between characters, the initial range has zero length. We expand it to the word enclosing the imaginary insertion point, then extract the text, then print it.

Note that the Text pattern is typically supported only by editable text controls, so you will need to practice with Notepad or some other text editor.

Just to show how annoying you can push things, let's secretly select the text under the cursor.

using System;
using System.Windows;
using System.Windows.Forms;
using System.Windows.Automation;
using System.Windows.Automation.Text;

class Program
{
 static Point MousePos {
  get { var pos = Control.MousePosition;
        return new Point(pos.X, pos.Y); }
 }

 public static void Main()
 {
  for (;;) {
   AutomationElement e = AutomationElement.FromPoint(MousePos);
   if (e != null) {
    object o;
    if (e.TryGetCurrentPattern(TextPattern.Pattern, out o)) {
     var textPattern = (TextPattern)o;
     var range = textPattern.RangeFromPoint(MousePos);
     range.ExpandToEnclosingUnit(TextUnit.Word);
     // Console.WriteLine(range.GetText(-1));
     range.Select();
    }
   }
   System.Threading.Thread.Sleep(1000);
  }
 }
}

Once a second, the program autoselects the word under the cursor. This gets really annoying fast, but it's just a demonstration.

Comments (12)
  1. Wow…I find all kinds of automation quite interesting, but I did not know that this is possible. Thanks for that!

  2. 0xDEADC0DE says:

    I've tried all your examples and I like them. Very useful. There is so much you can do with it.

  3. lurker says:

    Why are you using C# and not C++ for these kinds of examples?

    [Because C# lets me focus on demonstrating the technique. C++ would get all bogged down in unimportant details. -Raymond]
  4. @lurker says:

    Because C# is so much better than C++. And the automation framework is very easy to use in C#.

  5. Nick says:

    @lurker: That's like asking why he used JavaScript for the set enumerations: because he wanted to. Little program.

  6. Anon says:

    @lurker

    Why reinvent the world when someone else has done nearly all the work for you?

  7. Vaibhav says:

    I encounter the following error on running this code:

    The type or namespace name 'Point' could not be found (are you missing a using directive or an assembly reference?)

    What is missing?

  8. Drak says:

    @Vaibhav: Possible a reference and a using of 'System.Drawing'.

  9. John says:

    Hey, there are some subtle differences with this: stackoverflow.com/…/4734275 can you explain them?

    [How about you identify the subtle differences for me. This is like saying "I found some subtle differences between these two translations of Anna Karenina. Can you explain them?" -Raymond]
  10. Nick says:

    @Vaibhav: You're probably targeting the wrong framework or missing some references. The code listed has a using that will bring in System.Windows.Point. msdn.microsoft.com/…/system.windows.point(v=vs.110).aspx

  11. MuiBienCarlota says:

    It works with notepad but there is a swift in visual studio editor. Just like if we need to ScreenToClient the Control.MousePosition.

    But according to documentation, point (in Control.MousePosition and in AutomationElement.FromPoint) are supposed to be desktop coordinate.

    Strange.

  12. Lexicon Valley Fan says:

    I can't let the Anna Karenina reference go uncommented-upon: That was an awesome podcast.

Comments are closed.

Skip to main content