Enumerating notification icons via UI Automation


Today's Little Program uses accessibility to enumerate the current notification icons (and possibly click on one of them). This could be done manually via IAccessible, but the BCL folks conveniently created the System.Windows.Automation namespace which contains classes that take a lot of the grunt work out of walking the accessibility tree.

While it's true that the System.Windows.Automation namespace takes a lot of the grunt work out of accessibility, it is still rather verbose, so I'm going to start with some helper functions. They're all one-liners since they simply pass their parameters through with a little bit of extra typing.

using System.Collections.Generic;
using System.Windows.Automation;
using System.Linq;

static class AutomationElementHelpers
{
 public static AutomationElement
 Find(this AutomationElement root, string name)
 {
  return root.FindFirst(
   TreeScope.Descendants,
   new PropertyCondition(AutomationElement.NameProperty, name));
 }

The Automation­Element.Find extension method searches for a descendant of an accessible element with a particular name.

 public static IEnumerable<AutomationElement>
 EnumChildButtons(this AutomationElement parent)
 {
  return parent == null ? Enumerable.Empty<AutomationElement>()
                        : parent.FindAll(TreeScope.Children,
    new PropertyCondition(AutomationElement.ControlTypeProperty,
                          ControlType.Button)).Cast<AutomationElement>();
 }

The Automation­Element.Enum­Child­Buttons extension method enumerates the button controls which are immediate children of a parent element.

 public static bool
 InvokeButton(this AutomationElement button)
 {
  var invokePattern = button.GetCurrentPattern(InvokePattern.Pattern)
                     as InvokePattern;
  if (invokePattern != null) {
   invokePattern.Invoke();
  }
  return invokePattern != null;
 }
}

The Automation­Element.Invoke­Button extension method checks if the element is invokable (which buttons are), and if so invokes its default action.

Okay, given those helpers, we can write the actual enumerator.

class Program {

 public static IEnumerable<AutomationElement> EnumNotificationIcons()
 {
  foreach (var button in AutomationElement.RootElement.Find(
                  "User Promoted Notification Area").EnumChildButtons()) {
   yield return button;
  }

  foreach (var button in AutomationElement.RootElement.Find(
                "System Promoted Notification Area").EnumChildButtons()) {
   yield return button;
  }

  var chevron = AutomationElement.RootElement.Find("Notification Chevron");
  if (chevron != null && chevron.InvokeButton()) {
   foreach (var button in AutomationElement.RootElement.Find(
                      "Overflow Notification Area").EnumChildButtons()) {
    yield return button;
   }
  }
 }

Okay, here's what's going on.

First, we enumerate all the buttons that are children of an object called User Promoted Notification Area.

Next, we enumerate all the buttons that are children of an object called System Promoted Notification Area. This object is usually empty, but it may contain an icon if a demoted icon is temporarily promoted because it is showing a balloon.

Finally, if we are asked to enumerate hidden icons, we also find the Notification Chevron button and push it. That pops up a dialog called Overflow Notification Area, and we enumerate all the buttons from that dialog as well.

Okay, let's take this function out for a spin.

 public static void Main()
 {
  foreach (var icon in EnumNotificationIcons())
  {
   var name = icon.GetCurrentPropertyValue(AutomationElement.NameProperty)
              as string;
   System.Console.WriteLine(name);
   System.Console.WriteLine("---");
  }
 }
}

When you run this program, it should print the names of all the icons in your notification area, including the hidden ones (for which it needs to open the overflow dialog).

You may have noticed that it takes a long time to generate the icons in the System Promoted Notification Area; that's because the accessibility system is going crazy looking for something that usually doesn't exist. Let's speed things up by reducing the scope of the search. Once we find the User Promoted Notification Area, we will search for the System Promoted Notification Area inside the same window. That should save a lot of time.

// in static class AutomationElementHelpers

 static public AutomationElement
 GetTopLevelElement(this AutomationElement element)
 {
  AutomationElement parent;
  while ((parent = TreeWalker.ControlViewWalker.GetParent(element)) !=
       AutomationElement.RootElement) {
   element = parent;
  }
  return element;
 }

The Automation­Element.Get­Top­Level­Element extension method walks up the control view and returns the ancestor element that is a direct child of the root.

 public static IEnumerable<AutomationElement> EnumNotificationIcons()
 {
  var userArea = AutomationElement.RootElement.Find(
                  "User Promoted Notification Area");
  if (userArea != null) {
   foreach (var button in userArea.EnumChildButtons()) {
    yield return button;
   }

   foreach (var button in userArea.GetTopLevelElement().Find(
                 "System Promoted Notification Area").EnumChildButtons()) {
     yield return button;
   }
  }

  var chevron = AutomationElement.RootElement.Find("Notification Chevron");
  if (chevron != null && chevron.InvokeButton()) {
   foreach (var button in AutomationElement.RootElement.Find(
                      "Overflow Notification Area").EnumChildButtons()) {
    yield return button;
   }
  }
 }

Of course, what's the point of enumerating the icons if you can't also click them? Let's go look for the volume control icon and click it.

 public static void Main()
 {
  foreach (var icon in EnumNotificationIcons())
  {
   var name = icon.GetCurrentPropertyValue(AutomationElement.NameProperty) as string;
   if (name.StartsWith("Speakers:")) {
    icon.InvokeButton();
    break;
   }
  }
 }

So far, this program relies on the accessible tree structure used by Windows 7 and Windows 8. If you run the program on earlier versions of Windows (or later ones), it may not work. That's because the accessible tree changed. The accessibiliy tree is not part of the API. It's something that is exposed for accessibility purposes to the end user, and the fact that there is a programmatic interface to it is an artifact of the accessibility.

In Windows Vista, there is not a separate overflow area for notification icons. Instead, you use the chevron button to expand or contract the notification area. Let's tweak our program to work on Windows Vista instead of Windows 7:

 public static IEnumerable<AutomationElement> EnumNotificationIcons()
 {
  var userArea = AutomationElement.RootElement.Find("Notification Area");
  if (userArea != null) {
   // If there is a chevron, click it. There may not be a chevron if no
   // icons are hidden.
   var chevron = userArea.GetTopLevelElement().Find("NotificationChevron");
   if (chevron != null) {
    chevron.InvokeButton();
   }
   foreach (var button in userArea.EnumChildButtons()) {
    yield return button;
   }
  }
 }

The name of the notification area in Windows Vista is simply Notification Area, and the name of the chevron is Notification­Chevron with no space. (Somebody was apparently trying to save two bytes.)

Okay, now that you see the general idea, I'll leave Windows XP, Windows 2000, and Windows NT support as an exercise.

Comments (19)
  1. skSdnW says:

    Shell_NotifyIconGetRect was added in Win7 so people creating toasts should use that if all they need is the location. This API did not exist in Vista but Vista had toasts/flyouts for several of its system icons, was automation used to get the location on Vista?

    [Hardly. Automation names are not part of the API, so if you use automation for business logic, you are going to break the next time the UI changes. System icons which used the automation API would have been broken in Windows 7. -Raymond]
  2. parkrrrr says:

    UIA was not backported to Windows 2000, Windows NT, or pre-SP3 Windows XP. That might make support for those versions of the OS problematic.

    msdn.microsoft.com/.../ee684009%28v=vs.85%29.aspx

  3. Joshua says:

    Funny I remember doing some kind of UIA on XP SP1. Obviously not the same one as it literally moved the cursor across the screen.

  4. Azarien says:

    What's the point of doing "foreach ... yield return" instead of just "return"?

  5. Joshua says:

    Since Azarian can't be bothered to use Google: stackoverflow.com/.../proper-use-of-yield-return

  6. Nick says:

    @Azarien: It's the easiest way to merge three Enumerables together plus you don't have to fill all three sets before you start doing work on the first one.

  7. Azarien says:

    @Joshua: that doesn't answer my question. Which is:

    why

       foreach (var item in collection) yield return item;

    instead of just

       return collection;

    Both are lazy evaluated if collection is lazy evaluated, so why bother with additional overhead?

  8. Azarien says:

    Okay, ignore last comment. Now I see there are three iterations in one function. I thought there are separate functions.

  9. DWalker says:

    @parkrrrr:  Support for those versions of Windows (Windows 2000, Windows NT, or pre-SP3 Windows XP) is "problematic" anyway.  What is your point?  

  10. JW says:

    So I have been wondering for a while (read: pretty much every UI Automation post you've posted), and it makes me wonder 'Why?'...

    I understand UI Automation makes things possible and slightly more accessible for the right uses. But all of your posts and examples just make me feel like these are mis-uses and abuses of the API.

    Ever since the old days of Windows, there's been problems with compatibility, people finding window class names and manipulating the UI that manner. Yet in this post we get something really similar: version specific strings and tree structures that are supposed to make those same old things + more possible. You say these will differ per version, but how do you prevent the must-stay-the-same compatibility issues as before from creeping in? And similarly, how is this going to keep coders from messing with things they shouldn't mess with? (Unless I misunderstand your post, clicking a taskbar chevron and similar Windows UI are things I would not expect to be touched by other programs.)

    Simply put... do your posts on this subject not encourage the wrong kinds of uses for this API? Or is it really intended as a FindWindow()-style replacement to poke and prod everything on the system with by every application out there?

  11. Baltasar says:

    Is this the proper way to recognize the system volume notifier:

    var name = icon.GetCurrentPropertyValue(AutomationElement.NameProperty) as string;

    if (name.StartsWith("Speakers:")) {

       // ...

    }

    Looks clumsy to me. I hope that i18n does not affect the code, i.e., the name for the icons is stable among all international versions of Windows.

  12. Gabe says:

    Baltasar: If you're looking for something that wouldn't be affected by i18n, see AutomationIdProperty. The documentation msdn.microsoft.com/.../system.windows.automation.automationelement.automationelementinformation.automationid.aspx states this:

    "While support of an AutomationId is always recommended for better testability, this property is not mandatory. Where it is supported, an AutomationId is useful for creating test automation scripts that run regardless of UI language. Clients should make no assumptions regarding the AutomationIds exposed by other applications. An AutomationId is not necessarily guaranteed to be stable across different releases or builds of an application."

    In other words, you would use it to find a given control regardless of what the UI language is, but don't expect to be able to find the Speakers icon the same way across different versions of Windows.

  13. mg says:

    Baltasar: It breaks even before i18n. For me the full name is "Speakers (USB): 2%" so it would not match with the search. UI Automation API is very clumsy when the target application is not carefully designed for the automation. However, it is usually better than the alternatives when you have to automate interaction with the user interface.

  14. parkrrrr says:

    @DWalker:

    My point is, Raymond specifically mentioned support for those operating systems as "an exercise," presumably on the assumption that the names and tree structure and whatnot would be somewhat different, but manageable. But the fact that the entire UIA subsystem is missing in those versions makes it a rather more difficult exercise.

    You could use Active Accessibility, the predecessor to UIA, but even that will fail on Windows NT if it isn't at least NT4 SP6.

    I know you're trying to say that all of those versions are EOLed, anyway, so there's no point in supporting them. But some of us live in the real world where we still encounter those versions - especially XP but even NT4, believe it or not - on a regular basis.

  15. John says:

    @JW

    We use the UIAutomation Framework heavily here for an in house Automated Testing Framework (similar to White and other UI Test Frameworks/products) to perform integration testing of our software. Combined with good ol' FindWindow and EnumThreadWindows its very powerful, and allows us to automate repetitive testing tasks freeing up our testers to focus on more actively developed portions of our product.

    [UIAutomation is for accessibility and automated testing. In this case, it's so you can write an automated test to verify that your program is showing its notification icon when it should. It is not for shipping business logic because the accessibility tree is not contractual. (If a service pack breaks your tests, then you update your tests. If a service pack breaks your app, then you have a lot of unhappy customers.) -Raymond]
  16. JW says:

    So, I take from that then that no, UI Automation is not intended for production code used by end-users. I really hope that is how programmers will use it, but I've got a feeling it is going to be popular in a lot of products to 'unlock special functionality' regardless. :/

  17. parkrrrr says:

    UI Automation is absolutely intended for production code used by end-users. As long as your production code is accessibility or test software, and that's what your end users are using it for.

  18. John Doe says:

    The enumeration should definitely not be lazily evaluated.  What happens if you take so much time with an item that user interaction, which typically close the overflow area, gets in the way of the enumeration?  Just take a snapshot of it with ToArray or ToList.

    It gets more unreliable if you think that the area might actually be open to start with, so by clicking on it will close it!  Actually, if you run the enumeration twice fast, the second time will yield fewer items.  You must also check if the overflow area is open or not.

  19. mg says:

    @John Doe:

    Lazy evaluation is good enough considering that the properties are not cached. Taking a snapshot of automation elements does not help if you are later going to ask their property values or you try to interact with them (i.e. the element must exist at that point anyway). There are other, more significant problems with the implementation. For example, FindFirst() should not be used with TreeScope.Descendants when enumerating root element, because it may iterate over thousands of elements and lead to a stack overflow (according to MSDN).

    More robust implementation could use CacheRequest to get the property values and then take a snapshot of all needed automation elements. Also a custom implementation of FindFirst() method with breadth-first algorithm and an optional maxDepth parameter would be a good idea, but the implementation is a bit tricky because the tree traversal may fail whenever the automation tree changes.

    However, the automation element of speaker icon must still be alive when you try to click it so these actions have limited benefits for that scenario.

Comments are closed.

Skip to main content