UI Automation Math Text Support

  • Article

Microsoft products expose their contents for accessibility purposes via a set of interfaces known as UI Automation (UIA). Currently UIA has no special support for math text. Either the assistive technology program (AT) has to figure out if math is involved or the application has to return math-specific speech text as done with Office math speech. To support math more generally, UIA needs three additions:

  1. A document (ITextProvider) property that specifies the AT’s desired default math format (MathML, LaTeX, UnicodeMath, …) for math-zone text returned by calling ITextRangeProvider::GetText()
  2. A parameter that specifies the math format for an individual ITextRangeProvider::GetText() call
  3. Ways to navigate and select math zones

An AT like NVDA that handles all math accessibility (speech and braille of various verbosities and options) would use addition 1. With MathML containing the <maction> entity for UI, NVDA could generate math speech and braille enabling both speaking and editing of math. An AT like Narrator that doesn’t understand MathML would use addition 2, getting math speech with one call and math braille with another. This post describes these ways for improving UIA math support.

Recognizing Math Text

For the first two additions, it’s easy for an AT to recognize most math formats in a text string returned by UIA, especially since the AT chooses which format to return and can be looking for it. MathML math zones are XML strings that start with <mml:math> and end with </mml:math>. A LaTeX inline math zone starts with “$” or “\(“ and ends with “$” or “\)”. A LaTeX display math zone starts with “$$” or “\[“ and ends with “$$” or “\]”. A UnicodeMath math zone starts with “⁅” (U+2045) and ends with “⁆” (U+2046). Math braille is given by characters in the Unicode U+2800..U+28FF braille block. Non-math text uses other Unicode characters since braille engines can braille natural languages.

Math speech supplied by Office apps usually doesn’t have start and end math speech text. It might be worthwhile to have a math speech format type that includes the delimiters <mathspeech> and </mathspeech>. These delimiters wouldn’t be spoken but could be cues to speak the text as is and afterward to call for math braille if brailling is active. If a <mathspeech> XML element is added, it’d be worthwhile to support the Speech Synthesis Markup Language (SSML) more generally so that math character styles could be spoken with a different pitch, for example. Another possibility is to add <mathspeech> to SSML.

The sections below define methods that provide this UIA math functionality. Note that unless Narrator wants to take advantage of Office Nemeth math-braille capabilities, Windows doesn’t need to do anything other than document the new methods and include them in UIAutomationCore.h. Math speech already works well with Narrator, although it doesn’t offer math verbosity options (which differ from natural-language verbosity options).

Document Math Text Format Property

Typically, UIA doesn’t have UIA state properties. The properties it exposes are properties of the source content. But to define which math format ITextRangeProvider::GetText() should use by default, UIA needs to set a document property that specifies the math format. Accordingly, we define ITextProvider3 as follows

 MIDL_INTERFACE("242A2469-3CAB-403E-9DA6-FAF1327C7FC6")
ITextProvider3 : public ITextProvider2
{
   public:
   virtual HRESULT STDMETHODCALLTYPE get_Property(
   /* [in] */ int Type,
   /* [retval][out] */ int *pValue) = 0;

   virtual HRESULT STDMETHODCALLTYPE set_Property(
   /* [in] */ int Type,
   /* [in] */ int Value) = 0;
};

Type specifies the property type. For now, there’s only the default math format: TextProperty_MathFormat (1). Its values are given by

 enum MathFormatType
{
   MathFormatType_Default     = 0, // Same as GetText
   MathFormatType_MathML      = 1, // Math zones in MathML
   MathFormatType_Nemeth      = 2, // Math zones in Nemeth braille (U+2800 block)
   MathFormatType_LaTeX       = 3, // Math zones in LaTeX
   MathFormatType_UnicodeMath = 4, // Math zones in UnicodeMath
   MathFormatType_Speech      = 5, // Math zones with speech
};

Another property type could be TextProperty_MathVerbosity. To get an ITextProvider3 interface, the client calls

 ITextProvider::QueryInterface(__uuidof(ITextProvider3), (LPVOID *)ppTextProvider3)

If this call fails, the program doesn’t have math-format support.

Range Math Text Property

To enable getting more than one math format, e.g., speech and braille, we define the range-level interface

 MIDL_INTERFACE("724258C8-8A0D-407D-9622-E5E75D307513")
ITextRangeProvider3 : public ITextRangeProvider2
{
   public:
   virtual HRESULT STDMETHODCALLTYPE GetText2(
   /* [in] */ int maxLength,
   /* [in] */ int Flags,
   /* [retval][out] */ __RPC__deref_out_opt BSTR *pRetVal) = 0;
};

The arguments maxLength and pRetVal are the same as for ITextRangeProvider::GetText(maxLength, pRetVal). The low four bits of Flags are given by the MathFormatType enum above. The AT calls

 ITextRangeProvider::QueryInterface(__uuidof(ITextRangeProvider3),
      (LPVOID *)ppTextRangeProvider3)

If this call fails, the program doesn’t have range-level math-format support.

Math Zone Navigation

There are two general kinds of math-zone navigation: 1) from one math zone to another, and 2) within a math zone. The latter can be accomplished with existing functionality, typically by following the program selection changes or by moving by UIA TextUnit_Character and TextUnit_Word.

To navigate up to a math zone or skip onto the next math zone, UIA needs to have a math-zone unit. UIA annotation and attribute values are distinct from the defined UIA TextUnit’s, since attributes are in the 40000 range and annotations are in the 60000 range, while the TextUnit’s are < 10. If we enable the ITextRangeProvider ExpandToEnclosingUnit() and Move(), etc., methods to treat AnnotationType_Mathematics as another unit, then an AT could move by math zones, expand to a math zone, etc. If the AT calls for moving by TextUnit_Format, a math zone would be a format break point. If the QueryInterface for an ITextProvider3 succeeds, then a client could expect that navigation by AnnotationType_Mathematics would work. If a call returns an HRESULT error, navigation and selection by AnnotationType_Mathematics isn’t supported.

Math Control

People have thought about an alternative approach that uses a UIA math control like a UIA hyperlink or table control. This approach is discussed in Math Accessibility Trees. While it makes sense theoretically, math zones can be numerous and are often very small such as the variable 𝑥, which is only a single character. So, it’s simpler and more efficient to treat math zones as text with a math-zone attribute. AnnotationType_Mathematics seems to fit the bill and it has been defined for several years.

Note that the Text Object Model (TOM) uses a math-zone attribute (tomMathZone—see tom.h) as a unit for navigation and selection. Version 2 of TOM has advanced support for math text processing some of which is described in the post Setting and Getting Math Speech, Braille, UnicodeMath, LaTeX… For UIA purposes, the relatively simple approaches given here seem to suffice.