Look below the UI for more effective and robust UI automated test case designs


Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.


Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.


For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!


To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.


In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below



   1: using System;
   2: using System.Runtime.InteropServices;
   3:  
   4: namespace TestingMentor.PInvoke
   5: {
   6:   class NativeMethod
   7:   {
   8:     [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
   9:     public static extern bool SetLocaleInfo(
  10:       uint locale, 
  11:       uint localeType,
  12:       string localeData);
  13:  
  14:     [DllImport("kernel32.dll", SetLastError = true)]
  15:     public static extern bool SetLocaleInfo(
  16:       uint locale,
  17:       uint localeType,
  18:       int localeData);
  19:   }
  20:  
  21:   class NlsConstant
  22:   {
  23:     public enum Locale : uint
  24:     {
  25:       Invariant = 0x007F,
  26:       SystemDefault = 0x0800, // use system default for setlocaleinfo
  27:       UserDefault = 0x0400,
  28:       Neutral = 0x0000,
  29:       CustomDefault = 0X0C00, // Vista and later
  30:       CustomUiDefault = 0x1400, // Vista and later
  31:       CustomUnspecified = 0X1000  // Vista and later
  32:     };
  33:  
  34:     public enum LocaleType : uint
  35:     {
  36:       // VALUE LCDATA TYPES
  37:       CalendarType = 0x00001009,  // type of calendar specifier
  38:       CurrencyDigits = 0x00000019,  // local monetary fractional digits
  39:       CurrencySymbol = 0x0000001B,  // position of positive currency symbol 
  40:       FractionalDigits = 0x00000011,  // number of fractional digits
  41:       NativeDigitSubstitution = 0x00001014,  // native digit substitution
  42:       FirstDayOfWeek = 0x0000100C,  // first day of week specifier
  43:       FirstWeekOfYear = 0x0000100D,  // first week of year specifier
  44:       LeadingZeros = 0x00000012,  // leading zeros for decimal
  45:       Measure = 0x0000000D,  // 0 = metric, 1 = US
  46:       NegativeCurrency = 0x0000001C,  // negative currency mode
  47:       NegativeNumber = 0x00001010,  // negative number mode
  48:       PaperSize = 0x0000100A,  // paper size
  49:       TimeFormat = 0x00000023,  // time format specifier
  50:       
  51:       // STRING LCDATA TYPES
  52:       // Valid Unicode characters 
  53:       AM = 0X00000028,  // AM designator
  54:       PM = 0x00000029,  // PM designator
  55:       CurrencySymbol = 0x00000014,  // local monetary symbol
  56:       DecimalSeparator = 0x0000000E,  // decimal separator
  57:       DigitGrouping = 0x00000010,  // digit grouping
  58:       ListSeparator = 0x0000000C,  // list item separator
  59:       LongDate = 0x00000020,  // long date format string
  60:       MonetaryDecimalSeparator = 0x00000016,  // monetary decimal separator
  61:       MonetaryGrouping = 0x00000018,  // monetary grouping
  62:       MonetaryThousandSeparator = 0x00000017,  // monetary thousand separator
  63:       NativeDigits = 0x00000013,  // native ascii 0-9
  64:       NegativeSign = 0x00000051,  // negative sign
  65:       PositiveSign = 0x00000050,  // positive sign
  66:       ShortDate = 0x0000001D,  // short date format string
  67:       ThousandSeparator = 0x0000000F,  // thousand separator
  68:       TimeSeparator = 0x0000001E,  // time separator
  69:       TimeFormat = 0x00001003,  // time format string
  70:       YearMonthFormat = 0x00001006   // year month format string
  71:     };
  72:  
  73:  
  74:     public enum LocaleData : int
  75:     {
  76:       // LOCALE_ICALENDARTYPE VALUES
  77:       Gregorian = 1, // Gregorian (localized)
  78:       GregorianUS = 2, // Gregorian(Always English)
  79:       GregorianMEFrench = 9, // Middle East French
  80:       GregorianArabic = 10,
  81:       GregorianXlitEnglish = 11, // transliterated English
  82:       GregorianXlitFrench = 12, // transliterated French
  83:       Japan = 3,
  84:       Taiwan = 4,
  85:       Korea = 5,
  86:       Hijri = 6,
  87:       Thai = 7,
  88:       Hebrew = 8,
  89:       Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later
  90:  
  91:       // LOCALE_ICURRENCY
  92:       PositiveCurrencyPrefixNoSeparation = 0,
  93:       PositiveCurrencySuffixNoSeparation = 1,
  94:       PositiveCurrencyPrefixSeparation = 2, // one character separation
  95:       PositiveCurrencySuffixSeparation = 3, // one character separation
  96:  
  97:       // LOCALE_IDIGITSUBSTITUTION
  98:       DigitSubstitutionContextBased = 0,
  99:       DigitSubstitutionNone = 1, // use this setting for full unicode support
 100:       DigitSubstitutionNative = 2, // uses digits based on national conventions
 101:                                     // according to LOCALE_SNATIVEDIGITS
 102:  
 103:       //LOCALE_IFIRSTDAYOFWEEK
 104:       Monday = 0, // LOCALE_SDAYNAME1
 105:       Tuesday = 1, // LOCALE_SDAYNAME2
 106:       Wednesday = 2, // LOCALE_SDAYNAME3
 107:       Thursday = 3, // LOCALE_SDAYNAME4
 108:       Friday = 4, // LOCALE_SDAYNAME5
 109:       Saturday = 5, // LOCALE_SDAYNAME6
 110:       Sunday = 6, // LOCALE_SDAYNAME7
 111:  
 112:       // LOCALE_IFRISTWEEKOFYEAR
 113:       FirstDay = 0, // Week containing 1/1 even if single day
 114:       FirstFullWeek = 1, // first full week following 1/1
 115:       FirstWeek = 2, // first week with at least 4 days after 1/1
 116:  
 117:       // LOCALE_ILZERO
 118:       NoLeadingZero = 0,  // .975
 119:       LeadingZero = 1,     // 0.975
 120:  
 121:       // LOCALE_IMEASURE
 122:       Metric = 0,
 123:       US = 1,
 124:  
 125:       // LOCALE_INEGCURR
 126:       ParenthesisSymbolNumber = 0,   // ($1.1)
 127:       NegativeSignSymbolNumber = 1,  // -$1.1
 128:       SymbolNegativeSignNumber = 2,  // $-1.1
 129:       SymbolNumberNegativeSign = 3,  // $1.1-
 130:       ParenthesisNumberSymbol = 4,   // (1.1$)
 131:       NegativeSignNumberSymbol = 5,  // -1.1$
 132:       NumberNegativeSignSymbol = 6,  // 1.1-$
 133:       NumberSymbolNegativeSign = 7,  // 1.1$-
 134:       NegativeSignNumberSpaceSymbol = 8,  // -1.1 $
 135:       NegativeSignSymbolSpaceNumber = 9,  // -$ 1.1
 136:       NumberSpaceSymbolNegativeSign = 10,  // 1.1 $-
 137:       SymbolSpaceNumberNegativeSign = 11,  // $ 1.1-
 138:       SymbolSpaceNegativeSignNumber = 12,  // $ -1.1
 139:       NumberNegativeSignSpaceSymbol = 13,  // 1.1- $
 140:       ParenthesisSymbolSpaceNumber = 14,  // ($ 1.1)
 141:       ParenthesisNumberSpaceSymbol = 15,  // (1.1 $)
 142:  
 143:       // LOCALE_INEGNUMBER
 144:       Parenthesis = 0,  // (1)
 145:       NegativeSignNumber = 1,  // -1
 146:       NegativeSignSpaceNumber = 2, // - 1
 147:       NumberNegativeSign = 3,  // 1-
 148:       NumberSpaceNegativeSign = 4,  // 1 -
 149:  
 150:       // LOCALE_PAPERSIZE
 151:       USLetter = 1,
 152:       USLegal = 5,
 153:       A3 = 8,
 154:       A4 = 9,
 155:  
 156:       // LOCALE_ITIME
 157:       FormatAM_PM = 0,
 158:       Format24Hour = 1
 160:     };
 161:   }
 162: }

You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-­­dd. The automated oracle verified the year, month and day  values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.


I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals.  Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and  keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.


Skip to main content