Look below the UI for more effective and robust UI automated test case designs

Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.

Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.

For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!

To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.

In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below

    1: using System;
    2: using System.Runtime.InteropServices;
    3:  
    4: namespace TestingMentor.PInvoke
    5: {
    6:   class NativeMethod
    7:   {
    8:     [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    9:     public static extern bool SetLocaleInfo(
   10:       uint locale, 
   11:       uint localeType,
   12:       string localeData);
   13:  
   14:     [DllImport("kernel32.dll", SetLastError = true)]
   15:     public static extern bool SetLocaleInfo(
   16:       uint locale,
   17:       uint localeType,
   18:       int localeData);
   19:   }
   20:  
   21:   class NlsConstant
   22:   {
   23:     public enum Locale : uint
   24:     {
   25:       Invariant = 0x007F,
   26:       SystemDefault = 0x0800, // use system default for setlocaleinfo
   27:       UserDefault = 0x0400,
   28:       Neutral = 0x0000,
   29:       CustomDefault = 0X0C00, // Vista and later
   30:       CustomUiDefault = 0x1400, // Vista and later
   31:       CustomUnspecified = 0X1000  // Vista and later
   32:     };
   33:  
   34:     public enum LocaleType : uint
   35:     {
   36:       // VALUE LCDATA TYPES
   37:       CalendarType = 0x00001009,  // type of calendar specifier
   38:       CurrencyDigits = 0x00000019,  // local monetary fractional digits
   39:       CurrencySymbol = 0x0000001B,  // position of positive currency symbol 
   40:       FractionalDigits = 0x00000011,  // number of fractional digits
   41:       NativeDigitSubstitution = 0x00001014,  // native digit substitution
   42:       FirstDayOfWeek = 0x0000100C,  // first day of week specifier
   43:       FirstWeekOfYear = 0x0000100D,  // first week of year specifier
   44:       LeadingZeros = 0x00000012,  // leading zeros for decimal
   45:       Measure = 0x0000000D,  // 0 = metric, 1 = US
   46:       NegativeCurrency = 0x0000001C,  // negative currency mode
   47:       NegativeNumber = 0x00001010,  // negative number mode
   48:       PaperSize = 0x0000100A,  // paper size
   49:       TimeFormat = 0x00000023,  // time format specifier
   50:       
   51:       // STRING LCDATA TYPES
   52:       // Valid Unicode characters 
   53:       AM = 0X00000028,  // AM designator
   54:       PM = 0x00000029,  // PM designator
   55:       CurrencySymbol = 0x00000014,  // local monetary symbol
   56:       DecimalSeparator = 0x0000000E,  // decimal separator
   57:       DigitGrouping = 0x00000010,  // digit grouping
   58:       ListSeparator = 0x0000000C,  // list item separator
   59:       LongDate = 0x00000020,  // long date format string
   60:       MonetaryDecimalSeparator = 0x00000016,  // monetary decimal separator
   61:       MonetaryGrouping = 0x00000018,  // monetary grouping
   62:       MonetaryThousandSeparator = 0x00000017,  // monetary thousand separator
   63:       NativeDigits = 0x00000013,  // native ascii 0-9
   64:       NegativeSign = 0x00000051,  // negative sign
   65:       PositiveSign = 0x00000050,  // positive sign
   66:       ShortDate = 0x0000001D,  // short date format string
   67:       ThousandSeparator = 0x0000000F,  // thousand separator
   68:       TimeSeparator = 0x0000001E,  // time separator
   69:       TimeFormat = 0x00001003,  // time format string
   70:       YearMonthFormat = 0x00001006   // year month format string
   71:     };
   72:  
   73:  
   74:     public enum LocaleData : int
   75:     {
   76:       // LOCALE_ICALENDARTYPE VALUES
   77:       Gregorian = 1, // Gregorian (localized)
   78:       GregorianUS = 2, // Gregorian(Always English)
   79:       GregorianMEFrench = 9, // Middle East French
   80:       GregorianArabic = 10,
   81:       GregorianXlitEnglish = 11, // transliterated English
   82:       GregorianXlitFrench = 12, // transliterated French
   83:       Japan = 3,
   84:       Taiwan = 4,
   85:       Korea = 5,
   86:       Hijri = 6,
   87:       Thai = 7,
   88:       Hebrew = 8,
   89:       Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later
   90:  
   91:       // LOCALE_ICURRENCY
   92:       PositiveCurrencyPrefixNoSeparation = 0,
   93:       PositiveCurrencySuffixNoSeparation = 1,
   94:       PositiveCurrencyPrefixSeparation = 2, // one character separation
   95:       PositiveCurrencySuffixSeparation = 3, // one character separation
   96:  
   97:       // LOCALE_IDIGITSUBSTITUTION
   98:       DigitSubstitutionContextBased = 0,
   99:       DigitSubstitutionNone = 1, // use this setting for full unicode support
  100:       DigitSubstitutionNative = 2, // uses digits based on national conventions
  101:                                     // according to LOCALE_SNATIVEDIGITS
  102:  
  103:       //LOCALE_IFIRSTDAYOFWEEK
  104:       Monday = 0, // LOCALE_SDAYNAME1
  105:       Tuesday = 1, // LOCALE_SDAYNAME2
  106:       Wednesday = 2, // LOCALE_SDAYNAME3
  107:       Thursday = 3, // LOCALE_SDAYNAME4
  108:       Friday = 4, // LOCALE_SDAYNAME5
  109:       Saturday = 5, // LOCALE_SDAYNAME6
  110:       Sunday = 6, // LOCALE_SDAYNAME7
  111:  
  112:       // LOCALE_IFRISTWEEKOFYEAR
  113:       FirstDay = 0, // Week containing 1/1 even if single day
  114:       FirstFullWeek = 1, // first full week following 1/1
  115:       FirstWeek = 2, // first week with at least 4 days after 1/1
  116:  
  117:       // LOCALE_ILZERO
  118:       NoLeadingZero = 0,  // .975
  119:       LeadingZero = 1,     // 0.975
  120:  
  121:       // LOCALE_IMEASURE
  122:       Metric = 0,
  123:       US = 1,
  124:  
  125:       // LOCALE_INEGCURR
  126:       ParenthesisSymbolNumber = 0,   // ($1.1)
  127:       NegativeSignSymbolNumber = 1,  // -$1.1
  128:       SymbolNegativeSignNumber = 2,  // $-1.1
  129:       SymbolNumberNegativeSign = 3,  // $1.1-
  130:       ParenthesisNumberSymbol = 4,   // (1.1$)
  131:       NegativeSignNumberSymbol = 5,  // -1.1$
  132:       NumberNegativeSignSymbol = 6,  // 1.1-$
  133:       NumberSymbolNegativeSign = 7,  // 1.1$-
  134:       NegativeSignNumberSpaceSymbol = 8,  // -1.1 $
  135:       NegativeSignSymbolSpaceNumber = 9,  // -$ 1.1
  136:       NumberSpaceSymbolNegativeSign = 10,  // 1.1 $-
  137:       SymbolSpaceNumberNegativeSign = 11,  // $ 1.1-
  138:       SymbolSpaceNegativeSignNumber = 12,  // $ -1.1
  139:       NumberNegativeSignSpaceSymbol = 13,  // 1.1- $
  140:       ParenthesisSymbolSpaceNumber = 14,  // ($ 1.1)
  141:       ParenthesisNumberSpaceSymbol = 15,  // (1.1 $)
  142:  
  143:       // LOCALE_INEGNUMBER
  144:       Parenthesis = 0,  // (1)
  145:       NegativeSignNumber = 1,  // -1
  146:       NegativeSignSpaceNumber = 2, // - 1
  147:       NumberNegativeSign = 3,  // 1-
  148:       NumberSpaceNegativeSign = 4,  // 1 -
  149:  
  150:       // LOCALE_PAPERSIZE
  151:       USLetter = 1,
  152:       USLegal = 5,
  153:       A3 = 8,
  154:       A4 = 9,
  155:  
  156:       // LOCALE_ITIME
  157:       FormatAM_PM = 0,
  158:       Format24Hour = 1
  160:     };
  161:   }
  162: }

You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-­­dd. The automated oracle verified the year, month and day  values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.

I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals.  Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and  keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.