Did I Remember To


[29 Mar 2010: An expanded and annotated version of this is available on my website at http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf.]

Accessibility

  • Check the following MSAA property settings for every control on every dialog, command bar, and other UI feature: NAME, ROLE, STATE, VALUE, KBSHORTCUT, and DEFACTION
  • Change values in edit boxes and verify those values are reflected in the MSAA property set
  • Run in high contrast mode
  • Run in large font mode
  • Run with Sound Sentry
  • Run with sticky keys and other mouse key settings
  • Verify focus events are passed when controls receive focus
  • Verify that all audio, video, and animations have supporting textual scripts, screen narration, and/or closed captioning
  • Verify that text is rendered as text, not as bitmaps
  • Verify tabbing order
  • Verify shortcut keys exist for each control
  • Verify that each actionable color item can have its color customized (e.g., red squiggly lines)
  • Verify that all flashing objects flash to the system cursor blink rate

Alerts

  • Verify each alert identifies the problem first and then a solution
  • Verify alert text is not unnecessarily long (i.e., more than 3 lines)
  • Verify alert text is consistently worded and consistent in style
  • Verify alert text is correct and appropriate for the situation
  • Verify alert text contains complete sentences with correct capitalization and punctuation
  • Verify alert text avoids using abbreviations and acronyms
  • Verify alert text does not use an accusatory tone but rather is polite and helpful
  • Verify alert text uses the product name where necessary, not “we”
  • Verify the buttons work correctly
  • Verify the buttons have unique access keys
  • Verify alert text is localized correctly
  • Verify the correct icon (Information, Warning, Critical) is displayed on the left hand side
  • Verify the title bar contains the name of the product (e.g., “Microsoft Bob”)
  • Verify the buttons are centered below the message text

Anti-Virus

  • Test with and without anti-virus software installed

Application Interoperability 

  • Verify clipboard operations (i.e., cut/copy/paste) between this and other features, between this and other applications
  • Verify drag and drop operations between this and other features, between this and other applications

Boundary Cases 

  • Text length (e.g., of a list entry, in a field)
  • Extra-long filenames
  • Spaces in the text (e.g., of a list entry, field name)
  • Duplicate items
  • Large selection (e.g., the entire document or list)
  • Giant files
  • 4096 x 4096 pixel image
  • Maximum number of objects (e.g., shapes, columns)
  • Maximum and minimum values (e.g., for margins, values in edit boxes)

CPU

  • Multiple processors
  • Multiple vendors CPU’s (e.g., Intel, AMD)
  • Laptop configurations

Dialogs

  • Verify default edit focus and default button focus is correct
  • Verify tab order
  • Verify dialog and terms used therein are consistent with those in the rest of the application
  • Verify ninched controls are shaded
  • Verify mutually exclusive controls work correctly
  • Verify any samples in the dialog reflect the actual formatting in the document
  • Verify all commands that launch dialogs (e.g., F12 to Save As)
  • Verify every control has a short-cut letter, that all short-cuts work, and that every action can be executed via the keyboard
  • Verify every accelerator is unique within the dialog
  • Verify the dialog’s title is correct
  • Verify the menu command to show dialog is followed with “…”
  • Verify the sizing and spacing of all controls matches the design standard
  • Verify all dialog states, especially including more or less controls being visible due to application state or settings
  • Verify the correct system controls display on the title bar, and that they work correctly
  • Verify help topics are correct
  • Verify dialog tips (right-click on a control) are correct
  • Verify status bar text is correct
  • Verify the dialog displays and functions correctly with different color, font, and high contrast settings
  • Verify the dialog displays and functions correctly in high DPI mode
  • Verify all bitmaps being used with the text in the dialog are correctly localized
  • Verify controls respond correctly to valid and invalid input
  • Verify controls respond correctly to boundary case input (e.g., correct alerts are shown)
  • Verify Escape cancels the dialog
  • Verify the dialog is sticky (or not, as appropriate); i.e., it displays in same position as when last dismissed
  • Verify settings in the dialg are saved (or not, as appropriate) the next time the dialog is invoked
  • Verify the dialog is sized correctly in relation to its controls

File Names

  • With and without the complete path
  • With and without the extension
  • Long file names
  • Short file names
  • Deep directory levels
  • International characters
  • Invalid characters (e.g., ‘?’)
  • Invalid filenames (e.g., COM1, COM1.txt)

Hardware

  • Tablet PC
  • Removable Media
  • Pocket PC
  • Run with Power Management settings enabled and disabled

International Sufficiency 

  • Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
  • Run with Complex scripts and Right to Left (RTL) support
  • Verify all controls function correctly with the East Asian (IME)
  • Verify switching the keyboard to a different mapping works correctly
  • Verify system locale, unicode, and non-active code page support functions correctly
  • Verify ANSI, double-byte, and Unicode text are handled correctly for display, input, and elsewhere (e.g., double-byte characters will be split if treated as ANSI)
  • Verify any language-dependant features work correctly
  • Verify extended and non-standard characters are displayed and handled correctly
  • Verify the executable and other core files function correctly on any code page and any supported OS language
  • Verify the correct sorting order is used.
  • Verify user locale or system locale are used as appropriate.

Localization 

  • Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
  • Verify you know which languages to test
  • Verify controls in dialogs and elsewhere are aligned and sized correctly (e.g., text is not truncated)
  • Verify any functionality affected by the localization process works correctly
  • Verify data is ordered correctly
  • Verify tab order is correct
  • Verify no unlocalized strings remain
  • Verify accelerators are localized
  • Verify hot keys are localzied
  • Verify each accelerator is unique and consistent
  • Verify each hot key is unique and consistent

Menus and Command Bars 

  • Verify all commands work from menus and command bars
  • Verify keyboard shortcuts are correct on all platforms
  • Verify built-in menu items on a custom commandbar work correctly
  • Verify commands are visible and/or enabled only when appropriate
  • Verify command captions are correct and consistent with similar terms used elsewhere
  • Verify custom commands work when dragged onto built-in menus and toolbars
  • Verify voice command works correctly
  • Verify menu/command bar item context menus work correctly
  • Verify status bar text is correct and not truncated

Miscellaneous — Have you

  • Completed the spec inspection has been completed
  • Discussed the feature design and target users with your Program Manager
  • Done real world testing (i.e., used the application the way the user will)
  • Talked with your developer about the areas they think need special attention
  • Brainstormed test cases with the rest of your team
  • Discussed cross-feature implications with the rest of your team
  • Discussed the feature/application with Product Support
  • Perused bugs and other feedback from beta releases
  • Considered backward compatibility problems resulting from something that has been moved or modified changing default values or breaking something in a previous version

Networking

  • Verify correct handling of different levels of user access and permissions
  • Verify correct handling of loss of network access
  • Verify correct handling of user authentication requests
  • Verify correct handling of file access via mapped drive (mapped root drive where supported)
  • Verify correct handling of file access via UNC
  • Verify correct handling of Print Preview when the network printer is disconnected
  • Verify correct handling of long file name and long-file-name-to-short-file-name conversion correctly

Open/Save

  • Open/Save from/to Sharepoint Document Library
  • Context menu | Open, New, Print
  • Drag-and-drop
  • Double-click a shortcut on the Desktop
  • Most Recently Used list
  • Command line argument
  • Read-only files, directories, floppies, drives
  • Open from/save to over various types and speeds of network connections
  • Open from/save to different types of prototols (e.g., FAT, NTFS, UNC, http://, ftp://, floppy)
  • Open from/save to previous versions; roundtrip from/to previous versions
  • Open/save documents created in a different language-version of the application
  • Interrupted
  • File/Path name (e.g., long filenames, spaces in name)
  • Auto-save
  • Save when no documents are dirty, one document is dirty, multiple documents are dirty
  • Save when multiple documents are dirty but the user chooses to not save some of them
  • Save As

Out Of Memory (OOM)/Stress Testing 

  • Low memory
  • Low disk space
  • OOM caused via automation
  • OOM caused via real world scenarios
  • Minimum requirements machine
  • Flaky network
  • Too may users
  • Working with floppies
  • Aborting operations (e.g., user cancels, network cable pulled)
  • Pen And Speech Input
  • Speech Tips
  • Handwriting Tips (non-TabletPC)
  • Microsoft and non-Microsoft Input Method Editors (IMEs)
  • Unaware application support enabled and disabled (Windows SP1 or later only)
  • Handwriting/speech/IME on Tablet PC Windows

Performance

  • Verify perf testing has been done
  • Verify perf targets exist and are being met
  • Verify the correct data is being benchmarked
  • Verify performance with Clear Type turned on
  • Verify performance optimizations actually have the intended effect
  • Compare performance with previous versions
  • Compare performance with similar applications

Platforms

  • Run on every supported platform; at a minimum, probably, WinXP SP1, WinXP SP2, Windows Server 2003 Gold

Printing

  • Verify changing orientation works properly
  • Verify printing to a file works properly
  • Verify collation works properly
  • Verify “Number of Copies” works properly
  • Verify color printing works properly
  • Verify canceling printing works properly
  • Verify scaling works properly
  • Verify line or page numbering works properly (if supported)
  • Print to both local and network printers
  • Print to Adobe Acrobat to create PDF files

Registry / Configuration File Settings

  • Verify settings that should modify behavior do
  • Verify the appropriate settings are available in policy templates
  • Verify settings roam with the user

Security 

  • Consider what buffer overrun attacks might be possible
  • Consider what denial of service attacks might be possible
  • Consider what SQL injection attacks might be possible
  • Consider what virus attacks might be possible
  • Verify no NULL DACLS are created or used.
  • Verify security for links and macros is sufficient and works correctly
  • Verify filenames such as “……file” are handled correctly
  • Verify any temporary files are created in appropriate locations and have appropriate permissions
  • Verify any user-created files have appropriate permissions
  • Verify the application functions correctly under different user rights and roles
  • Verify user-specified input is handled correctly (e.g., are SQL injection attacks possible)
  • Verify every input is bounds-checked
  • Verify user privacy is not violated (e.g., file paths are stripped, user aliases and other identifying data can be removed)

Setup – General

  • Consider what problems could appear when installing as Minimum, Typical, Install on First Use, Run from Network, Custom, and Not Available.
  • Verify registry items are created/modified appropriately, and that other registry items are not created or modified.
  • Verify the application works correctly when portions of it have been disabled by an administrator
  • Verify any files shared with another feature or application are handled correctly
  • Verify installing and uninstalling the application or feature has the correct effect (which may be “nothing should change) on other features and applications
  • Verify “normal” users can use the application when it is installed by an administrator
  • Verify the correct files are installed when the application is installed via Install On First Use
  • Verify mass deployment (e.g., via Systems Management Server) works correctly
  • Verify the application works correctly on Terminal Server
  • Verify the application works correctly on Virtual PC and Virtual Server

Setup – Local Caching 

  • Verify the correct CABs are cached
  • Verify any files shared with another feature or appliation are handled correctly by install/uninstall/reinstall
  • Verify multiple SKUs share the cache correctly
  • Verify deleting a CAB from the cache causes a reinstall and the CAB is recached

Setup – MSI Authoring

  • Verify the MSI includes the correct components, files, and registry settings
  • Verify custom actions, conditions, shortcut creation, and other special authoring works correctly
  • Verify the default installation type (e.g., Typical, Minimum) is correct
  • Verify the correct install states are available
  • Verify the application works correctly in all possible install states

Setup – Multi-User

  • Verify the application works correctly for User2 after User1 installs it
  • Verify per-user features must be installed by User2 even after User1 has installed them
  • Verify User2’s per-user settings do not change when User1 changed them

Setup – Network Setup 

  • Does your feature uninstall cleanly when installed from the network (Post-Admin install)
  • Verify the correct files are installed when the application is installed via Run From Network
  • Verify the application can be uninstalled

Special Modes and States 

  • Safe Mode (e.g., booting with /safe to disable some settings) works correctly (if supported)
  • Zoomed (e.g., 100%, 25%, 200%)
  • Send To
  • Sharing documents between multiple users/machines
  • Cut, copy, delete
  • MDI, SDI
  • No file open (just app shell)
  • Full screen view
  • No command bars visible
  • Application and document (e.g., minimized, maximized, sized really small)
  • OLE
  • Autosave on
  • System background color changed
  • Multiple selection

Undo/Redo 

  • Edit/Undo, Edit/Redo
  • Multi-level Undo/Redo
  • Menu command text strings

Upgrade and Migration – Application Upgrade

  • Verify upgrading over a previous version replaces appropriate files but no others
  • Verify side-by-side installs work correctly
  • Verify the correct files (and versions thereof) exist after an upgrade
  • Verify default settings are correct
  • Verify previously existing settings and files are modified correctly by an upgrade
  • Verify the application functions correctly when the previous and/or new version are set to Run From Network
  • Verify any features or applications dependent on files or functionality affected by the upgrade work correctly after the upgrade

Upgrade and Migration – OS Upgrade 

  • Verify upgrading over a previous version replaces appropriate files but no others
  • Verify the application works correctly after the upgrade
  • Verify any features dependent on OS files or functionality work correctly after the upgrade
  • Verify the application works correctly after upgrading both the application and the OS

User Assistance/Help

  • Verify each error message is accurate, easily understood, and actionable
  • Verify input validation error messages refer to the correct problem
  • Verify the tutorial is correct (e.g., the UI it depicts matches the actual UI, the steps it takes are correct)
  • Verify all help topics have been reviewed for technical accuracy
  • Verify all context sensitive help is correct

Video

  • Multi-monitor environments (two, three and four monitors)
  • Video resolutions (1024×768 through 1600×1200)
  • color depths (8-, 16-, 24-, and 32-bit)
  • Color themes

Window Interactions 

  • IME
  • Z-ordering, especially Help on top
  • Modal, modeless dialogs
  • Window and dialog focus
  • Window size after a restore-from-minimize, restore-from-maximize
  • Window size on first launch, on subsequent launches
  • Window size after a manual resize
  • Multiple windows (MDI)
  • Multiple instances of the application
  • Arrange Vertical, Horizontal, All, Tiled, Cascade (both MDI children and application windows on the Desktop)

Windows Logo Requirements

  • Run all applicable Application Verifier tests (especially check that NULL DACLs are not used or created)
  • Install and run under Application Verifier with the “High Version Lie” test enabled
  • Install and run with Fast User Switching enabled, while Fast User Switched
  • Install and run on Windows 2000, then upgrade to Windows XP
  • Install and run in a highly secure Windows environment (“%system%securitytemplateshisecwsinf)
  • Run as a limited user
  • Verify temp files are placed properly
  • Verify System File Protection files are not overwritten
  • Verify that My Documents is the initial default location for Open and Save
  • Verify all UI inherits Windows XP themes
  • Verify files outside of the application folder are associated with a particular file type (i.e., double-clicking does the right thing)
  • Verify all executable components have a (correct) digital signature
  • Verify all UI is legible and accessible at 133DPI (high DPI)

Year 2000 

  • Verify dates entered with a two-digit year are interpreted as:
  •  1/1/00 through 12/31/29 are interpreted as 1/1/2000 through 12/31/2029
  •  1/1/30 through 12/31/99 are interpreted as 1/1/1930 through 12/31/1999
  • Verify dates on leap years are correctly interpreted (e.g., 2/29/2000, 2/29/2004)
  • Verify dates at least to 2035 are supported

 

[29 Mar 2010: An expanded and annotated version of this is available on my website at http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf.]

Comments (19)

  1. Siva says:

    Excellent post. Adding this post to my blog.

  2. Check list for a tester

  3. Anonymous says:

    S?curit? Web – Une checklist de tests – Antonio Fontes

  4. Gareth Brown says:

    This is a wicked Testing Checklist of application testing from The Braidy Tester

  5. "new-tester" commented on my Hallmarks of a Great Tester post to ask

    "I was wondering if you know where…

  6. Jeke Green asks me via the Contact me link

    "Hi Anutthara, I enjoyed reading your Spec Reviews for world…

  7. Next time you get asked how to test software in an interview, take some suggestions from the Braidy Tester

Skip to main content