Automated UI regression testing

Over the past year year or so I’ve been pretty passionate about trying to make automated UI regression testing easier for the Windows Mobile team. As such I helped to develop an automated system that will compare bitmap snapshots of an app’s UI from 2 builds and will notify our testers if anything has changed. Other groups at Microsoft have expressed their interest in using this tool and this has got thinking of how others may be approaching this same testing challenge (or if anyone even cares about this sort of test work). Anyone have any comments?

[Author: patderks]

Comments (11)

  1. Alex Odintsov says:

    This is very interesting information. I wish MS would have such tool available for the customers.

    I’m not sure if you can give more details, but if you can, can you describe how it works? Does it make screenshots and them compares pixels? Does it go from the beginning to the end of application automatically or it controlled by a person?

  2. SpiderMan says:

    Hey Patderks, this sounds very interesting. We are testing .NET WinForm controls by a tool made by our testers, which is extended from the NUnit framework. But the GUI test is still a problem to us. Could you share some more information about your tool?


  3. I would love to see more Windows Mobile testing tools. Maybe this could be an open source project somewhere?

    I don’t know if such a thing exists, but I believe its called acceptance testing. Basically record all my actions (ie: clicked button a, then menu x, then tapped there…) for playback during a test to make sure all the actions execute successfully.

  4. jevans says:

    Ship it!

  5. patderks says:

    The tool I mentioned doesn’t drive the UI at all, I’m not talking about a UI automation system that drives the UI (I know there are some tools currently that can help you automate your apps UI for testing purposes, check out this article on MSDN for some interesting ideas for testing a .NET app using reflection I can’t go into detail on how the tool I mentioned works other than what I said already but I am very interested to find out how other testers outside of Microsoft approach purely visual/UI regression testing (i.e. making sure an app "looks" good and doesn’t regress through the product life cycle). I’ll take any comments to heart and see what I can do to share ideas/tools or suggestions 😉

  6. jevans says:

    We’re stuck with manual testing which of course means our regression testing is extremely weak or nearly non-existant. Any leads on automated testing would be much appreciated!

    P.S. Would be most helpful if it doesn’t depend on .NET

  7. Jun Ma says:

    I am using AutoItX, a freeware COM object, to drive the Installation GUI tests. It looks much easier than those solutions listed in the MSDN. My driver is written in Visual Basic. It can detect the window, checkm the text on a window, send mouse events or keystrokes to the window, and make screenshots of the window. I am pretty happy with its capabilities. However, for a complex application, such as Microsoft Word, it may not grab all the strings on the window. That could be an issue.

    Check out the AutoItX if you are interested.

  8. Phil says:

    Pat, I’m glad you’re passionate about creating tools for the Windows Mobile team, but could you and your fellow Microsoft employees perhaps extend that passion to supporting your  customers?  I’m a Windows Mobile application tester, and there are virtually no useful testing tools in the Windows Mobile SDKs at all.  And the tools that are there are terrible (e.g. App Verifier just crashes on me).

    Mobile device devs and testers desperately need the following:

    1) A resource-constraint tool that allows testers to limit resources on the device and then see how their apps handle these situations

    2) UI automation.  I know this exists inside Microsoft (DATK), so why won’t Microsoft share these bits with the world?

    If Microsoft wants apps to be developed AND TESTED for Windows Mobile, it had better start producing some tools for doing this.  Visual Studio is great for cranking out apps, but there are no testing resources whatsoever.

  9. Andy says:

    For automated regression testing of GUI on mobile devices, we use a tool called CountDown from TestQuest.


  10. Mohan Babu.J says:

    I cant find any link for automated tools in this Forum, Could you please provide me the link for automated testing tools description, bcoz iam new to this windows mobile testing so iam not even understanding you discussion. If u have any useful information please drop me mail to :