Are you a Software Test Professional? If so, what test tool do you use? As most of people when they start in QA, you may have to become proficient with only one or two test tools because you are restricted to what is being used in your company, or what was best to test a specific application or perhaps because it aligned better with your teams' ALM tool stack.
Whichever is the reason, if you are an experienced professional you may benefit from having a bigger picture of what is out there on the Enterprise-level software test tool market. If you are a QA automation newcomer the goal of this guide would be to provide guidance in finding a good tool that matches your testing needs.
I will list the most common and popular test tools, what features makes them popular and their limitations.
MOST POPULAR UI AUTOMATION TOOLS
The list of below shows the most common UI Automation tools used for Functional Testing Automation and Regression testing (no particular order), their descriptions, features, pros and cos:
UFT/QTP - (official site here)
Quick Test Professional (QTP) now called United Functional Testing (UFT) was originally created by Mercury Interactive in the 90s and it became the de facto industry standard back then. Mercury was acquired by HP in 2006 and later HP sold its software division to Micro Focus in 2016. Therefore it's called Micro-Focus UFT now and is still one of the most used tools. However it has lost its leader position to Selenium in the last years. UFT uses VBScript as it's scripting language, it supports Web and Desktop and has protocols that ease scripting of some CRM/ERP applications like SAP and Oracle.
- It has been around for long time, it's the most mature offering.
- It does reporting much better that any other competitors.
- Its add-in protocols do speed up automation of commonly used Enterprise Management software.
- Easy to install, it's all in one package: IDE, test runner, Reporting, integration, etc.
- Integrates with Quality Center for test/bug management and it's part of HP/Micro-focus ALM tool stack.
- For simple test cases, it is the easier to learn and use even without programming skills because of its Keyword view. For more complex needs, you will need to script using VBScript.
- It covers some niches that others don't such as image-based scripting for Citrix protocol testing.
- UFT does well things like object repository, script recording and playback and reports.
- Price. Its cost-model based on protocols and frequent renewals makes its licenses much more expensive compared to every other offering.
- Limited browser and browser versions support. Windows only.
- Based on old-technology: VBScript which is a non-OOP language. ActiveX, etc.
- It was not originally designed for remote or multiple parallel execution. It is possible now but still it's not a straight forward process.
Selenium - (project site here)
Selenium has been around since 2004 but I cannot find it as being a challenger to UFT until 2008-2010. It has progressively become the industry standard since then because of its flexibility to be used on several third-party IDEs and using some of the most popular modern programing languages. It supports most Web Browsers and OS but it cannot be used for Desktop testing.
Selenium WebDriver is the main component of this offering but there is also Selenium IDE which helps record and playback scripts in a declarative easier to use view, and Selenium Remote Control and Grid that allows for remote execution of multiple tests in parallel.
- Supports modern OOP programing languages such as C#, Groovy, Java, Perl, PHP, Python, Ruby and Scala.
- It can be used from third party IDEs such as Visual Studio, Eclipse.
- It's Open-source and you can find many flavors, libraries and whole spin-off projects to help you automate almost any technology.
- Remote test execution.
- Server Hub that allows remote test execution of multiple tests in parallel.
- Not sure if a Pro or Con but: as for ALM integration, it integrates well with Jira, Jenkins and even VS Team Services, however this integration is not as straightforward as other in-stack alternatives (CodedUI with VSTS, or UTC with Quality Center)
- Runs on Windows, Linux and Mac.
- Compared to UFT and even CodedUi it requires skills to install, use and integrate with other tools.
- Selenium has an IDE (a Firefox-only Browser plugin) that it supposed to help Record actions, edit and playback but it's known to be unreliable and it was announced that it will be deprecated. Other than that it doesn't have a real IDE to help you record or edit scripts in a visual/declarative way. Testers have to manually write test scripts from their own IDE (Visual Studio, Eclipse, etc.) and therefore Selenium requires more programing skills. There isn't also a report or test runner feature, it relies on your IDE's runner. UI Object repository has to be implemented manually.
- Selenium tests are somehow more unstable compared to other test tools which makes it harder for developers to make solid automated scripts. However good, maintainable tests are achievable with good development practices.
- As with other open-source offerings, the tool is supported by communities and not a company, so bugs may not be addressed promptly.
- In general it requires more programming skills than UFT and even CodedUi (because CodedUI does have some IDE features that help you edit scripts).
Since Selenium is open-source, several other projects have spun-off for purposes not originally covered by plain Selenium. Below is a list of some of the most popular, these projects often require higher skills, longer initial setup and more programming knowledge than plain Selenium, but these are the reason behind Selenium becoming so popular::
Protractor for AngularJS (project site here)
Keep in mind:
Protector is an evolving community project, most documentation out there is obsolete, so you have to figure out issues by yourself most of the time.
Appium for Mobile Apps (project sit here)
Appium is a Selenium based UI Automation tool for Mobile apps, either native or hybrid on iOS and Android. This is a fairly developed project that has its own IDE that includes an Emulator, UI Element Inspector, etc., you would still need to write and edit your scripts on your own IDE (VS, Eclipse, etc.). It supports Selenium Grid, so parallel test execution on remote hosts is possible.
Keep in mind:
Mobile UI automation has a more complex initial setup, you need to be able to install and configure the mobile OS SDKs, download and configure Device Emulators, have a developer account with iOS, be able to provision the code, etc. This is true for all Mobile testing not only for Appium (e.g. Xamarin UI Test have the same initial hurdles).
Winium for Windows Desktop
Winium also based on Selenium is an UI automation framework designed to test Desktop applications on Windows. Just like selenium, it is open-source, community supported (which has its own pros and cons as stated above). It supports WinForms, WPF, Universal Apps (Windows Store) and even Silverlight apps. One core limitation is that it only support working on top of Windows 10.
Keep in mind:
Winium is a recent project and it's not as mature as other Desktop UI automation offers, people complain about struggling with the API and having to resort to do a hybrid solution (e.g. mixing it with CodedUI).
There are many other projects based on Selenium that are popular, like Selendroid and Robotium.
CodedUI - (MSDN documentation here)
By Microsoft, CodedUI is a powerful test framework that is superior in some aspects to other tools. For example it supports a wide range of technologies such as several IE Browsers, Desktop apps, SOAP, Windows Phone apps, etc. It also does integration with ALM very well: it integrates with VS Team Services for bug management, Agile planning, Continuous testing, etc. It supports remote execution and running multiple tests in parallel but the setup can be a bit more lengthy than for Selenium.
Even though it's easy to do get started with, It is notably difficult to maintain for larger projects, especially if using UIMaps which are auto-built UI control repositories. Also it has been reported that Microsoft won't release new versions of CodedUI (existing version will still be supported)
- If your team is already on Microsoft tool stack, CodedUI comes with any recent version of Visual Studio, otherwise Visual Studio can be pricy.
- Seamless integration with VS Team Services for all ALM processes, or with TFS and MTM for on-premise.
- Integration with MTM allows to record exploratory test that can be edited as CodedUI tests and can be associated with Backlog items and bugs.
- CodedUI IDE in Visual Studio is pretty solid with features such as Recorder, Test Explorer, Test Runner, CodedUI Maps, UI Spy, record and playback. This is somehow similar to UFT but superior to Selenium.
- Has one of the wider range of supported technologies with IE, Desktop applications (WPF, WinForms, etc.), Web Services, Windows Phone, etc.
- Supports C# and VB programing.
- Windows only. Out of the box, it doesn't support other Browsers like Chrome (you can do a hybrid solution by adding the Selenium libraries).
- Limited scripting languages compared to Selenium.
- Remote test execution and running multiple tests in parallel is supported thru MTM or VS Team Services but the configuration is a bit more complex.
- Even though, the UIMaps allow for a quicker start, programming skills are needed to have good, maintainable tests and for larger projects it's easier to drop UIMaps and implement object repository yourself.
- Mobile testing is Windows Phone only.
MS UI Automation Framework (UIA) - (documentation overview here)
Tip: UIA is an accessibility framework for development purposes. It is a component of the larger CodedUI framework, but for legacy desktop applications it is sometimes easier to go directly to UIA and use the appropriate accessibility pattern to drive actions for UI controls.
TestComplete - (official site here)
TestComplete is proprietary a UI Automation tool by SmartBear, it has been around since early 2000s, it is has pricy has UFT but it became increasingly popular when compared to it because it advertises that it doesn't need programming skills (although just like UFT you do need programming skills for advance scripts) and because it was a wider range of supported programing languages and technologies.
- Supports scripting with VB, Python, C#, Delphi, Angular, Ruby and PHP.
- Support testing Web, Mobile and Desktop.
- Requires less or no programming skills for simple test cases. It has a less steep learning curve compared to other offerings.
- It has its own UI with solid features.
- It has good documentation, supported and updated regularly.
- Windows only. No support for other OS.
- As mentioned, Price.
Katalon Studio - (official site here)
This is a relative newcomer, it has been around since 2015. It has a more reduced range of technologies it supports, but it's free and easy to use without programming skills for simple test cases.
- It has its own UI that allows for script recording and editing on a declarative view without programming skills. But it also has a scripting view for programmers to work on more advanced scripts.
- All features unified in the same UI package: Reports, Editor, Test Runner, etc.
- It's free and also has a paid dedicated support.
- It only supports Java and Grovy as programming languages.
- It has a limited range of technologies that are supported for testing: Web and Mobile.
- It doesn't have a large community for a free tool, so documentation and community support is scarse.
That is where I'm going to stop, if I describe every UI tool out there this blog post will never end. Just know that there are other tools that are very popular:
Tosca UI - official site
Rational Functional Tester (a.k.a. Rational Robot) - officinal site
Xamarin Test Studio - (official site for Visual Studio App Center here)
I'm going to mention one more alternative on the Mobile UI Test automation space because I believe that it has some features for that are pretty good compared with the others. I was not able to find a market research that had Xamarin as one of the offerings for UI Automation, either because Xamarin is relatively new or because it was rebranded when it was acquired by Microsoft. I really hope this changes in the future. Here are the reasons why I like Xamarin and Xamarin Tests Studio:
- Xamarin is a cross-platform development technology that allows you to deliver iOS, Android and Windows apps. Xamarin provides an integration framework where you can have your presentation layer code for each platform that you support and a single logic and data layer with Xamarin that would work for all platforms. This eases maintenance. Also Xamarin is fully integrated into Visual Studio for Windows and Mac.
- Xamarin Test Studio, now called Visual Studio App Center provides continuous-integration with your Mobile project, it integrates with VSTS which allows for requirement planning, automated builds, deployments, continues delivery to Stores, etc.
- Xamarin Tests are UI automation tests that allow you to do functional tests and regression of your Mobile app (iOS, Android or Windows), you create them on Visual Studio on a Xamarin Test Project. You get on the IDE some features that help you record and edit scripts such as an Emulator and Inspector and you can run the tests locally on your workstation or you can use the Xamarin Test Cloud, which is now also part of Visual Studio App Center.
- Visual Studio App Center lets you run multiple Xamarin UI Tests in parallel on hundreds of devices and configurations on the Test Cloud. These are not emulators, these are real devices.
- Also, it doesn't have to be Xamarin UI Tests, it can also be Appium or Expresso tests.
- VS App Center has some nice features such as reporting, dashboards, tracking against requirements, changesets, builds, deployments, etc.
MARKET SHARE OF UI AUTOMATION TOOLS
After a quick research, I found that most of the market share information for UI automation tools is a bit dated. And even when it is not dated, the business companies in charge of the research often mix Unit test tools and Performance Test tools with UI automation. Based on the more recent sources (from 2016-7), it seems that the following conclusions can be drawn:
- Selenium is the most used tool both by small and large companies. It has become almost an industry standard.
- Then depending on the need: UFT and CodedUI are normally the second most used, especially for larger companies and mainly for Browser and Desktop UI automation. Smaller companies tend to lean more to Selenium.
- UFT is still the tool most used to automate testing for ERPs/CRMs such as SAP, Oracle, Peoplesoft, etc.
- For Mobile testing, Appium seems to be the most used. Other offerings such as Perfecto, Expresso, Selendroid and Xamarin UI Tests are out but are much smaller players.
- On Mobile, many companies don't do UI Testing and limit themselves to unit testing with tools such as Junit, TestNG and Mockito. Therefore many studies compare these tools with UI test tools. In reality all software developments should go thru both Unit test and UI tests for functional and regression testing.
- Also let's keep in mind that there are many companies that have their own internal tools for UI testing.
SO WHICH ONE SHOULD YOU USE?
When selecting a UI Automation test tool:
- Take in consideration your programming skills, for example UFT is VbScript only, CodedUI can be used with C#, VB, Java, etc. but Selenium is the most flexible on this category.
- Do you have a large QA team, do you need a lot of collaboration and integration with ALM? What is your team using? It may be better to align with the tool stack that your team is using: e.g. CodedUI integrates seamlessly with VS Team Services. Selenium integrates well with third party tools like Jira and VSTS but needs a bit more configuration to do so. UFT also connects seamlessly with Quality Center.
- What type of application are you testing? Desktop? Mobile? Browser-based? On what OS?
- Price and support. UFT/CodedUI are well supported by their proprietors, Selenium is open-source.
- Lastly, take a look at the Charts above, borrowed from Quality Center Group, if you are new and don't have any of the restrictions mentioned before, it may be smartest to learn what most people are using since you are likely to find more work and information on them, or alternatively, you may want to specialize on a market niche.
- I think it is time for QTP to die*. (March 14, 2011). Retrieved from Paul Hammans' blog: https://paulhammant.com/blog/die-qtp-die.html
- Test Automation Tool comparison – Selenium vs QTP. (Nov 25, 2013). Retrieved from Aspyre Systems: https://blog.aspiresys.com/testing/test-automation-tool-comparison-selenium-vs-qtp/
- Test Automation Tools and Trends for 2016. (March 16, 2016). Retrieved from Test Project: https://blog.testproject.io/2016/03/16/test-automation-survey-2016/
- The QA job market is a killer, would you stand a chance? (Jan 4, 2016). Retrieved from Quality Test Group: https://www.qualitestgroup.com/blog/future-of-software-testing/the-qa-job-market-is-a-killer-would-you-stand-a-chance/
- UI Testing Automation Tools and Frameworks. (March 7, 2018). Retrieved from SCREENSTER: https://screenster.io/ui-testing-automation-tools-and-frameworks/#uft