Automated UI Testing for Native Windows Applications

Yesware has a robust culture of automated testing. We currently have no QA department where handing over manual testing bloats the “testing” phase of our build, test, release cycle. Instead, engineers at Yesware write unit tests and UI tests along with any patch they are trying to merge into our master branch. This allows us to confidently and quickly add features, refactor code, clear up tech debt, and update underlying dependencies, relying on our suite of tests to point out the exact issue(s) on any specific set of changes. We avoid lengthy development schedules that are slow to complete, cumbersome to change or adapt, and fragile in the face of the unexpected.

Automated UI tests are finicky and can be expensive to write and maintain. The choice in tooling makes a huge difference in the tests’ efficiency, reliability, and maintainability. The tooling should hide general but nitty gritty details of UI automation while allowing the developer to extend and customize to meet their specific requirements.

Yesware has much experience in writing and running automated UI tests for our web applications, and we have benefited much from the fruit of the open source community for creating, maintaining, and resolving issues for tools like capybaraSelenium WebDriver, WebKit implementations, and so many more. Something we didn’t have much experience in was automated testing for native Windows applications, or in the case of Yesware for Outlook, automated testing for a plugin to Outlook.

While trying to choose technologies to adopt, we recognized that writing automated UI tests when we didn’t control the host application would pose a challenge, so any tool that came with documentation and sample UI automation for Microsoft Office Addins would have a large leg up in our evaluation. So while we became aware of TestStack/White which builds on top of the UI Automation framework, we favored Microsoft’s offering: Coded UI. It had ample documentation and videos, and we found two sample UI automation projects covering how to use Coded UI for Office Addins.

On the surface, Coded UI looked very promising.

  • It was an official Microsoft offering
  • Visual Studio Premium already came with it bundled
  • Lots of documentation, including the aforementioned samples, videos, and MSDN documentation and walkthroughs
  • Promises to hide and abstract away the accessibility and automation layers
  • Mitigated the risk of targeting Office addins by demonstrating with working examples

However, Coded UI turned out to be a very difficult option.

The generated code was very verbose. For example, Coded UI generated 500 lines of code to

  • Launch Outlook
  • Open a message composer
  • Compose to “someone@example.com”, subject “hi”, body “hello”
  • Send the email

500_lines_of_code

500 lines of code to compose and send an email

Such verbose code was not a problem on its own, but since the code was so unreliable for our use case, its verbosity made diagnosing and resolving issues a nightmare.

The accessibility properties in Outlook changed in very subtle ways when our code changed. The code verbosity made it a challenge both to pinpoint these subtle discrepancies and to code an elegant resolution. Even very small changes to our application, Yesware for Outlook, could require a subtle correction in our automated UI test that was difficult to identify and implement. Larger changes to our application could require very extensive changes to the UI test.

The Coded UI search engine was a flaky black box that would intermittently fail. These failures appeared maddeningly similar to subtly incorrect search criteria. However, we ruled out that the error was in our search criteria by successfully finding the target element in another instance of the Coded UI search engine with the exact same search criteria.

As a silver lining, since we needed to adjust Coded UI’s behavior at the Windows automation layer, we became more familiar with Windows automation. We were able to leverage this experience as we moved to White.

White is much more expressive and readable than Coded UI. The previous example where Coded UI generated over 500 lines of code could be written in 70 lines using White for the exact same functionality.

automated_ui_test

Running automated UI test written in White. Try out the demo.

As you can imagine, maintaining 70 lines of code is much easier than maintaining 500 lines.

As daunting as the UI automation may appear (try checking out the MSDN UI Automation overview), we have found that searching by the “class name” and by the “text” accessibility properties has been consistently sufficient to retrieve the desired UI elements. White did a good job hiding the many details of Windows accessibility that were none of our concern.

White doesn’t generate the search properties for you, so use the Windows Inspect tool to help you identify the values of the search properties of the elements you are trying to interact with.

As we’ve mentioned, we run our tests continuously, so we also have our continuous integration server run our UI tests on every candidate patch. Jake Ginnivan wrote a great guide on how to setup a TeamCity build agent on Azure to run automated UI tests. While some details may be outdated, the crux consists of the following broad strokes

  • set up the build agent on an Azure VM
  • set up a persistent graphical UI session (necessary for Windows UI automation)
  • run the TeamCity build agent within that UI session

TestStack.White is not without its flaws. While it is a vast improvement over Coded UI for us, we’ve stumbled over a couple of parts of White. Their issues page is a pretty comprehensive list. Development isn’t particularly active, but the project has great potential and plenty of areas to improve. For example, Coded UI had means to catch playback issues and retry while White does not.

The UI automation tools for native Windows testing have lagged behind web-based tools. While there is a large ecosystem of tools to choose from to run automated UI tests against your web application, we found only two major contenders for automated UI testing against native Windows applications. We hope this article helps you make a more informed decision on how to UI test your Windows application. Writing and maintaining these UI tests have been a long road for our team, but it has gone a long way to catching UI bugs in our code before they are deployed and caught by our users.