Automated UI Testing for Native Windows Applications

Yesware has a robust culture of automated testing. We currently have no QA department where handing over manual testing bloats the “testing” phase of our build, test, release cycle. Instead, engineers at Yesware write unit tests and UI tests along with any patch they are trying to merge into our master branch. This allows us to confidently and quickly add features, refactor code, clear up tech debt, and update underlying dependencies, relying on our suite of tests to point out the exact issue(s) on any specific set of changes. We avoid lengthy development schedules that are slow to complete, cumbersome to change or adapt, and fragile in the face of the unexpected.

Automated UI tests are finicky and can be expensive to write and maintain. The choice in tooling makes a huge difference in the tests’ efficiency, reliability, and maintainability. The tooling should hide general but nitty gritty details of UI automation while allowing the developer to extend and customize to meet their specific requirements.

Yesware has much experience in writing and running automated UI tests for our web applications, and we have benefited much from the fruit of the open source community for creating, maintaining, and resolving issues for tools like capybaraSelenium WebDriver, WebKit implementations, and so many more. Something we didn’t have much experience in was automated testing for native Windows applications, or in the case of Yesware for Outlook, automated testing for a plugin to Outlook.

While trying to choose technologies to adopt, we recognized that writing automated UI tests when we didn’t control the host application would pose a challenge, so any tool that came with documentation and sample UI automation for Microsoft Office Addins would have a large leg up in our evaluation. So while we became aware of TestStack/White which builds on top of the UI Automation framework, we favored Microsoft’s offering: Coded UI. It had ample documentation and videos, and we found two sample UI automation projects covering how to use Coded UI for Office Addins.

On the surface, Coded UI looked very promising.

  • It was an official Microsoft offering
  • Visual Studio Premium already came with it bundled
  • Lots of documentation, including the aforementioned samples, videos, and MSDN documentation and walkthroughs
  • Promises to hide and abstract away the accessibility and automation layers
  • Mitigated the risk of targeting Office addins by demonstrating with working examples

However, Coded UI turned out to be a very difficult option.

The generated code was very verbose. For example, Coded UI generated 500 lines of code to

  • Launch Outlook
  • Open a message composer
  • Compose to “”, subject “hi”, body “hello”
  • Send the email


500 lines of code to compose and send an email

Such verbose code was not a problem on its own, but since the code was so unreliable for our use case, its verbosity made diagnosing and resolving issues a nightmare.

The accessibility properties in Outlook changed in very subtle ways when our code changed. The code verbosity made it a challenge both to pinpoint these subtle discrepancies and to code an elegant resolution. Even very small changes to our application, Yesware for Outlook, could require a subtle correction in our automated UI test that was difficult to identify and implement. Larger changes to our application could require very extensive changes to the UI test.

The Coded UI search engine was a flaky black box that would intermittently fail. These failures appeared maddeningly similar to subtly incorrect search criteria. However, we ruled out that the error was in our search criteria by successfully finding the target element in another instance of the Coded UI search engine with the exact same search criteria.

As a silver lining, since we needed to adjust Coded UI’s behavior at the Windows automation layer, we became more familiar with Windows automation. We were able to leverage this experience as we moved to White.

White is much more expressive and readable than Coded UI. The previous example where Coded UI generated over 500 lines of code could be written in 70 lines using White for the exact same functionality.


Running automated UI test written in White. Try out the demo.

As you can imagine, maintaining 70 lines of code is much easier than maintaining 500 lines.

As daunting as the UI automation may appear (try checking out the MSDN UI Automation overview), we have found that searching by the “class name” and by the “text” accessibility properties has been consistently sufficient to retrieve the desired UI elements. White did a good job hiding the many details of Windows accessibility that were none of our concern.

White doesn’t generate the search properties for you, so use the Windows Inspect tool to help you identify the values of the search properties of the elements you are trying to interact with.

As we’ve mentioned, we run our tests continuously, so we also have our continuous integration server run our UI tests on every candidate patch. Jake Ginnivan wrote a great guide on how to setup a TeamCity build agent on Azure to run automated UI tests. While some details may be outdated, the crux consists of the following broad strokes

  • set up the build agent on an Azure VM
  • set up a persistent graphical UI session (necessary for Windows UI automation)
  • run the TeamCity build agent within that UI session

TestStack.White is not without its flaws. While it is a vast improvement over Coded UI for us, we’ve stumbled over a couple of parts of White. Their issues page is a pretty comprehensive list. Development isn’t particularly active, but the project has great potential and plenty of areas to improve. For example, Coded UI had means to catch playback issues and retry while White does not.

The UI automation tools for native Windows testing have lagged behind web-based tools. While there is a large ecosystem of tools to choose from to run automated UI tests against your web application, we found only two major contenders for automated UI testing against native Windows applications. We hope this article helps you make a more informed decision on how to UI test your Windows application. Writing and maintaining these UI tests have been a long road for our team, but it has gone a long way to catching UI bugs in our code before they are deployed and caught by our users.

VSTO Lessons Learned

On the MSDN website, the Office and SharePoint Development in Visual Studio page discusses two primary options for developing an Office add-in: 1) use the latest and greatest Office add-in technologies targeting Office 2013, and SharePoint 2013, or 2) use Visual Studio Tools for Office (VSTO) to presumably target versions of the Office 2010 and older.

If you intend to distribute your Office add-in to the general public, don’t use VSTO.

VSTO offers many features to develop an add-in, including graphical editors to define Office UI customizations. It has extremely simple tools to manage deployment, and it magically loads code whenever the user starts your targeted Office application.

The problem with deploying it to the general public is that the VSTO Runtime (VSTOR) is a prerequisite that the user must have installed, and dealing with the VSTOR in the wild sucks.  In our experience, a number of users simply could not install the VSTOR, and there were issues even with those who could.

VSTOR upgrades can leave behind problematic artifacts, and resulting failures often produce opaque error messages. You can find a number of complaints on the MDSN Forum about one particular artifact that VSTOR upgrades leave behind. Below is one example, with no apparent, official resolution—just an incomprehensible error message:

The value of the property ‘type’ cannot be parsed. The error is: Could not load file or assembly ‘Microsoft.Office.BusinessApplications.Fba, Version=, Culture=neutral, PublicKeyToken=71e9bce111e9429c’ or one of its dependencies. The system cannot find the file specified. (C:\Program Files (x86)\Common Files\Microsoft Shared\VSTO\10.0\VSTOInstaller.exe.Config line 10)

MSDN Forum

Additionally, if your VSTO add-in takes too long to load, office applications can disable it. This is likely to happen, because the timer starts before the CPU reaches the add-in’s instructions. That’s right – your application can be penalized for “performance reasons” before your add-in is even loaded, nevermind starting to run. And if any of your potential users are on a slow computer, there is little that optimizing your code can do. Imagine our frustration trying to trace performance issues in our application when our users reported that it was disabled due to “being too slow” when the many reasons that it may be “too slow” didn’t involve our code at all.

The deployment mechanisms that VSTO borrows from ClickOnce can also fail, again with inscrutable error messages such as “Value does not fall within the expected range.” Here is another product’s support page that discusses a workaround that isn’t particularly user friendly.

Apparently, the VSTOR is prone to becoming corrupt. Our customer service representatives communicated that while troubleshooting with the user, our application started to work correctly when the user did nothing more than run the “Repair” tool for the VSTOR.

Common themes for these issues involve the development team spending too many hours on the following actions:

  • Investigating incidents with paltry and inscrutable logs
  • Identifying potential causes and fixes
  • Experimenting with fixes under various foreseeable circumstances
  • Experimenting with fixes in the wild
  • Refining those fixes as users give us feedback

Some of these issues seemed intractable, even as users went to great lengths to avail themselves for troubleshooting. Others went silent after prolonged communications with our support team going through our support script. I imagine a large portion of folks simply walked away after encountering the first issue or two trying to set up our product; with the number of possible issues, many folks could fall into this category.

With all this doom and gloom, what alternative is there? There is some talk about approach that involves implementing the IDTExtensibility2 interface. There is also the suggestion to use a tool like Add In for Microsoft Office and .net, which seems to do the heavy lifting implementing IDTExtensibility and providing features that have allowed us to replace VSTO and overcome the aforementioned problems by avoiding them (the VSTOR) altogether.

Moving to Add In Express has been undeniably the right move for Yesware for Outlook product development rather than sticking with VSTO. No difficult-to-install, buggy prerequisite exists. Therefore, prospective users’ ability to install our product has greatly improved. The ADX loader has resolved the frequent complaints about our Yesware for Outlook being disabled due to slow loading.

There are still some issues for us to improve. We kept the ClickOnce deployment mechanism since we focused on removing the VSTO dependency first, but ClickOnce can sometimes fail in the same user unfriendly ways that were already mentioned before. Add In Express offers a deployment mechanism which leverages the Windows Installer which is likely to make our installer reliable. We are always working on improving our product, so expect an MSI option for Yesware for Outlook soon!