UI Automation at Belvedere

The Challenge

Here at Belvedere, our goal is to provide high-quality, specialized, robust software. This can come in many forms: servers that communicate efficiently with the various exchanges at which we trade, client programs that allow traders to react to the markets as quickly and as profitably as possible, or algorithms that drive our own market making decisions. Each and any of these components must work seamlessly and predictably. To this end, we've adopted numerous testing protocols and practices to ensure that our software meets our own quality goals, including unit tests, behave tests, regression testing, functional testing, build checks, code analysis and more. Unfortunately, the testing becomes more complicated when you try to cover more of the process tree. Most of our testing patterns completely leave off any form of UI coverage. For example, we can test that a ViewModel method call will raise an event, however, we can't test that the binding between a button and that method is clear, or that clicking a button ultimately connects you to a gateway. The best way we have to do this now is manually, with QAs, developers, and BAs working together to test behaviors functionally. The problem with manual testing is the time that it takes for everyone involved.

Belvedere Tech is a constant hive of activity and busyness. QAs are testing new functionality. BAs are ensuring that teams run effectively. Developers are breaking things coding exciting new features. These processes are necessarily slowed down to keep quality high. Perhaps the biggest drag on team speed is regression. To ensure that new features don't break existing ones, you have to test every existing feature. For two of our biggest and most complicated client applications, this would take days. By the time that regression was fully completed, you'd already be behind on the next one. As a result, we make educated guesses at what specific features will be affected by any new change and supplement those guesses with a basic suite of hot spot test cases. It's maybe TK% effective at catching bugs. The remaining TK% can cost us thousands of dollars and hundreds of man hours.

In order to mitigate this, we need to be able to quickly and reliably run a suite of tests that cover as much of our system as possible. This run must be easy to augment, clearly report success or failure, minimally knowledgeable about the components behind the UI, repeatable, and portable. The solution we implemented: coded UI testing with a custom test runner.

UI Automation Architecture

In the 90s, Microsoft developed the Common Object Model (COM) for Windows to facilitate inter- and intra-process communication. The libraries in this model are language-agnostic as long as the callers implement matching interfaces. .NET is the framework that Microsoft developed to provide access to these libraries via C# (and I suppose VB). This is why you can code a button in WPF, WinForms, Python's TKinter, etc, and it will always behave like a Windows button; at its core, they're all the same button. One of the COM communication libraries is UIAutomation, which in .NET is exposed by UIAutomationClient, UIAutomationProvider, UIAutomationTypes, and UIAutomationClientsideProviders. Automation clients can make use of these libraries to query a layout tree element exposed by a provider and interact with it. The UIAutomation libraries will handle the communication between the two processes and make the calls and returns as necessary. This is the basic architecture as defined by Microsoft:

At its core it's very simple, especially because Microsoft has baked a UI Automation Provider (right) into each one of the controls in WPF. Every button, grid, list, table, panel, and scroll bar is a UI Automation Provider. Since 99% of our applications are now converted to WPF, the work that we then need to do is minimal. Virtually no provider-side work is required to automate it. All that is required from the provider is that the element you want to automate exists in the layout root (i.e. visible, on your desktop). From there, it's a simple matter of writing a coded UI test, which has two components.

AutomationElement

An AutomationElement is the client side representation of a UI automation provider, which as we've noted can be any WPF control. When you make a call to AutomationElement.FindFirst, it will return the AutomationElement that matches the property you specified. Here, we use the AutomationId property, given they are less volatile than, say, the Name property, which we can change at will in code, or any other property which is either defined by its behavior on the desktop or too generic to be predictive. Once you have the AutomationElement representing the provider you're interested in you can find out basic information about it, such as its position, its control type, its logical parent and children, its enabled state, its name, and more. Additionally, the AutomationElement provides the client with available AutomationPatterns.

AutomationPattern

An AutomationPattern is the control interface that allows you to manipulate and get the state of an AutomationElement, and by extension an automation provider. Let's say you have a TextBox which simply contains the string "Hello World". When you get the AutomationElement representing this TextBox you can know things like where it is in the layout, its name, or any other number of things common to all elements; however, you can't know its text without first getting the corresponding AutomationPattern. In this case that AutomationPattern is a "ValuePattern", is ascertained through AutomationElement.GetCurrentPattern(ValuePattern.Pattern). You can then call Assert.AreEqual("Hello World", Pattern.Current.Value). If you want to change the text of the box, you can call Pattern.SetValue("New Text").

Bringing all of this together, you have a basic coded UI test that looks like this:

//Find the window that the text box lives in
PropertyCondition windowCondition = new PropertyCondition(AutomationElement.AutomationIdProperty, "WindowOnDesktopId");
AutomationElement windowElement = AutomationElement.RootElement.FindFirst(TreeScope.Children, windowCondition);

//find the text box in the window we just found
PropertyCondition textBoxCondition = new PropertyCondition(AutomationElement.AutomationIdProperty, "TextBoxId");
AutomationElement textBoxElement = windowElement.FindFirst(TreeScope.Descendants, textBoxCondition);

//set the value of the text box to "New Text"
ValuePattern textBoxValuePattern = textBoxElement.GetCurrentPattern(ValuePattern.Pattern) as ValuePattern;
textBoxValuePattern.SetValue("New Text");

//Make sure it worked
Assert.AreEqual("New Text", textBoxValuePattern.Current.Value);

Customizing UIAutomationCore

All of this is very well and good, until we take a closer look at the patterns available to us. You'll notice that they are completely static, so any coded UI test must be querying AutomationElements (and therefore providers) that conform to one or more of the known patterns. The problem arises when you start developing custom controls that don't behave like your standard text box, or button, or expander. In most cases, it's relatively simple to override the pattern provided by a custom control. We've already done this; in fact, the master switch in our UI has connection status lights, which is a unique control that has a custom implementation of the ValuePattern. However, consider the UI Spreadsheet grid. From the UI automation library's point of view, it's purely a UserControl, and doesn't implement any useful AutomationPattern. You can't click on it unless you programmatically move the mouse and send a click signal. This isn't useful though, because you have no idea what you're clicking on! It can't be typed into, its data can't be queried, there's no state to report. All we can do is see the data. What we really need is a custom "SpreadsheetPattern" that allows us to interact with the spreadsheet in very specific ways. But how do we inject a new pattern into a library that's been pre-compiled with static properties for each pattern that .NET thinks it knows about? The solution is to register our new pattern with the underlying COM library that UIAutomationCore wraps in .NET, and UIAutomationClient and friends expose to our programs. Thankfully, someone has already done most of this work for us. The basic steps are as follows:

• Disassemble UIAutomationCore

• Replace portions of its linker instructions to accommodate a customized register method and property accessors and method calls.

• Reassemble UIAutomationCore with our SNK

• Wrap the whole thing in a .NET-accessible library (in this case, UIAComWrapper)

• Develop base classes that provide attributes which define pattern properties

If we augment the architecture diagram provided by Microsoft, we get something that looks a little bit more like this:

The implementation looks exactly the same, but the definition and registration changes are the secret ingredients that make this work. From here, we can implement an AutomationPattern that, very basically, looks like this:

public class SpreadsheetPattern : CustomPatternBase<ISpreadsheetProvider, ISpreadsheetPattern>
{
private SpreadsheetPattern()
: base(usedInWpf: true)
{
}

public static void Initialize()
{
if (PatternSchema == null)
{
PatternSchema = new SpreadsheetPattern();
}
}

public static SpreadsheetPattern PatternSchema;
public static AutomationPattern Pattern;
public static AutomationProperty SpreadsheetModeProperty;
}

Behind the scenes, CustomPatternBase will call Register on construction, which then scans for property and method attributes in the ISpreadsheetPattern and ISpreadsheetProvider interfaces and registers them with the UIAutomation COM library. At that point, your provider's AutomationPeer merely needs to implement ISpreadsheetProvider, and your client code needs to call the static Initialize method and cast any returned instance of the pattern as ISpreadsheetPattern to call into it.

Project Avalanche

The UIAutomation framework the piece of the puzzle lets us write repeatable and reliable tests of our UI, but on its own there is a lot of information to understand, and many moving pieces to ultimately make a test happen. Additionally, extending the test suite without any kind of framework would prove incredibly daunting on its own. With no other helpers, it would take the following steps in order to run a test:

• Write the test

• Launch the application being tested

• Run the test

On its own that doesn't sound so bad, but what if you wanted to run hundreds of tests? Would you want to launch the program being tested each time? If not, how do you coordinate the state of the tested application? For running the test, would you do that with NUnit and its built-in runner? If so, do you put every test in one place regardless of the test component? What if a QA wants to run these tests and doesn't have Visual Studio? To solve that, instead of NUnit, do you write a test runner? Does a dev have to write a new test runner for every test they write? Or modify the existing one to accommodate their new tests? All of these potential solutions are a fairly cumbersome barrier, and may lead to a fall-off of motivation to write new UI tests. In light of that, we decided that we would write a custom test discovery and execution tool which would do all of the heavy lifting. We then supplemented that with a framework that would allow developers to write a coded UI test in the same way they would any other unit test. This tool is called Avalanche and is composed of several key parts which take advantage of MEF, Salt, and of course UIAutomationCore.

BT.Infrastructure.AutomatedTesting

This is the project that defines the architecture of our custom coded UI tests. Here, we define the test runner base, the test attributes, and the configurations that will need to be shared by the Avalanche runner and the various test classes. Saber is an inherently configurable application, so it's necessary to provide the linkage behind defining a configuration and using it in a test.

AutomatedTestRunnerBase

The AutomatedTestRunnerBase is akin to a NUnit TestFixture. The abstract base class contains the properties that define the category of tests being run, as well as a RunTests method and associated events. This class provides the application executing the tests with a list of tests, the application required to run said tests, and the TestRail cases affiliated with said tests, should they exist. When RunTests is called, the runner base will scan itself for methods that handle class setup, test setup, test teardown, and class teardown and execute them in the proper sequence, while also calling each test that is defined in the implementation. All of these methods are marked with attributes in the implementation. Additionally and crucially, this base class has an InheritedExport attribute that will expose any implementation to MEF as an instance of the IAutomatedTestRunner interface.

AutomationElementFinder

In addition to defining the elements necessary for defining tests, we've provided a couple wrappers that make finding an AutomationElement much easier. Our UI is a tricky client in that a lot of the things it does happen asynchronously. Therefore, clicking a button might not result in the immediate appearance of an AutomationElement in the layout tree. To accommodate this, you'd have to do a while loop with a timeout each time you wanted to reliably find an AutomationElement. These are many lines of repeated code that a coder might forget to do, so we wrapped those calls up into three easy methods: FindProcess(procName), FindWindowOnDesktop(windowAutomationId, timeout), and FindDecendentElement(elementAutomationId, rootElement, timeout). The combination of these three methods will allow for the discovery of any automation element on the desktop quickly and reliably.

There are of course other pieces that glue all this together and make working with our custom framework simpler.

BT.SomeExistingSolution.Tests.AutomatedTests

When writing a standard unit test, it doesn't matter where the test is defined. All that matters is the test itself contains the proper test attributes that announce it to the process that executes the test. In order to keep this paradigm the same for coded UI tests, we've given each test runner a MEF export tag that allows for injection into the executor. This means that you can define your coded UI test in an arbitrary project, and the test will still be discoverable. We have imposed some minor limitations, though, to enforce logical naming standards and ensure a reasonable size of the resulting object catalog. Primarily, the tests must be defined in an assembly that ends with ".AutomatedTests". When scanning the directory that the executor lives in, it will only register exports that live in those assemblies to ensure we don't bloat the catalog. Additionally, by enforcing that automation tests live in their own projects, existing references won't tempt developers into accessing the underlying code of the application directly. This in turn allows the test collection and the application to be completely separate, despite potentially living in the same solution.

BT.Avalanche

This brings us to Avalanche, the application that wraps all of this together. Avalanche discovers exported UI test files, handles defining of which sets to run, displays run results, manages run configurations, and handles reporting of the results to TestRail, our QA test management platform.

Test Discovery

The very first thing that Avalanche must do is discover which tests it knows about. I mentioned briefly earlier that it does this through MEF and registering assembly catalogs. To be more specific, though, when we run a build all of our custom libraries are dumped into a single folder. Since Avalanche itself is part of this build, we can piggy-back off of the dump location to know where to look for tests. This ensures that a run of Avalanche will be testing the same versions of libraries and applications. Additionally, since its version number is the same as its target application versions, we don't need to do any special referencing to know which version we're testing. Finally, this allows the application to be somewhat portable. You can dump the avalanche .exe into any folder and it will be able to discover our custom UI tests.

Configuration

Since our software is highly configurable to allow our traders to connect with multiple products using multiple technologies, we need to be able to specify which particular configuration we're going to be testing in any given run. Fortunately, a few years ago we wrote the logic necessary to launch our applications by way of a command line. This gives us the ability to pass in our configuration arguments programmatically, rather than needing to automate the log on form for every application. Since Avalanche itself is responsible for launching the application that each test requires and managing the state of that particular application, we can define a configuration within Avalanche to launch the software. These can either be defined manually by the user or pulled from a Salt pillar tied to the user's machine. A user simply selects the configuration required for the test and runs the tests. Avalanche will launch the required applications with that configuration.

Test Execution

Test execution in Avalanche couldn't be simpler. If you're familiar with Visual Studio's Test Explorer, the behavior is very similar. The tests that Avalanche discovered are listed on the left hand column of the program, and their most recent state is indicated with a red x (failed) a green check (passed) or no icon (not run). Clicking on a test will show more detailed information about that test result. In order to run tests, you simply enter your username and password, choose a configuration, and click "Run Tests". To run a subset of tests, select them, and chose right click -> Run Selected Tests. The tests will then run on their own and update with the appropriate result.

Communication with TestRail

After completing a test run, it's clear to the user whether or not it was successful. One of the more important tenants of a successful business is documentation. It's difficult to log and share the result of a test run when the results are only as lasting as the lifecycle of the Avalanche application. In order to make these results visible to the rest of the department and to bring Avalanche in line with the role of a traditional QA, we've built on the TestRail API to be able to submit test results to a defined test plan. Once a user has successfully connected to TestRail, she or he will have the ability to enter a test plan ID into Avalanche and send the results. Each coded UI test has an optional argument for TestRail Case ID. When a test run is submitted, Avalanche will pull the test plan from TestRail, match the coded test's case ID with the ID of a case in the plan, then push the result for that particular TestRail case.

The result is turning this—

Into this.

Each case in TestRail will then show its passed/failed state, and if applicable the reason why it failed. If the test plan in TestRail has a Gerrit ID affiliated with it, that result will then be picked up in Gerrit as part of our QA review. A TestRail plan doesn't need to consist of 100% Avalanche tests for this to work, nor does Avalanche need to report completed tests only if all of them have been run. The whole system is very flexible.

Conclusion

So that's it! At Belvedere we've developed a framework that does the menial tasks for a QA. It runs a predefined set of automated tests quickly, reliably, and repeatedly. From a developer's perspective, writing a UI test is no harder than writing a unit test. From a QAs perspective, executing a regression takes minutes instead of hours. This means that we can cover far more test cases that we ever would have before. All that we need to do now is write UI tests with the same diligence that we write unit tests, and given the improvements we've made in that area in the past few years, I see no reason that we can't get this to the same level of robustness and comprehensiveness for which we're constantly striving.