Elements of GUI Testing

Applying what I learned with JUnit to writing GUI tests, and building a better 'bot.

Contents

The Pain (Problems and Goals)

On the subject of why testing is so important, I won't repeat here what has been aptly described in Test Infected. If you haven't read that document, please do so before proceeding here.

Often a developer will want to make some optimizations to a piece ofcode on which a lot of user-interface behavior depends. Sure, the newtable sorter will be ten times faster and more memory efficient, butare you sure the changes won't affect the report generator? Run thetests and find out. A developer is much more likely to run a test thathas encapsulated how to set up and test a report than he is to launchthe application by hand and try to track down who knows how to do thatparticular test.

To test a GUI, you need a reliable method of finding components ofinterest, clicking buttons, selecting cells in a table, dragging thingsabout. If you've ever used java.awt.Robot, you know that you need ABetter 'Bot in order to perform any user-level actions. The events theRobot provides are like assembly language for GUI testing. Tofacilitate and encourage testing, you need a higher-levelrepresentation.

Describing expected behavior before writing the code canclarify the developer's goals and avoid overbuilding useless featuresets. This principle applies to GUI component development as well ascomposite GUI development, since the process of writing tests againstthe API can elucidate and clarify what is required from the client/userpoint of view.

GUI Design

First of all, any GUI component should provide a public API which canbe invoked in the same manner via a system user event orprogrammatically. Keep this in mind when writing new components. In thecase of Java's Swing components, the event handling is mixed up withcomplex component behavior triggers in the Look and Feel code. It's notpossible to execute the same code paths without triggering theoriginating events. A better design would be to have the Look and Feelcode simply translate arbitrary, platform-specific event sequences intopublic API calls provided by the underlying component. Such a designenables the component to be operated equally as well by code, whetherfor accessibility or testing purposes.

GUI Testing

GUIs need testing. Contrary to some opinion, the problem is not always(or even commonly) solvable by making the GUI as stupid as possible.GUIs that are sufficiently simple to not require testing are alsouninteresting, so they do not play into this discussion. Any GUI ofsufficient utility will have some level of complexity, and even if thatcomplexity is limited to listeners (code responding to GUI changes)and updates (GUI listening to code state changes), those hookups needto be verified.

Getting developers to test is not easy, especially if the testingitself requires additional learning. Developers will not want to learnthe details of specific application procedures when it has no bearingon their immediate work, nor should they have to. If I'm working on apie chart graph, I don't really want to know the details of connectingto the database and making queries simply to get an environment set upfor testing. So the framework for testing GUIs should require no morespecial knowledge than you might need to use the GUI manually. Thatmeans

• Look up a component, usually by some obvious attribute like its label.
• Perform some user action on it, e.g. "click" or "select row".

Scripts vs compiled code

How can I test a GUI prior to writing code? One alternative (and usefulin certain cases) is developing a mockup in a RAD tool. Unfortunately,the usefulness is relatively short-lived; it's not really possible (atthis point in time) to automatically generate a very interestinginterface. If your entire interface consists of buttons and forms, youmay not really need a gui tester anyway. Mockups don't convert well totests for the actual developed code, and RAD tool output usuallyrequires some hand modification afterwards.

What a test script could plainly describe the GUI components ofinterest, and simply describe the actions to take on those components?Providing you know the basic building blocks, you can edit the scriptsby hand or if you don't know the building blocks, in a script editor.No compilation necessary, which speeds development and maintenance oftests.

I wanted scripts to be hand-editable, with no separate compilationstep. I wanted to be able to drop scripts into directory hierarchies asneeded, and have the test framework pick them up automatically,similar to how JUnit auto-generates test cases.

How to Test

One issue in defining a GUI test is that you need to map from asemantic action (select the second item in the records table) onto aprogrammatic action (myTable.setSelectedIndex(1)). Thiscomprises two separate problems. First, the target of the action,"records table", must be somehow translated into an actual code object.Second, the semantic event "select the second item" must be translatedinto a programmatic action on that code object.

Tracking components

There are many methods of identifying a component in a GUI hierarchy.We want the tests to be flexible enough to not break simply becauseanother button or panel was added to the GUI layout.

• Component Names Java provides for naming components, but since no one ever names them, this method of identification is mostly useless. Worse, some auto-generated components (frames for otherwise frameless windows, windows for popup menus, and most dialog instantiations) have auto-generated names, which aren't particularly helpful, or even downright misleading if components get created in a different order than expected.
• Position in hierarchy This method guarantess a unique match, but also guarantees your script will break when that hierarchy changes, even for otherwise trivial modifications (like inserting a scrollpane). Each component would need to store its parent reference and index within that parent. Note that this implies each parent reference might need to store the same information for itself.
• Parent window title This is useful as a first-pass discriminator in multi-window applications.
• Component class This helps discriminate from the available components list, but is not likely sufficient on its own to identify a component.
• Custom tags Here's where the real component resolution is done. Many component classes will have some aspect or property that can be used to uniquely identify them, typically a label (the text of a button or menu, a labelFor component, a window's title), but potentially any identifying element may be used. Abbot uses custom component testers to get the unique tag for any given component. These testers are dynamically loaded, so we don't have to know a priori how to get a tag for a given component class. Combined with all the previous attributes, Abbot does a pretty good job of tracking components.

Recording events

Ideally, there would be a programmatic function on every component that performed a given user action, similar to the doClick method on AbstractButton.Given that this is not the case, we want to provide for bothimplementing such functions and loading them dynamically, andconstructing semantic events from low-level events when correspondingfunctions are not available. Having dedicated semantic functions makeswriting scripts easier, but supporting the low-level events means theyare not absolutely required.

The most basic events to support are those that correspond to low-level OS events for user input, namely

• Pointer motion
• Button press/release
• Key press/release

In addition, the most common semantic events that affect user input arewindows opening and closing, so we throw in support for those(Alternatively, the time delta between events could be preserved forreplay, but doing so has not yet proved useful in very many cases)

• Wait for window open/closed

The easiest way to write a script is to record a series of actions forlater replay, adding checkpoints or assertions along the way to verifycorrectness. Eventually, you might want to add heuristics that stripout meaningless events, but keeping track of a few basic events issufficient for good functionality.

Over and above event stream support, it's useful to have custom,class-based component functions that provide higher-level semanticevents. For example, a custom table might export a select(int row, int col) action to select a cell within the table, or sortByColumn(int col)to invoke a click on the column header which would cause a sort in thetable. The custom actions use existing building blocks (low-levelevents or existing semantic actions) to construct the higher-levelsemantic event.