How do I automate "service-level" testing for a GUI app

Question

I have extensive experience automating GUI testing with tools like TestComplete or Microsoft's CodedUI. However, I am trying to get out of the game of testing at the UI layer and instead would like to get started trying to test at the "service" or "API" or "logical" level as advocated widely throughout the internet. My understanding of this is that I need to somehow access the functions being called underneath the hood when I am clicking buttons or otherwise interacting with the UI. However, I am not really sure how to do go about doing this.

With my GUI automation scripts, basically I just wrote a separate application that was run side by side with the application under test to click buttons and whatnot. Since there isn't really a test API available to touch the service layer externally, how do I go about testing this layer? Do I just go into Eclipse and make some "Unit Tests" to call functions from the application directly? Do I need to bother the developers and ask them to write me a test API so I can access the service layer externally? Or is there a better way to go about doing this?

EDIT: Here's an example of what I wish to accomplish. Let's say I have some application wherein I am supposed to take some sales order with ID SO12345 through some business workflow with three separate screens (Initiate Order, Process Order, Complete Order). In each of these screens, I basically just enter the sales order ID into a text box and click save, which then updates the status of this order in the backend SQL database (insert new sales order record with status "New Order", update status to "Order Processed", update status to "Order Complete").

Basically all I want to do in this example is move a sales order through these three screens and test the database to see if the correct updates have been made along the way. With UI automation, I would make an entirely new application and write a method for each screen and write code in each of those methods to navigate to the appropriate screen, enter the sales order ID into a text box, then click save. Then with these three methods in hand, I could write code calling these three methods and check the database in between each automated step. But now if I want to instead test at the service level and just call the underlying methods underneath each button click, where should I be doing that? Can I make a unit test project and write my automation code in there (even though it's not actually a true unit test) or is it better to get the developers to write me a library that I can reference in a separate app to access these navigation methods directly?

Answer 1

Paul,

A lot depends on what's available in your Application Under Test: often if there is no API available, the functions are only accessible when you have the correct GUI controls open. This is, alas, all too common, particularly with older applications (I spent a lot of time working with an application that still had code from 25 years ago grandfathered in, where a lot of the functional logic happened at the presentation layer. You can imagine the fun that was to test! (The devs hated it too, but between the complexity of it and the company's business decisions there was never time to clean it up properly)).

Where you can access the service layer, it's easy to work with no matter what tool you're dealing with. I've done API-level testing with TestComplete, dynamically building XML files to send to the service and parsing the responses. When you can't, what I've found to be the best option is to include data layer tests in your code, checking your database for correct data storage after you've exercised the GUI for the tests you're performing (which with your experience you probably know already).

Some of the things I've done to make this less painful when the inevitable GUI changes occur include:

• object oriented test code
• heavily data-driven tests (at my previous employer I worked with a relational database of tests and test data that had evolved from the first data driven tests - and was maintained entirely in CSV files. Functional, but not exactly easy for a new automator to follow).
• shared library code to handle navigation within the AUT. Typically in the code I worked with at my last employer, the automation code had a class for the data of each object, and a library to handle the navigation functions involving manipulating that object (I'm starting clean at the current employer, so there's nothing there right now, but this is the model I'm going to use).
• Shared definition of all the AUT objects manipulated. At my last employer this was a shared list of constants defining string constants for the form names and menu choices, so that instead of calling Appname.FormName.Section.Section.Fieldname.SetText we could call CUSTOMER_FORM_CONSTANT.Fieldname.SetText - and in the most commonly used fields, CUSTOMER_FORM_FIELD_NAME_CONST.SetText. That meant if there was a need to change a form field, at best the change could happen in a single line of code. At worst, it might require a search/replace in one unit (Except that the oldest automation code was over ten years old and still in use, and that was mostly record/playback with some parameterization and library use).
• Data validation per test and at the end of each run. This part needs to happen with API testing too, since I'm sure you've seen tests that appear to complete without any problems - and store incorrect data. At my last employer, there was typically key validation of data at the end of each test in the suite, then when the GUI tests completed, every database table the test changed was checked against a set of baseline files.
• You probably already know this one, but it never hurts to be reminded to start from a known data set. The first thing any test involving complex data should do is set the AUT database to a known starting point. Because the applications I work with are so complex it's not feasible to input all the data as part of a test (it would be hours before there was enough data in place to run the test) I'll start by restoring a known database and build tests from that.
• Where you can make function calls directly, do that - if your AUT is compiled with debug info TestComplete can access most of its internal functions and members (with the caveat that Embarcadero introduced a nasty bug into Delphi XE where building a sufficiently large and complex application (the one I was working with had over 4 million lines of code and counting) corrupted debug info. Since I'm no longer with that company, I don't know which version of the Delphi IDE contains the fix for that one - but you can imagine the havoc it caused in a half-million line automation code base utterly dependent on the debug info) and can often call them directly. Developers can use this to expose internal process calls for your automation.

Good luck.

UPDATE: Adding to this to reflect your updated question:

I wouldn't call what you want to do "unit tests" - they're functional or API tests, since none of them exercise only one single unit.

Here's how I would approach this, if no API exists -

• I'd start with a database containing a number of orders in various states of fulfilment. The key thing here is knowing what the orders are and what state they're in.
• I'd write my script code so that it could take an order ID as a parameter, check what status it should be (this could be maintained in a second database or a CSV file, or within the test tool), and run the sequence based on the order ID. I'd loop through them, and have the test code set up to expect an error message if I tried to create a new order for an existing order ID, and to check the database after each step. If you organize your script to treat each order ID as its own object with whatever properties it needs, this becomes a trivial exercise and one which allows you to perform a large number of tests with the same code base.

Without the API access it's slower, but still feasible.

Answer 2

Welcome to SQA, Paul. It is probably a good idea to have some API/service-level tests. The way your question is phrased raises some red flags though.

If you are a tester, you should not write unit tests. I don't mean you aren't capable of writing unit tests, or even that you aren't the most qualified person to write unit tests. If anyone writes unit tests, it should be the developers. There are a number of reasons for this. If you write unit tests for someone else, they need to either explain the intent of their changes (which usually doesn't happen) or you need to guess at them yourself (which carries its own risks). Developers also need to be invested in unit tests. They need to run the appropriate tests before they decide their changes are correct. They also need to fix the tests when they break, e.g. because of API changes. Developers are less likely to behave that way if a tester writes the unit tests.

(Of course, testing is just a means to an end. In some organizations, it may not make sense for a developer to write unit tests at all. This is not to say the developer should not test their code -- of course they should test their code -- but they do not necessarily need to write automated unit tests that perform reliable setup/tear-down, run on a certain test framework, run automatically after every checkin, inject results into a nicely formatted report, and so on.)

Testers should focus on writing integration/system tests. If your developers write unit tests, the right kind of API may already be available. If it isn't, someone will need to write one. Whether you write it yourself or ask the developers to write it for you depends on your skills, your familiarity with the code, the product architecture, your relationship with the developers, and how your developers perceive the value of those tests.

I would start by deciding what kind of integration/system test you want to write. I would start with something low-effort and high-value: low-effort so you're more likely to finish it, and high-value so its value will be apparent to the people you work with. Once you decide on the kind of test, you can determine what kind of API you need, and then you can decide who should write it and when it should be written.

Another advantage of getting the developers involved in the test API is that you lower the risk that the functions in that API will change or go away. If you simply use some functions that look useful, you have no guarantee that a developer won't change or remove that code next week.

Answer 3

Talk to development!

An understanding of your architecture is needed properly answer the question. How does your application communicate with the databases? If you have a service orientated architecture odds are you can test there.

The most common service testing tool out there is soapUI, which generally deals with soap or rest. I use it extensively and it is awesome, but a key to the awesomeness is clear open communication and team work with development, especially if you don't have documented API.