Automated testing: focusing on testing controls vs. workflow

Question

I'm in charge of regression automation on my team, and I need to add some new tests into our regression. Before I get started, though, there's an issue I'm struggling with: where is the line between focusing on automating user workflow through a system, and focusing on only ensuring that the controls are working as they should be?

For example: imagine a system can set up new users, and later edit their information. On the screen that accesses users, there's a list of recently added users at the top, and there's a search function at the bottom. The workflow of a client using our system would be to create a user, then access this screen to edit the user's information immediately afterward. I could write my automation that way, but I would like to write short tests that only test one thing each. I'm trying to stay away from behemoth tests that take a long time to run and test many things, and when one fails you need to wade through many many checkpoints to see exactly which one failed - this is inefficient. A test that follows this workflow would have to either test two things (creating the user, then editing the user's information), which would make it one of those behemoth tests I'm trying to stay away from, or I would have to write two tests that have a dependency between them (Test1 creates the user, Test2 depends on the user being on that screen to click).

It seems to me the better solution is to write a test that focuses on creating a user and write a separate test that uses the search function I mentioned to look for an existing user (we always start our tests with a pre-set database of users) to edit their information. The problem is that this tests purely the controls, not the workflow. I could always throw in a third test that tests the functionality of the recent users screen, so that's covered by the automation too. It seems to me that manual testers should focus more on workflow, whereas my automation should focus more on purely how the controls work. Is my reasoning correct?

Answer 1

Welcome to SQA, Ana. I see at least two questions:

• Does it make sense to distinguish between focused tests and workflow tests? Yes, it does. A focused test concentrates on a particular feature, including things like edge conditions. It attempts to answer the question, "Does this feature work in all circumstances?" A workflow test verifies that the system as a whole is usable for its intended purpose. Both kinds of tests are necessary, but it is a better use of time to execute them separately.
• Should focused tests tend to be automated, and workflow tests executed manually? That depends on your circumstances. Here are some question that might help you decide what to automate. Which tests tend to be error-prone? Which tests do you want to run after every build? Which tests would developers run before every check-in if they were fast enough? Which tests are easy to automate and maintain? (The maintenance part is admittedly hard to predict.) Which tests would you use for smoke-testing?

Answer 2

Essentially, this is a common dilemma in designing an automated test. IMHO, the design of a test is heavily dependent on it's intended purpose in the development lifecycle.

You stated that these are regression tests...implying that the current functionality is working correctly and the primary purpose of the test is to expose bugs caused by changes in the product code. Essentially, IMHO regression tests are a superset of unit tests, but are broader in scope.

In my team we design our "regression tests" with a broad scope to cover as much of an end-to-end workflow as possible. We run regression tests on every new build. This means the tests are more complicated and there could be several points of failure which is why ensuring the machine is in the appropriate state during test execution and verbose logging is critical. Also, time is a factor, but our none of our tests take more than 1 minute to complete, and most e-2-e scenario tests execute in a few seconds. (To be transparent...my team tests a set of APIs that transports and parses data between different clients and different services so our end-to-end workflows are essentially without a UI and mocking (or sometimes using real) web services.)

We also design a separate set of functional tests (not regression tests) that focus on the functionality/capabilities of individual APIs or components. These tests are intended to hit edge cases, error conditions, use malformed data, use variable data, or inject systemic anomalies to expose weaknesses in the system. Our functional tests tend to be more discrete or smaller in scope because we are tying to find previously undiscovered bugs, and test the robustness of that particular component is a variety of conditions.

Answer 3

It's a similar dilemma which I am currently going through.

I have decided to predominantly create my test cases following a similar flow to the user story. This means they're several dependencies that have to be assumed before I get to the core part of the test. I try and reduce these where possible but I accept that there will be repetition. I guess in an ideal situation I could remove that repetition by separating out some steps into their own tests and just referencing each step.

I don't see creating these system tests, like they are integration tests as a worthwhile use of time. I also don't think it is useful to create big tests that could fail at multiple steps before you get to the thing you are actually trying to test as being useful either.

It's a balancing act and I don't believe i've found the right answer yet.