2

After working with UI automation for some time and having to maintain many system tests (huge end-to-end tests that start all the way from software installation to testing via UI), I finally realized that it is a bad investment to have many UI tests. I saw some diagrams in the past showing how many tests should exist on Unit/Integration/System level (around 70/25/5).

Question: if you were to find correct and strong arguments for management to persuade them not to invest a lot into UI automation, but more into Unit/Integration tests, what arguments would you use (maybe some links to case studies, test books, etc.)?

  • Other than cost? Honestly though why do you feel you need less automation? Is the UI not stable and not ready? In which case I wouldn't be automating at all at this point. Pushing away from automation these days seems to be the opposite track many of us are on as Testers/Checkers/QA, why are you going the other way? You say its a bad investment, but not why. – MichaelF Apr 22 '14 at 12:26
  • What tool do you use to automate UI? – Jose Pita Apr 22 '14 at 13:02
  • I made changes to the question to clarify that I meant to reduce UI automation by investing into Unit/Integration tests – oldbam Apr 22 '14 at 15:35
  • 1
    I want to add another distinction: You're struggling not with automating the UI per se, but with testing end to end through the UI. (In contrast, you could test the UI itself without the back end, with fewer problems and less cost.) – Dale Emery Apr 22 '14 at 16:22
1

I'm assuming you mean to push back in terms of reducing the quantity of full-system automated tests and increasing the quantity of lower level tests. I've been pushing the same thing at my organization for a little while, so I've got some anecdotal advice.

I think the test pyramid is the best place to start. You sound familiar with it, so I'd continue talking about that. If you need more background on it, Martin Fowler wrote a good article about it http://martinfowler.com/bliki/TestPyramid.html

Recently I've been trying to ask people what they think the purpose of different types of tests are. If you have a reasonably decent test pyramid, you essentially know that your application is functioning correctly. So then, what are the automated UI tests actually testing? In my experience they're mostly testing that the application was deployed/installed correctly and is actually running.

Another concept I really like that goes along with this is Depth of Test. If you're doing all or most of your testing through the UI it's not possible to have any useful focus to your tests. This reduces the usefulness of your tests because when one fails it's not immediately obvious what caused the failure.

Along with this I like to say that the usefulness of a test decreases exponentially with the time it takes to run. This is more related to how often a test runs and less about the execution time of the actual test although they're usually related. Unit tests are very useful because they run quickly and developers are used to running them. Integration tests should run fairly quickly such that the barrier to a developer running them pre-checkin shouldn't be very painful. Maybe they don't want to run them every time they compile. Automated UI tests, especially a lot of them, take on the order of 10s of minutes, so you don't want to run them that often, if at all. This makes the test less useful because the feedback it provides is significantly disconnected from the actual change it's supposed to be testing.

3

This is often a difficult argument, not least because it frequently shifts "test" responsibility to developers, who may not wish to deal with writing unit or integration tests and see no value to either.

Some points I've found helpful here are:

  • It's much cheaper in terms of licenses needed and time required to keep business logic tests at the unit test or integration test level where they can run in seconds every time a build runs. If you have a large number of UI level tests that are covering business logic/calculations, this one is (almost) a gimme. (As an example, at a prior workplace, there were thousands of UI-based tests that took a grand total of nearly 12 hours to run. They could - and should - have been unit or integration tests of tax calculations with different input parameters. There was NO unit testing around that code, and worse, it was so spaghettified it was untestable without major refactoring.)
  • the more unit testing and integration testing that's built into the codebase by developers, the more stable the application will be and the easier it will be to modify without breaking existing functionality - and if you get a lot of problems with regression, this is a big deal (in the company with no unit testing, emergency patch releases consumed more than half the test team's time - which meant that new development testing and regression testing got short-changed, leading to more regression bugs leading to... you get the picture)
  • even the most well-designed UI automation is fragile and high-maintenance. Even if the issues caused by record/playback have been dealt with, something as simple as making a field non-editable can cause hours of automation refactoring
  • UI automation tends to run into one of two problems: either there are a lot of repeated actions with corresponding problems with repeated code, or the test run is highly dependent on prior tests having completed correctly. In the first case, you can have a small change to the UI causing massive refactoring issues, and in the second a failure early in the run can cause cascading failures further along the run. I have yet to encounter a clean way to maximize DRY automation code while minimizing dependency.
  • The closer a test is to the thing it's testing, the easier (and faster - and therefore cheaper) it is to identify the problem. If a unit test fails, the problem is in that unit or the unit test. If an integration test fails, the problem could be in any of the units involved, the communication interfaces between the units, or the test. If a UI test fails, you can add in the user interface, anything else the computer happened to be doing at the time (a scheduled virus scan can wreak havoc on a UI automation run), and communication between the UI and the back end. Even with the best logging and test design in the world, this is a challenge for UI test automation.

The short version is that if you're running a lot of UI tests to validate business logic, those tests would be better handled as unit or integration tests for time, cost, and effectiveness.

0

The main thing can be the initial cost. But why one should ignore automation, if there's the technology available for the same and can ease the work. I support automation, though we can't solely depend on any one tool, because that might not give accuracy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.