How to push people to use automatic tests?

Question

I am a test automation engineer.

My main work is to automate UI tests for projects. My tests are suppose to be launch after every release in "qualification", "pre production" and "production". Then, the person in charge (usually the "Product Owner"), takes a look at the tests results and acts accordingly to said result.

On the paper, this process works fine. It also works fine with some projects depending on who is in charge of said project.

Sometimes, the person in charge of a project will simply not use my tests. Or will launch them but not look at the test results (what's the point of launching them, then?). Or will launch them, look at the results but not tell me if the tests are broken because of a new feature (so basically the test needs fixing but they don't tell me so I don't fix them and then they become useless).

I do realise that I have a major communication issue here (and it's probably made worst by my remote working), but I don't know how to fix it.

How do I push people to use my tests?

I make sure to present my tests to every person who could use them. I also tell them to contact me anytime they need my help or if they have any questions. But it doesn't seem to be enough. I have written several documentation on how to launch the tests and then look at the tests results. I don't know what else to do.

How can I reduce the probability that my tests won't be used? Especially when my tests are here to make their life easier (so they don't need to test so much by hand anymore).

Answer 1

You make your tests part of the build pipeline, and have them email you and everyone else who matters the results. That way, you're not relying on anyone else to run them or check the results.

As an automation engineer, writing the tests is only part of your job. You are also responsible for your part of the CI/CD pipeline, up to and including configuring the automated tests to run after each build with the appropriate label and checking the results afterwards.

Taking responsibility for ensuring the tests run and checking the results also means that you will know, quickly, whether changes have broken your automation and be able to make updates in a more timely way. You will also be able to direct information about failures to the appropriate project manager and raise bugs regarding failures.

If this is too much to add to your other duties, it's time to find a willing assistant in your fellow testers in the organization. It's very likely that there'll be one or more who are interested in expanding their skill sets.

Answer 2

Let's consider a couple of realities:

1. Developers will not run tests that they believe are rubbish.

In all the years I've been building and testing applications, I have never seen a third party write tests that were any good. The most common flaw I've seen is that the third party uses bad techniques to identify the UI elements that must be interacted with or observed. Concretely, this means looking at HTML IDs, and CSS class names that are presentational rather than semantic. Very frequently, the third party will even demand that the UI developer add IDs (or test-ids) to the HTML to facilitate the testing.

All of that is bad, and it inevitably leads to test that are fragile (i.e. start failing when the application changes in irrelevant ways), and to a UI codebase that doesn't make sense to future developers because it is littered with bizarre HTML attributes that misstate their own significance.

The fact is that QA engineers have no place deciding how to build UI, and every developer grasps that intuitively. Fighting it will not only fail, but make enemies. But this does not mean that QA engineers have no power.

2. Having quality tests is the only way for developers to have confidence in their own releases.

In all but the smallest applications, it's hard to make changes without breaking something else. Discovering bugs through manual regression testing can be hard and is always time-consuming and error-prone. But if a regression does get released, users will inevitably find it (and usually pretty fast), and that will create a "production fire": a high-visibility emergency that requires developers to immediately scramble to find and fix the bug, and then release it. This is always a stressful situation, as any developer who has lived through one will admit. These things have a way of happening during evenings and weekends -- which means devs get called at odd hours and forced to tackle these problems. (If that is not happening at your org, start doing it.)

But simply having a bunch of tests, or even having pretty high code-coverage, is no guarantee against regressions. The tests themselves must be high-quality.

There is only one way to create such a test suite: the UI developers themselves must write it. It is 100% the job of UI developers to write the tests of their own software. Why?

• Only UI developers will know which parts of the DOM are stable with respect to the elements that must be interacted with or observed.

  They have that information because they are the ones who made it so! It's a big part of the planning phase. No other person in the organization will even be remotely equipped to figure this stuff out just by looking at the page at runtime. But, with power comes responsibility.

• Only UI developers can spot breakage by running the tests before their breaking patch hits the CI pipeline.

  Changes to the codebase happen on developer workstations. Ideally, broken code should never make it off the dev's computer. The only way for that to happen is for the developers to be able to run the tests locally (and in the same way as the pipeline, to guarantee identical results). This obviously means that devs must be capable of running the tests, and they should do so before they push their code (or, before they merge their feature branch such that it will be deployed).

• The automated test setup needs to fit into developers' workflow.

  Devs must be running the tests all the time, locally. That's not going to happen if running the tests is hard or impossible to do on a typical developer workstation. But developers are (rightly) responsible for their own workflow. Again: with power comes responsibility. Devs get to decide what their workflow is, but that also means they must bridge the gap between their preferred workflow and the running of tests.

The only reliable way to make all of this stuff happen right is for the devs to write their own tests. They should choose the tech and tools, but it has to be done.

And by the way, being good at writing tests makes it a lot easier to design and build good app code in the first place. I'm no TDD fanatic (although I know a few), but there are still some cases where even I think it works best to write the tests first and then write app code that passes those tests. Clearly there's no room for a third-party in that flow!

So, where do you fit in?

3. Writing tests is hard, just like writing app code is hard.

The test environment is importantly different from the regular browser environment, and there's only one way to handle that difference: the dev has to learn the testing environment just like they learned the browser environment when they first started doing UI work.

That means every dev will have to suffer through that initial, painful learning phase, where they write a test but it doesn't do what they want and they don't know why. Where they have to ask for help with seemingly simple stuff, e.g. "how do I write a test for something that uses timers?"

One thing a QA engineer has that most devs don't is that the QA engineer has line-of-sight on tests from many different teams and projects. That means the QA engineer is ideally positioned to help teams notice shared problems, and to circulate good practices.

This is more of a leadership/mentoring role than a coding role, but no less important.

Try to get yourself added to every code review that includes tests. Even if you don't spot problems or have questions, it'll help you keep abreast of what devs are doing in their tests. (It'll also help you notice when devs start failing to write tests.)

4. Make sure the pipeline runs the tests and fails the build if anything fails.

   Using email to notify people is all well and good, but nothing will get results like putting a brick wall in front of every failed test. Devs probably won't do this. They need someone to do this for their own good.

---

One thing you may want to consider: make it your org's policy that all new devs must begin their tenure with a significant stint only writing tests. This is something I had never encountered before my current role, but it absolutely improved my testing abilities by an order of magnitude.

In my case, I spent the first 2 months on this job only fixing old tests and writing new tests. And I'm no rookie: I've been a professional dev for over 20 years, and I've shipped software for half a dozen platforms, for clients and of my own design. That first 2 months still made a huge difference in my testing skill and comfort level. And my employer got something out of it, too: upgrades to their old testing codebase, plus a chance to learn whether I was any good without having to take the risk of letting me break their user-facing software.

---

I don't think you or your org will have much success if the goal is "make the devs use your tests." I think the best sustainable path is for the devs to own their own tests. And it will undoubtedly make them better as devs.

Answer 3

People who are supposed to write tests, i.e., programmers who are not dedicated test automation programmers but are supposed to be working in a test-driven manner (in the widest sense of the word) can be pushed to use (write) automatic tests with these instruments:

• Teach them about the tangible benefits they reap from automatic tests.
  • Safety: when changing or refactoring code, or even just implementing new code, they are safe(r) from inadvertent changes to the existing behaviour.
  • Ease: a well-done test-driven development framework makes it easier, not harder, for a developer to write their code. For example in Ruby on Rails, which due to its architecture needs a lot of different little parts for a new feature to be created (i.e., controller, view, models, configuration, helper functions/libraries etc.) it is easy for a not so experienced programmer to get the effect where you are overwhelmed and don't exactly know where to start, or are just confused by all the little areas to touch. Starting with a test first (i.e., by "visiting" the route/URL you are supposed to implement in a test), the error messages will tell the programmer exactly each step that is missing.
  • Inbuilt documentation: especially when working on code from other devs, having a well-defined suite of automated tests makes it easy(er) for a developer to get into the grove, and find out, for example, which actions an actual users would perform to reach some goal in the application.
  • Having your test suite "green" is a great psychological boon - you know that you did not break the system, and there is little risk that you will pull the anger of your colleagues when you are found the one breaking an important part of it and pushing...
  • Actually teach them how to write tests - e.g. have regular brownbag sessions or something like that where you or a colleague highlights good techniques, or practice using the test suite together. Make it a team effort.
• Enforce it by requiring successful tests (and test coverage) in the CI/CD pipeline, and making said pipeline the only technical possibility to deploy onto the servers. This is decidedly a secondary way to push the devs - if you cannot get their buy-in through any other reasonable method, this is kind of a last-ditch effort, and may easily lead to attrition.

For people who are tasked with running tests and checking the results:

• The positive motivation is of course to be sure that a deploy does not break the target system. Frankly, if a person responsible for this task does not care about that aspect, as in your case, I don't really see what kind of "benevolent" motivation you could employ. Presuming the tests actually work, and actually do find errors, and that actual bugs popping up on the production deployment are actually "red" in the test suite, there is just no arguing about this. Did you actually ask them why they are doing it? Are they forgetting it? Is it taking too long? Are the results inconclusive?
• A neutral measure is to make the process easy. If the suite is running for a long time, either there must be enough time between the end of the development cycle / sprint and the deployment deadline; or the run-time must be reduced, i.e. by investing in more technical resources.
• An addendum to the previous is to make sure that the tests are run in CI/CD whenever a dev pushes something, to get early warning when tests are red. Require a green test result before merging pull requests, etc.
• Finally, again, you can make running the tests, and the tests being green, a required step in the CI/CD pipeline; and make it impossible to deploy from outside of the CI/CD pipeline. This might be more appropriate than for the previous group of developers. The deployment process is the last barrier before damaging the production servers, so running the tests should be the default behaviour, not something optional.

Answer 4

Successful UI automation tests require :

Buy in by management to spend time/resources
Solid and robust infrastructure Good test framework,
Quality tests design,
Easier to understand test reports
Discipline in the development process

There are many considerations. It all depends on how complex the product is, how much resources your organization have. Test automation effort could be larger than product implementation.

WHO should write tests

I disagree that only UI developers should write their tests.
UI developers should write their unit, functional, component tests as part of development requirement.
There should be some amount of user/scenario driven tests written from the perspectives of users. Top down, bottom up, meet in the middle in all sorts of directions. Goal of different types of tests serves different purposes and all are crucial to a successful product.

What Tests and when to run

What type of tests ? unit, functional, component, integration, e2e, stress and performance user driven are written at which level, by whom and how frequent they should be run, per PR, per build, nightly, weekly etc.

Unit and Sanity tests in the pipeline is to catch regressions, but it does not mean you have a good usable product.

Long running tests, performance tests may consume a lot of your resources.

When stress tests are run, will it affect other parts of the system?

Other UI automation issue

Need constant monitoring and analysis Tests flakiness

Reporting - Even though report say PASS they may be false positives Even though report say DO NOT PASS they may be false negatives Even if you can auto-generate issues and all sort of reports from test failures, if no-one is monitoring, analyzing debugging and resolve the problems, it just defeats the purpose. Very quickly, the pipeline will be abandoned.

More considerations

reporting retention period logging cleanup strategies enhance/add/remove/streamline

As your product develop more features and enhancements, your tests will keep increasing. The time to run the tests will increase too. Need to constantly to revisit existing sanity tests strategy in the pipeline to make it more efficient.

Answer 5

I think a lot of the answers are about developers and tests, but your question is about Product Owners and people-in-charge.

Perhaps these people just don't have the enthusiasm for reviewing detailed test results?

So, I'll ask you, is it possible for your test result presentation/report/view to indicate if a test result is different to the previous run of the test in an unexpected manner?

If you want people to contact you, maybe have a link in the report they can click on to create an email/slack/chat message.

Then there people-in-charge would only need to look at the exceptions. So they can concentrate on the signal in your report. rather than what I'm guessing might be a larger more comprehensive report - that they might consider noisy.