16

One test automator quit his job a few days ago.

We noticed today all most all of his test programs produced fake results. What he did was either producing random test results or assert true == true.

This sound very stupid, but how do we prevent test automator from creating fake results?

Specifically: what are the industry-recommended practices to audit test automation code and prevent a test automator from generating code that doesn't test what it is supposed to test?

  • This is a valuable question: I have edited to remove the opinion-based aspect. – Kate Paulk Mar 7 '17 at 12:54
  • 1
    @KatePaulk, thank you. Current title is much clear. – oscar Mar 7 '17 at 14:18
  • 2
    Code reviews are the obvious answer here. Also... relevant xkcd:xkcd.com/221 – Reinstate Monica Mar 7 '17 at 15:36
4

Code review will do it.

  1. Management, senior managers need to be aware of the value of having a review. They have to actively allow effort and time for reviews. This is normally the hardest part. Some senior managers consider reviews to be waste of time.
  2. Team leader, they need to supervise reviews to be carried out.
  3. Team members need to perform technical reviews, peer reviews, walkthrough and inspection.
  • yeah, senior management want short term results at once. It is really a pain. – oscar Mar 7 '17 at 14:20
17

As others have said: code review.

It is not uncommon for code like assert true == true to be used as a placeholder during test automation development (I personally would use assert true == false or assert fail or similar as my placeholder so my code could not pass until I was ready to write the correct assert, and so there was no chance I would forget the placeholder). Code reviews will catch placeholder code as well as outright bad code.

The critical thing that you need to do is to treat automation code as production code. That includes coding standards, code reviews, and every other code quality standard in your organization.

  • 1
    I agree with "code review". I disagree with "treat automation code a production code". Test automation is a tool, it's not production. – Joe Strazzere Mar 7 '17 at 14:02
  • 7
    I 100% agree with "treat automations as production code". For example when using continuous deployment your test code is as much part of production as the other code. It should be the same quality level. Also you need to refactor it regularly and keep it high quality. Test code is code and it needs to be maintained and extended just like any other code. Welcome to the next century. :) – Niels van Reijmersdal Mar 7 '17 at 14:42
  • @JoeStrazzere - fair enough. I've found if test automation code isn't held to the same standards as application code, it quickly becomes the poor cousin and has quality issues as a result - perhaps there's a better way to phrase things? – Kate Paulk Mar 7 '17 at 15:32
  • 1
    @KatePaulk - I understand what you mean, but realistically test automation code never goes through the same stringent process as production code. Otherwise you'd have a QA team testing the test automation code, using test automation... in an endless loop. In my experience, test automation needs to be a far lighter process in order to be efficient and cost effective. – Joe Strazzere Mar 7 '17 at 15:38
  • 1
    In addition to code review, one could invest in Mutation Testing. With MT, a machine would change the SUT code in ways to make a "correct" test failed. If you have a test that does not test a feature, the tests would continue to pass; therefore, spotting an error in your test code. MT is still limited, but can be a powerful tool to complete code review - specially when the automation code gets more abstractions. – João Farias Mar 9 '17 at 14:16
6
  1. Review test automation code.
  2. Run each automated test at least once before you write the code it will test. (Of course, this means you'll have to write the test code before you write the code it will test.)
  • yeah, this will effectively prevent false automated tests. – oscar Mar 7 '17 at 14:19
4

I didn't see this one, so I wanted to add it "Management Issue".

Having someone who willingly left without ever doing real work is a management issue. They should have been fired by management for not doing their job. Management should eliminate silos and enforce bad practices as well while creating an environment for success which would prevent this situation in the first place. As almost every answer mentioned code reviews and accountability should be a natural part of any team (and employment), but management and team leads should set that up.

In this case I'd say the manager needs to be hstrong texteld responsible for wasting budget and not automating successfully. My guess is that a manager in this position blames the automation team for bad work instead of taking responsibility for their own job and role in the process. They likely don't understand what it takes to actually do it either. I would seriously question the competency of a manager who keeps an employee who does nothing and doesn't even know about it. I'd also wonder about a manager who hires someone who clearly isn't honest and doesn't help the team.

If you want a technical answer go with the Code Review up top by Kate Paulk, but the underlying issue here is incompetent management/leadership.

2

Although code-reviews seems the most obvious one. I would like to vote for test-first and pair-testing.

Test-first:

Making sure you write a failing test first will go a long way preventing irrelevant tests.

One way XP’ers would keep themselves honest is to insist they write a failing unit test (demonstrating the need for complexity) BEFORE adding the extra complexity to the system.

The same goes for adding unneeded tests. Demonstrate the need for writing an extra test. I also tend to write one or two end-2-end and or integration tests before starting on new code.

Pair-testing:

You could always practise pair-testing. Let a developer and a tester gang-up and test some! :)

Other reads:

2

In addition to on-going code review I would consider:

Existing code audit(code review)

Code review is normally done for new and changed features. Based on what you have already found I would take a step back and review all the existing test code as if it is new. You'll often need to develop a strategy to review them in chunks alongside existing feature development.

Champion good practices

If you document, promote and encourage good practices they are more likely to be followed.

Red, Green, Refactor

Learn about and practice this technique.

Education

From lunch and learns to half-day seminars provide plenty of learning material on doing the right thing.

Pay for performance

Include code quality items in personal performance reviews. Reward the bottom line ($) for doing the right thing.

Tech Debt weeks

Much as I try to do the right thing now I also balance this with the business need to deliver features. My outlet is that I sometimes create debt tech tickets. We do 1 or 2 weeks of tech debt every 6 weeks so these are not 'wishful' tickets that aren't ever done but actually queuing up of real work to be done soon.

Pairing

From dev-dev to dev-qa to qa-qa it helps to keep folks more honest, open and questioning assumptions. Group-think can be an issue but generally I find the benefit to be positive.

Code Quality Tools

From code climate to codecov there are a lot of code analysis tools out there. They help you manage the process of having quality code by analyzing your code base and test coverage. They can show you improvements over time and can let you set and manage levels of code quality.

Test your tests

Consider adding code that looks at the structure of your tests themselves and looks for things like: duplicate page objects, unused page object, not using page objects, use of true == true statements, etc. Make these into actual tests so that you can let the tests help enforce the standards, not you through constant persuasion arguments in code review.

btw 'industry standard' doesn't work for me here. The software quality industry is still too varied and undefined.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.