One test automator quit his job a few days ago.
We noticed today all most all of his test programs produced fake results. What he did was either producing random test results or assert true == true.
This sound very stupid, but how do we prevent test automator from creating fake results?
Specifically: what are the industry-recommended practices to audit test automation code and prevent a test automator from generating code that doesn't test what it is supposed to test?
Code review will do it.
As others have said: code review.
It is not uncommon for code like assert true == true to be used as a placeholder during test automation development (I personally would use assert true == false or assert fail or similar as my placeholder so my code could not pass until I was ready to write the correct assert, and so there was no chance I would forget the placeholder). Code reviews will catch placeholder code as well as outright bad code.
The critical thing that you need to do is to treat automation code as production code. That includes coding standards, code reviews, and every other code quality standard in your organization.
I didn't see this one, so I wanted to add it "Management Issue".
Having someone who willingly left without ever doing real work is a management issue. They should have been fired by management for not doing their job. Management should eliminate silos and enforce bad practices as well while creating an environment for success which would prevent this situation in the first place. As almost every answer mentioned code reviews and accountability should be a natural part of any team (and employment), but management and team leads should set that up.
In this case I'd say the manager needs to be hstrong texteld responsible for wasting budget and not automating successfully. My guess is that a manager in this position blames the automation team for bad work instead of taking responsibility for their own job and role in the process. They likely don't understand what it takes to actually do it either. I would seriously question the competency of a manager who keeps an employee who does nothing and doesn't even know about it. I'd also wonder about a manager who hires someone who clearly isn't honest and doesn't help the team.
If you want a technical answer go with the Code Review up top by Kate Paulk, but the underlying issue here is incompetent management/leadership.
Although code-reviews seems the most obvious one. I would like to vote for test-first and pair-testing.
Test-first:
Making sure you write a failing test first will go a long way preventing irrelevant tests.
One way XP’ers would keep themselves honest is to insist they write a failing unit test (demonstrating the need for complexity) BEFORE adding the extra complexity to the system.
The same goes for adding unneeded tests. Demonstrate the need for writing an extra test. I also tend to write one or two end-2-end and or integration tests before starting on new code.
Pair-testing:
You could always practise pair-testing. Let a developer and a tester gang-up and test some! :)
Other reads:
In addition to on-going code review I would consider:
Code review is normally done for new and changed features. Based on what you have already found I would take a step back and review all the existing test code as if it is new. You'll often need to develop a strategy to review them in chunks alongside existing feature development.
If you document, promote and encourage good practices they are more likely to be followed.
Learn about and practice this technique.
From lunch and learns to half-day seminars provide plenty of learning material on doing the right thing.
Include code quality items in personal performance reviews. Reward the bottom line ($) for doing the right thing.
Much as I try to do the right thing now I also balance this with the business need to deliver features. My outlet is that I sometimes create debt tech tickets. We do 1 or 2 weeks of tech debt every 6 weeks so these are not 'wishful' tickets that aren't ever done but actually queuing up of real work to be done soon.
From dev-dev to dev-qa to qa-qa it helps to keep folks more honest, open and questioning assumptions. Group-think can be an issue but generally I find the benefit to be positive.
From code climate to codecov there are a lot of code analysis tools out there. They help you manage the process of having quality code by analyzing your code base and test coverage. They can show you improvements over time and can let you set and manage levels of code quality.
Consider adding code that looks at the structure of your tests themselves and looks for things like: duplicate page objects, unused page object, not using page objects, use of true == true statements, etc. Make these into actual tests so that you can let the tests help enforce the standards, not you through constant persuasion arguments in code review.
btw 'industry standard' doesn't work for me here. The software quality industry is still too varied and undefined.
