Automation engineers- general purpose vs. dedicated

Question

As part of our recent recruiting effort we need to decide on the best approach for choosing automation engineers. On the one hand software engineers, or programmers, can write great automation software, faster and better leaving the "what to test" to others.

On the other hand engineers with solid testing background but only with basic programming skills might be able to write good-enough automation but with much better "what to test" in it.

I worked with both types and seen advantages to both, but with no conclusive winner. Am I missing something ?

Answer 1

The hiring decision really between a developer with limited testing knowledge and someone with testing experience and limited coding skills.

In my experience, hiring developers with no/limited testing knowledge will get you "automated tests" that are barely a notch above unit tests in complexity and value because the "test automator" lacks an understanding of testing paradigms which will limit the overall effectiveness of their automated test designs. (so while they may write code faster it may not be necessarily be 'better.')

On the other extreme, testers with little programming background tend to automate things that replicate a bunch of steps through the UI attempting to emulate a rudimentary sequence of repetitive actions. This automation also provides little long term value and tends to require a lot of maintenance costs.

I would like to think that there are more folks who have a solid understanding of software testing coupled with solid programming skills.

The pool of potential candidates should not be limited to these "either/or" profiles. But, if you need software testers and you can't find well rounded software testers who are multi-faceted then the best course of action may be to hire people who are a good fit for the team and then invest in that person and help them either obtain the knowledge/skills they lack.

Answer 2

You are not missing anything. "Software engineer", "programmer", and "engineer with solid testing background" are general terms that could describe a wide spectrum of personalities and abilities. Any of those terms could describe a great automation engineer or a terrible automation engineer. And of course there are many other aspects of the candidate that you did not mention but which are just as important to evaluate during the interview, e.g. curiosity, interest in the job, compatibility with the test of the team, ability to communicate clearly in person and in writing, and organizational skills.

Sometimes you just have to pick something, pay attention to what happens, and adjust as you go along.

Answer 3

For the past three years, I've spent all of my time consulting to test automation groups in a variety of organizations. My clients' test automation focuses entirely on basic happy path and boundary case tests. In that context, I would go with good programmers with little testing skill over good testers with little programming skill.

Test automation is software engineering. Automated tests are software. Like other software, the majority of the cost happens after the code is initially written. Writing software that reduces those ongoing costs is hard. It's real programming that takes just as much skill as writing applications.

My experience is this: Testers who have a little programming skill can usually write automated tests that work. Their code tends to be procedural, and filled with lots of incidental details, such as GUI terminology, test framework terminology, or literal data values.

Generally they have not developed skill in abstracting important concepts from the details, such as classes, "page objects," and other common test automation patterns.

Generally, they have not learned about how to write code that is expressive, with low coupling and high cohesion. As a result, their code poorly describes the responsibility being tested, and it's difficult to change. Given that automated tests have to change in response to changes in both requirements and implementation, and that these changes happen regularly, the difficulty of changing the tests creates a great deal of pain. Further, when the test fails, the code and the stack traces give little aid in diagnosing the failure.

I've talked with scores of people about successful and failed test automation programs. The ongoing cost of automation is one a huge contributor to test automation program failure. (Actually, the biggest factor is not the cost, but people's surprise at the cost, but that's a story for another day.) Initial cost matters. Ongoing cost matters more.

On the other hand, not all programmers are great at programming. If you want a great test automator, you'll need some reasonable way to assess programming skill.

Of course you will also need testing skill. Given the ongoing expense of test automation, you'll want to make sure the tests you automate are worth the cost. Programming skills can reduce the costs of automated tests, but you'll need testing skills to make sure the automated tests test provide value.

With a few rare and delightful exceptions, I have not been able to help testers improve their programming skills to any great degree. I have, on the other hand, been able to help programmers improve their testing skills and practices, at least for the kinds of basic tests that are the core of most test automation efforts. Perhaps some or all of the difference can be attributed to my coaching skill and style.

I want to reiterate that my experience is entirely with people automating fairly basic happy path and boundary tests. If you're doing performance testing, security testing, or some other kind of test that requires specialized skills beyond the basics, I don't know how to advise there.

And no matter what amount and type of test automation you're doing, you will also want people with excellent testing skills to do exploratory testing.