How should you interview for QA *Automation* positions?

Question

A variant on the How should you interview for QA positions? I am interviewing candidates for an Automation position, we don't use commercial automation tools, but rather use a framework using mainly Perl, so knowing Perl or some other programming language is required, but being a "computer geek" is a great plus. Another problem is that good programmers tend not to have testing state of mind, they will follow the needed design but will have a hard time going from test case to test code without forgetting something in the way.

So... besides some basic programming questions, maybe a single very basic testing question, what else should I ask ?

Answer 1

In my experience, if you hire an "automator", you're going to get someone who can write code, but has no skill or desire to test. And that's fine if that's what you want.

However, if you just want tips on how to hire a tester than can automate, then ask testing questions that have potential to be solved more efficiently with code. An example I use frequently is "You're in charge of testing a fuction that returns a random number between 1-6 (esentially a die rolling program) - how would you test it?"

I won't go into the details of the answer here, but testers who can program, are good at recognizing the scope of challenges in this problem and will have ideas on how writing automation and analysis tools will help them address the challenges.

Answer 2

You must define the role before you can determine the interview questions. You need to ask yourself - what will this new hire be expected to do?

Will your Automator actually be required to do any testing? Or just build automation tools? Will your Automator analyze your needs and create her own requirements? Or is someone else creating the requirements? Will your Automator be required to conceive, design, and develop a test automation framework? Or is that already in place, and this person will just write the code to fill in the missing pieces?

Knowing that answers to that will help steer you in the right direction.

You may need only a developer, who happens to be developing test automation code. Or you may need a tester, who can build automation to aid in her testing. Or you may need a designer/developer with domain expertise in testing and test automation who can analyze your department's needs, design an effective solution, and develop the code to implement that solution.

The sourcing and interviewing of the candidate will change depending on your choice of role.

Answer 3

We faced this during recent hiring and it took 6 months. We finally learned:

Test their automated test writing skills

i.e. the thing we want them to do on a daily basis!

Yes we started off with the puzzles and algorithm questions provided by our development group. However these consistently turned out to be poor predictors for the good people we wanted to hire. I personally knew two outstanding candidates that we rejected due to this process.

So we finally turned to testing their automated test writing.
We created a mini application in a few minutes using our development framework - Ruby on Rails.
We wrote a couple of simple ui tests for it.

We then asked them to:

• Write a couple more tests cases for a model
• Write a couple of test cases for one of the views
• Dry up some test cases that had sections of duplicated code
• Use page objects for a couple of UI tests to make them more readable
• Determine why intermittent failures were happening for one of the field validations
  (the issue was a test that used 1..31 values for day even for months like February)

Answer 4

I've been very successful in asking candidates to write code for some simple algorithm and then have them test the code as if someone else wrote it. I try to look for tests that go well beyond unit tests and into integration or system tests. If they are good coders and good testers, they will start to go into more complex areas like boundary, stress, performance testing, and lots of white box. See if they come up with good ways of looking into the code they are testing and that they are comfortable with it. Do they know how to debug, interpret code, understand complexity, understand callpaths, think about code coverage? And echoing some of the answers in How should you interview for QA positions, make sure they don't give up - if they are good testers they will figure out lots of crazy ways to break what they're testing and shouldn't stop until the interviewer says so. Good coders will stop with unit tests. Good testers will stop at black box. Good coders + good testers are ninjas on fire.

Answer 5

This is a good question, from my experience you should be breaking down the technical skills you expect the candidate to master into 3-5 essential must have skills. For example:

• Knowledge in scripting/programming language, such as Perl/JavaScript/Java/Bash
• Knowledge in relational database/no SQL, such as Oracle/MongoDB/Hadoop/SQL
• Knowledge in testing framework, such as Selenium Webdriver/JUnit/LoadRunner
• Knowledge in operating system, such as Linux/Windows/MacOS
• Knowledge in QA methodologies, such as STP/STD/STR

After doing so, you need to prepare a test and look for relevant interview questions from the Web, or try to use some automated tools which already done that for you, such as Codelect.net, LuGo-Test.com.

Note, there are many platforms that can help you with interviewing developers, although there are only few which covers QA topics.

Many web sites contains free interview questions, although you need to double check their content correctness and relevancy, since only a small part of them are gathering relevant content.

Answer 6

To evaluate the candidate's creativity give them some exercise involving lateral thinking skills like testing a new feature that is not yet implemented on Facebook. Are they able to think about the risk areas?, Are they able to ask you good questions to go about the exercise? What different test methodologies come to their mind while testing it?

In testing you could ask about the framework and test methodologies. This way we can evaluate if they are aware and really understand why they apply certain test methodologies in certain situations and the benefits and disadvantages of using one approach over another. For example you could ask how to design test data while testing "post comments" feature in facebook. What are the pros and cons of using data-driven testing approach or random test data generation approach in this context?