Should we reject QA candidates based on basic reasoning and math tests?

Question

Short version:

When hiring experienced QA people, do you want someone who has good technical skills, or do you want someone who has good aptitude skills? My boss says that if the guy can't solve puzzles and aptitude questions, then he doesn't have the required thinking skills to justify the QA position. Is this true?

Long Version:

I am a junior recruiter in a software firm. Irrespective of the experience of the QA guy, my company follows a process with very heavy emphasis on aptitude skills. We usually give a lot of puzzles, reasoning questions and basic maths stuff to check their thinking skills. If a person doesn't perform well in this first "aptitude" round, he is usually sent back. Last sunday, my technical panel interviewed some of these "average" aptitude skills candidates and found their testing/QA skills to be top-notch. My question is simple : Should we make aptitude/problem solving/puzzles as the only criteria for hiring experienced QA people (anywhere between 1-6 years of experience)? or, should we give more weight to their technical skills?

Answer 1

Testing aptitude is a tricky thing, because testers do different kinds of things that all have their own kind of aptitude. There's a big difference between being a world-class tester and a key member of a world-class testing organization.

For instance, I am considered by many who don't know me (but have heard of me) to be a great tester. And it's true that 1) I hope that I am, 2) I've worked hard to be great, 3) many people who HAVE seen my work would testify to my competence. But even so, there are some important aspects of testing that I'm not very good at. For instance, most of my admirers would admit that my diplomacy skills are not great, nor do I have much patience for managers who don't know shit about testing, and yet presume to tell testers how they should do their jobs. (They make me... sooo... angry....) Um, anyway, I'm at my best when I'm working with people whom I can vent to and who will forgive me for my venting.

One charming, docile, patient, stable person + me = a much better test team than me alone. So, we should not be too excited about any narrow idea of allowing only great puzzle solvers into our teams.

Having said that, I would suggest that these are some interesting aptitudes of a tactical software tester (meaning someone whose main job is designing and performing tests):

• A talent for gently, cheerfully, and insistently getting information out of people through interviews, observation, mind-melds, truth serums, or whatever means necessary. The ability to ask questions in a way that makes people want to answer. I evaluate this by posing testing questions that are insufficiently specified, and observing the questions that are asked.

• A talent for thinking laterally. Technically, I mean a talent for (and patience for) producing and using multiple alternative mental models to interpret and cope with technical situations that occur. This aptitude often shows itself by another name: a sense of humor. I evaluate this by posing realistic puzzles about the behavior of a product or the state of a spec, and observing how the tester entertains different possibilities during the course of analysis.

• A talent for thinking logically. This means reasoning from within the constraints of a particular model. To evaluate this, I give testers a problem in test framing. I get them to describe a test, and then I walk them forward and backward through the test, asking them to explain exactly why they did what they did, and what the implications would be of doing it differently.

• A talent for loving puzzles. Notice I did not say a talent for solving puzzles. This is because many puzzles we must cope with are too difficult to solve. And yet we must try, anyway. Testers who don't have the patience and courage to throw themselves at puzzles are doomed to miss a great many bugs.

• A talent for rapid learning. Great testing can be described as rapid learning. Some obscure technology is thrown in your direction, and you have to jump on it like a hungry piranha. You must be excited about learning new things. To evaluate this, I look for evidence that the tester has taught himself something complicated and interesting. However, I would prefer to observe this directly by giving the tester a few hours to test a website I specify.

• A talent for story-telling. The credibility of a tester comes, often, through their ability to explain things that happened. To evaluate this, of course I interview. But I'm looking for someone who is able to tell compelling technical anecdotes.

• A talent for identifying and using tools. This does not mean programming, but programming skill certainly speaks to this.

• A talent for writing. This is hard for a lot of testers, since a lot of testers are writing in English, and English is not their first language. Furthermore, testers who got into testing via the computer science route tend to hate writing. This is partly why I'm suspicious of companies that are gleefully technocratic, such as Microsoft and Google.

• A talent for service. Testing is a support activity. A tester who loves to help others do better work (as opposed to attacking people for making mistakes) will be treasured by the programmers. Personally, I'm pretty good at this. I give developers who work with me a 13 point document outlining my unilateral commitment to their success.

I don't need to see all these aptitudes to be excited about a tester. But I need to see at least a few. Out of all of them, I think rapid learning is the most important... Though perhaps that's my bias, considering I'm a high school dropout who wrote a book about education. Self-education seems, to me, like a key that unlocks many doors.

-- james

Answer 2

We usually give a lot of puzzles, reasoning questions and basic maths stuff to check their thinking skills. If a person doesn't perform well in this first "aptitude" round, he is usually sent back.

Why even ask basic math and reasoning questions at all? Why not start with tests that, although simple, at least mimic the actual, specific real world QA situations they will encounter on the job?

That is, rather than a question like:

If an object travels at five feet per second, how many feet does it travel in one hour?

How about one like:

You are presented with a form that has two text boxes, one for first name, and one for last name. List ten things you would type in each of these text boxes to try to break the application, and explain why.

Answer 3

My boss says that if the guy can't solve puzzles and aptitude questions, then he doesn't have the required thinking skills to justify the QA position. Is this true?

It may be true, but how does your boss know? Has he hired both good and poor puzzlers and found that their QA abilities within your shop track well with their puzzler abilities? Or is it just folklore?

I know terrific QAers who hate puzzles. And I know terrific puzzlers who are obviously unqualified to be QAers.

I believe critical thinking skills are essential for a good QAer. I'm just not sure they are measured well by a candidate's ability or desire to solve puzzles. When I see reliance on puzzle questions during the interview process, I have to question the interviewer's interviewing skills.

You may also want to have a look at this Blog about best and worst technical interviews

Answer 4

Word puzzles and math problems may seem to be convenient proxies for assessing testing talent. Indeed, one may argue that universities use word puzzles and math problems to assess technical ability. For a recent graduate, or for someone who is good at solving word puzzles and math problems, it may seem natural to expect QA candidates to also have the same talent. Moreover, it is easy to find lists of these kinds of questions and it is easy to decide whether the answers are right or wrong.

However, software testers are not paid to solve word puzzles or math problems. The abilities these questions appear to measure help with only a small part of a tester's job. A tester must be organized, detail-oriented, amenable to tedium, good at communicating, thorough, respecting of their co-workers, and tolerant of pressure. A tester who is good at word puzzles and math problems but deficient at these other skills will be a bad tester.

Interviewing is hard work. It requires deliberate questioning and careful listening. An interviewer who relies solely on word puzzles and math problems is doing a disservice to the employer and to the candidate.

Answer 5

I think people should be hired so their skills match the actually position and daily tasks as closely as possible. If the role could use someone with the drive and problem solving skills to solve "puzzle-like" tasks, then it's probably a good requirement. If the role requires hard testing experience and technical abilities, then perhaps posing questions about knights and knaves wouldn't be too helpful.

When I was hired for my current role as an automated tester, my interview had a written test with two components: a programming component and a testing/QA component. The programming component was pretty basic. Write a function to add some numbers, write a recursive function to computer a factorial, and analyze some Java-like code. The QA component was writing a bug report, write a test plan for testing a dialog box, and find and describe bugs in a buggy program. These are tasks that I more or less do every day (including recursion, it comes up plenty in the GUI testing I do). I did well on the test, I got the job, and it turns out I'm pretty good at it. I have a Master's in applied math, and those specific math skills I gained wouldn't really help me in this role. Even if I aced a puzzle question, it might not have indicated I'd be good.

Also keep in mind that posing "test-like" questions might be stressful for some candidates. Some people who appear quite inept with a written, timed test would appear nothing but in a more open-ended discussion, or vice versa. Placing too much emphasis on test results might unfairly rule out some good candidates from the role (which is a bit ironic I suppose).

Answer 6

If the puzzles and math problems reflect the actual product, then I'd agree they are valid interview tools to assess whether a candidate can handle the job. If these are due to a bias of education and culture on the part of those in charge, then they are not really necessary and maybe a hinderance. Or is this a tool that was just chosen because someone thinks that these questions can give an indication of test ability.

I've seen companies where many of the senior management were from specific types of schools, anyone not from such schools were usually expected to perform over a bias before consideration was given. In many cases it depends on what the reliance on puzzles and math questions are for.

Answer 7

I have worked for a similar company in the past. In my opinion an ideal QA candidate would have a bit of both skills in order to succeed.

A person may be technically very sound but what good are the skills if he cannot present the idea to the world when needed.

In pressure situations people may lose composure and do something unexpected. Testing math and analytic skills does not mean that technical skills won't be tested.

I agree to your manager to some extent, if a person cannot do 2+2 in a normal environment, how can he come up with creative test scenarios when needed?

But then again, technical skills are also very important and they cannot be ignored, no matter how big a mathematician the candidate is.

Answer 8

Based on my experience, you don't want to hire a QA whose only asset is strong technical knowledge who may be lacking in the problem-solving, question-asking, quick-learning skills that many other posts have described. After all, a QA is going to have to interact with developers, business analysts and perhaps end-users. Also, as another post pointed out, those who are heavily technical are often not very good with the written word.

At the same time, you would also not want to hire a QA who lacked the necessary technical skills but had good people skills, writing skills and was perhaps able to solve puzzles/problems during an interview. Although, it could be argued that a person with good math and problem-solving skills could easily develop the technical skills.

And it goes without saying that you would not want to hire a QA who doesn't demonstrate technical ability or problem-solving abilities.

I guess my answer to your question is that I wouldn't rule out the use of puzzles and tests but don't weight them too heavily. You are looking for a well-rounded candidate and some people who may fail a test during an interview will still excel in a real work environment.

Answer 9

I like James Bach’s explanation. I also believe I am a world class tester though I have 5-6 years of experience and I love to solve the puzzles. You guys have not mentioned that if you hire a technical and puzzle solver person in your organization, how a puzzle solver survives in the organization? I worked some good companies in MN as a contractor. Since I am a creative person and puzzle solver, I can do test design for any new technology. When I solve the puzzle, other QA team members start sharing their brain and criticizing my work. Before I solve the puzzle, everyone of my team (FT employee) was busy with other project , actually they were just pretending to see how do I work and if I can figure out something, they will continue from there. But I solved the testing design. Suddenly the project was on hold because of not sufficient budget. Rests of the Team members were so happy because they got a chance to let me go. How? Because Manager is always political position- so manager will not upset their 5-10yers full time employee. As a result, I finished my project within 5 months. Next project, QA lead was managing the project. Again I did a test design for a new UI and once I finished writing scripts to demonstrate how to test. Other senior team member jumped and corrected something. So that I do not get the whole credit. In my team, 5 members, all are copy past tester. They wait for my action to test some different module first, once I tested and logged defect, they start jumping and got the credit. My lead knows everything. I worked here for almost 10 months I would not mind what happens to me if they would give me just recognition. Most of place, I worked my team members and lead –they got pain in their chest to say “you did a good job in front of people”. I hate this testing job. Now thinking of a career change.