While hiring can I ask a QA candidate to find bugs on this game as an interview homework

Question

While interviewing a QA candidate, we showed him our company's game from the Google Play Store.

We asked him to play a bit and tell what the game is about. The game is simple enough, and we expect him to quickly understand when the game is over, what each item gives the player, etc. This is a test to see his logical thinking.

After that, we told him to test the game at home and find critical bugs, such as crashes, game freezes, wrong game-play and other situations that can frustrate the player.

This is the second candidate that sends results for about 6-9 issues found. The issues are mostly UI specific, such as when you press 2 buttons simultaneously or 2 buttons in a sequence in the menu, or when you turn the internet on and off and then press "Show an ad". Nothing related to the core gameplay, and nothing really critical.

At this point we are unsure whether the candidates aren't good enough or we aren't good enough interviewers to give the right task. We are developers and we have just started trying to hire a QA for our team. Should we fully describe the game and only then ask to find bugs? Or should we do anything else?

Answer 1

This may be culture-specific, but from my point of view, you’re asking for someone to work for free. The interview should be the interview; you don’t ask someone to do more work after the interview. And from your point of view, how do you know you can trust that the person in question was the one who did the homework? If you want them to do some live testing, which I’m not convinced is a good idea, have them do it in your presence.

Also, from a testing point of view, it’s very hard to do that in isolation. They’d need a very complete spec, or someone to ask questions of, to understand what the proper behavior is.

And, finally, it's not clear to me what the point of the exercise is. An interview is a way for you to find out if the candidate is the right fit for the job, and the job is the right fit for the candidate. How does having someone test an app, not in your presence, help with that? If you were having them do testing in your presence, while asking them to describe their thought process, or ask them to put a test approach together, I would see that as having value. But (I'm guessing) wanting to hire whoever finds the most bugs in your app is problematic at best.

Answer 2

IMO, your interview process shows that your company doesn't really understand QA to begin with.

QA has specific goals. It isn't "sit here with our software for x hours and figure things out". It needs to be clearly categorized, and there should be a pass/fail list of objectives.

Your scenario is unrealistic, because there is no way to ever finish. At how many hours is the QA complete? 10? 100? 1000? How many bugs are enough? Did you put bugs in the software for your candidates to find?

This is in addition to working for free.

Answer 3

You know, the funny thing is, this almost became a good interview tool.

If I walked in to a QA interview, and the interviewer said, "By the way, when you're done, we'd like you to bug-hunt our production product when you go home," my first thought would be: You clearly don't care about your QA programmers (Why would you? If you feel you can get random strangers in interviews to do their job for free, you clearly don't respect them!)

If, instead, the interviewer said, "Here's a version of our product, before it hit QA, 2 years ago. QA found a number of bugs, and later, users found a few more. We'd like you to do a trial QA run on this for 30 minutes, focusing on ABC and XYZ, and see what issues you're able to spot."

... the difference? Instead of thinking, "Oh, I see - you view me as free labor" - I'd instead think, "I'm being compared with their existing/prior QA staff." Best thing is, it's a lot more clear-cut from your perspective: you can compare their list of found issues versus a historical baseline (and even better, you can see if they have a fresh perspective and would've caught issues that the existing QA staff missed.)

As you currently handled it? I'll second what Lightness said: the good, talented employees are saying, "No way".

Answer 4

And as a homework we have asked to test the game and find critical bugs

No.

No, no, no.

It is not your place to give "homework" from a job interview, least of all "homework" that involves doing your job for you for free.

No!

Nothing related to the core gameplay, and nothing really critical.

I'm not surprised — you put in place no structure. No quality or functional targets specified, no adequate issue reporting procedure, no way to follow-up with devs, and er no pay.

So at this point we are unclear, do we have not enough good candidates

Your good candidates did not get back to you because they would not want to work for you.

We are not enough good interviewers to give the right task?

No.

However, to add something positive, I do like the face-to-face technique you described.

Answer 5

I think it's a good practice to give some demo Game & ask the tester to find some bugs in the game it helps to figure out these basic qualities of the candidate like.

• Judgement Capability
• Decision making capability (Bug or not)
• Attention to details (Instruction sheet)
• Efficiency of the candidate
• Focus on random application (Layman approach)
• Flexibility with the situation & work
• Time management

• Observation of the random application....& much more

  here you made some mistake which I think you should improve

  1. This is a wrong application as this game is in production, you may give some of your game(not in production) or any beta version game.

  2. Don't ask someone for homework, better to give application and ask to find bugs in your office otherwise you will never come to know that whether candidate finds these bug or some professional game tester at home.

  3. It is one of the tuff jobs to find the bug before the interview so don't restrict tester to find only "critical" bug.

  4. Give them one hour time & then conduct the interview, ask about there experience judge the qualities of the candidate as mention above.

  5. As you say that you are developer interview a tester in such a way that like he/she is an end user.

  6. Ask some basic manual testing question to make sure about testing concepts(as per resume).

  7. It's not required to explain the game to tester let him/her understand the game figure out his layman approach and understanding about new things.

Answer 6

I think it works better if you use an application that you know has issues as that helps you see if people find the deep issues or only the shallow ones.

I for instance use applications that I got from the RST training with Michael Bolton.

It also helps if you ask yourself what you would expect beforehand when they do a challenge like this. For me, in your story, the more you need to hint & explain, the more the red flag is waving, TBH.

The homework is less useful, just let somebody work for an hour or so and then discuss what they did. Let them explain their thinking. If you are looking to hard to find something useful in a candidate it is likely not the one you are looking for. Unless you want to spent a lot of time teaching them, that could be different...

A good read is this one from Cem Kaner and another one from Paul Carvalho. Or see this blog. But there are many more ;-)

Note: I like the way you are addressing this, including the critical self reflection! I wish more teams/companies would do that.

Answer 7

2 candidates is a small sample pool to go on as already mentioned. Another thing to consider (which I never did before until it was pointed out to me) was:

1. Do you let the candidate know in advance that they are going to have a practical test to do?
2. Do you let them know they will have some take home work?

Reasons for asking this:

1. Depending on the candidate and their nervousness, springing a test on them unannounced may cause them to be more nervous and perhaps not perform at their best. Letting them know in advance helps them to be less nervous about it.
2. If they didn't know they had some take home work after the interview, they may not have allowed time for it. If the candidate is interviewing in other companies they will have that to contend with along with a full time job (most likely). If the homework is unexpected, it may well be that the candidate is struggling to find the time to fit it in.

I think you have the right idea for how to interview but I would suggest removing the homework and instead do it all in the interview session paired as JoãoGFarias already suggested. In my own experience, paired sessions definitely helped me to find the candidate I was looking for. I thought about what did I actually want from the candidate? Did I want lots of bugs reported or to see how they thought critically about the application? How did the candidate interact with the team. It is also hugely positive as an interviewee to get that real time interaction with your potential future co-workers to see how you would work with them.

Should we fully describe the game and only then ask to find bugs?

I don't think you need to do this. I would suggest giving some background (which it sounds like you've already done). A strong tester will usually start asking questions pretty quickly trying to clarify any uncertainties they might have. This is another good interaction experience for them and you.

I would also recommend checking out this advice from a now tester who used to be a recruiter.

Answer 8

Your practice is great, many companies do not do it.

When doing exploratory tests, one should take notes so he can map what he is seeing and what he want to explore further and get clarifications about. Also, a good testers should be able to explain how and why he is testing, its techniques and goals. Communication is crucial for good testing.

As far for your concerns, two candidates is not so great sample to drawn conclusions, but maybe change the task from a homework to a live exercise can improve results. Doing pair-testing, making the candidate work together with others candidates or some employee**, may clarify to you if the candidate is able to do structure and professional testing, and communicate it to others.

On my opinion, looking for the mid/long term in the company, these characteristics are more important than the specific errors the he is able to find in the game. People have different backgrounds in different domains - unless your company is a game company, you should try to avoid this bias through making the found bugs something secondary.

** In this case, the candidate should drive the testing a bit more.

Answer 9

In the interview you concentrate to ask questions related to specific app only related to you.I strongly believe this is a wrong way,briefly you must assume and do this a part from the interview not at all.First ask questions related to testing techniques and the projects he was working in if the candidate has working experience, then give him general knowledge about the task that he will test it for example you explain what the game do,after that let him thinking to write test plan and test cases to start testing the game.Finally rate the candidate according to all above.