16

While interviewing a QA candidate, we showed him our company's game from the Google Play Store.

We asked him to play a bit and tell what the game is about. The game is simple enough, and we expect him to quickly understand when the game is over, what each item gives the player, etc. This is a test to see his logical thinking.

After that, we told him to test the game at home and find critical bugs, such as crashes, game freezes, wrong game-play and other situations that can frustrate the player.

This is the second candidate that sends results for about 6-9 issues found. The issues are mostly UI specific, such as when you press 2 buttons simultaneously or 2 buttons in a sequence in the menu, or when you turn the internet on and off and then press "Show an ad". Nothing related to the core gameplay, and nothing really critical.

At this point we are unsure whether the candidates aren't good enough or we aren't good enough interviewers to give the right task. We are developers and we have just started trying to hire a QA for our team. Should we fully describe the game and only then ask to find bugs? Or should we do anything else?

  • 18
    Any interview task/question should be designed to extract information about a specific characteristic or skill (it could be multiple specific skills). What specific information are you trying to glean from this assignment? The answer may help others to better address your question. – VanderLinden Jan 15 '18 at 21:41
  • 5
    Is the specific game relevant? And is it your game? – corsiKa Jan 16 '18 at 4:06
  • 26
    They spend enough time to find 6-9 issues. Do you expect them to spend like 15 hours on your issues or what? – Džuris Jan 16 '18 at 8:43
  • 17
    To make this a proper test of a candidate you should already know what bugs you expect them to find. – Ywapom Jan 16 '18 at 18:46
  • 10
    You also mention that this is the first QA position at your company. This is just my inference, but based on your question, it seems like you may not have a well-defined role for your QA position. This will make it difficult to design a useful interview/evaluation process, and will also increase the likelihood that the eventual hire fails to meet expectations and/or becomes discontented and leaves. "QA" needs to be defined for your company's needs beyond "test the product and make sure there are no bugs". – VanderLinden Jan 17 '18 at 0:00
76

This may be culture-specific, but from my point of view, you’re asking for someone to work for free. The interview should be the interview; you don’t ask someone to do more work after the interview. And from your point of view, how do you know you can trust that the person in question was the one who did the homework? If you want them to do some live testing, which I’m not convinced is a good idea, have them do it in your presence.

Also, from a testing point of view, it’s very hard to do that in isolation. They’d need a very complete spec, or someone to ask questions of, to understand what the proper behavior is.

And, finally, it's not clear to me what the point of the exercise is. An interview is a way for you to find out if the candidate is the right fit for the job, and the job is the right fit for the candidate. How does having someone test an app, not in your presence, help with that? If you were having them do testing in your presence, while asking them to describe their thought process, or ask them to put a test approach together, I would see that as having value. But (I'm guessing) wanting to hire whoever finds the most bugs in your app is problematic at best.

  • I totally disagree. Didn't down vote, though.... – Ray Oei Jan 15 '18 at 16:06
  • 8
    OK sorry, I need to clarify. On first read I thought it was just an example app. Not their product. I agree with you: free testing is not a good thing... I do think live challenging is really useful, though ;-) – Ray Oei Jan 15 '18 at 18:50
  • 2
    Working for free is against the law. If the candidate's tasks add value to the company then you should pay the candidate at least minimum wage for doing them. – emory Jan 16 '18 at 20:00
  • 1
    @jskrwyk it is not illegal to ask someone to do something in the context of an interview (before they are employed). it is illegal to ask someone to do something of value before they are employed. ask your candidates to find bugs in your fizz-buzz implementation not in your product. – emory Jan 17 '18 at 11:19
  • 3
    Even it it is not illegal asking for someone to do productive work for a chance of being hired it is a very shoddy business. – Paulo Scardine Jan 17 '18 at 18:30
35

IMO, your interview process shows that your company doesn't really understand QA to begin with.

QA has specific goals. It isn't "sit here with our software for x hours and figure things out". It needs to be clearly categorized, and there should be a pass/fail list of objectives.

Your scenario is unrealistic, because there is no way to ever finish. At how many hours is the QA complete? 10? 100? 1000? How many bugs are enough? Did you put bugs in the software for your candidates to find?

This is in addition to working for free.

  • 1
    I commented on the question with a similar point and then saw your answer. While this doesn't directly answer the question as posed, I think this is the most valuable answer in terms of helping the OP. – VanderLinden Jan 17 '18 at 0:28
  • I want to upvote this answer. I really agree on QA not simply beeing about finding bugs. It contains an important part about making sure there are no bugs (well, fewer at least) to start with. Testing is one way to find bugs after the fact -- but QA includes an important part before testing even starts. – ghellquist Jan 20 '18 at 9:58
30

You know, the funny thing is, this almost became a good interview tool.

If I walked in to a QA interview, and the interviewer said, "By the way, when you're done, we'd like you to bug-hunt our production product when you go home," my first thought would be: You clearly don't care about your QA programmers (Why would you? If you feel you can get random strangers in interviews to do their job for free, you clearly don't respect them!)

If, instead, the interviewer said, "Here's a version of our product, before it hit QA, 2 years ago. QA found a number of bugs, and later, users found a few more. We'd like you to do a trial QA run on this for 30 minutes, focusing on ABC and XYZ, and see what issues you're able to spot."

... the difference? Instead of thinking, "Oh, I see - you view me as free labor" - I'd instead think, "I'm being compared with their existing/prior QA staff." Best thing is, it's a lot more clear-cut from your perspective: you can compare their list of found issues versus a historical baseline (and even better, you can see if they have a fresh perspective and would've caught issues that the existing QA staff missed.)

As you currently handled it? I'll second what Lightness said: the good, talented employees are saying, "No way".

  • 1
    I like this answer. The key is to provide some set of requirements for the applicant to test against, and the areas where those requirements are not met should be known in advance - unless the question is intended to determine whether the applicant asks about requirements before diving into testing in the first place. – VanderLinden Jan 17 '18 at 0:32
  • I also wanted to add that the question seems to imply that this is going to be their first QA hire, so the portion of the answer about defects identified by QA in the past may not apply - although it certainly applies if the answer is applied in a more general way to candidate evaluation practices. – VanderLinden Jan 17 '18 at 0:34
  • 2
    I would assume the developers were doing the QA... I hate to see a product that genuinely had zero QA... they can easily pull out an older build and simply change it from "QA found a number of bugs..." to "as sole developers, we found a number of bugs..." – Nelson Jan 18 '18 at 7:36
10

I think it's a good practice to give some demo Game & ask the tester to find some bugs in the game it helps to figure out these basic qualities of the candidate like.

  • Judgement Capability
  • Decision making capability (Bug or not)
  • Attention to details (Instruction sheet)
  • Efficiency of the candidate
  • Focus on random application (Layman approach)
  • Flexibility with the situation & work
  • Time management
  • Observation of the random application....& much more

    here you made some mistake which I think you should improve

    1. This is a wrong application as this game is in production, you may give some of your game(not in production) or any beta version game.

    2. Don't ask someone for homework, better to give application and ask to find bugs in your office otherwise you will never come to know that whether candidate finds these bug or some professional game tester at home.

    3. It is one of the tuff jobs to find the bug before the interview so don't restrict tester to find only "critical" bug.

    4. Give them one hour time & then conduct the interview, ask about there experience judge the qualities of the candidate as mention above.

    5. As you say that you are developer interview a tester in such a way that like he/she is an end user.

    6. Ask some basic manual testing question to make sure about testing concepts(as per resume).

    7. It's not required to explain the game to tester let him/her understand the game figure out his layman approach and understanding about new things.

10

And as a homework we have asked to test the game and find critical bugs

No.

No, no, no.

It is not your place to give "homework" from a job interview, least of all "homework" that involves doing your job for you for free.

No!

Nothing related to the core gameplay, and nothing really critical.

I'm not surprised — you put in place no structure. No quality or functional targets specified, no adequate issue reporting procedure, no way to follow-up with devs, and er no pay.

So at this point we are unclear, do we have not enough good candidates

Your good candidates did not get back to you because they would not want to work for you.

We are not enough good interviewers to give the right task?

No.

However, to add something positive, I do like the face-to-face technique you described.

6

I think it works better if you use an application that you know has issues as that helps you see if people find the deep issues or only the shallow ones.

I for instance use applications that I got from the RST training with Michael Bolton.

It also helps if you ask yourself what you would expect beforehand when they do a challenge like this. For me, in your story, the more you need to hint & explain, the more the red flag is waving, TBH.

The homework is less useful, just let somebody work for an hour or so and then discuss what they did. Let them explain their thinking. If you are looking to hard to find something useful in a candidate it is likely not the one you are looking for. Unless you want to spent a lot of time teaching them, that could be different...

A good read is this one from Cem Kaner and another one from Paul Carvalho. Or see this blog. But there are many more ;-)

Note: I like the way you are addressing this, including the critical self reflection! I wish more teams/companies would do that.

4

2 candidates is a small sample pool to go on as already mentioned. Another thing to consider (which I never did before until it was pointed out to me) was:

  1. Do you let the candidate know in advance that they are going to have a practical test to do?
  2. Do you let them know they will have some take home work?

Reasons for asking this:

  1. Depending on the candidate and their nervousness, springing a test on them unannounced may cause them to be more nervous and perhaps not perform at their best. Letting them know in advance helps them to be less nervous about it.
  2. If they didn't know they had some take home work after the interview, they may not have allowed time for it. If the candidate is interviewing in other companies they will have that to contend with along with a full time job (most likely). If the homework is unexpected, it may well be that the candidate is struggling to find the time to fit it in.

I think you have the right idea for how to interview but I would suggest removing the homework and instead do it all in the interview session paired as JoãoGFarias already suggested. In my own experience, paired sessions definitely helped me to find the candidate I was looking for. I thought about what did I actually want from the candidate? Did I want lots of bugs reported or to see how they thought critically about the application? How did the candidate interact with the team. It is also hugely positive as an interviewee to get that real time interaction with your potential future co-workers to see how you would work with them.

Should we fully describe the game and only then ask to find bugs?

I don't think you need to do this. I would suggest giving some background (which it sounds like you've already done). A strong tester will usually start asking questions pretty quickly trying to clarify any uncertainties they might have. This is another good interaction experience for them and you.

I would also recommend checking out this advice from a now tester who used to be a recruiter.

3

Your practice is great, many companies do not do it.

When doing exploratory tests, one should take notes so he can map what he is seeing and what he want to explore further and get clarifications about. Also, a good testers should be able to explain how and why he is testing, its techniques and goals. Communication is crucial for good testing.

As far for your concerns, two candidates is not so great sample to drawn conclusions, but maybe change the task from a homework to a live exercise can improve results. Doing pair-testing, making the candidate work together with others candidates or some employee**, may clarify to you if the candidate is able to do structure and professional testing, and communicate it to others.

On my opinion, looking for the mid/long term in the company, these characteristics are more important than the specific errors the he is able to find in the game. People have different backgrounds in different domains - unless your company is a game company, you should try to avoid this bias through making the found bugs something secondary.

** In this case, the candidate should drive the testing a bit more.

2

In the interview you concentrate to ask questions related to specific app only related to you.I strongly believe this is a wrong way,briefly you must assume and do this a part from the interview not at all.First ask questions related to testing techniques and the projects he was working in if the candidate has working experience, then give him general knowledge about the task that he will test it for example you explain what the game do,after that let him thinking to write test plan and test cases to start testing the game.Finally rate the candidate according to all above.

Not the answer you're looking for? Browse other questions tagged or ask your own question.