6

Would it be a good idea to have an interview in a format of a paired exploratory testing session with a candidate on a manual/automated testing position? A thing under test might be an internal web application or a publicly opened web resource.

What things to look for in a candidate and how to quantify the results of such an interview? (I don't think that number of newly-found bugs is a good and relevant measurement)

  • I hope that you would inform the candidate ahead that pair testing would be a part of the interview. Personally I would be confused what is expected of me if required to do pair testing of a 100% unknown system on the spot. And as you feel, it is not easy to interpret the results of such testing. I wish that all positions were filled by 3-6 month contract-to-permanent. Such short time contract is best way to evaluate a candidate, IMHO. – Peter M. - stands for Monica Aug 25 '17 at 13:21
  • @PeterMasiar right, good points - it could be confusing if done incorrectly. I imagine this as explaining what an app under test is supposed to be doing on a high-level, technologies used to build it and major features and then ask a candidate about things he or she wants to explore and take a closer look at, see what exploratory testing ideas would be generated and then check them out together. It could, I am afraid, take a long time to conduct this kind of interview and it feels like there can potentially be a number of things that can go wrong there :) Thanks! – alecxe Aug 25 '17 at 13:35
  • @alecxe Is there anything that you would like to see more in my answer? Given the bounty you're probably not satisfied, so if I can give a more complete answer I'd be happy to. – FDM Aug 28 '17 at 12:40
  • @FDM nono, that is a very good answer and I'm happy to have it! Just want to bring more attention and opinions to potentially enrich the topic. Thanks! – alecxe Aug 28 '17 at 12:42
  • Ok, cool. Yes, more takes on the topic would be an interesting read. :) – FDM Aug 28 '17 at 12:49
1

Yes, it would. I like teams that execute exploratory testing in pairs, I think it is easier to write charters, create mind-maps and decide on heuristics together then alone. It will keep you sharper, you will go faster in executing and analysing the results.

I would like to find answers on the following questions during an interview:

  • Will your team like working with him/her?
  • Can they design good test-cases?
  • Can the produce reproducible results in a free-format?

Seeing how someone would approach a pair-testing sessions is worth a lot. It is about communication and test design. Also you will notice if someone is a quick learner. Can they lead the session and get some results while working together with someone else. How do they respond to critical feedback during the session.

Exploratory testing sessions are time-boxed, making it a great candidate for a second interview test. You can perform it with a time-box of lets say 45 minutes and work with a tester colleague. Afterwards review the results together with the hiring manager. Giving the candidates room to explain what they would do different next time for example. Resulting in a interview of around 1,5 hours, which I think would be acceptable.

Alternatively I would do pair-programming of test automation on a simple application. Just like I would always do a pair-programming interview for developers.

3
+50

Great idea. Some things that might help:

  • Instead of pair testing, just explain the basic context and let the candidate figure it out. A big advantage here is you will get a feel of:
    • How many questions does this person ask?
    • What type of questions? Relevant or not?
    • How independent and confident is this tester?
  • Quantifying the result isn't hard science, but:
    • There might be some (critical or high) defects that you would expect to be found. How many did he find?
    • With regular interviews, if we don't take into account technical tests such as for developers, you usually follow your instinct (will this personality fit in our team? Does he show the right attitude?). If you actually see and hear someone testing, you've got a lot more to go by: now it'll become clear if the CV and talking match the expected skill!

Additionally, if this person is going to do test automation, I've listed some questions you can ask here.

  • 2
    We do very similar to this - asking folks to test various things - everything from whiteboard/thought exercises (e.g. we're designing a system to do foo, the current design is this, how would you test this? what issues do you see? risk points? etc) There are some obvious answers to most of our exercises (way too nebulous requirements, why have both a black list/whitelist, etc). That being said, we're still trying to come up with better questions/exercises as even these don't seem to go deep enough. – ernie Aug 25 '17 at 17:39
1

I wouldn't refuse number of found issues as a relevant measurement. After all you're looking for an engineer that would be finding defects in your applications.

As to me, any interview should spot a light onto two kind of person's skills: hard and soft ones. I tend to consider such kind of interview spotting both types: on one hand a person shows you how they would test the particular component (and this is your responsibility to pick the representative application for an interview), on other hand the person shows how they would comunicate their thoughts to teammates.

Since you cannot measure (using any objective scale) a person's soft skill, you can only rely on:

  • number of found issues.
  • velocity (if you have the target value for the issues found)
  • number of issues requiring any forums exploration
  • time taken for such the exploration
  • number of issues from previous point with the succefully found solution
  • time taken for such the exploration
  • Time taken to discuss an issue that would make a partner convinced about the issue is actually a defect
  • Time taken to discuss an issue that would eventually considered as not-a-bug

However if this is the only method you're going to apply in your interview you should take care of the "quality" of the application for the interview since it might not cover all the representative cases that one would meet in the real project life.

1

I have done something like this for a manual tester position. I whipped up a quick MVC site with 3 pages and put "issues" into each page (something like 11 issues total). I made it clear that I wasn't looking for the interviewee to find all the issues. I just wanted to watch how they go about it.

  • Did they open DevTools?
  • Did they thoroughly test an input?
  • Did they notice non-functional issues?

I sat with each candidate for 30 minutes and tried to be as friendly and helpful as possible. This is a high stress situation, which is also something I was looking for the candidate to handle.

In the end I only did this once as it was very time consuming and I started to only fill more Sr level positions on my team.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.