12

Would it be a good idea to have an interview in a format of a paired exploratory testing session with a candidate on a manual/automated testing position? A thing under test might be an internal web application or a publicly opened web resource.

What things to look for in a candidate and how to quantify the results of such an interview? (I don't think that number of newly-found bugs is a good and relevant measurement)

  • I hope that you would inform the candidate ahead that pair testing would be a part of the interview. Personally I would be confused what is expected of me if required to do pair testing of a 100% unknown system on the spot. And as you feel, it is not easy to interpret the results of such testing. I wish that all positions were filled by 3-6 month contract-to-permanent. Such short time contract is best way to evaluate a candidate, IMHO. – Peter M. - stands for Monica Aug 25 '17 at 13:21
  • @PeterMasiar right, good points - it could be confusing if done incorrectly. I imagine this as explaining what an app under test is supposed to be doing on a high-level, technologies used to build it and major features and then ask a candidate about things he or she wants to explore and take a closer look at, see what exploratory testing ideas would be generated and then check them out together. It could, I am afraid, take a long time to conduct this kind of interview and it feels like there can potentially be a number of things that can go wrong there :) Thanks! – alecxe Aug 25 '17 at 13:35
  • @alecxe Is there anything that you would like to see more in my answer? Given the bounty you're probably not satisfied, so if I can give a more complete answer I'd be happy to. – FDM Aug 28 '17 at 12:40
  • @FDM nono, that is a very good answer and I'm happy to have it! Just want to bring more attention and opinions to potentially enrich the topic. Thanks! – alecxe Aug 28 '17 at 12:42
  • Ok, cool. Yes, more takes on the topic would be an interesting read. :) – FDM Aug 28 '17 at 12:49
7

Yes, it would. I like teams that execute exploratory testing in pairs, I think it is easier to write charters, create mind-maps and decide on heuristics together than alone. It will keep you sharper, you will go faster in executing and analysing the results.

I would like to find answers on the following questions during an interview:

  • Will your team like working with him/her?
  • Can they design good test-cases?
  • Can they produce reproducible results in a free-format?

Seeing how someone would approach a pair-testing session is worth a lot. It is about communication and test design. Also, you will notice if someone is a quick learner. Can they lead the session and get some results while working together with someone else. How do they respond to critical feedback during the session.

Exploratory testing sessions are time-boxed, making it a great candidate for a second interview test. You can perform it with a time-box of let's say 45 minutes and work with a tester colleague. Afterwards, review the results together with the hiring manager. Giving the candidates room to explain what they would do differently next time for example. Resulting in an interview of around 1.5 hours, which I think would be acceptable.

Alternatively, I would do pair-programming of test automation on a simple application. Just like I would always do a pair-programming interview for developers.

|improve this answer|||||
6
+50

Great idea. Some things that might help:

  • Instead of pair testing, just explain the basic context and let the candidate figure it out. A big advantage here is you will get a feel of:
    • How many questions does this person ask?
    • What type of questions? Relevant or not?
    • How independent and confident is this tester?
  • Quantifying the result isn't hard science, but:
    • There might be some (critical or high) defects that you would expect to be found. How many did he find?
    • With regular interviews, if we don't take into account technical tests such as for developers, you usually follow your instinct (will this personality fit in our team? Does he show the right attitude?). If you actually see and hear someone testing, you've got a lot more to go by: now it'll become clear if the CV and talking match the expected skill!

Additionally, if this person is going to do test automation, I've listed some questions you can ask here.

|improve this answer|||||
  • 2
    We do very similar to this - asking folks to test various things - everything from whiteboard/thought exercises (e.g. we're designing a system to do foo, the current design is this, how would you test this? what issues do you see? risk points? etc) There are some obvious answers to most of our exercises (way too nebulous requirements, why have both a black list/whitelist, etc). That being said, we're still trying to come up with better questions/exercises as even these don't seem to go deep enough. – ernie Aug 25 '17 at 17:39
3

I have done something like this for a manual tester position. I whipped up a quick MVC site with 3 pages and put "issues" into each page (something like 11 issues total). I made it clear that I wasn't looking for the interviewee to find all the issues. I just wanted to watch how they go about it.

  • Did they open DevTools?
  • Did they thoroughly test an input?
  • Did they notice non-functional issues?

I sat with each candidate for 30 minutes and tried to be as friendly and helpful as possible. This is a high stress situation, which is also something I was looking for the candidate to handle.

In the end I only did this once as it was very time consuming and I started to only fill more Sr level positions on my team.

|improve this answer|||||
3
+100

Quite interesting topic and although this question has been raised two years ago it is still a current topic! Hence I would like to share our experience / thoughts.

Preparation from our side as business department:

  • We used a web application
  • Before starting we explained the product
  • We encouraged the applicant to speak all his steps "loudly" so that we can follow him during the testing procedure
  • We prepared also to show the screen on the wall and prepared also some mobile devices if the user wants to test with mobile devices (e.g. if one user doing this you can ask him after the interview why he used also mobile devices. Sometimes the user explains about the difference about mobile browser testing and "normal" browser applications so this means that the user thinks also somehow in a "broader" way of testing)
  • We also provided a board where the candidate can made some notes. One of the candidates wrote some kind of End-to-End testing (some short notes) and it was interesting to see how the candidates are working

What did we learned from paired exploratory testing & what things to look for in a candidate?

  • Personal testing way of the candidate: How does the person tests the web application? (by speaking it out loudly we could also follow him and understand his procedure and thoughts) and sometimes we added the thoughts of the person in our test cases afterwards :-)
  • Communication of the candidate: How many questions does the person asks before starting with the test? (in a daily life as a tester you also raise questions to Product Owner, developer, requirements manager etc. So here we could also figure out how communicative this person is)
  • Additional nice thinks for us: And when testing the application sometimes we can also figure out what we did wrong or which new requirement is missing. So we noted down the nice to have requirements and discussed this later with the Product Owner. Sometimes you can create good ideas when doing pairing testing at interviews
  • Bugs: Detecting defects in the meantime was indeed not the main task since we tested on (pre-)productive environment
  • Testing approach of the candidate: At the end it was interesting for us to see how the candidate is approaching. This helped us also to get a feedback whether the candidate fits with our team (preferably you should invite one of our testmanagers/testers in exploratory pair testing)
  • Familiar with tools: there was also one candidate who was opening a free dev-tool (writing logs automatically) and there you also can see, that these candidates are also working with tools and are familiar with it
|improve this answer|||||
2

I wouldn't refuse number of found issues as a relevant measurement. After all you're looking for an engineer that would be finding defects in your applications.

As to me, any interview should spot a light onto two kind of person's skills: hard and soft ones. I tend to consider such kind of interview spotting both types: on one hand a person shows you how they would test the particular component (and this is your responsibility to pick the representative application for an interview), on other hand the person shows how they would comunicate their thoughts to teammates.

Since you cannot measure (using any objective scale) a person's soft skill, you can only rely on:

  • number of found issues.
  • velocity (if you have the target value for the issues found)
  • number of issues requiring any forums exploration
  • time taken for such the exploration
  • number of issues from previous point with the succefully found solution
  • time taken for such the exploration
  • Time taken to discuss an issue that would make a partner convinced about the issue is actually a defect
  • Time taken to discuss an issue that would eventually considered as not-a-bug

However if this is the only method you're going to apply in your interview you should take care of the "quality" of the application for the interview since it might not cover all the representative cases that one would meet in the real project life.

|improve this answer|||||
2

This is a very interesting topic and I have conducted a few interviews similar to this and here is my feedback based on the interviews:

  • You can evaluate candidate's ability to find out bugs within the given timeline.
  • You can evaluate the candidate's ability to focuses on functional or non-functional issues first.
  • You can judge the testing approach of the candidate
  • Communication of the candidates, how would they communicate bugs to the dev team.
  • Interest of the candidate, how is the candidate interacting with the app and the way he is asking questions about the product and are they able to think out of the box

However, these are not only the points, but you can also evaluate the candidates on other criteria based on the position requirements.

|improve this answer|||||
2

I think this is a much better idea than asking questions such as "Where do you see yourself in 5 years?" etc. :)

Let me share a few points I find important:

  • Regarding the application under test. I'd personally choose one I know and one where at least some bugs could be found. That's because I'd like to see the candidate reporting issues or at least talking about (possible) issues as well.

  • The session should have a particular time slot, for your situation, I'd go for something in the range of 30 - 60 minutes.

  • Before even starting and/or presenting this to the candidate, I'd let them share their ideas about such exploratory sessions and how they think this could be done. You can find out from their answers if they know how to orginise such exploratory sessions.

  • During the session, let them talk. It's important because you can see how they communicate, how they go about exploration, how they discuss problems, how they ask. A tester should be able to communicate well, and this is a way to find out.

  • I'm a bit sceptical about quantifying the results. Why do you need to do this? Is it because HR asks you to? Just like exploratory testing cannot be fully quantified, and you don't want it to be, I wouldn't necessarily try to make the interview quntifiable. If you really need to quantify something, it shouldn't be the number of bugs. I'd rather create a checklist of imporatant things/skills I want the tester to know (and I can find out about during the exploratory session and interview) and count the number of checks I make with each candidate. Or you can create your own scale with well-defined levels of a skill. You probably might ask HR about how to go about this, I assume they have some experience in this area.

All in all, I think that seeing a candidate at work brings valuable information about how they fit in your team/company. Such a session is a short peek into exactly this very thing.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.