Is asking about "The most interesting bug in your career" a good interview question?

Question

We've been interviewing for quite some time and I've been asking candidates:

What is the most interesting bug you've found in your career?

Either the most interesting, or the one they are proud of finding, or because it was challenging to reproduce, or the symptoms were unusual.

So far, not a lot of people could actually recall something during the interview. It is understandable that sometimes it's difficult to recall something not from the top of the head under the stress of an interview. We are not pushing hard on this, saying that it's okay if you cannot remember one now.

My thinking is that we can get some signal from the answer - it may provide insights into how critical, serious and diligent one is about the detected issues and give some idea about the ways of candidate's exploratory thinking.

But, is it overall an appropriate question to ask, is our motivating logic correct? Should I modify it or build on top of it adding some similar questions?

---

Addressing a comment:

what would be your model answer?

Not sure about whether I can give a good "model" answer myself, but, I would highlight a few of the bugs and problems I've helped to discover trying to give samples from different projects and types of issues:

• interesting UI issue, this one is a bit silly, I understand - but may help to "break the ice" during an interview. Once I've discovered that clicking on a certain button in one of our web applications behaved weirdly - the more often you've clicked the faster is started to rotate around its initial position - not sure if it violated the Kepler's laws of Orbital Dynamics but that was a fun one
• lack of proper validation in an input field. There was a "comments" field in one of our screens that were allowed to be edited. But, as I've discovered - it was vulnerable to HTML injection attacks - the HTML you've saved into the field would be interpreted when the field will be displayed. And, it would be interpreted for any user that would "see" this field. That was a major security flaw since it allowed to inject javascript as well.
• SQL injection attack. Using sqlmap and manual testing with PostMan, I've discovered that some of our API endpoints were vulnerable to SQL injection attacks. This allowed me to get access to the information I was not supposed to see and in other case put a serious load on the backend service (making it full scan a huge table)

I have some more examples from my experience along this line, but this is probably something I would love to hear in response. This would show that a candidate has experience with different types of testing, cares and knows about where lack of proper user input validation may lead, can and use tools to help out him/herself in testing, is capable of looking under the hood when needed...

But, I clearly understand, that is not an easy thing to do under the pressure, stress and time constraints of an interview.

Answer 1

I would push back hard on this question.

An interview question is a machine designed to extract a signal from a candidate. Let's examine the parts of this machine.

---

"The most" has already been commented on. Why is it important that the answer be the most interesting? Why are you asking the candidate to solve an optimization problem in your interview? If they give you the second-most interesting bug, have they failed? When I ask a coding problem asking for the most something, I expect the maximizing item to be produced. These are computer professionals you're talking to; don't make them solve optimization problems unless you are testing them on solving optimization problems.

---

Why "interesting"? What is so important about a bug being "interesting"? "Interesting" is vague. Interesting is a predicate that has an argument that you have not supplied: interesting to who? You? Now you're asking the candidate to read your mind. To them? Now you're testing them on their ability to find things interesting. People typically find "interesting" things that are out of the ordinary, so now you're asking about completely atypical work. Why is asking about rare, atypical work high signal for you? Shouldn't you be seeking signal on typical work?

What signal are you attempting to extract with "interesting"? Why is that signal important to you?

Also, whatever their answer is to whatever your question is, the answer will be something they remember. People are more likely to remember interesting things, recent things and important things, so responses are already going to skew towards interesting and important. There's not so much need to call it out.

---

"bug". Why bugs? The question presupposes that the purpose of QA is the discovery of defects. It is not; the purpose of Quality Assurance is... wait for it... assuring quality on behalf of the customer. There are lots of ways of assuring quality that do not fall into the category of discovering defects. There are lots of defects discovered that do not affect customer experience.

I've discovered lots of interesting (to me!) bugs in compilers that have no practical impact whatsoever on the quality of the end product, since they are in scenarios that no sane developer would encounter. What signal would you derive from my detailed exegesis of a bizarre and interesting bug that no customer will ever be affected by?

---

"you found" -- again, we are committing the error that the job of QA is to find defects. And why is it important that the candidate found the bug? There are lots of interesting defects that led me to hone my craft that were found by other people. Is the signal you're attempting to extract here ability to find interesting defects?

I have had people on my team before who were hired specifically because of their proven ability to find unusual defects. These were specialists who had deep knowledge of compiler technologies who were specifically hired to find the sorts of defects that the usual QA team would not. Are you looking to hire such a specialist? They are expensive, and they are not particularly useful for run-of-the-mill QA tasks; that's a waste of their talents.

---

"in your career" -- the question presumes first that the candidate has had a career; the question is not suitable for entry level. The question also presumes that the signal is to be derived from the specific work that the candidate has performed. Most of the defects I've learned from in my work were discovered by someone else, or discovered by static analysis. What is the signal you hope to get by asking the candidate narrowly about defects that they discovered in their career?

---

The TLDR is: your question sounds like you're asking for a "war story" to break the ice; it is not clear that it elicits a good signal.

Now that I've torn apart every word in your question: what would I prefer? I'd prefer to break it down into questions each of which elicits clear signal:

• Describe for me a software defect that you learned something important from. What was the defect, and what did you learn? Signals: what does the candidate think is important? Are they learning from their work? Example: I remember the first bug I ever found as a full-time Microsoft employee; the bug was in an API had been bugged and marked "fixed" six times already. There were six if statements in the code, each one handling a special case! I learned a lot from that bug that I still apply every day: check the bug history, look for code that is bug farms, examine the root cause and address it, rather than writing a tailored patch just to make a test case pass, don't just write a regression for the bug that was fixed; look for more bugs, and many more.

• Describe for me a defect that you thought was important to fix. What was the defect? What factors made you think it should be prioritized? Signal: Is the candidate focussed on QA's mission to be a quality advocate for the customer? What do they prioritize against? Do they consider the cost of the fix vs the harm to the customer? Is there a good cost-benefit tradeoff? How do they deal with balancing the risk of introducing a new defect with fixing an existing bad defect?, and so on.

And so on. Don't just ask for a war story. Don't ask for a solution to an optimization problem. Figure out what signal you are really trying to elicit, and ask directly about it.

Answer 2

To be honest, I wouldn't be able to come up with any defect in particular.

If I were you, I'd rephrase my questions as situations, for example:

• What actions would you take if you had to reopen a defect for the fifth time?
• You need to retest a defect which has been resolved by a developer without any further comments or explanation. What actions would you take, if any, besides retesting?
  • Personal favorite answer: looking at the fixed code via the changeset
• What do you consider a good plan of attack for a defect that's almost never reproducible (but does sometimes occur in production environment)?
• How would you most efficiently test a field that should become red on 30 out of 500 possible values?
  • A good answer for me would be: testing one value in UI to see if the system works, and then check the list of 30 values in database/code/config.

Answer 3

I am terrible at recalling names, places, restaurants, and... bugs I have investigated. I usually ask my girlfriend or my friend to give me a name of a place we have been together or an actress in a movie with have seen. That's how my brain is wired. I don't like it but I get used to it. I guess some candidates might be like that.

But if you task me with investigating a bug during an interview -- then I might demonstrate my skills.

Some people might be better at doing things than describing them.

Answer 4

I was asked a question like this at my last interview, and it took me a bit of time to think back and come up with a good answer. This was partly due to a dearth of experience - at that time I had only held internship positions.

After I read your question, I repeated the exercise, trying to come up with my favorite bug in a span of time that would be acceptable for an interview response. I was easily able to recall some interesting bugs from the last week or two, but it took a bit longer to remember the details of anything beyond that. Of course, this exercise was done without the pressure of an interview environment.

I think the question certainly has value in that it can provide insight about a candidate's curiosity. However, it may need to be tailored to the candidate. For example, someone with at least a few years of experience in the industry should have no problem answering, even if it takes them a minute to think about it.

If the candidate is a new CS grad, perhaps it could be asked in a way that would lead them in a direction that is relevant to their experiences: "Tell us about a time during development for school or a side project where you were stumped by a bug or defect, and how you eventually solved the issue."

Answer 5

I don't know if I'm just too literal, but when I get asked these sort of questions, I get hung up on the word "most". It's not too hard to come with a bug, but making sure that I've never seen any bugs that were more interesting is a tall order. So my advice would be to get rid of the word "most". Also, if you're looking for general aptitude rather than specific software experience, you can broaden your question to design flaws in general. You can also give them an example of a bug you've found (or a made up one) and see what suggestions they ask and what questions they ask.

Answer 6

I would say such a question might spotlight how deep the candidate understands the technologies, analyze the root-causes and is able to troubleshoot issues.

As per my experience I can remember several interesting defects, but I consider them interesting because I did find the root-causes of them in code, or in environment or in human-factor etc.

If a candidate just found and reported the defects (but didn't try to figure out why those defects existed) those defects would unlikely have any special properties to hook one's mind.

Answer 7

I think it's an excellent question. I think it's likely to help you understand:

(a) what kind of technical challenges the interviewee is accustomed to dealing with. (Do they choose a programming problem, a human interface problem, or a systems architecture problem? Do they interpret "bug" to mean not meeting the spec, or not meeting the business requirements?)

(b) what level of personal responsibility they assume for quality and defect prevention. (Do they take ownership of the problem, or was it someone else's fault?)

(c) how deep their analysis goes of the root cause of the problem (were our processes at fault? How should we have prevented this happening?)

If the question were thrown at me at interview I would have difficulty choosing from half a dozen incidents that quickly come to mind, but whichever I chose, I think the question would give me a good opportunity to present my particular skills and experience in a favourable light.

I'm answering this from the perspective of a developer / designer / architect, not a QA tester. But I still think the answers would be informative.

Answer 8

If you aren't getting good answers, then perhaps you can modify the question slightly to help the interviewee. The word "interesting" can be interpreted in so many different ways, and perhaps that is where your candidates are getting stuck?

Maybe you are looking for the "hardest to find" bug, maybe the "hardest to fix" bug, maybe the "most off the wall" bug. Maybe it is really a chance to "discuss some of the nastiest, weirdest bugs you've found". Is it your own bug or someone else's?

Back in the days of C++ standardization, I submitted a few bug reports to companies who build the compilers (and got bug fixes, even). Those were definitely "interesting". But perhaps you want to know about my bug in FORTRAN when I just started and was trying to write a scripting language so our users could run scripts in our product? Or the bug I found in the IBM tape drives embedded software (pre-release, of course)? Or that nasty one which turned out to be a certain sequence of clicks and keystrokes that a particular user was doing differently than anyone else? All these bring up different aspects of my skills.

Of course, my first reaction was to figure out a bug that would let me bring up one of my talking points for the interview (either one I hadn't gotten to yet or one I felt needed reinforcing), so depending on when you asked that question, you might get a completely different answer from me. Because I can model my answer on talking points, you're not going to learn anything about me that I don't want you to know, or haven't already told you. All thanks to that word "interesting".

Answer 9

It's a great question because it separates code copiers/script kiddies from actual software engineers.

Example: A well-known, very expensive piece of analytical software owned by a large company (I won't say their name, but their initials are I, B and M) would refuse to allow drill-down access to data via SSL.

The company isn't going to but another service contract, etc. and the programmer is stuck trying to solve a problem with 3rd party compiled code. Despite the documentation, nothing works. And yes, it must be solved due to law.

There's nothing a script kiddy can do with this. There's no copy/paste code in stack exchange to fix this. All the Scrum and Agile skills someone has won't do a darn thing to fix this.

But a programmer - an engineer - would say that he duplicated the environment and looked at the server logs and saw the errors that weren't making it to the screen, and that he confirmed these errors on the production logs. The programmer would say that he decompiled the code and found the IF statement handling the protocol. The programmer would see how the IF statement was failing to properly handle the HTTPS protocol and implemented a workaround outside of the code: use URLs with port numbers that were encrypted (other than 443) and configure the web server accordingly.

Which one do you hire?

Answer 10

I think it is an appropriate question to ask in an interview. I would restructure the question as what is the most interesting bug that you or your team found and what's the lesson learned? That way it may not be just the individual who found it but he is also learning from other people in the team.

I don't remember the most interesting bug I found and fixed, but I remember several of them we found and fixed as a team.

Answer 11

I've had a long enough career that I've seen and fixed many defects. It takes a real whopper these days to pique any genuine interest beyond that appropriate for efficiently characterizing and correcting the issue. To the (miniscule) extent that I reminisce on bugs of the past, I tend to see them from my present-day perspective. Thus, although I account myself fairly good at tracking down and squashing bugs, but I would have trouble answering the question.

Additionally, inasmuch as of late I have usually been a team of one, the issues freshest in my mind are are ones that I was responsible for creating myself. In that light, the question has a bit of the flavor of "What's your greatest weakness?" I really don't want to tell an interviewer all about what a great mess I can make, even in the context of explaining how good I am at cleaning up after myself (and others).

Answer 12

Yes, this is a good question to ask.

If the person you are asking is really fond of testing he would be very excited to tell you the story. It does not matter whether he is a Manual Tester or an Automation Engineer, both should have something to be proud of (but we should consider that he is telling the story from his point of view, I mean this bug could be not that impressive for a Automation Engineer but a kind of great bug among Manual Testers team)

And here is my example: memory leak defect.

When I had just been employed there was a problem in software: sometimes it crashed in an unpredictable manner (sometimes during one action, sometimes during another action). At that moment I was testing mainly with QTP (now UFT) and started to try Java-tests. The advantage was great: I could check JVM-memory during execution and dump memory usage right at the crash time (with MemoryAnalyzer-1.6.0). After analyzing memory it was clear that there were 8 objects that were duplicated many times during program execution. So I reported defect with certain object names. Devs liked that because it was supposed that there is nothing to do with this unpredictable crashes and localization of the bug was of a top level.

Answer 13

I think the answers and comment are fulfilled the idea and this question.

I am thinking that this question has to be change in other way, Even in this condition that I am sitting in home and stressed out, I can even hardly to answer to this question, it is even hard to understand what is hat asking for... I guess it would better if you had a question that were mostly related to the situation not condition because:

1. Our mind are in that way so it is visualizes everything, meaning, we are tend to remember anything from our visualize memory not every word or sentence from that condition, so if we were in the specific situation we will more likely to remember it!

I guess you got the point, this is the way we think and the question has to be more specific and as simple as possible as well.

Answer 14

Interviewing testers is very tricky, the ones you want are the ones that are interviewing you.

Be aware that client confidentiality will prevent them from telling you about their most juicy bug, usually only beer will get you that but I wouldn't recommend it in a interview.

I would instead focus on questions that allow you to focus on comparison of candidates results , anecdotes are far harder to compare than straight questions.

A technique ive used with success in the past is asking questions that lead people to STAR answers, How To Master the STAR Method For Interview Questions.

A good tester should pick up on this pattern and you can see if they bothered to research for the interview.

Getting people to write code for interviews is only really suitable for developers where it matters that they are sharp on algorithm design for speed and extensibility. Testers would just look this up because its reliability and accuracy that is more important. Ask testers how they know and where they learn.

Answer 15

Great question for seniors, maybe not so good for junior testers. I think being able to explain risks to managers, developers and other testers with some good bug stories is mandatory for seniors. If you cannot teach from experience than you are probably just a medior.

Just finding the defects is only part of the job, understanding how they happend and preventing similar issues in the future is just as important.

Some stories that come to mind:

• The missing where clause: After the first time I ran into an defect which deleted/updated more data than was expected I started to using database-diffs, during my testing steps. Making sure no more or less data was updated than I expected. I have a good story how data was lost in production, which we spotted during testing but could not reproduce. Afterwards it was a complex relationship issue, where in some cases the where-clause could be missing, updating thousands of records. If only I had database-diffs during my testing sessions I could have proven the issues, although I could not reproduce it at the time.
• The admin password in plain-sight: I did a basic security test on our Citrix hosted application. After some hours I was logged into the main control-server. First I could escape the application into PowerShell, from there I did a search on all files and network share for the word password. Seemed the unattended install scripts did not clean-up after themselves. I am not a great hacker, I just followed a how to break into Citrix guide from the web.
• It works, but it isn't what the client needed: I have lots of stories where we got defects reports from the field. We build what was "required", but when used it was faulty. The case with the label printing was one. We added a new form of printing labels, but they needed two labels, but after printing the first label we would hide it in the systems. Now it was impossible to print it again, unless the whole work-order was processed again, leading to lot of manual labour. Easy fix, but the client didn't want to upgrade, because they just started a high risk maintenance job of two weeks.

I think I can come up with lots more of compelling defect stories, but I am sure I could NOT come up with a new one on the spot under pressure. Maybe my most recent found defect would come to mind.

Still it seems fine to ask an interviewee to prepare a good defect story before he/she comes in. Testing their story-telling skills, but also how they handled the situation. Differencing between click-monkeys and people who investigate mysteries and grow from them.

Answer 16

My ready to go answer for something like this isinteresting, in that, from a support/faultfinding perspective, it's an interplay between software and user experience, and thinking about how users actually do things, rather than the "this is how it should work" perspective.

Trying to replicate an observed user fault in one screen of the software at issue, we just couldn't get the error to happen. We remoted in, went through their replication steps and it worked fine every time. We watched them do the process, and it also worked every time. Yet every time they were left to their own devices, it would eventually throw the intermittent error. Finally, thanks to someone actually being on site and observing without the user being directly told we were watching, we caught what was happening. It turns out the end user, being so completely au fait with the software, was typing, flipping through screens, options and keyboard shortcuts significantly faster than almost anyone else could manage. The program in place wasn't quite able to keep up with the user input, and this exposed a couple of bugs in input handling. The kicker was that every time we watched them, the user would slow down some, both consciously to make sure we could see what they were doing, and unconciously taking more time and care, to avoid the insinuation they were doing something wrong. So of course it never did it when they knew we were watching.

Answer 17

Most interesting Defect(IMHO) is Lack of team communication

As people already have given excellent answers about whether is it a good question or not so I will not go into it but share one interesting experience around it.

I remember from a long time back during one waterfall project , I came upon a minor issue(which was hardly touching the real iceberg problem) but which sparked a lengthy discussion in team meeting which exposed an interesting revelation that a functionality was entirely functional and working fine as per the requirement but the actual usage scenarios were entirely different than what developers & testers envisioned.

Finally, it was found the screen business scenarios and user input were entirely different and the current implemented feature was fully functional and tested but entirely useless for the business.

The lesson I learned from that (from quality perspective) ,above any technical maturity in a team is "team communication".

*A well communicating team can prevent lot of defects in the first place itself before implementing anything.