Skip to main content
Software Quality Assurance & Testing

Return to Answer

added 322 characters in body
Source Link
Eric Lippert
Eric Lippert
  • 1.3k
  • 1
  • 8
  • 8

Also, whatever their answer is to whatever your question is, the answer will be something they remember. People are more likely to remember interesting things, recent things and important things, so responses are already going to skew towards interesting and important. There's not so much need to call it out.

Also, whatever their answer is to whatever your question is, the answer will be something they remember. People are more likely to remember interesting things, recent things and important things, so responses are already going to skew towards interesting and important. There's not so much need to call it out.

edited body
Source Link
Eric Lippert
Eric Lippert
  • 1.3k
  • 1
  • 8
  • 8

I've discovered lots of interesting (to me!) bugs in compilers that have no practical affectimpact whatsoever on the quality of the end product, since they are in scenarios that no sane developer would encounter. What signal would you derive from my detailed exegesis of a bizarre and interesting bug that no customer will ever be affected by?

I've discovered lots of interesting (to me!) bugs in compilers that have no practical affect whatsoever on the quality of the end product, since they are in scenarios that no sane developer would encounter. What signal would you derive from my detailed exegesis of a bizarre and interesting bug that no customer will ever be affected by?

I've discovered lots of interesting (to me!) bugs in compilers that have no practical impact whatsoever on the quality of the end product, since they are in scenarios that no sane developer would encounter. What signal would you derive from my detailed exegesis of a bizarre and interesting bug that no customer will ever be affected by?

Source Link
Eric Lippert
Eric Lippert
  • 1.3k
  • 1
  • 8
  • 8

I would push back hard on this question.

An interview question is a machine designed to extract a signal from a candidate. Let's examine the parts of this machine.

"The most" has already been commented on. Why is it important that the answer be the most interesting? Why are you asking the candidate to solve an optimization problem in your interview? If they give you the second-most interesting bug, have they failed? When I ask a coding problem asking for the most something, I expect the maximizing item to be produced. These are computer professionals you're talking to; don't make them solve optimization problems unless you are testing them on solving optimization problems.

Why "interesting"? What is so important about a bug being "interesting"? "Interesting" is vague. Interesting is a predicate that has an argument that you have not supplied: interesting to who? You? Now you're asking the candidate to read your mind. To them? Now you're testing them on their ability to find things interesting. People typically find "interesting" things that are out of the ordinary, so now you're asking about completely atypical work. Why is asking about rare, atypical work high signal for you? Shouldn't you be seeking signal on typical work?

What signal are you attempting to extract with "interesting"? Why is that signal important to you?

"bug". Why bugs? The question presupposes that the purpose of QA is the discovery of defects. It is not; the purpose of Quality Assurance is... wait for it... assuring quality on behalf of the customer. There are lots of ways of assuring quality that do not fall into the category of discovering defects. There are lots of defects discovered that do not affect customer experience.

I've discovered lots of interesting (to me!) bugs in compilers that have no practical affect whatsoever on the quality of the end product, since they are in scenarios that no sane developer would encounter. What signal would you derive from my detailed exegesis of a bizarre and interesting bug that no customer will ever be affected by?

"you found" -- again, we are committing the error that the job of QA is to find defects. And why is it important that the candidate found the bug? There are lots of interesting defects that led me to hone my craft that were found by other people. Is the signal you're attempting to extract here ability to find interesting defects?

I have had people on my team before who were hired specifically because of their proven ability to find unusual defects. These were specialists who had deep knowledge of compiler technologies who were specifically hired to find the sorts of defects that the usual QA team would not. Are you looking to hire such a specialist? They are expensive, and they are not particularly useful for run-of-the-mill QA tasks; that's a waste of their talents.

"in your career" -- the question presumes first that the candidate has had a career; the question is not suitable for entry level. The question also presumes that the signal is to be derived from the specific work that the candidate has performed. Most of the defects I've learned from in my work were discovered by someone else, or discovered by static analysis. What is the signal you hope to get by asking the candidate narrowly about defects that they discovered in their career?

The TLDR is: your question sounds like you're asking for a "war story" to break the ice; it is not clear that it elicits a good signal.

Now that I've torn apart every word in your question: what would I prefer? I'd prefer to break it down into questions each of which elicits clear signal:

  • Describe for me a software defect that you learned something important from. What was the defect, and what did you learn? Signals: what does the candidate think is important? Are they learning from their work? Example: I remember the first bug I ever found as a full-time Microsoft employee; the bug was in an API had been bugged and marked "fixed" six times already. There were six if statements in the code, each one handling a special case! I learned a lot from that bug that I still apply every day: check the bug history, look for code that is bug farms, examine the root cause and address it, rather than writing a tailored patch just to make a test case pass, don't just write a regression for the bug that was fixed; look for more bugs, and many more.

  • Describe for me a defect that you thought was important to fix. What was the defect? What factors made you think it should be prioritized? Signal: Is the candidate focussed on QA's mission to be a quality advocate for the customer? What do they prioritize against? Do they consider the cost of the fix vs the harm to the customer? Is there a good cost-benefit tradeoff? How do they deal with balancing the risk of introducing a new defect with fixing an existing bad defect?, and so on.

And so on. Don't just ask for a war story. Don't ask for a solution to an optimization problem. Figure out what signal you are really trying to elicit, and ask directly about it.