5

I have a question regarding a feature which is not working as expected because the KPIs decided are not reached. As we are working with machine learning, this is not a bug in the code but for example - missing training data.

Example: we want our algorithm to detect correctly a person in 95% of the cases but he detects correctly only 85% of the time.

Because it is not related to the code, I should not create a bug ticket but then what should I do? Should I create a task for the developer to improve his model?

And even more tricky what should I do in case of a degradation (our algorithm has an accuracy of 85% in version 1 and has 75% in version 2)?

Thanks already for your feedback!

Improve this question
1
  • What about F1 score?
    – dzieciou
    Feb 28, 2020 at 13:15

2 Answers 2

Reset to default
10

I don't understand why it would not be a bug, if it clearly does not match the expected result. See the following definitions:

defect: a shortcoming, fault, or imperfection

https://www.dictionary.com/browse/defect

A software bug is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways

https://en.wikipedia.org/wiki/Software_bug

Even if the solution is not a code change, it still should be researched, and actions could be taken. The test data and the model are also part of the computer system.

I always report unexpected degradation in performance or accuracy. Again you probably need to research it and decide if this the new acceptable baseline.

Should I create a task for the developer to improve his model?

Do you represent the stakeholders/business? Does the developer? If yes, decide if the model needs improvement. If no, consult the person(s) that use this feature. Do you have enough data to assess the risks? Don't make quick assumptions.

Reporting defects is not about right or wrong, following the specs or not, but about does this still adds the right value to our product, business, etc..

Keep these modern testing principles in mind:

  • Our priority is improving the business
  • We believe that the customer is the only one capable to judge and evaluate the quality of our product
  • We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
Improve this answer
5
  • Thanks for your answer. However based on past experience, I still have issue regarding creating a bug ticket because: * I know that we won't be able to close that ticket for a very long time (if we can close it). * For each class, I will have to create multiples bug tickets which will not be helpful for anyone. I think that our main issue is that we don't really define any intermediate / final acceptance criteria. i.e: it is acceptable to have a response time of 201ms for a requirement of 200ms but you will tell me that even in this case I should create a bug ticket, right ? =)
    – Damien
    Feb 28, 2020 at 12:56
  • 2
    No, I tell you to use commons sense from a business perspective, not blindly follow rules. :) Not creating defects because they never get priority is a real problem, consider implementing a zero-bug policy: medium.com/qualityfaster/the-zero-bug-policy-b0bd987be684
    – Niels van Reijmersdal
    Feb 28, 2020 at 13:06
  • 2
    Maybe you should try to do a root-cause analysis on why you do not think it is a good idea to create defects for these issues, because I get the feeling it is more complex than a yes/no answer. My assumption is that quality is at the end of the development cycle instead of the beginning. Try jamesshore.com/Agile-Book/root_cause_analysis.html
    – Niels van Reijmersdal
    Feb 28, 2020 at 13:09
  • 1
    @Damien I know the hardship of working in a place where there are no proper quality process . But as Niels recommended , discussing the issues , process improvement etc also comes under the duty of QA. And mostly , don't hesitate to raise issues , even if they remains open for ever , you have something to show when someone questions the team why this was not detected during testing . Emails doesn't help when it's time for blame game
    – PDHide ♦
    Feb 28, 2020 at 13:33
  • 1
    Thanks all for your answers, I do understand what to do now. And yes Niels, like you said this is not a yes/no answer, I have the confirmation when I ask to each colleague here around. There is even no answer at all ! I see that theorically I should raise a bug but practically it won't really be possible (in some cases). I will start with some strategy to deal with these cases and what i should do when it happen (root cause analysis, what if degradation, communication, tracking the problem,...) We also try to establish some intermediate objectives to reach but it's not always followed!
    – Damien
    Feb 28, 2020 at 14:35
3

A bug report is a document in which someone communicates someone else about a problem, possibly suggesting paths to correction.

Breaking it down:

1 - where someone communicates

That's anyone who is able to detect a problem, by using an oracle and an experiment process.

2 - someone else

That's anyone who can solve such a problem.

3 - a problem

That's an incompatibility between reality and an oracle.

So, if the developer is (2), then you can write a document to him/her showing how was your experiment and the oracle to use to detect a problem.

The developer may say your experiment was flawed by some reason, or maybe the oracle is not appropriated for some reason or the developer realizes that she/he can't figure out what needs to be done. That will be consequence of your document - and these conversations will only happen if you document your experiment.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.