Pareto Efficiency in Software Testing

Question

I've stumbled upon the Pareto Principle and Pareto Efficiency terms used in different contexts.

What does Pareto Efficiency mean in the context of software testing? My understanding is that, it relates to a sufficient number of test cases that is enough to examine functionality of a system under test. Is my understanding correct?

A bonus question: is the Pareto Principle related to Property-based testing concept that allows you to specify the "scope" of possible inputs instead of a set of example input test cases?

Answer 1

Pareto Principle, originally described by Vilfredo Pareto and later formalized by Joseph Juran. In software testing this principle here is just a rule of thumb, but an important one. 80% of errors being concentrated in 20% of the developed product functionality. So errors in the software (modules, functional business processes, etc.) are distributed unevenly.

the percentages could be anything like 70/30 or 90/10, the reality is that most things are caused by a few underlying factors.

Not only, it relates to a finding a sufficient number of test cases that is enough to examine functionality of application or smart test case selection.

But For QA in software testing company, knowing this fact can offer tremendous value. If a tester is simply looking at a list of 100 bugs, it may not be clear if there is any underlying meaning. But if the tester were to combine those bugs based on some kind of category, it may be possible to see that a very large number of bugs come from very few places.

Here are a few recommendations for getting the most out of this principle:

For example, 80% of the program's bugs may result from calling the same underlying library. However, that may not be readily apparent from where the bugs occur within the program. Remember that bugs may result from flawed procedures. For example, a large number of bugs could be present because a developer is using out of date specifications.

This principle can be tremendously powerful in reducing the bug counts within a program because solving just a few things can make a program much more stable.

Answer 2

Pareto efficiency in software testing often refers to a smart test case selection. Let's say there is not enough time to run all test cases (e.g. before a hotfix release), then you can leverage the Pareto principle to reduce the overall risk by addressing multiple objectives. Have a look at "Pareto Efficient Multi-Objective Test Case Selection" by Shin Yoo and Mark Harman:

Previous work has treated test case selection as a single objective optimisation problem. This paper introduces the concept of Pareto efficiency to test case selection. The Pareto efficient approach takes multiple objectives such as code coverage, past fault-detection history and execution cost, and constructs a group of non-dominating, equivalently optimal test case subsets. The paper describes the potential benefits of Pareto efficient multi-objective test case selection, illustrating with empirical studies of two and three objective formulations.

Regarding property-based testing: I think they are orthogonal as you can combine both.

Answer 3

Risk/Priority Based Testing: 80% Of Users use 20% key functionality of any application under test.

Answer 4

I am extremely glad that you brought this question and I am feeling so excited to define the term. Pareto efficiency or Pareto principle in software testing defines that 80 percent of the software defect can be identified from 20 percent of test modules. This means if you are repeating the same test cases, it will not bring you any new errors, bugs, or defects. However, if you need to locate the remaining or new bugs within the given system, then you may need to review the existing test cases or work on adding or updating new test cases.