2

I am teaching SW testing in a Computer Science department. As expected, I am conducting equivalence partitioning testing. For an exercise, I gave the class a set of 5 rules for code that grades password strength and asked them to list the equivalence classes for it. It's a simple set of rules; they are not that critical to my question - I am listing them here just so you see it's a rather simple set of rules:

  • The password score (grade) can be between 0 and 10 (10 is a strong password)
  • A password should be longer than 12 characters. The score is reduced by the distance from the actual length and 12.
  • A password should include at least one upper case letter, one lower case letter, and one number. The score is reduced by 2 for every infraction.
  • A password shouldn't be the words “password”, "admin" or "root". Violating this rule results in a score of 0.
  • A password shouldn't be the same letter or number repeated for its entire length. This deducts 7 points.

The resulting number of equivalence classes is... 20. This is after I added some restrictions to avoid the number being even higher.

So my question: If for such simple code, I need 20 ECs, for an extensive system, the ECs will be an enormous number - rendering the EP test technique not very helpful.

Am I, and all the SW test textbooks, overplaying the usefulness of this technique? Is it useful except for trivial examples?

4 Answers 4

2

for an extensive system

That's why we have the principles of modularization and composition as code quality criteria.

Conducting checking in a big module is not sound engineering.

Taking your own example, you probably would not create checks for the String library or the assert library - you are assuming they are bug-free (not really, but sound from an engineering point of view and your goal of checking your password evaluation code).

Going one step further, when you want to check your frontend displays an error when the password is weak, your check will just have an example - with a weak password - your password evaluation module quality is not the point of the frontend checks.

Dijkstra tried to promote the possibility of "proving" software, using structural analysis but admitted that for practical purposes is not useful.

Engineering != Math. Do what is necessary to build what you want to build. Use equivalence partitioning to fulfill your mission, not as a wall you have to overcome.

2
  • thanks - Your last sentence summarized is well. - Michael Apr 18, 2023 at 3:40
  • You are welcome, Michael. If that's the case, can you mark this answer as Accepted, so other's will know it satisfies the question? Apr 21, 2023 at 7:29
2

Ah the joys of combinatorial math.

As the other answers have implied, it's very easy for pretty much any technique to generate a massive number of test cases for any real-world application.

Equivalence partitioning is a way to group those test cases to help reduce the burden of testing. That doesn't change the fact that a real-world application is likely to have a large number of equivalence partitions - it's still easier to test (e.g.) that a password consisting of 12x the same character gets a low score than it is to test that a password of 12x0 gets a low score and a password of 12x1 gets a low score through every character allowed by the password input.

Regardless of the software, you're dealing with the fact that there are effectively an infinite number of pathways through it. In your example of a password strength module, the number of possible passwords may be finite, but with a minimum length of 12 and an unknown maximum length, it will be an unmanageably large number, particularly when you include all the potential invalid values. Compared to the many millions of possible password values, is 20 equivalence partitions such a burden?

2
  • 1
    Hi - Agree that EP can reduce the number of tests - but as you indicated, in many cases the number of ECs is practically infinite. In these cases, I think the value of EP is not so much in deciding the test cases, but it helps in thinking about the problem and how to partition it. -Michael Apr 18, 2023 at 3:43
  • 1
    @MichaelStahl - exactly. It's a tool to help think about the problem and how to manage it. When you're working with real software, that's as good as it gets.
    – Kate Paulk
    Apr 18, 2023 at 14:59
1

Equivalence partitioning is a powerful test design technique that can help testers develop test cases efficiently by reducing the number of test cases required to achieve a specific level of test coverage. It involves dividing the input space into different partitions, with each partition containing a group of inputs that exhibit the same behavior. For instance, when testing a password strength checker, a partition could be created for passwords with different lengths, passwords containing uppercase letters, passwords containing numbers, and so on.

While it's true that the number of equivalence classes can quickly grow for complex systems, the technique remains useful. It can be combined with other test design techniques such as boundary value analysis, decision tables, and state transition diagrams to generate a comprehensive set of test cases that provide thorough test coverage.

In the case of password strength checker, for example, it's possible to combine equivalence partitioning with boundary value analysis to identify test cases that check the behavior of the system at the boundaries of each partition. Additionally, decision tables can be used to identify different combinations of inputs that lead to different outputs, while state transition diagrams can be used to capture the behavior of the system under different conditions.

Therefore, it is not accurate to say that the equivalence partitioning technique is not helpful for extensive systems. When used in combination with other test design techniques, it can help testers generate a comprehensive set of test cases that provide good test coverage. However, it's essential to recognize that no test design technique can generate a perfect set of test cases, and manual testing should be used to supplement the automated tests.

Reference:


Edit:

Hi Michael, thank you for your follow-up question. It's true that for complex systems, the number of equivalence classes can grow exponentially, which can make it challenging to apply the technique. However, this is not unique to equivalence partitioning but is a common issue with most test design techniques. The complexity of the system under test is often proportional to the number of test cases required to achieve adequate test coverage.

One way to address this challenge is to prioritize the equivalence classes based on their significance and the risk they pose to the system under test. By doing so, you can focus on the most critical test scenarios, which can help reduce the number of test cases required.

Another approach is to combine equivalence partitioning with other test design techniques, as I mentioned above. This can help you generate a more comprehensive set of test cases while still keeping the number of test cases manageable.

Overall, it's important to recognize that no test design technique can guarantee complete test coverage, and a combination of techniques and exploratory testing is often required to achieve the desired level of quality. Equivalence partitioning remains a useful and widely adopted technique that can help testers develop test cases efficiently and systematically.

I hope this answers your question. If you have any further questions or concerns, please feel free to ask.

2
  • Hi Milin - thanks for responding. Most of what you write is basically what the textbooks say; I am of course aware of BV testing, decisions tables etc. Still - if for a simple example I end up with 20 ECs, then for any real system will have thousands of ECs, most of which I can't even imagine... so how realistic is this technique? MIchael Apr 18, 2023 at 3:01
  • Hope the addendum in the answer helps you. Apr 18, 2023 at 4:29
1

If for such simple code, I need 20 ECs, for an extensive system, the ECs will be an enormous number - rendering the EP test technique not very helpful.

Am I, and all the SW test textbooks, overplaying the usefulness of this technique? Is it useful except for trivial examples?

First, great question! And I wish more people would teach these topics in Computer Science courses. My CS courses never did!

No, you are not overplaying the usefulness of ECs. Even when you use it in combination with other techniques, like Boundary Value Analysis, these techniques can be very powerful. I've often taught these techniques to the dev teams I've worked on.

To me, what perspective are you coming from? A lot of devs dislike anything related to testing, so they throw out phrases like:

  • "Testing is easy"
  • "If it works, it works"
  • "Why do we need tests? Users can find the bugs."
  • "It works on my machine"
  • "It works according to spec"

And a lot more phrases that escape me but are just as diminutive and minimizing in nature. I think all testers have heard phrases like these.

Are you trying to achieve "perfection?" Some level of exhaustive testing? Do you want to test via brute force? Trying to find a "magic number" that proves the software is completely tested (like O notation)?

Are you thinking testing is a bottleneck and takes too long? If we can't find all the tests or all the bugs, then we shouldn't do any testing?

Keep in mind, you'll never reach perfection and it's impossible to do exhaustive testing as it takes too much time!

The ultimate purpose of these techniques is to help reduce the amount of testing done while still proclaiming the software is tested at a reasonable coverage level. They are used to make the complex more simplified. These techniques speed up test cycles.

These testing techniques are engineering techniques. Are they simple to understand? Yep! So is a for loop and those get used all the time! I had CS professors that professed that "simple code is better, more secure, more performant, etc." Why make a technique more complicated when it works?

I'll leave you with this: Think of Testers like they are Editors (book editors, newspaper editors, film editors, etc)!

It's the job of a tester to ensure the final product matches the objective and goal; ensure consistency; features are functional by themselves and in combination with other features; to find mistakes; to ask questions, etc

Would you want to read a book or watch a movie without editing?

2
  • Hi Lee. What I read from your answer is that (a) EC is a powerful tool and (b) that it helps reduce the amount of testing done. Right. This is what I teach... but if this technique ends up with (a) long time spent to REALLY understand the correct partition to ECs, and (b) end up with a very large number of ECs... I may as well go with "run tests based on the expected use" (which is what you would do after doing the EC analysis for hours, end up with 300000 ECs and realizing you can't test them all). Apr 22, 2023 at 19:17
  • @MichaelStahl It sounds like your argument is very academic, which is fine! There's a place for that. In practice though, EC is rarely used by itself. It's commonly paired with another technique, usually BVA, to help reduce the partitions further. Are you teaching EC in combination with other techniques? Also, because in practice we have time constraints, we aim for as much test coverage as possible, fully knowing we can't guarantee we've tested it all (software is complex) nor found all the bugs. We have to get to a point of "it's good enough for most use cases."
    – Lee Jensen
    Apr 26, 2023 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.