Is it a best practice to use loops for a test case to run it multiple times?

Question

When writing a test case is it a best practice to use loops for several test cases?
For an example consider following two code segments

Code 1

public class testing{

    public void testLogin(){
       fillCredentials();
       clickLoginButton();
    }

    public void updateprofile(){
       viewprofile();
       changeMobileNumber();
       clickUpdateButton();
    }
}            

Code 2

public class testing{

    for(int =0;i<5;i++){
        public void testLogin(){
           fillCredentials();
           clickLoginButton();
        }

        public void updateprofile(){
            viewprofile();
            changeMobileNumber();
            clickUpdateButton();
        }
   }
}        

Only difference in this code is I have used a loop a in second code segment. Is it OK to use loops over multiple test cases?(With the intention of testing multiple users continuously)

Assume all the methods are properly declared and no error in the code

When writing test cases like this using more loops it's some times easier to do testing. But the code gets complex.
This is my direct question, should I have to keep test cases simple as much as possible( by avoiding loops as much as I can) or is it OK to use loops and make the work done?

Answer 1

Apart from the fact your second piece of code will not even be compiled and also pretty much not representative since you're not showing us how exactly you would use the information of which particular iteration in the loop is currently going on, I would not recommend to use the loops in data-driven testing approach.

Much better is to use well-know test-execution frameworks like JUnit. They have parameterization mechanism, so you will be able to run your tests for the set of input parameters without increasing complexity.

There is a nice article from automationrhapsody.com on how to implement parameterization in JUnit. There is another from tutorialspoint.com that shows an example for testing.

Answer 2

Generally speaking I consider it a bad practice if you are using different data combinations to run UI tests.

I refer to this as data combinatorial testing.

Generally speaking I look to be sure that the UI works. That a user can tab through the fields, can pick one value from a dropdown menu, can get to the submit submit button page, etc.

I avoid different data combination to do this. If I found out a page where I am testing 5 different database entries - like 5 different logins, this is a clue that I need unit and/or integration tests. These are the 'data driven' tests that KDM refers to. These data combinatorial tests should exist as unit and/or integrated tests. The main reason at the end of the day is really speed. 50 UI tests might take 30 minutes to run. 50 unit/integrated tests might take 1 second to run!

The main exception to this is when I am checking a tests reliability and 'flakiness'. In this situation I use loops to run the same test multiple times. However this is a one-off data gathering exercise and does not describe the situation you are in.

Answer 3

If you're looping 5 times and the test fails, did it fail on the first occurrence? The last occurrence? Is it failing on all occurrences?

I like to have one assertion executed per test method if at all possible. Sometimes it's not possible because two properties have to be asserted at the same time.

Tests should use obviously good conditions, obviously bad conditions and as many non-obvious edge cases as possible. The next programmer (who could be you!) to read the tests should get a good idea of the desired behaviour just from reading the test setup and the expected results of each one. Looping make this less clear because you can't as easily trace through just one thing in all the data.

E.g. if you're testing date validation, you don't want to check 29 February in every year, just the ones that add value to the test:

• Years that are not multiples of four, e.g. assertFalse(is_valid_date('29/2/2018'))
• Years that are multiples of four, e.g. assertTrue(is_valid_date('29/2/2020'))
• Centuries that are not multiples of four, e.g. assertFalse(is_valid_date('29/2/2100'))
• Centuries that are multiples of four, e.g. assertTrue(is_valid_date('29/2/2400'))

Once you have these in place and they all pass, you only need to add further tests if bugs are found to ensure that a) they are fixed and b) they get caught if they ever revert.

This seems simple enough that they could be looped over, but what if you're validating a form with 10 inputs? It will be really hard to see what each value is for and why it should pass or fail validation. The same goes for performing calculations and any other sort of test.

Also, you should be testing different things each time, i.e. each test should have a different reason for passing or failing. The date validation example shows how the order affects comprehension. Most years are not leap years. Every fourth year is. Centuries are not leap years even though they are divisible by four. Centuries that are multiples of four are leap years even though most centuries aren't. Each test builds on the understanding of the previous test.

Grouping them by true and false (e.g. by looping over all true conditions, then all false conditions) makes the pattern harder to see. Why are we testing centuries that are multiples of four when all years that are multiple of four are already leap years? It's not until you get to the next set that the penny drops. Oh! I get it! It's because there's an exception between them! Usually centuries aren't leap years.

Good unit tests run quickly. If they don't, they wont be run very often, or at all. You should be doing the minimum that consistently adds value. Once adding another test no longer adds value because it's a repeat of a previous test, don't add it.

Answer 4

In the end it boils down to the personal preferences. Personally, I do not like loops in the test cases because of the following reasons:

1. Loops make the test case more complex
2. In most cases a loop can be replaced with a data driven test which is more readable
3. Loops break assert-for-one-thing thumb rule. I don't mean a single assert statement.
4. When a test fails, knowing the reason is more complicated.

Answer 5

It Depends

Testing in a loop is a best practice when you're testing for things that require a lot of repetitive testing to come up with a positive or negative result. There is no One Right Answer to Every Situation, but there is usually One Right Answer to a Specific Situation. Each scenario has to be examined on a case-by-case basis.

Test Once

When you're just testing the form to make sure you can log in, there's really no difference if you do it once or a thousand times. Either it works or it doesn't. This is what people mean when they say "insanity is doing the same thing over and over and expecting a different result." If you're testing something that's ultimately idempotent, test it once. If there are two possible branches (e.g. login and failure), test each one.

Test Lots

Sometimes you're out to make sure that the system is imposing reasonable limits or performs well under stress. You should probably be doing this. In those cases, it makes sense to log in a thousand times to see what happens. There's two particular categories here I can think of immediately.

Testing Limits

Perhaps a user is only allowed to log in 5 times an hour, perhaps because it would otherwise create too many sessions and bog down the authentication server. Or whatever the case may be. Run the test and try to log in 6 times immediately. Did it work? Limits testing is important when there are limits to test. Or maybe a user is supposed to be locked out after 5 failed attempts. Are they? Test the boundaries of your system to make sure that they are, in fact, boundaries.

Testing Resource Use

Some frameworks, libraries, languages, etc can leak resources over time. The two main resources that are likely to leak are memory and event handlers, although others are certainly possible, and if you know about any potential leaks, you should write unit tests once they're fixed as part of a regression test suite.

Memory leaks will eventually cause the application to consume enormous amounts of memory, and may result in the OS slowing down to fulfill the requests and/or terminate the program. If a person logs in three times, then, in theory, memory usage should be more or less the same than if they logged in 300 times. This can be especially important in a browser, as many users only close their browsers once every few months; the leaks actually could cause a crash.

Event handlers can also leak, in a sense; the same event gets registered over and over to a single element, causing duplicated (possibly harmful) behavior, and as they stack up, eventually the CPU will be pegged simply trying to service all the events, making it seem as if your app is frozen, even if it doesn't cause any other obviously harmful effects, like saving a file multiple times. In this case, it makes perfect since to loop a thousand times and check the resulting event handlers. There should be approximately the same number after the first login as the thousandth.

At the end of the day, use common sense, and think through the logic. Test no more than you need to, but definitely test as least as much as you need to. There is a balance, and using loops, or not, is a balance between compile/deployment times and catching otherwise obscure errors.