11

We have a large set of test cases but limited time and resources to execute/test them all. We thought but it's not possible to execute/test everything.

I want to know strategies which will be beneficial in the above situation to cover more tests with limited time and resources.

Can someone suggest any strategy or way to move ahead for the above situation?

  • @AlexeyR. No, that doesn't answer my question. My question is to execute test cases manually. We don't have automation in place for those test cases. – JAINAM Feb 17 at 13:10
  • What did make you think that question is about test automation? – Alexey R. Feb 17 at 13:45
  • @AlexeyR. Have a look at the question and it's answers. You will understand how both questions ate different. – JAINAM Feb 17 at 15:27
  • Yes, indeed. I missed that it tagged with automated testing tag. However I still believe there is no much difference between how to effectively pick subset of automated tests or subset of manual tests. Both imply techniques of picking tests which maximize the coverage and minimize execution time. – Alexey R. Feb 17 at 15:45
  • 2
    @AlexeyR. I believe picking set of test cases for automation tests and manual tests are not always the same. As most of the time automation tests are only of stable modules, and majorly for regression test. And I'm not trying to pick set of test cases for regression testing. So, I hope this will help you understand how both questions are different. – JAINAM Feb 18 at 9:10

11 Answers 11

10
+50

Good question. Here I am going to explain some steps from my experience.

1) We need good teamwork for that.

2) Here I just want to clarify that "Execute All Test Case/Suite" term. We need to prioritize test cases in four quadrants like below.

Using a Priority Quadrant

3) Do a good amount of Exploratory Testing instead of totally scripted approach.

4) Try to cover all business-critical flow first.

5) Try to test all previous production defects and cover in the current regression cycle.

6) Give yourself some time for proper planning before starting with a sequential approach. What type of modules and test cases you can eliminate first and remove it first so now you have a list of all important tasks. Again use Priority Quadrant [Step-2]

6) And last but not least. Again we need Step-1 :)

Always be open for feedback and suggestion.

| improve this answer | |
  • 2
    It's a good approach initially but over time it leads to an over-growing pile of 'should but not important enough' and although each individual item was not important, the accumulated effect of not doing them leads to a pile of technical debt that slows you down. And then a competitor comes in. It is a incredibly hard problem. – Michael Durrant Mar 12 at 8:22
7

Basically you never have enough time and resources to test everything, your test cases are already a subset of this infinite "everything".

What should you do then ? prioritize. A common heuristic is RCRCRC:

Recent: new features

Core: essential functionality of your product

Risk: risk is defined as the Probability of occurrence (how likely is it to happen) multiplied by the affect (what's the damage, in money hours lost or anything relevant), Risk hides under it a lot and can be calculated at different levels from module to system to environments.

Configuration sensitive: this represents both internal configuration and environment settings, for example the type of the device or operating system

Repaired: recently repaired bugs represents untested area

Chronic: some areas are known to be vulnerable. This can be due to complexity, the level of the developers or an area that falls in between responsibilities

Take you list of test cases and grade them for each item, let's say 1 - 5 (don't use zero since it will interfere with the next step), multiply then the six figure and sort by the result. This will give you a first estimation as to what should be tested first and why.

| improve this answer | |
5

Feedback also from my side what I learned from projects.

1. Prioritize the test cases

In my past projects we prioritized the test cases. We used HP ALM and there we had also couple of test cases and it was impossible to execute the all. So what we did is just to prioritize the test cases e.g. Critical, Very High, High, Medium, Low - same as you would do it for defects. All created test cases were based on release policy which we also created. This helped just to focus on the most important test cases

2. Involve others in testing scope e.g. Product Owners and shift some test cases to them

Involve the otherst in the test team. Testing doesn't mean that you just test alone with your testing mates. It means that you can also involve other team members. In our project we hence involved also the Product Owner for testing. And not just at the end (e.g. when deployment is done and UAT should be done via business department and Product Owner). In Janets book I found the hint that UAT is done mostly after post deployment. and that tis does not mean that it has to be left to the end game (Janet Gregory "More Agile Testing" page 202). So when testing a product of course you can involve other stakeholders to test a product. In our case our Product Owners helped to test some test cases! By the way, you also can ask UX department to support you. We asked also our UX department to support us to execute some test cases and we got also valuable feedback from them.

Organize somehow Mob testing and act as test coordinator who throws test cases to them!

You can also try to do a Mob testing session. Just give them some test cases (which are easily understandable) and some instructions (and prepare e.g. test data) and just let them run the test cases! We did this when we invited some stakeholders from other department just to support us! This worked and the inputs was very helpful for us (because we also got feedback for new Change requests). The good thing was, that we just send one from our testanalysts there who was responsible for four-five groups of two persons. Somehow this is like a accelerator for proceeding with test cases.

Ask the business department for support

We asked also the business deparmtent for support. Generally the business department only takes action when the UAT should be done. But we asked them to support us, and we gave the reason that the help would be valuable in order to deliver a (testing) stable product.

Use a logging tool for exploratory testing

We reduced also our amount of time to create test cases when we made the decision to use a capture-replay tool. In our case it was tricentis/quasymphony. During test execution this tool creates test cases and during that time we did somehow exploratory testing. Means execution and creation of test cases. So this reduced somehow time for creating concrete test cases.

You see, there are a lot of ideas. But the most important one is that testing is a whole team approach! So you can invite all relevant stakeholders to support you - and based from our experience mostly were willing to support us.

| improve this answer | |
4

Start by analyzing what would be the impact regarding change. So you know the scope to be covered.

Out of the large set of Test Cases, you must be able to prioritize test cases as not all of them will have same priority. You may end up with High Priority, Medium Priority, and Low Priority set of Test Cases.

You might want to execute all the Test Cases but it will not be possible in this situation where there are time and resource constraints.

Do estimation of the time and effort need to perform each set of Test Cases. Then there will be few scenarios:

  • If you have time to cover High and Medium Priority. It’s good news.
  • If you have time to cover only High Priority. It’s still fine.
  • If you don’t even have time to cover High Priority, you need to discuss with your Manager or the one who need to make decision about release to communicate that you really need time to atleast finish the Test Execution for High Priority Test Cases. Otherwise there is the risk that main functionality will not working properly and will impact production environment when deploy.

If they are insist to go ahead with same time line then you already did your job to provide information and what will be happen is the responsibility of the whole team.

When you don’t have much experiences in Testing - everything is so important and need to be tested to ensure that it’s work.

When you have years of experiences in Testing - you will know that we will never have enough time to do the work. Challenge will be, How to deliver the best on time within time and resources given?

The key is to be aware of the objectives and goals. It will guide us through and make sure that we don’t get lost.

| improve this answer | |
4

In such situations, the most important thing is to prioritize your tests. Risk-Based Testing is a good way to prioritize your tests. Look at the risks you foresee and base your tests based on the mitigation plan for the tests. For every risk analyze the impact and the likelihood of occurrence and assign a priority to the corresponding tests. Run the high priority tests first and if time permits, take up the other tests.

Another approach could be to use data combination techniques for test design such as "Classification Tree Method, Decision Table or Pairwise Testing, etc." to help you group tests together. However, in order to assess the priority, look at the risks and prioritize based on the requirements and their business impact and priorities. Tools such as Hexawise, Testona and Razorcat etc., can actually help you with data combination tests.

Bottom line, Testing should be done prioritizing the features. It should ensure that common flows are all tested as much as possible.

| improve this answer | |
3

Find some measure then rank.

Lets say your measure is, say, line coverage; or exprssion coverage, or ...

  1. Run each test collecting coverage measure for that individual test.
  2. Rank the tests. (I'll expand on ranking after).
  3. Next time around only run the ranked tests + tests that previousely found a bug + randomly selected other tests to fill out the resources you have for testing in this "round".
  4. Repeat for the next incremental update to your program being tested.

Ranking:

I mention an algorithm for selecting those subset of tests that give the same total coverage, but with less tests as many tests will give coverage of similar areas.

Note:

This is the approach used in chip design where creating tests to execute is much easier than actually running the test on simulated hardware. It allows you to monitor coverage through design (in your case, program), evolutions keeping tests known to advance coverage whilst also allowing the addition of new tests to fill coverage gaps/randomly explore the tests-space.

| improve this answer | |
2

How do I execute/test a large set of test cases with limited time and resources?

  • For example you have 500 test scenario or testcases for each module, As a tester first we have to decide which feature is important in this module than you have to go for that particular feature test scenario.( For Manual Testing )

  • When you have large number of scenario that time you have to ignore those testcases, less important features.

  • For example you have 500 test scenario or testcases for each module,As a tester best option is Automation testing, In automation we will use TestNG for executing large number of testcases in a one time.its reduce your work time. ( For Automation testing )

| improve this answer | |
2

The answer depends on whether you are doing manual or automated testing

For all testing make sure you follow the guidelines in:

  • The agile testing pyramid
  • The agile testing quadrants

Most of all you make sure you

Have a conversation with the product owner and follow their advice

If you are doing manual testing my advice is:

  • Use risk based assessment
  • Use an exploratory charter
  • Hire more people to do the work
  • Hire experienced people to help and guide the people doing the work

If you are doing automated testing my advice is:

  • Use parallelization
  • Use browser testing services such as Browserstack, Sauce Labs, etc.
  • Hire automation engineers who are passionate about quality
  • Do NOT use developer interviews which focus on performance and efficiency*

* The main focus should be readability and maintainability. Performance (of the test code itself) should NOT be a primary consideration in selecting an automation engineer. Unfortunately most app devs that I encounter performing interviews have not learned this yet. This is my opinion based on my experience.

| improve this answer | |
2

Large number of test cases generally means some of them are ignorable or plenty of them depends on your test case making and extent of the effect of feature on the system you are testing.

Firstly, you will look for the major cases in regression if they pass or not, you will try to check things are not exploding when running major things. That may compile to 20–30% of your testing. Then you will go towards semi-major cases which may have direct or indirect effect on the efficiency of the system. Thus you can perform 50–60% testing and make a go for it.

Major test cases refers to the main functionalities of the system

Semi major refers to functionalities that are dependent on main functionalities or some how important to the client or system efficiency.

That’s what I suggest. But it is totally situation dependent.

| improve this answer | |
1

For most test cases depend upon your choice to which one you have to execute first and later on. For large test cases, mostly it needs to be prioritized. First: prioritize the test cases. Second: Check which cases/issue affects most of the module. Third: Go for newly fixed issues for modules.

| improve this answer | |
1
  1. Prioritize the critical test cases and time consuming test cases. Share those test cases to the testing persons who executes fast and correct.
  2. Then give the medium test cases to other persons who have good knowledge of the project/ product.
  3. Then aggregate the low priority test cases and give to the remaining testing persons to execute.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.