Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
We have a large set of test cases but limited time and resources to execute/test them all. We thought but it's not possible to execute/test everything.
I want to know strategies which will be beneficial in the above situation to cover more tests with limited time and resources.
Can someone suggest any strategy or way to move ahead for the above situation?
Good question. Here I am going to explain some steps from my experience.
1) We need good teamwork for that.
2) Here I just want to clarify that "Execute All Test Case/Suite" term. We need to prioritize test cases in four quadrants like below.
3) Do a good amount of Exploratory Testing instead of totally scripted approach.
4) Try to cover all business-critical flow first.
5) Try to test all previous production defects and cover in the current regression cycle.
6) Give yourself some time for proper planning before starting with a sequential approach. What type of modules and test cases you can eliminate first and remove it first so now you have a list of all important tasks. Again use Priority Quadrant [Step-2]
6) And last but not least. Again we need Step-1 :)
Always be open for feedback and suggestion.
Basically you never have enough time and resources to test everything, your test cases are already a subset of this infinite "everything".
What should you do then ? prioritize. A common heuristic is RCRCRC:
Recent: new features
Core: essential functionality of your product
Risk: risk is defined as the Probability of occurrence (how likely is it to happen) multiplied by the affect (what's the damage, in money hours lost or anything relevant), Risk hides under it a lot and can be calculated at different levels from module to system to environments.
Configuration sensitive: this represents both internal configuration and environment settings, for example the type of the device or operating system
Repaired: recently repaired bugs represents untested area
Chronic: some areas are known to be vulnerable. This can be due to complexity, the level of the developers or an area that falls in between responsibilities
Take you list of test cases and grade them for each item, let's say 1 - 5 (don't use zero since it will interfere with the next step), multiply then the six figure and sort by the result. This will give you a first estimation as to what should be tested first and why.
Feedback also from my side what I learned from projects.
1. Prioritize the test cases
In my past projects we prioritized the test cases. We used HP ALM and there we had also couple of test cases and it was impossible to execute the all. So what we did is just to prioritize the test cases e.g. Critical, Very High, High, Medium, Low - same as you would do it for defects. All created test cases were based on release policy which we also created. This helped just to focus on the most important test cases
2. Involve others in testing scope e.g. Product Owners and shift some test cases to them
Involve the otherst in the test team. Testing doesn't mean that you just test alone with your testing mates. It means that you can also involve other team members. In our project we hence involved also the Product Owner for testing. And not just at the end (e.g. when deployment is done and UAT should be done via business department and Product Owner). In Janets book I found the hint that UAT is done mostly after post deployment. and that tis does not mean that it has to be left to the end game (Janet Gregory "More Agile Testing" page 202). So when testing a product of course you can involve other stakeholders to test a product. In our case our Product Owners helped to test some test cases! By the way, you also can ask UX department to support you. We asked also our UX department to support us to execute some test cases and we got also valuable feedback from them.
Organize somehow Mob testing and act as test coordinator who throws test cases to them!
You can also try to do a Mob testing session. Just give them some test cases (which are easily understandable) and some instructions (and prepare e.g. test data) and just let them run the test cases! We did this when we invited some stakeholders from other department just to support us! This worked and the inputs was very helpful for us (because we also got feedback for new Change requests). The good thing was, that we just send one from our testanalysts there who was responsible for four-five groups of two persons. Somehow this is like a accelerator for proceeding with test cases.
Ask the business department for support
We asked also the business deparmtent for support. Generally the business department only takes action when the UAT should be done. But we asked them to support us, and we gave the reason that the help would be valuable in order to deliver a (testing) stable product.
Use a logging tool for exploratory testing
We reduced also our amount of time to create test cases when we made the decision to use a capture-replay tool. In our case it was tricentis/quasymphony. During test execution this tool creates test cases and during that time we did somehow exploratory testing. Means execution and creation of test cases. So this reduced somehow time for creating concrete test cases.
You see, there are a lot of ideas. But the most important one is that testing is a whole team approach! So you can invite all relevant stakeholders to support you - and based from our experience mostly were willing to support us.
Start by analyzing what would be the impact regarding change. So you know the scope to be covered.
Out of the large set of Test Cases, you must be able to prioritize test cases as not all of them will have same priority. You may end up with High Priority, Medium Priority, and Low Priority set of Test Cases.
You might want to execute all the Test Cases but it will not be possible in this situation where there are time and resource constraints.
Do estimation of the time and effort need to perform each set of Test Cases. Then there will be few scenarios:
If they are insist to go ahead with same time line then you already did your job to provide information and what will be happen is the responsibility of the whole team.
When you don’t have much experiences in Testing - everything is so important and need to be tested to ensure that it’s work.
When you have years of experiences in Testing - you will know that we will never have enough time to do the work. Challenge will be, How to deliver the best on time within time and resources given?
The key is to be aware of the objectives and goals. It will guide us through and make sure that we don’t get lost.
In such situations, the most important thing is to prioritize your tests. Risk-Based Testing is a good way to prioritize your tests. Look at the risks you foresee and base your tests based on the mitigation plan for the tests. For every risk analyze the impact and the likelihood of occurrence and assign a priority to the corresponding tests. Run the high priority tests first and if time permits, take up the other tests.
Another approach could be to use data combination techniques for test design such as "Classification Tree Method, Decision Table or Pairwise Testing, etc." to help you group tests together. However, in order to assess the priority, look at the risks and prioritize based on the requirements and their business impact and priorities. Tools such as Hexawise, Testona and Razorcat etc., can actually help you with data combination tests.
Bottom line, Testing should be done prioritizing the features. It should ensure that common flows are all tested as much as possible.
Find some measure then rank.
Lets say your measure is, say, line coverage; or exprssion coverage, or ...
Ranking:
I mention an algorithm for selecting those subset of tests that give the same total coverage, but with less tests as many tests will give coverage of similar areas.
Note:
This is the approach used in chip design where creating tests to execute is much easier than actually running the test on simulated hardware. It allows you to monitor coverage through design (in your case, program), evolutions keeping tests known to advance coverage whilst also allowing the addition of new tests to fill coverage gaps/randomly explore the tests-space.
How do I execute/test a large set of test cases with limited time and resources?
For example you have 500 test scenario or testcases for each module, As a tester first we have to decide which feature is important in this module than you have to go for that particular feature test scenario.( For Manual Testing )
When you have large number of scenario that time you have to ignore those testcases, less important features.
For example you have 500 test scenario or testcases for each module,As a tester best option is Automation testing, In automation we will use TestNG for executing large number of testcases in a one time.its reduce your work time. ( For Automation testing )
For all testing make sure you follow the guidelines in:
Most of all you make sure you
If you are doing manual testing my advice is:
If you are doing automated testing my advice is:
* The main focus should be readability and maintainability. Performance (of the test code itself) should NOT be a primary consideration in selecting an automation engineer. Unfortunately most app devs that I encounter performing interviews have not learned this yet. This is my opinion based on my experience.
Large number of test cases generally means some of them are ignorable or plenty of them depends on your test case making and extent of the effect of feature on the system you are testing.
Firstly, you will look for the major cases in regression if they pass or not, you will try to check things are not exploding when running major things. That may compile to 20–30% of your testing. Then you will go towards semi-major cases which may have direct or indirect effect on the efficiency of the system. Thus you can perform 50–60% testing and make a go for it.
Major test cases refers to the main functionalities of the system
Semi major refers to functionalities that are dependent on main functionalities or some how important to the client or system efficiency.
That’s what I suggest. But it is totally situation dependent.
There are a few strategies that can be used to test a large set of test cases with limited time and resources:
Prioritize test cases based on risk: Identify the most critical test cases that need to be executed first and focus on those. This will ensure that any critical issues are identified and addressed early on.
Use automated testing: Automated testing can help you cover a large number of test cases quickly and efficiently. This can also reduce the need for manual testing, which can be time-consuming and resource-intensive.
Use test case reduction techniques: Test case reduction techniques can help you reduce the number of test cases that need to be executed by identifying redundant or unnecessary test cases.
Use parallel testing: Parallel testing allows you to run multiple test cases at the same time, which can help you cover more test cases in a shorter amount of time.
Use regression testing: Regression testing can help you identify any new issues that may have been introduced as a result of code changes. This will allow you to focus on testing only the areas of the code that have been modified.
Use cloud-based testing: Cloud-based testing allows you to run your test cases on different devices and configurations without having to purchase and maintain physical devices.
Use exploratory testing: Exploratory testing is a technique that allows you to discover new features and bugs. This approach will allow you to cover more test cases with limited resources.
Use crowd testing: Crowd testing allows you to outsource your testing to a large number of testers. This can help you cover a large number of test cases in a shorter amount of time.
For most test cases depend upon your choice to which one you have to execute first and later on. For large test cases, mostly it needs to be prioritized. First: prioritize the test cases. Second: Check which cases/issue affects most of the module. Third: Go for newly fixed issues for modules.
