Test case preparation and execution time

Question

I usually estimate the test execution time in a sprint by identifying the test scenarios related to requirements. (Test case point analysis)

1. Is there any techniques that we can use for estimate the time for execution of one particular testcase? (As all test cases will not have same set of steps and will take different time.)

2. Is there a technique which receives highly accurate estimate that we can use for preparation of testcases? (Currently I estimate the time for testcases preparation by my past experiences.)

Answer 1

Firstly, I would be careful when communicating this type of relationship between test cases and your testing, because, test cases are not testing. Promoting otherwise may cause issues both for the stakeholders who won't understand what is happening and for the tester who will have less leverage to promote deep testing.

Testing is the process of evaluating a product by learning about it through experiencing, exploring, and experimenting, which includes to some degree: questioning, study, modeling, observation, inference, etc., not the process of making evaluations by applying algorithmic decision rules to specific observations of a product (aka pure scripting - test cases).

If they know a bit about testing, stakeholders don't want to know about test cases, but about your discoveries about the status of the product. Thus what you need to track is what makes the time you spend trying to discover important facts about the status of the product.

One framework to do so is the Session-Based Testing Management. You can summarize your testing session using a sheet as below. In it, you can register how much time you spent exploring the product, applying sensemaking of the behavior of the product. how much time you spent preparing your environment and product for testing, and how much time you were focused on the charter versus exploring areas that seemed more interesting.

This information can then be aggregated to uncover if some of these activities are usually a bottleneck for testing (Setup and sensemaking are usually bottlenecks in environments with low testability).

Having this summary can help you to forecast the testing time for future similar activities (investigation on same code base area, same time of refactoring, same downstream dependencies, etc).

Answer 2

Some common estimation techniques can help.

Having historical data can help. For each test case, measure the time it takes to complete. Given a large enough body of test cases, you can determine how long it takes to run a test case. You may be able to extrapolate how long it takes to run an average test step, as well. However, the time it takes to run a test case or a test step depends on the complexity of the case (or the step) and the experience of the tester running the particular case.

If multiple testers are gathering the data, you can expand this to look at organizational averages for test cases and steps.

Comparing test cases that are similar in complexity can also be helpful, regardless of the number of steps. Experts - people who are well-versed in testing different parts of the system - can be useful in creating these comparisons. Estimating in groups instead of individually can further enhance the estimate quality, and having historical data can also be helpful.

As far as preparation of test cases go, having clear requirements and comparing the complexity of testing similar requirements coupled with expert insights can be helpful. Creating test cases is far more of a creative endeavor than executing test cases, so it can be more difficult to estimate since there are more variables. Having more historical data, more experience, and more expert insights can help reduce the uncertainty.

Answer 3

Test case estimation is always tricky and should always be on the basis of User story complexity. As Software Testing Companies that provide security testing services, usually estimating a 5 points user story (per Acceptance Criteria), Testing estimate (for Test case creation + Execution) should be approx. 20-24 hours of QA effort. We also need to ensure that we always have room for defect reporting and regression testing while estimating because in case we don't have optimum bandwidth than product quality might get hampered as we have to rush to achieve the deliverables.