What should be the critieria for the selection of new feature test cases to be moved into build acceptance test suite?

Question

I am working on a project to move new feature test cases to build acceptance test suite. Later on I will also be selecting which test cases should be moved to regression suite.

While I was selecting test cases I had the below factors in mind:

1. Is the test case robust enough?
2. Is it critical to ensure functionality of new feature?
3. Test Coverage - Having one test case covering multiple business rules rather than having 2 test cases cover a business rule each
4. Sampling of positive use case
5. Boundary value tests
6. Feature interaction/Integration test
7. Sampling of negative use case

I was told that build acceptance suite generally tests only happy paths not negative scenarios. Is that true? If I have missed any other crucial factor for selection please suggest.

Answer 1

I think you answer this by asking, "Why do I need a build acceptance test suite?" There is no universal answer, but if your organization has a suite, someone should be able to answer the question.

For example, in a previous job, the developers on my team frequently made changes that rendered the build unusable: it would not install, or it would crash as soon as you tried to log in, or it was broken in some other fundamental way. The problem was so serious that developers were afraid to refresh their source trees. We addressed the problem by doing daily smoke tests in which someone on the team (we took turns) installed a build and ran through some essential tests that covered common tasks that had to work in order for the developers to be productive. The tests varied over time as the project progressed. If a build made it through the process with a serious problem, we considered adding tests for it in the next smoke test. And if we went a long time without a particular test failing, we considered using the test less often. Because we knew why we ran the smoke test, we could measure its effectiveness and adjust it accordingly.

There are several other questions about smoke tests on SQA. You may want to check them.

Answer 2

As the requirement is to move new feature test cases to build acceptance suite, I think it is important to note, what benefits the build acceptance tests provide usually:

1) Fast feedback to developers, if there are issues in the checked-in code 2) Tester gets a working build to start with their activities 3) Application should be usable every time

Now for feedback to be faster, following are few points that need to be considered before test cases being moved to build acceptance suite - (a) The test cases must be / can be automated 100%, so that later no manual intervention is required. (b) The test cases can be executed in less time (its relative term but then fast feedback is also the same)

For testers to get a working build: (a) The build should be install-able correctly, so installation related test cases (b) Basic functionality (e.g. login, export etc.) of new features should be working, so that testers can verify additional test cases (boundary / negative)

For application to be usable every time: (a) Test cases covering business critical scenarios should be part of the build acceptance suite.

So, as per the above analysis, I don't consider that negative test cases or even boundary test cases should be part of build acceptance tests. I would be eager to find more inputs though on these lines.

Answer 3

Along with understanding the need to have a build acceptance suite, which will help define parameters, I think the following things might universally apply:

1. Don't pick atomic test cases, i.e. TCs that do only one thing. You've partly covered this in #3 of your question. You want TCs that cover a basic scenario that you consider absolutely essential for your product.

2. Define who will own which TCs. If they fail, who is responsible to investigate? What is the expected turn around time?

3. Build acceptance TCs should run fairly quickly. Since you'll run them often (depending on whether you have nightlies, everyday), these should finish running anywhere from a few minutes to a couple of hours depending upon the size and complexity of your product.

4. TCs should be functional in nature, i.e. not perf, security or usability based.

5. Manual TCs for this purpose should be absolutely minimal IMO.

6. Avoid negative cases unless they are mission critical

7. Boundary TCs should be avoided unless mission critical

8. State who decides future TCs. Can everyone just mark their TC as required for build acceptance? You might want to have a person or two, depending upon the size of your org, to determine this. Giving free reign will mean you'll have too many TCs that take too long to run.

9. What is the expected accuracy? i.e. What percentage of times is it ok if TCs failed because of environment or test code reason?

10. What course of action will be taken if a TC fails too many times because of test code itself? You'd want to maintain high test code quality. Removing a TC from the suite temporarily upon say <90% accuracy might encourage better code quality and therefore higher build quality.

11. Means to identify and communicate means of failure, for instance code checkin that caused it. Hope that helps. Good luck!

Answer 4

"Yes, I am totally agreed with this statement. The decision on when and how to automate a suite of regression test cases can be challenging. The common problem with automated regression testing is that many people consider it as a good idea but they don’t know when and how to automate the things. Below are some key points that one should keep in mind before automating the regression test: a) If someone is testing a software frequently. b) It is also a must when we expect frequent releases and incremental changes to each release. c) And last but not least, regression testing is must when we have a large product with sporadic releases. "