Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I am working on a project to move new feature test cases to build acceptance test suite. Later on I will also be selecting which test cases should be moved to regression suite.

While I was selecting test cases I had the below factors in mind:

  1. Is the test case robust enough?
  2. Is it critical to ensure functionality of new feature?
  3. Test Coverage - Having one test case covering multiple business rules rather than having 2 test cases cover a business rule each
  4. Sampling of positive use case
  5. Boundary value tests
  6. Feature interaction/Integration test
  7. Sampling of negative use case

I was told that build acceptance suite generally tests only happy paths not negative scenarios. Is that true? If I have missed any other crucial factor for selection please suggest.

share|improve this question

4 Answers 4

up vote 7 down vote accepted

I think you answer this by asking, "Why do I need a build acceptance test suite?" There is no universal answer, but if your organization has a suite, someone should be able to answer the question.

For example, in a previous job, the developers on my team frequently made changes that rendered the build unusable: it would not install, or it would crash as soon as you tried to log in, or it was broken in some other fundamental way. The problem was so serious that developers were afraid to refresh their source trees. We addressed the problem by doing daily smoke tests in which someone on the team (we took turns) installed a build and ran through some essential tests that covered common tasks that had to work in order for the developers to be productive. The tests varied over time as the project progressed. If a build made it through the process with a serious problem, we considered adding tests for it in the next smoke test. And if we went a long time without a particular test failing, we considered using the test less often. Because we knew why we ran the smoke test, we could measure its effectiveness and adjust it accordingly.

There are several other questions about smoke tests on SQA. You may want to check them.

share|improve this answer
    
+1 This is a great answer! We have build acceptance test suite in our organization to test whether the buid is ready for testing. We are also planning to give this suite to the DEV for testing before they deliver any bug fixes to ensure their fix does not break existing functionality. So I think, negative use cases are not good candidates for BAT. Is that correct? –  Aruna Feb 24 '12 at 6:49
1  
+1 because it says exactly what I would. Aruna, yes. Negative test cases aren't good candidates. You're looking for something that verifies that core function isn't broken. If you can't use the core functions of the application, there isn't much point going further. Ideally a build that fails acceptance tests would also mean no further integration until the acceptance tests pass - but we live in the real world, and not every organization is willing or able to do that. –  Kate Paulk Feb 24 '12 at 11:32

As the requirement is to move new feature test cases to build acceptance suite, I think it is important to note, what benefits the build acceptance tests provide usually:

1) Fast feedback to developers, if there are issues in the checked-in code 2) Tester gets a working build to start with their activities 3) Application should be usable every time

Now for feedback to be faster, following are few points that need to be considered before test cases being moved to build acceptance suite - (a) The test cases must be / can be automated 100%, so that later no manual intervention is required. (b) The test cases can be executed in less time (its relative term but then fast feedback is also the same)

For testers to get a working build: (a) The build should be install-able correctly, so installation related test cases (b) Basic functionality (e.g. login, export etc.) of new features should be working, so that testers can verify additional test cases (boundary / negative)

For application to be usable every time: (a) Test cases covering business critical scenarios should be part of the build acceptance suite.

So, as per the above analysis, I don't consider that negative test cases or even boundary test cases should be part of build acceptance tests. I would be eager to find more inputs though on these lines.

share|improve this answer
    
+1 For emphasis on short feedback cycle which is very important –  Aruna Feb 24 '12 at 21:42

Along with understanding the need to have a build acceptance suite, which will help define parameters, I think the following things might universally apply:

  1. Don't pick atomic test cases, i.e. TCs that do only one thing. You've partly covered this in #3 of your question. You want TCs that cover a basic scenario that you consider absolutely essential for your product.

  2. Define who will own which TCs. If they fail, who is responsible to investigate? What is the expected turn around time?

  3. Build acceptance TCs should run fairly quickly. Since you'll run them often (depending on whether you have nightlies, everyday), these should finish running anywhere from a few minutes to a couple of hours depending upon the size and complexity of your product.

  4. TCs should be functional in nature, i.e. not perf, security or usability based.

  5. Manual TCs for this purpose should be absolutely minimal IMO.

  6. Avoid negative cases unless they are mission critical

  7. Boundary TCs should be avoided unless mission critical

  8. State who decides future TCs. Can everyone just mark their TC as required for build acceptance? You might want to have a person or two, depending upon the size of your org, to determine this. Giving free reign will mean you'll have too many TCs that take too long to run.

  9. What is the expected accuracy? i.e. What percentage of times is it ok if TCs failed because of environment or test code reason?

  10. What course of action will be taken if a TC fails too many times because of test code itself? You'd want to maintain high test code quality. Removing a TC from the suite temporarily upon say <90% accuracy might encourage better code quality and therefore higher build quality.

  11. Means to identify and communicate means of failure, for instance code checkin that caused it. Hope that helps. Good luck!

share|improve this answer
    
+1 These are very important points. Typically ownership of failures is a challenge because the resource who automated the new feature test cases might have moved to a different project when we are planning to move his test cases to BAT. The communication plan for failures, ownership and maintaining a high test code qualtiy are great points and deserves very high marks from me! –  Aruna Mar 4 '12 at 2:52
    
True, while testers might move between projects, IMO, so should ownership. If the existing tester moves, a new one should be handed ownership of their TCs. –  Aniket Mar 4 '12 at 6:33
    
I partially agree with that. After the test cases are moved to BAT the ownership does change. But before that, the new feature test cases author should be accountable for handover of a robust high quality suite to the BAT team. For the future releases the BAT team could be held accountable –  Aruna Mar 6 '12 at 0:59

"Yes, I am totally agreed with this statement. The decision on when and how to automate a suite of regression test cases can be challenging. The common problem with automated regression testing is that many people consider it as a good idea but they don’t know when and how to automate the things. Below are some key points that one should keep in mind before automating the regression test: a) If someone is testing a software frequently. b) It is also a must when we expect frequent releases and incremental changes to each release. c) And last but not least, regression testing is must when we have a large product with sporadic releases. "

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.