19

Okay, so I know we can't test every scenario possible, it's just not possible. But I want to get any ideas for making tests more comprehensive and getting the QA team to find problems before the code hits production.

We have a sizable and very configurable system and we're trying to update some major functionality with it. The problem is that every time we try to go to production something goes wrong and we hit the panic button and restore from backup.

We are now being asked why these problems didn't show up in QA or our unit tests and don't have any real good answers.

We currently have:

  • a build server that runs after ever check-in,
  • many unit tests (2000+ tests with over 60% coverage), but there's room for more,
  • a QA environment that is similar to production (3 servers instead of 4 and older hardware, but same config and OS) however it does not have as much data as production and what it does have can get out of date,
  • several in house people that we can have spend hours on QA,
  • and extensive error logging.

Are we just going to have to deal with the fact that we can't find every thing that could go wrong or are there techniques to make it more likely to find the problem before end users see it?

13
  • 1
    Define "many" for your unit tests. What's your code coverage? Are you doing boundary and negative testing or do you only cover the happy path?
    – Anthony Pegram
    Sep 21, 2011 at 16:40
  • "a QA environment that is similar to production", how similar? Ideally QA is nearly identical to production.
    – Chad
    Sep 21, 2011 at 16:43
  • 1
    It can be out of date, but it needs to represent real data properly. As for quantity, it's often good to have the same quantity, otherwise how do you know your app will be able to perform adequately on production volumes of data?
    – Chad
    Sep 21, 2011 at 17:08
  • 1
    @David Hogue: So these are regular users and not developers? That's fine, then. See also Joel's articles on testers here and there.
    – Falcon
    Sep 21, 2011 at 17:23
  • 1
    In fact, there are people who have already answered that question. A popular list of what you should test for is 32 pages, and it doesn't even touch the how. thebraidytester.com/downloads/YouAreNotDoneYet.pdf. Sep 24, 2011 at 21:43

10 Answers 10

11

I would have a staging enviroment thats an exact clone of live. Then when you have to rollback a bug you can also track it down and fix it on the staging enviroment.

Once you have a comprehensive list of bugs and their causes you can work on finding trends and stopping them happening again.

3
  • Definitely a good idea. We have made copies of the production data for reproducing bugs in the past. We do need to do more of that though.
    – David Hogue
    Sep 21, 2011 at 16:50
  • 1
    +1 for staging. I don't know how your system works, but from the user's viewpoint we don't have a highly interactive system - mostly just receiving files and processing. This allows us to forward production data simultaneously to the staging system and the production system, resulting in a very good test against live production data in the staging system. We can then compare the data in staging to the data in production in a useful way, too. This has greatly decreased "it slipped through" errors. Getting more samples of real customer data has also helped. Sep 21, 2011 at 20:53
  • 1
    Root cause analysis is key. Find out what failed, figure out why you didn't catch it, and fix that.
    – Bruce
    Sep 24, 2011 at 5:28
10

Errors occur when some user enters data that is different than the test data.

Fuzz testing in addition to boundary testing could help. Really any kind of randomly generated or randomly mutated data (by your business rules) can help find these kinds of errors. Dedicated testers who try to break the system and who do not just test that it is working properly are usually very good at finding malicious input or malicious data combinations, too.

It's mostly just some change that affects another area and slipped through testing.

Lately I've seen such errors mostly in the UI (with WPF) when there're already a bunch of unit tests for the business logic around. Someone changes a control template and some other things don't work properly anymore or do not look right. Automated UI tests can help with this but they're really tedious to create and maintain. Dedicated testers can find such things quite quickly, though. If it's the business logic that is failing, then create more unit tests and aim for a higher coverage. Code contracts and tighter constraints on the database might help, too.

3
  • I do like the idea of fuzz testing. Many of our problems come from odd user generated data or bad data in the database and that might help. We do have a few UI tests, but they've been time consuming to write and easily break when there's a change. We might have to put more effort into these.
    – David Hogue
    Sep 21, 2011 at 17:01
  • @David Hogue: The usual suspects. Tighter code contracts and harder constraints on the database can help with this, too.
    – Falcon
    Sep 21, 2011 at 17:05
  • If your UI Tests are brittle then do fewer, I had many UI changes coming in but as I started using the site as Page Objects rather than a recorded UI I had less breakage. Your mileage may vary. I give this one a +1 as it sounds more along the lines of what your problem is, data and inputs, not so much environmental.
    – MichaelF
    Sep 22, 2011 at 12:49
5

60% code coverage means there's lots of room for improvement in your unit tests. However, coming up with unit tests to handle situations far from the happy path can be rather tough. Your unit tests will never test all the ways things can go bump in the night.

A human can come pretty close. Hire a tester. They are cheap and incredibly cost effective. A good tester will emulate two key behaviors:

  • Sheer stupidity, to cover the stupid mistakes made by your less than optimal users: What happens if I do this? Well isn't that interesting. Time to open a bug report.
  • Sheer evil, to cover the stupid mistakes made by your less than optimal programmers: What happens if I do that? Well isn't that interesting. Time to open yet another bug report.

Let the tester work with the product under development and you won't have so many interesting problems at release time.

4

"several in house people that we can have spend hours on QA"

Are these several people professional QAers/testers? Or just folks with a few hours of free time?

How long did it take to design and develop the updates to your major functionality? Perhaps instead of "hours on QA", you need "days" or "weeks"?

Have you performed a root cause analysis for any of the problems you are seeing? It's difficult to determine a solution if you don't really know the cause.

2
  • 1
    There's definitely a difference between having a dedicated QA team and a few people who just happen to be around. Having dedicated testers would be quite helpful here. Sep 22, 2011 at 18:16
  • The internal QA Team would certainly be the ones who could most help answer the question "why these problems didn't show up in QA or our unit tests". Sep 23, 2011 at 13:42
3

David,

Great general question and great job backing it up with specific details.

I have been focused on answering the question "how can software testers find the most defects using the smallest possible amount of time / fewest number of tests?" for the last five years. Whenever testers are trying to address the problem of "too many possible things to test in a limited amount of time," I strongly recommend they give pairwise and/or more thorough combinatorial test design approaches a try. These test design approaches are custom-made to solve this specific challenge.

Here are three videos I posted last week that provide an introduction to this important (and, in my opinion, under-appreciated) topic:

Intro to pairwise and combinatorial software testing --> http://www.youtube.com/watch?v=kFH4QnymGGc

Intro to pairwise and combinatorial software testing (cont.) --> http://www.youtube.com/watch?v=Za5aQAMIq_g

How to think about test inputs in pairwise and combinatorial software testing --> http://www.youtube.com/watch?v=EZHha-nGvcM

These kinds of test design approaches can be made to work very well for virtually any System Under Test. And they can be particularly powerful in situations similar to yours (e.g., systems with multiple configurations options and a lot of complexity).

  • Will the tests generated through these test design approaches identify every single useful test? No. They will generate a fantastically varied and potent "starter set" of tests that you would want to supplement with additional tests you devise (particularly negative tests).

    Could they generate most of the tests you would want? Yes (provided you select test inputs thoughtfully).

    Would the tests generated by this approach find a disproportionately high number of defects as compared to most other manual approaches to selecting test conditions to use in test cases? Yes.

    How could you get started to quickly test out if this approach would be helpful in your context? One way would be to follow the "use a maximum of 3 values per parameter" approach described in the third video.

    Why are these test design approaches tend to be so powerful as compared to manually-selected test case selection methods?

    1. More variability is generated between tests (e.g., ...
    2. ... the amount of repetition from test case to test case is scientifically minimized), and
    3. You will achieve 100% coverage of every pair of test inputs with simple pairwise test plans; 84% of defects in production today could have been detected by simple pairwise tests.(1)
    4. You will achieve 100% coverage of all triples or all quadruples by quickly "turning the coverage dial up;" this approach to designing your tests allows you to focus your tests on where you'll most likely get the "biggest bang for your buck" and will allow you to make a conscious choice - with input from your business stakeholders - about whether the appropriate number of tests in a given situation is (a) the most powerful few dozen tests, or (b) the most powerful few hundred tests, or (c) the most powerful few thousand tests.

I hope these suggestions help.

  • Justin

(1) Findings from four studies supporting the finding that ~84% of defects in production today could be triggered during testing by executing well-thought-out pairwise tests are described in this article published in IEEE Computer magazine: http://app.hexawise.com/Combinatorial-Software-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf

2

Unfortunately, there's no silver bullet for solving the problem you have mentioned. This is a learning experience every testing team has to go through and improve upon their testing process from feedback from the customers.

James Whittaker's book Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design is an excellent resource. Implementing exploratory testing will certainly help in going a long way in improving quality.

Jerry Weinberg's book Perfect Software: And Other Illusions about Testing is also another good resource.

Unfortunately, unit tests do not address quality issues encountered by your customers. A good unit test suite is absolutely necessary for finding code-level regression issues. Training your QA staff to "think like the customer" is one of the important aspects for a successful testing effort. QA staff should be acting as customer advocates, questioning and scrutinizing every requirement, GUI, documentation and all the deliverables. Encourage the testing staff to go beyond the documented test cases and acceptance tests. James Whittaker's book provides a very good framework around which testers can perform these tasks.

Hope this answers the question to some degree.

1

We also have such a complicated system, and actually, I've asked another question regarding the same issue. What we do now, is that we have a pair of twin servers with up to the finest similarities, including installed applications, hardware specification, network capabilities, etc.

One of them is our production server and the other is testing server (AKA staging server by many developers).

However, we don't have unit tests at all. The only thing that increased our quality was that, we made our plans such that, each and every deploy should stay in testing phase for a sprint. This means that at the end of each sprint (production cycle), we deploy our new features to our test environment, and we take our testing server very seriously. I mean, we avoid any undo, or anything like that to the most strictest level.

Our testers also are not simply testers withing company. We actually have sold some of these applications (on our test server) to some of our clients free of charge, so that they can test it.

This way, we usually get lots of bugs during testing phase. I know that we have terms like beta, alpha, CTP, RM, R2, etc. But, hey, after all, what matters is that, we took a strategy that made everybody happy, and increased our quality.

So, my suggestion is that you can deploy to a test server (which is behaved just like a normal production server by you, and you would take it seriously and respect it) and define a period for testing cycle.

1

You really want to try test as much as possible? Try building a Chaos Monkey and see how fault tolerant your system is.

1
  • Ooh, I heard about that. I'm a little fearful about it breaking things badly, but that's exactly the state we need to get out of.
    – David Hogue
    Sep 21, 2011 at 18:09
1

Often unit tests verify that the happy path works, and that errors are handled correctly on a per-method basis. So it can be more fruitful to focus your testing on (a) less common paths, that nonetheless cause disproportionate customer dissatisfaction (your signup may work perfectly, but does your Unsubscribe have an endless loop?), and (b) end to end error behavior - if you actually let one of those unit test error states go through your stack to the user, are they going to get an error reporting and recovery experience you'd be proud of?

1
  • Oh, and if you happen to run your app in a data enter, go talk to your ops guys; they will give you plenty of ideas for additional testing.
    – Bruce
    Sep 24, 2011 at 5:40
1

If I were asked to help troubleshoot, I would begin by asking whether your production problems appear during deployment or after the system has been in deployment for a while.

If the problems appear during deployment, I would next ask whether you deploy your QA environment is exactly the same way you deploy the production system, taking into account the hardware differences you described.

After ruling out any deployment issues, I would ask whether your bugs are in specific areas or are systemic. Judging from your question, the bugs are systemic. In that case, for each bug, I would ask whether the bug should have been caught by an automated test or a manual test. If an automated test, I would ask whether the author(s) of the test knew why the test did not find the bug. If a manual test, I would ask whether the manual tests are documented.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.