6

Consider a modern web application that serves JSON through a REST API to a rich Javascript-based application with one or more single-page apps within it. Our own application is written using Angular JS but the framework is irrelevant to a large extent.

There are the following components where problems can occur:

Server-side

  • Database access (including configuration, reference data and data that always needs to be there such as an admin account)
  • REST API (business logic, configuration, routing)
  • Service bus implementation, which the API uses to serve the requests (components, workflows, configuration)

Client-side

  • View Models (state and UI interactions such as dialogs)
  • Some HTML rendering
  • Routing
  • Configuration

How much testing is sufficient? Meaning, what's the minimum effort required to test reliability?

Is it best practice to simply do integration/end-to-end testing (deploy the whole application to a real server with a real database and some sample data) and test all possible scenarios by checking UI-level responses?

Improve this question
5
  • Having the budget will you really be able to test all possible scenarios on the UI levels? Wouldn't you be able to cover more automating some tests just behind UI on backend level?
    – dzieciou
    Nov 6, 2012 at 19:49
  • We probably can't get away without doing at least some back-end testing but that still doesn't answer the question of what/how much. Integration testing is considered to be any testing which tests several components together, rather than isolating each one (unit tests)
    – georgiosd
    Nov 6, 2012 at 19:53
  • Sorry to be picky, but you're asking how much integration testing is enough, when you have already done unit testing, or rather how much testing is enough in general?
    – dzieciou
    Nov 6, 2012 at 19:57
  • We haven't done any testing yet. Hence the question. We're trying to figure out where to concentrate our efforts.
    – georgiosd
    Nov 6, 2012 at 19:58
  • 1
    Welcome to SQA, georgiosd. I think it will be hard to answer your question as it is phrased. Your test plan needs to take into account not only the the architectural components but also the complexity of those components and even the business problem you are trying to solve. It may be more helpful for you to propose a test plan (and a high level, of course) and let us review it.
    – user246
    Nov 6, 2012 at 21:36

4 Answers 4

Reset to default
2

Is it best practice to simply do integration/end-to-end testing (deploy the whole application to a real server with a real database and some sample data) and test all possible scenarios by checking UI-level responses?

When we talk about the relative merits of test strategies, it is usually better to discuss specifics than generalities. Your definition of integration test may differ from mine, both at a high level and in the specifics.

You may find that it is hard to test all possible scenarios using any one technique. I am not just being pedantic here; some scenarios require a lot less effort to check with an API-oriented unit test than with a test that goes through the UI. Tests blur the line between "did I build it the right way?" and "did I build the right thing?" may be better accomplished -- or only possible -- by testing with the UI.

Another way of thinking about this is in terms of short term vs. long term reliability. We tend to think of an application as a single unit built for a purpose, but of course it is really an aggregation of components that, when used in a certain way, serves a particular purpose. It is possible that each of those components is designed to do no more and no less than is required for that application. In practice, though, the component may have other behaviors that are not exercised by the application because they are not currently needed. If you only test at the UI level, you at most guarantee that the components behave correctly when used by the application as it exists now. As soon as the application uses a component in a different way, the component may break. Unit testing gives you an opportunity to find those problems early on.

To put it another way, UI testing does not necessarily cover all code paths. Some code paths are only accessible via unit tests. You should test those too because while they may be unnecessary to today's application, next month's application may need them.

Improve this answer
4
  • +1 for explanation why covering path in unit tests, that are not accessible in integration/UI tests, makes sense.
    – dzieciou
    Nov 9, 2012 at 9:59
  • On the other hand: what if the system fails on lower level (e.g., unit level) but does not fail on higher level (e.g. system level)? I've read once an article about random testing that found many bugs in OS drivers, but those bugs were ignored by OS authors, because libraries using those drivers check invalid input before it arrives to faulty drivers. Maybe it does not make sense to cover literally all paths?
    – dzieciou
    Nov 13, 2012 at 20:17
  • 1
    We have all worked with software that works perfectly when used in exactly the way it was intended, but which breaks as soon as we "get off the beaten path". If a component works today but breaks tomorrow because we changed how we used it, we call that component fragile. Using fragile components makes the development process less predictable. Of course, just as we triage application bugs to decide which deserve fixing, we should also triage component bugs. We may decide that a bug in a component is not worth fixing, but if we don't unit test, we won't even know about the component bugs.
    – user246
    Nov 13, 2012 at 20:28
  • Thank you - several years after reading this, it makes total sense to me. Tests are code too and it makes sense to prioritize by risk as you would prioritize by effort/value for features.
    – georgiosd
    Aug 25, 2015 at 15:25
10

How much testing is sufficient? Meaning, what's the minimum effort required to test reliability?

When looking at the question of "how much testing?" you have to consider "how lucky do you feel?" You could do no testing at all if you feel really lucky, or if the consequences to being wrong are extremely low. You could test everything for a really long time if you aren't feeling lucky, or if the consequences to being wrong are not low.

Your answer might be different if you are building a company blog rather than an air traffic control system.

Is this a one-off isolated application, where the layers/components aren't (and won't be) used by any other apps? How many of the layers/components are new (and thus perhaps deserve more attention)?

Is there a reason why you wouldn't test each component individually in addition to integration testing?

Do you have budget and/or schedule limits that you must meet?

Is it best practice to simply do integration/end-to-end testing (deploy the whole application to a real server with a real database and some sample data) and test all possible scenarios by checking UI-level responses?

That may not be best for you. It usually wouldn't be best for me.

Improve this answer
6
  • Joe, sure it depends on the context, but how much context would you need to have to answer such questions or give some guidelines? Is it feasible given the limited space for answer here?
    – dzieciou
    Nov 6, 2012 at 20:47
  • What sort of answer are you looking for ? Number of hours, percentage for each type ? There are so many variants that it's hard to be specific
    – Phil Kirkham
    Nov 6, 2012 at 21:08
  • I see a lot of both questions like this and answers like "well it depends on the context". So, I'm curious how one can learn to select the right thing for his or her context? One things is by try-observe-correct approach. If a bug in integration was discovered in production I may ask myself what wrong have I done. Another approach would be to write simple test plan with focus area and discuss with peers what is missing. How have you learned to define the focus? I don't mean learning precise numbers, I mean selecting areas of focus of testing.
    – dzieciou
    Nov 6, 2012 at 21:20
  • dzieciou, if I asked you "how long does it take to write an application?" wouldn't you want to know a bit about the application's requirements before you guesstimated? Testing is like that. allthingsquality.com/2011/01/estimation-guesstimation-and.html
    – Joe Strazzere
    Nov 6, 2012 at 22:17
  • Joe, good article. I see your point and I agree having requirements and constrains helps narrow down the context. But it is good to know what constrains matter. And your answer lists some good constrains. So, instead of giving definite answers, can't we just teach and learn what questions tester must ask himself when planning tests?
    – dzieciou
    Nov 6, 2012 at 23:57
2

I think unit tests are inevitable. And unless your application is a standalone and no other applications are using it, integration test is also inevitable. The question you should really ask is how deep? Do you think it is necessary to provide 90+% unit test coverage? Do you need to provide close to complete coverage for endpoints? That will always be a "depends" answer, and I think Joe has covered it very well.

Improve this answer
1
  • I agree. The clue is to understand what things the answer will depend on. Do you want to have quick feedback on any regression made? Unit tests will execute faster. There are hundreds of such factors and it is not possible to list them all in a single answer.
    – dzieciou
    Nov 6, 2012 at 23:50
2

Another way to look at it is "What is most likely to be dangerously broken by changes?"

So while "Database access (including configuration, reference data and data that always needs to be there such as an admin account)" might take down the Web site, it would be immediately obvious, and simple website monitoring (which you should have anyway) would notify you.

However you might have business logic that, if it was incorrect, could cost your company significant amounts of money (calculating quotes perhaps?) and might go unnoticed for some time.

Basically, you could do a cost benefit analysis.

Improve this answer
1
  • Yes, you can simply do a risk analysis. You may plan tests upfront but when you start tests, you may discover new risk areas and decide to update the plan, for instance, do more integration tests in risky areas.
    – dzieciou
    Nov 12, 2012 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.