Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I have been reading alot about automated acceptance testing. However, I have not found any information about how to manage external dependencies.

Let's say you have a web application that acts like a UI for various central business applications which expose services in different technologies.

The web application also has some reasonable amount of own business logic, but it's really related to the application and not to the core business of the company, all of that is handled by central services.

The issue is that trying to write an automated acceptance tests on the "real" system would yield to two main problems:

  • It's not possible to guarantee that a given dataset for the core systems will remain unaltered (maybe not even coherent anymore) the next time it's needed by the test. Adjusting it via direct data injection is not an option (too complex), adjusting via service calls is possible but will only cover the most common issues, meaning that it's always possible that someone will have to check tests to see what happened to the dataset (or analyze if it's a code issue).
  • It's also not possible to guarantee a stable development environment. At the end, other systems might be testing some features so central systems become unavailable or unreliable at least. One option is to run the test against a more stable environment (QA, Preproduction), but if the development depends on some change of a central service, we would loose the "quick feedback" nature of acceptance tests in agile development.

So, having these issues, it occured to me that if the team would already test all the services with automated integration tests against coherence at least, we could trust in the communication layer and in the input/output maps of the services we call. At least an error in these tests would clearly point at the problem (connection with the service or coherence of the service). We could skip the part of depending on business logic which is not on our side.

After that, we could mock the service layers on acceptance tests so that we isolate the behaviour of our application from the environment and focus to test what our system does and avoid having to validate the proper behaviour of other entities where we don't have direct control.

However, I am unable to find any literature from the widely known agile gurus on this subject.

It would be great to hear if someone has had success on such an approach, hear why such an approach would be moronic, or find some reading of similar approaches...

share|improve this question

2 Answers 2

I would agree with Laurent, and we have faced a very similar situation. Our team tests the APIs for data transport and parsing. The developer unit/component level tests were all against mock objects and data stores.

Unit tests are not always comprehensive so we built a 'fake server' to emulate various web services that we used for running additional functional tests. The pros of this approach are control over the data (contacts, feeds, photos, etc), and the ability to simulate network errors via fault injection.

But, while a fake server is good for evaluating raw functionality in a isolated environment, it doesn't test how the software performs and functions end-to-end in the "real world." The point of acceptance testing is to evaluate the software in a 'real-world' situation similar to what the customers will experience; and that is the biggest advantage of going against real servers.

One of the down-sides of going against real services is that some of your tests may become unreliable. Some sporadic issues dealing with real services can be overcome by re-designing the automated tests. For example, ping the service before the test begins to make sure it is running and abort the test if it is not ready after a brief period of time.

So, we still use a mix of both approaches, but our growing suite of customer scenario tests (acceptance tests) and our performance tests are now going against real servers.

share|improve this answer
    
OK, thank you very much... Do these tests offer "quick feedback" to the developer team? What approach do you take on tests that should be run on a daily basis and be as stable as possible, but still business-driven? –  Martin Dec 2 '13 at 13:02
    
I am not quite sure what you are asking. In my group all automated tests (unit, functional, non-functional) are ran at least daily on multiple branches. The entire team has immediate feedback to the results via internal/intranet web services. Any failing tests are investigated and triaged. –  Bj Rollison Dec 5 '13 at 15:15

I think "acceptance tests" and "mocking services layers" don't go along. My feeling is that when you do acceptance/functional testing, then you run your whole system in a manner close to the way it will be used although you try to do it in a QA/preprod environment to be confident of the state of your system. If, for the reasons you mention, you can not do that, and want to focus your tests on one component (the web app) without having the services running, then you will have to dig a little bit in the source code of your front end and it becomes a component-testing activity that could probably be better achieved by the development team of the web app.

I have a slightly similar situation than you and we decided to do:

  • "component tests" of the UI using QUnit and sinon.js as a mocking library (so that we don't need the services/backend running)

  • "end-2-end / acceptance testing" of the whole system using Selenium on QA environment that are similar to production environment.

This is not exactly answering your question, but might help you anyway

share|improve this answer
    
Hi, thank you very much for your thoughts. I know I might not get a specific answer, but hearing from experiences always helps. The issue is that we are trying to write acceptance tests that will help the team get quick feedback wether acceptance criteria are met. If we try to run it in QA environment, we loose the fast-feedback opportunity. For this kind of tests we already have some corporate tools where we can script such tests. The nature of these acceptance tests is to help developers validate acceptance criteria sooner. –  Martin Nov 29 '13 at 10:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.