As soon as a developer pushes a commit, our Jenkins run unit tests, then integration tests (e.g. just backend) and finally system tests (e.g the whole app: backend, frontend, platforms, etc). The system tests stage is fairly quick, about 3 minutes. Then, once a day, we run stability tests for 2 hours.

This has been good for some time, but we are starting to struggle with Android and iOS devices, so I'm considering migrate to AWS Device Farm. The problem? After a quick calculation, it would cost me around $10 / device / day, which is waaaaaaaaay too much.

That made me wonder whether the approach I've designed is not correct. If for example, I run system test once per day, it'd be $0.51 / device / day, which is far more acceptable.

So, the bottom line of this question is, in a continuous integration environment, should the product be constantly built?


It all comes down to finding a balance point between cost of achieving quality and quality.

  • Perhaps you can start by calculating average time it takes for system tests to find a bug? E.g. If it takes around 5 days for system tests to find a bug, then you can try running system tests every 5 days and keep your eyes on quality. If there is no drop in quality, then go with 5 days or even relax it even more to 6 days or etc.
  • Have you considered running your tests on Android and iOS emulators locally? This will help you out in reducing cost.

I personally believe in a continuous integration environment, the product should be constantly built, but there are always trade offs.

| improve this answer | |

You should narrow the number of devices you run your tests against. Run your tests on the devices which the majority of your users use.

You should probably run your test suite at least nightly and once before release.

Optimally, tests should run after each commit to give people feedback sooner. We built a local device farm because the device farm services in the industry were expensive and flakey. It was better for us to have the device in front of us while we debugged.

| improve this answer | |

Generally yes, in a continuous integration environment, the product should be constantly built. However every company will have slightly different needs that affect how that is implemented.

I would want to know more about these "stability" tests. "Stability tests" is not one of the 4 quadrants of Agile Testing so it sounds like it represents something specific to your company. My question is what kind of tests are these - unit? integrated? UI? Performance?

My company currently has its own term that it invented called "shakeout". We run "shakeout" tests to make sure the environment is up and working. Unfortunately this is also an invented term by an organization that is is more used to manual testing and the experience of finding environment issues after a few hours or a few days. Organizations with these sort of issues have not often not yet adjusted to the automated tests given feedback in minutes about the environment not being set up correctly. This may not be a cause in your particular case.

I recommend you classify the kind of test you have as:

  • Unit
  • Integrated
  • UAT
  • Performance and security

and that you classify the type of tests in those areas using:

  • Smoke
  • Happy
  • Sad
  • Optional
  • Compliance

As for the volume of running and cost I would consider maybe running Happy and Optional UI cases to reduce cost. Sad cases tend to be the majority and happy cases usually go 'though' the sad case paths anyway so happy cases can often represent the majority of the value when time and cost is also a factor. Just make sure the business backs this decision and understands it.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.