As soon as a developer pushes a commit, our Jenkins run unit tests, then integration tests (e.g. just backend) and finally system tests (e.g the whole app: backend, frontend, platforms, etc). The system tests stage is fairly quick, about 3 minutes. Then, once a day, we run stability tests for 2 hours.
This has been good for some time, but we are starting to struggle with Android and iOS devices, so I'm considering migrate to AWS Device Farm. The problem? After a quick calculation, it would cost me around $10 / device / day, which is waaaaaaaaay too much.
That made me wonder whether the approach I've designed is not correct. If for example, I run system test once per day, it'd be $0.51 / device / day, which is far more acceptable.
So, the bottom line of this question is, in a continuous integration environment, should the product be constantly built?