There are many situations when we realizing that Automation Testing must be done. I work in a product based company and we are going to start to work on that product from the scratch. In the product, we are going to add a lot of complexes functionality in the small chunks. Product function is that much complex that it may take more than two years to deploy the first version of the product as per plan. Here I want to know when should we start Automation Testing so that I can manage the cost of the testing.
20% of the features deliver 80% of the value. If you deliver in small chunks hopefully people can start using the most valuable features after the first 2-3 months already. Now you might get a paradigm-shift, probably you will not need the two years of imagined features. Let's say the business owners get new insights and make a small shift in functionality, but you want to keep using the already delivered features how are you going to make sure you didn't break anything? Manual regression testing?
I would always advise starting test automation from the start of the project. More and more common is a test-first approach in the form of Test-Driven Development. Per feature I would write:
- Lots of unit-tests
- A couple of integration tests
- One happy path end-to-end test
- One negative end-to-end test
There is another opinion in the test community. That creating test automation for software that is still in a lot of flux, e.g. the user interface changes a lot of the time, you should wait until the UI stabilises, because rewriting you tests are expensive. I think this is a false presumption and comes from the thought that most if not all automated tests are end-to-end. Which are indeed slow and expensive to develop and maintain. This happens when the developers have no experience with lower-levels of test automation and the testers have only experience with end-to-end test automation. I think this could be a result of not applying the test-pyramid correctly when gathering skills for the development team.
I think it is important that the developers can refactor their code in safety for longer running projects. They should be confident to make architectural changes to the project to keep it understandable and maintainable. To be able to do this in small steps you need lots of fast automated tests.
Start from the beginning, and consider using BDD practices for your UI so your automation can get updated along with the requirements. Go after the biggest stuff first - the login page for your end users, the main data entries they will make. This stuff is the least likely to change over time and you won't want to retest after two years!
If your application involves an api, go after that first. That section should stabilize first and there's several tools for api automation that's easy and direct - I think Postman is the most popular right now.