A standalone framework for testing multiple products/services or individual frameworks per product/service?

Question

I'd like to know what advantages/disadvantages to testing a range of different sites, applications or services using a single, standalone framework?

Let's say that some of these sites combine to form an end-to-end journey, and there are also API endpoints involved.

To me, having a single standalone framework that houses the API and acceptance tests for the different sites means we have a unified framework where we only need to solve and maintain consistent design patterns (PageObject, helper functions (e.g. date conversion)) once and only once.

However, most modern frameworks tend to have the API and acceptance tests sit inside the application itself. For instance, you generally define your unit tests in Karma and end-to-end tests in Protractor for an Angular app. This means that for each product, you are pushed towards using a particular set of tools or paradigm that is specific to that technology solution, and you have to write things from the 'ground up' so to speak.

In my view, it would be nicer to use an all-in-one framework that can run API tests and acceptance tests and recognise different projects. This could be done in many frameworks (I've been looking at Cypress.io, CodeceptJS and Webdriver.io for instance).

What are the downsides to having a unified, single repository and framework for housing all your tests?

Answer 1

Just to reiterate, a Framework is not just a set of tools or languages.

https://smartbear.com/learn/automated-testing/test-automation-frameworks/

A framework is comprised of a combination of practices and tools that are designed to help QA professionals test more efficiently.

These guidelines could include coding standards, test-data handling methods, object repositories, processes for storing test results, or information on how to access external resources.

While these are not mandatory rules and testers can still script or record tests without following them, using an organized framework typically provides additional benefits that would otherwise be missed out on.

So, Cypress.io, CodeceptJS and Webdriver.io for instance are just tools.You can create a all in one framework using any language, you can use requests module in python ,HttpURLConnection or restassured in java etc for API testing. Similar modules wil be available for database testing, performance testing etc. You can develop your on modules also if you like.

But, the already available tools are developed and designed to overcome the challenges in handling issues in SUT. For instance , protractor natively handles asynchronous nature of angularpages and there by makes test execution pretty fast.

If you try to reinvent the wheel, it might end up in following issues:

1. Cost overhead
2. More turnover time
3. Lesser efficiency
4. More learning curve
5. Increased complexity
6. More development time
7. More human resource requirments

From my experience, the best implementation time and flexibilty for API testing was provided by Robotframework as python natively handles json unlike in java. For UI , protractor-jasmine gave faster turn-around time as it didn't require complex frameworks like testng or maven etc.Protractor-jasmine also allows to play around with low level programing than making the entire thing complex using feature files like in cucumber or keyword tests in robotframework.

Summary:

Like tests are context depended , tools are too. Choose the best solution and not the common solution. Just like we have javascript , angular etc for front end, and , python,java ,.net etc for backend development.