What is the Best Way to Manage Test Automation?

Question

At our company we have multiple test automators and they all are using different tools to test their applications (mostly APIs and a web app). The problem I see with this is that:

1. Creates silos, hard for other testers to help out
2. Duplicate effort

I want to combine everyone's tests into one suite or to I guess combine all the frameworks to one place. This way testers can help out where needed and there is no steep learning curve. Is this the right approach or is it best to leave these 'silos' up?

Answer 1

There's belief that one standard framework will solve problems and that this is the only way to solve them. This is because initially frameworks tend to solve problems:

Contextual systems provide more scaffolding, better "out of the box" behavior, and contextual intelligence via that scaffolding. Thus, contextual systems tend to ease the friction of initial use by doing more for you.

However, one framework for all your teams may not be the best solution:

• There is no silver bullet: a wunder-framework that fits everyone. Your teams might have reasons to work differently: slightly different technologies, slightly difference processes.
• Frameworks are hard to maintain. Putting things related to Web API testing, UI testing, reporting, etc. into one framework make it heavy and hard to maintain.
• Frameworks are hard to customize. The initial scaffolding makes customization later harder. The frameworks enforces you to do things in a certain way, limiting what you can do or not. Changing a framework might be hard because such a change would be impact other teams as well.

Still, there are other ways for your teams to get out of silos:

• Build libraries. I recommend you to listen to the presentation "Building common libraries and not failing at it" from Devoxx Poland 2016 conference. It talks about experience of moving from one framework to the family of libraries. Basically, each library follows old Unix philosophy "Do one thing and do it well", so they are useful and easier to maintain.
• Open library market. Each team builds their libraries and share them with other teams. For instance, if I am working on authentication and another team has already build a library for authentication I can reuse it. But this doesn't force me to change also the way I generate reports, load test data, etc.
• Introduce one coding style so it is easy for new comers to read existing code.
• Present problems and how each team solved it, so teams are familiar with libraries of other teams.
• Introduce cross-team code reviews from time to time, so people familiarize with code of other people, build and enforce coding standards.

Answer 2

There are two answers.

1. You should move to one framework, of course.
   As you yourself pointed out, standardisation, flexibility and preventing duplicate effort makes this well worth the effort in the long run.

2. Keep the stove-pipes!
   Well-known fact that different environments have different strengths. No doubt in your company for every application the best tool was used. Main effort has gone into making it all work right and building up a nice suite of useful tests. No use throwing all that away.

But.
In real life nothing is cast in iron. Middle ground is to keep running the existing suites, but at the same time focus on one environment and try to move there exclusively for all new automation.

Once the new direction is established try to find room in the background and quietly convert the already existing suites to comply as well.

Step zero is to convince the testers to make the effort. They need to share your vision and see the benefits, like learning new stuff and improving the test scenarios. Enthusiasm is a requirement for an enterprise like this.

In the end you'll likely still end up with at least two environments, as web apps are a rather specialized kind of breed.

Answer 3

I think that it is important to consider another option as well. Let the Developers teach other their specific test framework(test automation tool), this way they can all see the strengths in the specific tool while learning something that they can bring into their testing framework/tool.

I do agree with @Bookeater's answer as well, but think it is important to Learn about new things because that technologies and languages that are being used are changing constantly, so to be stuck in a single language or framework isn't beneficial either, in my opinion.

I believe that the Testers themselves should help to make the decision as to what testing framework(s) that you use. letting them learn and look at each of the frameworks let's them figure out for themselves what the better framework is going to be to test the specific applications.

What I have been trying to get to is that they can help you decide which framework will work better for them and the applications, and also which one will be easier to standardize as well.

Answer 4

From my point of view there are no silver bullet solutions. API test and Web Test are different per nature. API test can be executed using standard Unit or integration test.

Web test are basically functional test. They are planned and executed manually or automated with some automation framework., like Selenium, Eggplant etc. Those test can be ported to any code base so they can be executed ("triggered") automatically alongside API test (unit/integration) on the same integration server. Optimally you will have to setup some smoke "unit tests" and parallel execution of long running Web tests. BrowserStack, SauceLabs or proprietary SeleniumGrid solution should help.

If you build some sort of abstraction on top of the automated Selenium scripts the same team can code and maintain both the test libraries in one standardised manner. Think also of having test plan defined with Gherkin syntax. This way the automation library can be scaffolded as well...