Before asking the actual question, let me describe the current situation.

The IT department consists of a few Scrum-like teams, with one or two software testers per team. Furthermore there are two overarching roles not bound to a team: the QA team lead, and a test automation expert.

The test automation expert has achieved the following in about 18 months:

  • Create a framework to write API tests
  • Create a framework to write UI tests
  • Create a framework to set up data (using SQL and events) for automated tests
  • Create a tool to collect and store results from all test runs
  • Over 2500 API tests have been written (including by the other testers)
  • Over 120 UI tests have been written

Additional notes:

  • Training of the team members is ongoing (basic coding skills to writing automated tests).
  • Tests are integrated in CI pipelines by the team lead and architectural team.
  • Maintenance of API tests is a responsibility being pushed to the teams.

Considering the above, the current 'test automation' work is now limited to:

  • maintenance of frameworks and tests (but that's decreasing)
  • training/coaching (limited in time of course)
  • writing additional API/UI tests on new features for the teams

The question

Is there a next level of challenges for this role - beyond the frameworks? And beyond simply adding more tests? In what other ways can the role support the QA process?

One idea was exploring AI test automation but that is quite a different beast. Especially because the ROI - for in-house tools - seems hard to defend.

Additional edit

  • The tests are run in pipelines (each team has at least one pipeline, and there are pipelines for consolidated Dev/Release branches). Especially for releases, tests provide a quick feedback and are fairly consistently at about 98,5% passing rate.
  • Test data management: currently, some pipelines deploy to both a "manual testing environment" having a full restored database, and and "automated testing environment" having an empty database. This database is first of all prefilled with fixed core data (specific scripts). Furthermore every test is self-sufficient in that it prepares its own customers, contracts etc.
  • 1
    I don't see much details on test data management. This is something very important for any automation effort and tells about the maturity of automation framework and areas of improvement. Could you please provide more details on that? – Vishal Aggarwal Sep 29 '18 at 13:42
  • 1
    Also, I have observed in my experience implementing CI pipelines could mean different things to different project teams in terms of maturity. Could you please elaborate on that as well as what's the role of tests in the overall CI pipeline structure? – Vishal Aggarwal Sep 29 '18 at 13:46
  • One more thing, its an excellent question :) !!! – Vishal Aggarwal Sep 29 '18 at 13:47
  • @V.A. Thanks, added two points at the bottom to clarify your questions. Hope that's sufficient. – FDM Oct 4 '18 at 19:36

What's the overall execution time? How does the feedback loop look like?

Talk On: Accelerate Automation Tests From 3 Hours to 3 Minutes

Can be detailed results provided in a few minutes(or even better in seconds without compromising quality) on the overall health of the build to the relevant stakeholders with relevant metrics?

I would ask myself these questions in a similar situation, as a dramatic decrease in execution time(like from a few minutes to a couple of seconds) may prove to be something 'game-changing' in a fast-paced environment.

This may indirectly increase the release frequency of the SUT and greatly aids to the development team to experiment/change and perform refactoring on a greater scale with the help of this increased feedback loop. Both are highly visible value -addition to the team and the management.

As with any idea, this may or may not be relevant for certain business domains and project teams but definitely an idea worth exploring.

  • Thanks, currently our 2500 API tests are running in about 35 minutes. Will certainly have a look at that link! – FDM Oct 7 '18 at 17:18
  • You are welcome.I hope you would find the link useful. – Vishal Aggarwal Oct 17 '18 at 23:10
  • Due to increased runtimes and complexer tests, reducing runtimes will be a main concern this year for us. Accepted answer as such, seeing the refactoring and impact involved in our case. – FDM Mar 4 '19 at 13:12

The value of this role will depend a lot on how complex or unusual your product is.

If your product is reasonably simple to test and doesn't require complex engineering then there may not be a lot of value in having a framework specialist. For example a website and IOT stack have very different test requirements.

As someone who is framework specialist what i tend to add to the process is:

  • documentation - A proper framework should be fully documented with each release , i do this with inline documentation that is extracted with sphinx. This ensures that test developers know how to use the tools.

  • releases - If there are more than 1 people using it then you should have releases of your test framework, if not test developers will struggle to keep tests stable as they will have to manage changes from the product and the framework.

  • packaging - A test framework is useless if its not packaged in a way that makes it easy to deploy and allows you to A/B test versions of it. It should be easy to deploy in CI and on developers machines in as few steps as possible otherwise debug and triage will eat all the testing time.

  • encapsulation - Ideally the framework should encapsulate enough of the product that tests are focused on the business and functional requirements without having to handle the mechanisms underneath which may be changing. Good encapsulation also makes it easier to provide mocks and doubles.

  • support - The test framework developer should have time to help triage bug reports from their framework as well as train test developers in using the framework.

  • tests - Every test framework should have tests, it will track fast moving code and have a lot of tricky dependencies therefore it needs a test suite that will quickly show breaking changes.

  • utilities - A good framework developer watches how people use their framework and adds tools that fit the needs of the people using it.

Its tricky to attach metrics here but a good framework should be measurable in the following ways :

  • reduced bug triage time - when bugs are reported from a framework that is well written they are unambiguous and contain all the information required as well as a simple method for reproduction.

  • reduced test development time - testers should be a lot more productive with a good framework as they will be assembling parts rather than writing code from scratch.

A big mistake is to measure the amount of tests or the amount of test passes , neither of these things will help and are very easy to distort.

The takeaway from this i think should be , only some orgs are big enough to sustain a test framework developer and sometimes they only need one at the beginning of a project.

  • Very insightful reply, thanks. However, the points you listed are all covered already, tried and tested by a year of usage. :) – FDM Sep 28 '18 at 6:49
  • sounds like you are doing it right then :) – Amias Sep 28 '18 at 10:48
  • 1
    something that is often overlooked in this role is how you can interface with the maintainers of the upstream open source test tools that you or your org are using. This can be code contributions , reviewing changes or just helping out with bug triage. All of this helps the team get the most from the tools and be aware of any changes coming. But again you need a sizeable org to make this worthwhile, i've not seen many people mange this as external consultants. – Amias Sep 28 '18 at 10:52

Create a tool to collect and store results from all test runs

At my team our next challenge was trying pinpoint the failure to a specific component and possibly to a function and not just the commit that triggered the CI.

Our system was complex and tests above the unit level had many environmental dependencies and consisted of many sub systems.

Logging was great, but each test resulted in huge multiple log files from the different sub modules. The idea was to build a clear picture of what happened in the system, compare it the what you would have expected from an healthy system.

The last part required machine learning algorithms, and if we had a chance to complete it we could also find problems that were not hard coded into the test's oracles.

tldr; use machine learning to analyze your results to find problems


Production monitoring. You can't test on prod, but you can monitor it. For instance, number of credit applications submitted. If you make a deployment and this number drops to zero, something is up. Optimally, you can monitor different endpoints over time and gather historical information. Eventually, you can use this as a baseline and start to establish a threshold to alert people to check if a service is functioning properly.

If you are really bored/have a lot of time, you can add some machine learning to and slowly train your thresholds to be more accurate.

You can also start parsing your logs for errors. Often times, UI tests will cause an error but your test misses it because you weren't looking for it specifically. If you get the backend logs from your website as your automation is going through the system and look for errors there, you will catch more bugs than your UI tests alone.


As you start to look at the bigger picture for more challenges I think it will help to consider the Agile Testing Quadrants - Unit, Integrated, UI and Performance/Security.

You will also want to look at them in respect to the Agile Testing pyramid.

Test Data is critical. Make sure you have isolation. Don't go overboard to create 'Joe Smith, 123 Main Street'

Increased and improved Visibilty of the CI pipeline (like on overhead and "public" (inside company) locations.

Performance testing using functional tests for the workflows is often a good thing to embark on to address that risk

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.