Test automation role: challenges beyond automated tests and frameworks?

Question

Before asking the actual question, let me describe the current situation.

The IT department consists of a few Scrum-like teams, with one or two software testers per team. Furthermore there are two overarching roles not bound to a team: the QA team lead, and a test automation expert.

The test automation expert has achieved the following in about 18 months:

• Create a framework to write API tests
• Create a framework to write UI tests
• Create a framework to set up data (using SQL and events) for automated tests
• Create a tool to collect and store results from all test runs
• Over 2500 API tests have been written (including by the other testers)
• Over 120 UI tests have been written

Additional notes:

• Training of the team members is ongoing (basic coding skills to writing automated tests).
• Tests are integrated in CI pipelines by the team lead and architectural team.
• Maintenance of API tests is a responsibility being pushed to the teams.

Considering the above, the current 'test automation' work is now limited to:

• maintenance of frameworks and tests (but that's decreasing)
• training/coaching (limited in time of course)
• writing additional API/UI tests on new features for the teams

The question

Is there a next level of challenges for this role - beyond the frameworks? And beyond simply adding more tests? In what other ways can the role support the QA process?

One idea was exploring AI test automation but that is quite a different beast. Especially because the ROI - for in-house tools - seems hard to defend.

Additional edit

• The tests are run in pipelines (each team has at least one pipeline, and there are pipelines for consolidated Dev/Release branches). Especially for releases, tests provide a quick feedback and are fairly consistently at about 98,5% passing rate.
• Test data management: currently, some pipelines deploy to both a "manual testing environment" having a full restored database, and and "automated testing environment" having an empty database. This database is first of all prefilled with fixed core data (specific scripts). Furthermore every test is self-sufficient in that it prepares its own customers, contracts etc.

Answer 1

What's the overall execution time? How does the feedback loop look like?

Talk On: Accelerate Automation Tests From 3 Hours to 3 Minutes

Can be detailed results provided in a few minutes(or even better in seconds without compromising quality) on the overall health of the build to the relevant stakeholders with relevant metrics?

I would ask myself these questions in a similar situation, as a dramatic decrease in execution time(like from a few minutes to a couple of seconds) may prove to be something 'game-changing' in a fast-paced environment.

This may indirectly increase the release frequency of the SUT and greatly aids to the development team to experiment/change and perform refactoring on a greater scale with the help of this increased feedback loop. Both are highly visible value -addition to the team and the management.

As with any idea, this may or may not be relevant for certain business domains and project teams but definitely an idea worth exploring.

Answer 2

The value of this role will depend a lot on how complex or unusual your product is.

If your product is reasonably simple to test and doesn't require complex engineering then there may not be a lot of value in having a framework specialist. For example a website and IOT stack have very different test requirements.

As someone who is framework specialist what i tend to add to the process is:

• documentation - A proper framework should be fully documented with each release , i do this with inline documentation that is extracted with sphinx. This ensures that test developers know how to use the tools.

• releases - If there are more than 1 people using it then you should have releases of your test framework, if not test developers will struggle to keep tests stable as they will have to manage changes from the product and the framework.

• packaging - A test framework is useless if its not packaged in a way that makes it easy to deploy and allows you to A/B test versions of it. It should be easy to deploy in CI and on developers machines in as few steps as possible otherwise debug and triage will eat all the testing time.

• encapsulation - Ideally the framework should encapsulate enough of the product that tests are focused on the business and functional requirements without having to handle the mechanisms underneath which may be changing. Good encapsulation also makes it easier to provide mocks and doubles.

• support - The test framework developer should have time to help triage bug reports from their framework as well as train test developers in using the framework.

• tests - Every test framework should have tests, it will track fast moving code and have a lot of tricky dependencies therefore it needs a test suite that will quickly show breaking changes.

• utilities - A good framework developer watches how people use their framework and adds tools that fit the needs of the people using it.

Its tricky to attach metrics here but a good framework should be measurable in the following ways :

• reduced bug triage time - when bugs are reported from a framework that is well written they are unambiguous and contain all the information required as well as a simple method for reproduction.

• reduced test development time - testers should be a lot more productive with a good framework as they will be assembling parts rather than writing code from scratch.

A big mistake is to measure the amount of tests or the amount of test passes , neither of these things will help and are very easy to distort.

The takeaway from this i think should be , only some orgs are big enough to sustain a test framework developer and sometimes they only need one at the beginning of a project.

Answer 3

Create a tool to collect and store results from all test runs

At my team our next challenge was trying pinpoint the failure to a specific component and possibly to a function and not just the commit that triggered the CI.

Our system was complex and tests above the unit level had many environmental dependencies and consisted of many sub systems.

Logging was great, but each test resulted in huge multiple log files from the different sub modules. The idea was to build a clear picture of what happened in the system, compare it the what you would have expected from an healthy system.

The last part required machine learning algorithms, and if we had a chance to complete it we could also find problems that were not hard coded into the test's oracles.

tldr; use machine learning to analyze your results to find problems

Answer 4

Production monitoring. You can't test on prod, but you can monitor it. For instance, number of credit applications submitted. If you make a deployment and this number drops to zero, something is up. Optimally, you can monitor different endpoints over time and gather historical information. Eventually, you can use this as a baseline and start to establish a threshold to alert people to check if a service is functioning properly.

If you are really bored/have a lot of time, you can add some machine learning to and slowly train your thresholds to be more accurate.

You can also start parsing your logs for errors. Often times, UI tests will cause an error but your test misses it because you weren't looking for it specifically. If you get the backend logs from your website as your automation is going through the system and look for errors there, you will catch more bugs than your UI tests alone.

Answer 5

As you start to look at the bigger picture for more challenges I think it will help to consider the Agile Testing Quadrants - Unit, Integrated, UI and Performance/Security.

You will also want to look at them in respect to the Agile Testing pyramid.

Test Data is critical. Make sure you have isolation. Don't go overboard to create 'Joe Smith, 123 Main Street'

Increased and improved Visibilty of the CI pipeline (like on overhead and "public" (inside company) locations.

Performance testing using functional tests for the workflows is often a good thing to embark on to address that risk

Answer 6

Move from verification to quality improvement

Typically tests, especially end-to-end ones, verify quality at the end of the process against what has already been made. Tests that are verifying quality will not, alone, be the artifacts that actually improve quality.

The next step in the quality journey is to focus on actual quality code activities. There are several key areas to focus on:

• Promote stronger linting. This is one of the cheapest 'shift-left' activity you can do. Strong linting provided guidelines for consistency and for following standards to make code more readable

• Advance quality CI processes. There is only so much you can do on your local machine. As infrastructure grows it becomes increasingly important to run your tests in the cloud so they can be parallelized to run faster and consistently in a way that is visible to all. This means getting proficient and knowledgeable about using tools such as Jenkins and circleCI and approaches such as IAC (Infrastructure as code).

• Promote quality PR practices. Make sure that PR's have healthy open discussions and are not a rubber-stamping process

• Grade your application code. Perform static analysis with tools such as code climate to rate your code and point out low quality issues such as duplicated code, long methods, too many parameters, etc.

• Focus more on code coverage during application development so that it is one of the key gating parts of adding code

• Talk more about quality issues including testing and automation during backlog refinement sessions.

• Champion development practices such as BDD and TDD to have quality baked in during development

• Focus on cycle time and approaches to reduce how long tests take including parallelization, deleting tests, moving tests to a lower level, mocking and stubbing dependencies, etc. test approaches to