Test Management Tool that recommends test cases for next test execution?

Question

I have just finished the Machine Learning course and was wondering if there is a Test Management tool that learns over time which test cases to run for the given project and eventually recommends a test suite for a next test execution?

Answer 1

Microsoft's Visual Studio can collect details of which pieces of code are executed by which tests. This data can then be used when code is changed to suggest tests that should be rerun. You could try some Web searches for terms such as "Microsoft test impact data" and "MTM recommended tests".

Answer 2

I am not aware of any.

But one project in python-coverage tool compares diff of the change with info about which unit tests cover which code and runs the unit test which cover the changed code only (instead of full suite).

Tool in not trying to be smart, because in Zen of Python we have koan: "In face of ambiguity, refuse the temptation to guess." Reason is: guess can be as often good as misleading, and there is no automated way to tell which is which. Report the ambiguity and exit instead.

Answer 3

I wanted to create one myself (worked with ML on a project before) but this had to be connected with source code and I figured many people would hesitate sharing their repositories.

I'll be interested too in looking an existing tool that can do this.

Answer 4

Currently Silk Central test management can analyze your keyword tests and check which ones are used most frequently together.

This then is used to recommend the next keyword in your keyword sequence during test creation.

All this is explained in the datasheet here:

https://www.microfocus.com/media/data-sheet/BDS-Silk-Central.pdf

Automatically analyzing the structure of existing keyword-driven tests, Silk Central recommends likely keywords for the currently edited keyword-driven test

The keyword library builder is supported using Java, so you can use this on your Selenium scripts if required.

Answer 5

I think that currently this is done mostly through tools providing analysis of

Code Coverage

Tools detailed at https://stackoverflow.com/a/195027/631619
Code Coverage is a measurement of how many lines/blocks/arcs of your code are executed while the automated tests are running.

Obviously you would be interested in the code that is not executed by any test so the tool needs to show that.

Sophisticated Tools will include examination of:

• scoping
• dependencies
• function code length
• core vs. optional processes
• number of times function is called
• whether a value is returned from a function
• use / abuse / suggestion of design patterns
• number and type of parameters and use in logic paths

so that they can grade code and not just provide pass/fail

Answer 6

Unfortunately, I'm not aware of any off-the-shelf software, but have a look at the field of search-based software engineering (SBSE), especially its subfield search-based software testing (SBST).

For instance, in his talk on SBST, Mark Harman explains how automatic test suite selection can be done with respect to multiple objectives:

• previously found faults
• code coverage
• execution time
• …

He also refers to Shin Yoo's work, which includes very good papers on that topic. As an example, "Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation" reports results from a project that integrated search-based test optimization into Google's regression testing processes.