Is there an industry-standard term for choosing which tests to run?

Question

I have been searching for information on mapping tests to code changes but I haven't had any luck. I suspect that there is some industry standard term that would have better google-juice.

Suppose you have tens of thousands of hours of tests and millions of lines of code. If you know that certain tests should be run whenever you change a piece of code, where others could be run only periodically. What would you call the process of picking tests to run?

We call it test charting, but the complete lack of google results leads me to assume there is a better term out there. Or is this just a rare-enough challenge that not many organizations face it?

Answer 1

I have a partial answer. I am fairly certain that while I was at Microsoft we called this "Test Selectivity" but Google searches for that term are pretty sparse as well. Possibly a better way to find information on executing only specific tests when certain product code changes is to search for the technology that make this possible. You need a code coverage tool that allows you to measure code coverage per test case such as Clover: http://www.atlassian.com/software/clover/tour/java-code-coverage.jsp (just an example I found on Google, there are many others). Then you need some way to associate a product code change with those test cases (diff the code and then pinpoint test cases that previously covered the same methods where the code changes happen). In a quick Google search I didn't find any information on specific technology or tools that allow you to make that association and filter your test cases that get executed. I know we had some in-house tools at Microsoft, but I'm not sure what exists out in the wild and you would need to do some more searching to find those. I would assume that some of the code coverage solutions would have these features bundled with it, but again, some more searching would be necessary.

Answer 2

I call the general practice test selection.

When I use a series of suites, run at different frequencies or under different conditions, I call it test staging.

Answer 3

In my group at MS we refer to this as targeted regression testing and I wrote about it here (http://www.testingmentor.com/imtesty/2009/11/18/the-minefield-myth-part-2-the-value-of-regression-testing/).

You will need a way to determine which binaries (exe, dll, etc) have changed binarily for each new build. A tool that will perform binary compare of the files in 2 directories is Diff2Dirs (http://www.testingmentor.com/tools/generaltools.htm).

Next you will need a tool such as Dependency Walker (http://dependencywalker.com/) to help identify other dependent modules.

Of course, this also assumes that your test cases are somehow mapped or associated with the product binary or binaries they are targeting. We organize our regression test dlls so we know which product code binaries they are hitting, but also by prioritization. So, we can target only changed code & dependencies, or we can run only pri 1's, or everything.

Answer 4

I'm not aware of any specific terminology for this. I personally call it "targeted regression" (as opposed to general regression where you're going over everything).

In the software my employer builds, there's a massive codebase with an immense level of complexity (the combinatorial math on well over 100 configuration flags scattered through the system is bad enough without considering the things the software actually does), so our regression scripts have, over the years, evolved into a series of feature-specific script runs each of which is more-or-less independent.

They aren't as granular as "changing line X of file Y triggers test Z", but we'll do targeted regression if, for instance, we know there's a change to a specific feature.