Supervised machine learning in a "one-time-test" classification problem

Question

Being inspired by the question "Good practices for identifying "one-time" tests?" I decided to make a proof of concept for the idea of applying supervised machine learning algorithms to a problem of "one-time-test" classification. The question such the classification would answer is When one creates a test, what class would that test be assigned? Whether that class would be a "one-time test" or not?.

In my idea that aproach would not set an exact instruction to remove a test after the new version of tested product is been released. It would rather flag a test to help a test manager revise it after some reasonable time to decide if it really needs to be removed.

In my comment I suggested to build up a Neural Network, however NN is not the only paradigm that can be used. Without respect to the particular classification algorithm, all supervised algorithms require a training data set that is composed of a set of "features" and the actual class that the feature vector (basically it means a set of test properties) represents.

Thus there are two problems I have to solve. They are:

1. What is a "one-time test" (it is rather not a problem but a question)
2. How to define the appropriate feature-vector.

Regarding the first question I would define a one-time test as: "a test that is executed for only a single release on the extent of, say, 5 sequential major releases". The second issue is actually what I would like to get your advice for. This might be considered as a dup of the question that I've been inspired with, however this one emphasizes the countability of the test properties and rather the objective analysis of the test properties. Yet, I have come up with the following (considering the properties should be countable) features (which I think impact the class the test would be assigned to):

1. Number of test steps
2. Estimated execution time
3. Number of high priority tests the test designer has introduced overall
4. Number of low priority tests the test designer has introduced overall
5. Number of reworks the tests designed by the particular test designer underwent
6. Ratio of one-time-to-non-one-time tests the test designer has introduced overall (this one and three previous features is the attempt to objectively define subjective test designer competence)
7. Number of letters in test description (since that could define how concrete the tester is)
8. The number of the day in a year the test is created (since there are some days when a test designer might put less effort to their job as they would usually do)
9. Number of the tests for this particular component that exist. (Represents how well the component is covered with the tests so far)

Hence my question is:

What countable test properties (which could be picked from a given regression historical test set) could potentially impact the test reusability rate?

Answer 1

Sounds like a reasonable approach. However, regardless of all the other features you may come up with, you probably want to take the test's past bug detection into account (production and test system, but be aware of flaky tests). This can be an absolute (e.g. total number of found bugs) or a relative (e.g. number of found bugs within the last n days/weeks/months/…) value. When developers tend to make a specific test fail, you probably don't want to label it as a one-time candidate.

There's an awesome and freely available course on ML by Professor Yaser Abu-Mostafa. He describes the essence of ML as follows:

• A pattern exists.
• We can pin it down mathematically.
• We have data on it.

Quite often the first part isn't obvious, but I think that if you can describe it mathematically and have data, simply start coding. ML involves a lot of experimenting. Therefore, the sooner you start, the sooner you will know what to adapt. To quote Rasmus Rothe:

[…] in real-world scenarios, it is less about showing that your new [ML] algorithm squeezes out an extra 1% in performance compared to another method. Instead it is about building a robust system which solves the required task with sufficient accuracy.