3

Being inspired by the question "Good practices for identifying "one-time" tests?" I decided to make a proof of concept for the idea of applying supervised machine learning algorithms to a problem of "one-time-test" classification. The question such the classification would answer is When one creates a test, what class would that test be assigned? Whether that class would be a "one-time test" or not?.

In my idea that aproach would not set an exact instruction to remove a test after the new version of tested product is been released. It would rather flag a test to help a test manager revise it after some reasonable time to decide if it really needs to be removed.

In my comment I suggested to build up a Neural Network, however NN is not the only paradigm that can be used. Without respect to the particular classification algorithm, all supervised algorithms require a training data set that is composed of a set of "features" and the actual class that the feature vector (basically it means a set of test properties) represents.

Thus there are two problems I have to solve. They are:

  1. What is a "one-time test" (it is rather not a problem but a question)
  2. How to define the appropriate feature-vector.

Regarding the first question I would define a one-time test as: "a test that is executed for only a single release on the extent of, say, 5 sequential major releases". The second issue is actually what I would like to get your advice for. This might be considered as a dup of the question that I've been inspired with, however this one emphasizes the countability of the test properties and rather the objective analysis of the test properties. Yet, I have come up with the following (considering the properties should be countable) features (which I think impact the class the test would be assigned to):

  1. Number of test steps
  2. Estimated execution time
  3. Number of high priority tests the test designer has introduced overall
  4. Number of low priority tests the test designer has introduced overall
  5. Number of reworks the tests designed by the particular test designer underwent
  6. Ratio of one-time-to-non-one-time tests the test designer has introduced overall (this one and three previous features is the attempt to objectively define subjective test designer competence)
  7. Number of letters in test description (since that could define how concrete the tester is)
  8. The number of the day in a year the test is created (since there are some days when a test designer might put less effort to their job as they would usually do)
  9. Number of the tests for this particular component that exist. (Represents how well the component is covered with the tests so far)

Hence my question is:

What countable test properties (which could be picked from a given regression historical test set) could potentially impact the test reusability rate?

  • I'm not sure the ones specific to the particular test designer are particularly helpful (or healthy). You don't want it to be a blame-assignment system, and often people get assigned to write low-value tests, etc. so it isn't necessarily their fault. And that assumes a one-time test is the same as a low-value test, which isn't necessarily true. – c32hedge Sep 1 '17 at 16:44
  • I really like 5 and 9 though. One I might add is the number of failures over time for the test, and whether each failure was a bug found or a test error, which would be somewhat related to the number of reworks. – c32hedge Sep 1 '17 at 16:46
2
+50

Sounds like a reasonable approach. However, regardless of all the other features you may come up with, you probably want to take the test's past bug detection into account (production and test system, but be aware of flaky tests). This can be an absolute (e.g. total number of found bugs) or a relative (e.g. number of found bugs within the last n days/weeks/months/…) value. When developers tend to make a specific test fail, you probably don't want to label it as a one-time candidate.

There's an awesome and freely available course on ML by Professor Yaser Abu-Mostafa. He describes the essence of ML as follows:

  • A pattern exists.
  • We can pin it down mathematically.
  • We have data on it.

Quite often the first part isn't obvious, but I think that if you can describe it mathematically and have data, simply start coding. ML involves a lot of experimenting. Therefore, the sooner you start, the sooner you will know what to adapt. To quote Rasmus Rothe:

[…] in real-world scenarios, it is less about showing that your new [ML] algorithm squeezes out an extra 1% in performance compared to another method. Instead it is about building a robust system which solves the required task with sufficient accuracy.

  • Thanks! I agree with professor, however imho "a pattern exists" cannot be considered as a precondition to start applying ML to a problem. The pattern does often exist but is not straigntforward. Moreover several patterns could exist. Unsupervised machine learning algorithms, for example, are intended to analyze the data without knowing any pattern at all. ML alogorithms are just intended to estimate the function that would produce statistically close results to the the ones produced by the real process. – Alexey R. Sep 1 '17 at 18:29
  • BTW here is a great course also. There is a lot of practice in implementing different ML learning algorithms on your own. coursera.org/learn/machine-learning/home/welcome – Alexey R. Sep 1 '17 at 18:31
  • @AlexeyR. I agree, it shouldn't be seen as a precondition. Usually, you only guess that there might a pattern you can leverage. But even unsupervised algorithms follow the GIGO principle. – beatngu13 Sep 1 '17 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.