I'm looking for ML usage in performance testing of a software. mainly Regression testing where we run same tests for every release to ensure there is no degradation. Can I get some guidance around usage, approach etc.
1 Answer
Can you be more specific?
You might be suggesting building a model that, given performance numbers for a specific release, can correctly identify performance degradation vs. random variation. For supervised learning, you will probably need at least several hundreds of data points, and perhaps orders of magnitude more, to build a model you can depend on. The size of the data set will depend on the number of variables.
You might be suggesting building a model that predicts performance degradation based on prior knowledge of code changes, unit test results, and corresponding performance outcomes. This is even more complex than what I described above. Very few companies even attempt something like this.
Machine learning is all the rage now -- everyone wants to be a data scientist -- but the truth is that lots of problems can be solved by less glamorous but effective means. If you're looking for an excuse to do machine learning, I suggest finding a different problem domain. If you're actually trying to solve a problem, I suggest looking for a simpler means of detecting performance degradation. For example, you could flag any tests that appear to be more than 5% slower than your benchmark numbers.