What prerequisites does a tester need before testing a Machine Learning based product?

Question

I have been asked to test a new project, which is a web application for performing job searches. I've tested these kinds of products before, but this new project implements machine learning to enhance a couple features. These features include suggesting suitable jobs to a profile, and sending out targeted jobs via emails.

I've not tried to test a system which relies on machine learning. How would I go about doing so? Specifically, how would I test this system to test that profiles are being sent 'correct' jobs via emails and through the website?

Answer 1

Don't worry too much about the idea of testing a product which includes machine learning algorithms - you just need to make sure that it's returning accurate results.

Primarily, testing ML functionality involves creating and uploading a training data-set (the developers may have one already), uploading a smaller more specific testing data-set, and then confirming the algorithm is learning/reacting correctly.

For example, the following steps could be taken if this was an ML algorithm used for job recommendations. You'd first want to enter a couple thousand jobs. Then you'd set up a profile on the application containing specific data such as preference and location data. Afterwards, you'd upload a testing data-set containing jobs that should be attributed to you, and jobs which shouldn't be attributed to you. Finally, you'd check that the 'good' jobs are recommended, while none of the 'bad' jobs are recommended.

This process of loading a large training data-set, and then a smaller testing data-set, should be applicable to most applications of black-box ML testing.

Answer 2

At the outset, the machine learning models have been termed as non-testable due to the absence of the test oracle. However, one could make use of pseudo oracle approach such as some of the following for testing the machine learning models:

• Use the metamorphic testing technique to test the predictions of the models. Here is a detailed one on performing black box testing of machine learning models.
• Use the dual coding technique to compare the predictions of the model created using different algorithms.
• Try using framework such as Lime to examine the predictions from the perspective of features contribution in every prediction.
• Create different data slices and test the predictions across these data slices

One could do white box testing of ML models by examining some of the following:

• Data
• Features
• Aspects of ML models
• ML pipeline

Answer 3

I assume that you are testing the responses from the app itself, not how it learns (process of the learning and the results of it).

Testing app which includes machine learning is like any other opaque/black box app.

My only concern would be that as application learns more, what previously was valid correct answer would become invalid (as better answer is provided), but you would have to evaluate it as any changed answer from a black box: Is it valid? You may ask ML group to evaluate if they like such changed answer, in app is learning "right way".

Answer 4

One thing I would add here is to separate deterministic part from a less deterministic one.

Your system has multiple functionalities:

• reading profile (deterministic)
• making recommendation (less-deterministic)
• sending email (deterministic)

Given that you can have two types of tests:

1. Unit tests, covering deterministic parts by mocking machine learning parts, e.g. a test preparing an email could rely on a fake recommendation model that always returns correct recommendations
2. Evaluation tests that make sure a recommendation model returns recommendations good enough, i.e., above certain accuracy score.