I have been asked to test a new project, which is a web application for performing job searches. I've tested these kinds of products before, but this new project implements machine learning to enhance a couple features. These features include suggesting suitable jobs to a profile, and sending out targeted jobs via emails.

I've not tried to test a system which relies on machine learning. How would I go about doing so? Specifically, how would I test this system to test that profiles are being sent 'correct' jobs via emails and through the website?

  • 1
    A bit too board, can you please provide more background information?
    – Yu Zhang
    Jun 12, 2017 at 10:14
  • Background about what application or my experience? Jun 12, 2017 at 10:16
  • It's a career related portal to search suitable jobs as per profile, some suggestion, auto mail notification for newly posted jobs ...etc ...in nutshell, all result will be generated based on machine learning algorithms. Jun 12, 2017 at 10:25
  • Ignore the ML stuff for now (but don't say that) and focus on the other ML - My Learning about what the business objectives are in more detail Jun 12, 2017 at 11:39
  • Voting to close this as too broad. Maybe re-asking about something specific. Jun 12, 2017 at 20:46

4 Answers 4


Don't worry too much about the idea of testing a product which includes machine learning algorithms - you just need to make sure that it's returning accurate results.

Primarily, testing ML functionality involves creating and uploading a training data-set (the developers may have one already), uploading a smaller more specific testing data-set, and then confirming the algorithm is learning/reacting correctly.

For example, the following steps could be taken if this was an ML algorithm used for job recommendations. You'd first want to enter a couple thousand jobs. Then you'd set up a profile on the application containing specific data such as preference and location data. Afterwards, you'd upload a testing data-set containing jobs that should be attributed to you, and jobs which shouldn't be attributed to you. Finally, you'd check that the 'good' jobs are recommended, while none of the 'bad' jobs are recommended.

This process of loading a large training data-set, and then a smaller testing data-set, should be applicable to most applications of black-box ML testing.

  • 1
    One thing worth mentioning is that you don't necessarily can and should assume 100% accuracy on a testing set, i.e., you don't expect the model to always return correct recommendations. You can assume some satisfying accuracy level (% of passing samples) and mark tests as green when actual accuracy level is above.
    – dzieciou
    Sep 7, 2021 at 11:09

At the outset, the machine learning models have been termed as non-testable due to the absence of the test oracle. However, one could make use of pseudo oracle approach such as some of the following for testing the machine learning models:

  • Use the metamorphic testing technique to test the predictions of the models. Here is a detailed one on performing black box testing of machine learning models.
  • Use the dual coding technique to compare the predictions of the model created using different algorithms.
  • Try using framework such as Lime to examine the predictions from the perspective of features contribution in every prediction.
  • Create different data slices and test the predictions across these data slices

One could do white box testing of ML models by examining some of the following:

  • Data
  • Features
  • Aspects of ML models
  • ML pipeline

I assume that you are testing the responses from the app itself, not how it learns (process of the learning and the results of it).

Testing app which includes machine learning is like any other opaque/black box app.

My only concern would be that as application learns more, what previously was valid correct answer would become invalid (as better answer is provided), but you would have to evaluate it as any changed answer from a black box: Is it valid? You may ask ML group to evaluate if they like such changed answer, in app is learning "right way".


One thing I would add here is to separate deterministic part from a less deterministic one.

Your system has multiple functionalities:

  • reading profile (deterministic)
  • making recommendation (less-deterministic)
  • sending email (deterministic)

Given that you can have two types of tests:

  1. Unit tests, covering deterministic parts by mocking machine learning parts, e.g. a test preparing an email could rely on a fake recommendation model that always returns correct recommendations
  2. Evaluation tests that make sure a recommendation model returns recommendations good enough, i.e., above certain accuracy score.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.