5

I have been asked to test a new project, which is a web application for performing job searches. I've tested these kinds of products before, but this new project implements machine learning to enhance a couple features. These features include suggesting suitable jobs to a profile, and sending out targeted jobs via emails.

I've not tried to test a system which relies on machine learning. How would I go about doing so? Specifically, how would I test this system to test that profiles are being sent 'correct' jobs via emails and through the website?

  • 1
    A bit too board, can you please provide more background information? – Yu Zhang Jun 12 '17 at 10:14
  • Background about what application or my experience? – Nitin Rastogi Jun 12 '17 at 10:16
  • It's a career related portal to search suitable jobs as per profile, some suggestion, auto mail notification for newly posted jobs ...etc ...in nutshell, all result will be generated based on machine learning algorithms. – Nitin Rastogi Jun 12 '17 at 10:25
  • Ignore the ML stuff for now (but don't say that) and focus on the other ML - My Learning about what the business objectives are in more detail – Michael Durrant Jun 12 '17 at 11:39
  • Voting to close this as too broad. Maybe re-asking about something specific. – Chris Kenst Jun 12 '17 at 20:46
6

Don't worry too much about the idea of testing a product which includes machine learning algorithms - you just need to make sure that it's returning accurate results.

Primarily, testing ML functionality involves creating and uploading a training data-set (the developers may have one already), uploading a smaller more specific testing data-set, and then confirming the algorithm is learning/reacting correctly.

For example, the following steps could be taken if this was an ML algorithm used for job recommendations. You'd first want to enter a couple thousand jobs. Then you'd set up a profile on the application containing specific data such as preference and location data. Afterwards, you'd upload a testing data-set containing jobs that should be attributed to you, and jobs which shouldn't be attributed to you. Finally, you'd check that the 'good' jobs are recommended, while none of the 'bad' jobs are recommended.

This process of loading a large training data-set, and then a smaller testing data-set, should be applicable to most applications of black-box ML testing.

3

At the outset, the machine learning models have been termed as non-testable due to the absence of the test oracle. However, one could make use of pseudo oracle approach such as some of the following for testing the machine learning models:

  • Use the metamorphic testing technique to test the predictions of the models. Here is a detailed one on performing black box testing of machine learning models.
  • Use the dual coding technique to compare the predictions of the model created using different algorithms.
  • Try using framework such as Lime to examine the predictions from the perspective of features contribution in every prediction.
  • Create different data slices and test the predictions across these data slices

One could do white box testing of ML models by examining some of the following:

  • Data
  • Features
  • Aspects of ML models
  • ML pipeline
1

I assume that you are testing the responses from the app itself, not how it learns (process of the learning and the results of it).

Testing app which includes machine learning is like any other opaque/black box app.

My only concern would be that as application learns more, what previously was valid correct answer would become invalid (as better answer is provided), but you would have to evaluate it as any changed answer from a black box: Is it valid? You may ask ML group to evaluate if they like such changed answer, in app is learning "right way".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.