1

I am developing a Q&A web application and have implemented an expert recommendation algorithm to recommend experts based on the posted questions.

  1. How do I test the reliability of the algorithm?
  2. What metrics should I use to measure reliability of the algorithm?
  • Can you define Reliability? Are you referring to performance based reliability or reliability as far as 'I ask about dentistry but get referred to a software engineer'? – Paul Muir Oct 31 '16 at 16:11
  • Accuracy and consistency – Navindren Baskaran Oct 31 '16 at 16:11
  • This is a math problem, not QA problem. QA is about the reliability of engineering – Peter M. - stands for Monica Oct 31 '16 at 17:01
  • Thank You! But can i use this metric: number of accurate results / total number of test cases. The higher the probability, the much reliable the recommendation is. – Navindren Baskaran Oct 31 '16 at 17:19
  • You can, but it's not going to help much because your test cases are by necessity going to be simpler than your actual user actions - you're talking about an expert system here. – Kate Paulk Oct 31 '16 at 17:36
2

First, you're talking about trying to build test cases to validate the accuracy of an expert system. That's not a simple exercise, and it's made more complex by your expert system referring users to human experts based on their input.

This kind of software is utterly dependent on accurate data. If it does not maintain an up-to-date database of expert qualifications, it will become less accurate over time. Testing the expert recommendation algorithm is subject to the same limits: you're trying to test that given an infinite or near-infinite possible input set your algorithm will return the best matched recommendation - but how do you know what that recommendation is?

Honestly, for something like this, I would look to implement extensive user monitoring: record each question, give the users the option to select alternative experts from a ranked list, and record their choice vs. the top-ranked expert. If there's a difference, ask them to say why they chose differently.

The data you'll need to be monitoring and tracking won't be easy or cheap to wade through, but I don't see a better way to "test" your algorithm than constantly monitoring how it performs against user choices.

For future iterations, you then have the advantage of a set of known inputs and outputs you can use as your starting point to ensure you don't break things.

I'd honestly also look into the math or software design stack exchanges because they're likely to have better ideas how to refine and improve your algorithm.

| improve this answer | |
  • +1 for consulting math or software design stack exchanges. – user246 Oct 31 '16 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.