I am developing a Q&A web application and have implemented an expert recommendation algorithm to recommend experts based on the posted questions.
First, you're talking about trying to build test cases to validate the accuracy of an expert system. That's not a simple exercise, and it's made more complex by your expert system referring users to human experts based on their input.
This kind of software is utterly dependent on accurate data. If it does not maintain an up-to-date database of expert qualifications, it will become less accurate over time. Testing the expert recommendation algorithm is subject to the same limits: you're trying to test that given an infinite or near-infinite possible input set your algorithm will return the best matched recommendation - but how do you know what that recommendation is?
Honestly, for something like this, I would look to implement extensive user monitoring: record each question, give the users the option to select alternative experts from a ranked list, and record their choice vs. the top-ranked expert. If there's a difference, ask them to say why they chose differently.
The data you'll need to be monitoring and tracking won't be easy or cheap to wade through, but I don't see a better way to "test" your algorithm than constantly monitoring how it performs against user choices.
For future iterations, you then have the advantage of a set of known inputs and outputs you can use as your starting point to ensure you don't break things.
I'd honestly also look into the math or software design stack exchanges because they're likely to have better ideas how to refine and improve your algorithm.
