How to test search engine results?

Question

I had an interview question:

You were told that a search engine result by target query does not satisfy a requirement by its position. How would you test it?

How would you test it?

I had some thoughts though:

• You need a list of input search queries
• You need a target position for each of the query
• You match each target position with actual

But it was not enough.

Answer 1

I am not expert in search engines, but there are many factors that contribute to how search results are displayed. The query is is essentially the starting point, and how the engine parses the query is important. Also some search engines will 'refactor' the query based on natural langauge algorithms. The order of returned results is often dependent on various metrics such as popularity, # of matching words, keywords, tags, etc. Also, some of these measures are given more weight than others.

So, this question by itself doesn't really provide you with enough context, or perhaps the purpose of this question was to get you to ask questions to derive more context. Sometimes interviewers will ask vague questions to see if the interviewee will start taking "stabs in the dark" without gaining sufficient context (e.g. start banging away at problems and hope to stumble on a solution, or think about the problem and consider potential solutions with forethought).

Answer 2

Actually, the question how to test search engines is part of information retrieval (IR) problem. I will list just a few approaches how they can be evaluated/tested.

Traditional IR approach

One way in traditional IR is to prepare test setup, including:

• Candidate queries
• Dataset of indexed pages
• Expected result set (included expected ranking) for each query

Next you obtain actual results by running your search engine over candidate queries and compare actual results with expected results using one of information retrieval metrics, e.g. precision/recall or nDCG. That would give you a quantitive answer how bad or good is your search engine with respect to expected result set (requirement).

Obviously, the answer will be context-specific. It will particularly depend on choice of candidate queries, they intention and wording, the dataset of indexed pages, and the way expected results were judged. So what was the context?

Goal-driven/Interactive IR approach

Another approach, maybe more practical, would be to involve potential users and let them assess quality of results for a given query. They could bring their own queries or you could define queries for them. Those could be a few people or a whole community, from which you could get more implicit feedback about results quality (e.g., results that are clicked are expected ones).

Problem isolation

Finally, the problem might be not to evaluate the overall quality of the search engine, but only narrow down the reason for low ranking of this particular query result. So, I would follow the problem you got:

You were told that a search engine result by target query does not satisfy a requirement by its position.

and asked myself:

• What were the other results that were ranked higher than this?
• Why were they ranked higher? Can I tell this from result snippets? Maybe they match query keywords but with a different meaning? So maybe the wording is incorrect? Would another wording of query intention improve the result?
• The same query maybe worded differently, and the same wordings may have different query intention behind them. Do I really know what was intention of a searcher behind this query?
• What precisely is the requirement? To boost higher sponsored pages? Or rank higher pages that matches query intention?

Those can help you reproduce the issue for other similar queries.

Answer 3

I worked for a SEO company in 2004. Our target was to get natural/organic search results on the first or second page of Google for the targeted key phrases (Natural/organic search results are the regular search results, not the ones sponsored by Google Ad Words). The company was a start-up and decided to focus on Google only since it had more than 80% of the Search Engine market at the time and studies showed that users very rarely looked past the first two pages of Google for a search result match.

The way we tested the results was to type in the targeted key phrases (combination of keywords) into Google and see if the listing showed up on the first or second page of Google, if yes then it was considered a success, if not then it was considered a failure.

Today things have changed with Google's targeted advertising so people might receive different results from different computers and if they are logged in to Google Plus this might affect the results too, so I don't know if its even possible to test this anymore.

Answer 4

Because the question mentions "search result by the query does not satisfy its requirement by position" , it looks like the test objective is to ensure that the right ranking / relevancy based sorting is in place on the search algorithm . To debug / drill down this further, I might ask more details on the query- was it just a simple text search on a single field or did it involve a combination of fields , did it involve any non-English characters ( diacritics etc). Sometimes the reason why a result might be ranked lower could be that some special characters are omitted in search or some specific fields are not indexed due to which the search omits those fields.

We could test by

• Use the same term / query with additional combination of fields
• See if the hitmap works fine - i.e does the first document in the search results really have more "search term count" than other docs in the field searched for - opening the document and counting would assess this.
• Eliminate one character at a time or one field ( for multi fielded search) and then check if results are relevant
• use a synonym of the term ( for term search) and see if it pics that doc and ranks higher
• Narrow down the search by date or any other field and see if a reduced result set ranks it higher

Another method would be just to call the Search API using a tool like fiddler or Postman and then compare the rank returned with the ranking in the UI. Sometimes while parsing and loading the client side page, due to some caching issues, sorting might be messed up.