I am a QA. This is the software chart:

Search Engine (Apache Solr) <-> Server (expose itself to UI by an API) <-> UI.

The server makes queries on the search engine and send the returned data to the UI. I want to test the correctness of the data, to make sure that the returned values is the expected returned values.

I don't manage the Search Engine. I do have an access for manage the data inside it, and making queries on that Search Engine.

As a tester I can use the server's API.

I see two options:

  1. Implementing the query on my own and make sure its the same results as the results of the developer’s query (which does not make sense in tens of tests and doesn't indicates 100% of correctness.).

  2. Insert specific data to the search enginge and check the retured values from the server or from the UI knowing the desired returned values for query in advance (which seems very fragile because sometimes new data would be inserted for new tests, and the new data may cause a failure to the existing results because of changing the assert's ecpected value).

What is the best practice from those options? I feel like I'm missing something because both of those ways are not perfect, is there a better way for doing this?

  • 1
    Is the search engine built by your organization or is it owned by a third party (like Google)? Your tags imply that the query is run against a database and that an API is involved - does the API belong to the search engine? How much control do you, as the tester, have over the database? Can you access the API directly? Without this information any answers would have to be quite general and might not be much help to you.
    – Kate Paulk
    Commented Sep 30, 2019 at 12:04
  • 1
    Is the purpose of this test to prove that the developer's query in the server is correct, or that the search engine works like it should?
    – deasa
    Commented Jan 7, 2020 at 18:17
  • Are the search engine results deterministic? Some big data searches might be stream processed, meaning you'll first get the first results that were found and later on you'll get the more meaningful ones when processing catches up. This means that your results might be inconsistent in a non predictable way.
    – Rsf
    Commented Sep 4, 2020 at 10:58

2 Answers 2


Assuming that you are working on an isolated test environment (ie, that the only data the whole system works with is your test data), I would recommend approach #2, using a specific dataset.

From your description of approach #2 it seems like the test data also contains data from tests other than yours. That is the problem. If you can control the dataset, then you will know exactly what answers your query should return. Your test case would be something like this: create your own dataset with a known set of answers, load it in a fresh empty copy of the system (ie, an empty database (or whatever other storage is used)), run the query, check that it produces the expected answers.

In this approach, you will need a separate test dataset for each query that you are testing.


To me it sounds like a typical Server-client test (Database <> Server <> Client) but you have a search engine instead of DB. Most of the time you don't want to test if DB itself is working as expected. Therefore, tests could be designed from different aspects:

  • Data validation: sequence of queries return the expected data. The focus is more on queries/macros etc. Some test scripts will do the job.
  • Logic of the server/API: GET/POST/PUT etc are returning the expected response. Of course, it is expected from developers to mock the DB and write good test for the API methods. if so, no extra test is needed.
  • UI test: focus is more on the flow and users' interactions with the UI. Many of them could be mocked, so frontend developers are able to write test more efficiently.
  • end-2-end: Run all components and do a full end to end test to see if they work well together. As you know it is usually flaky and expensive, so if previous tests are not enough, e2e makes sense. Typical high level test frameworks e.g. Selenium could help here.
  • There are many other tests depends on your need that could be added e.g. non-functional tests, contract test etc.

If any of the above tests are missing, try to agree with development team how and who can implement them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.