Component testing microservice that consumes logs from azure log analytics

Question

I am testing a microservice that is grabbing logs from an Azure Log Analytics (LA) resource and returning them as a digested response or placing them into a database.

My current solution is using MS' data-collector-api to inject logs into a real LA resource, wait until my injected logs have been consumed by LA, and then call the microservice endpoint to consume the injected logs, finally verifying that the output of the microservice is in line with what is expected. The SLA on LA is about 7 minutes, but can be much longer, and the data-collector-api is still in preview with some bugs around guid values.

Is there a better way to be testing this transaction of data?

Answer 1

From my point of view your tests are wearing too many hats. If I understood correctly, your tests are doing / checking all the following:

1. creating logs on LA
2. connecting to LA
3. grabbing the logs
4. "digest"
5. validate

Judging by your provided microservice description, points #1, #2, #3, besides being very time consuming to perform each and every time for every test / test suite, they are also are kind of useless as they do not seem to be within the scope of the said microservice.

Possible alternatives:

1. Contract Testing This way you can make sure that any changes in your service will not impact the reading of logs from Azure LA.
2. Have a separate job that runs periodically to create and add a basic log to Azure LA (since you mentioned the time constraint of 90 days, then having this separate job run once a month should do the trick) and then have some specific tests that only validate the retrieval and maybe integrity of logs.

Regardless of the above approach, if the purpose of the service is to digest/interpret/transform the logs retrieved, this should be done at the service level. An approach would be for your tests to generate the random data you mentioned and save it on the disk/in memory and then the service to read it from there and validate whatever is required.

The approach of having more atomic tests should considerably speed up your testing while also allowing you to use resources more efficient and get rid of any constraints Azure LA may impose (if any).