How to Automatically Test Statistic Data Which is not Aggregated in Real Time

Question

I want to write automated e2e / functional / acceptance tests for a statistic service which calculates the numbers only once a day.

For the tests I need to make sure that I know which numbers the test should expect.

I have one option in mind: Create an endpoint which calculates the data before running the expectations. Downside: This would introduce extra logic into the app only for the tests.

Do you have more / better options? Or am I on the wrong path, should I only care about numbers - but not exact numbers in the e2e tests - and cover the real numbers on a different test level?

Thanks a lot for your ideas / thoughts.

Answer 1

I hope I understand your question correctly. You have a system that looks something this:

• Input data sources
• Scheduled task to calculate a statistic
• Output store for scheduled task results
• A way to see the output, maybe a user interface

A test that verifies that process could look like

• Prep fixed input data
• Trigger the calculation process
• Verify the output in the user interface

I have some systems that use scheduled tasks in Windows todo calculations:

• We added a schedule service HTTP end-point to trigger the calculation
• The windows tasks uses a script to call the HTTP end-point
• Our tests use the same HTTP end-point to trigger the schedule
• We manual test that windows scheduling system works once

Some notes

• Test most of the calculations in lower level tests (unit/component tests)
• In the end-to-end test maybe only check the totals, not the details
• I like to extend applications to make it more testable, having some end-points only used by tests is not a bad thing. In this case we could also add a button to the user-interface to trigger the schedule. Maybe users might also like that.

Answer 2

I feel like there are actually two parts to verify based on your description, the calculation part, and the display of updated data.

For the calculation part, because you said it runs once a day, I assume it's a cron job, a script. Then some audits to verify the calculation will be enough. (test inputs with expected outputs).

I normally won't go with your original idea of creating an endpoint to calculate yourself, because the potential bug in your own calculation logic will create noise and not easy to maintain.

For the second part, the things you actually care about in your acceptance test are that the calculation job ran and the data is updated. The actual checkpoint to test is just "the data is updated". so if you have access to DB, test the data shown match the calculation result data in DB, if no access to DB, just make sure the data is updated.

These two parts should be sufficient for the feature, but the auditing part should be separate from your e2e test suite.