How to test data retention policy and cron job for deleting old data?

Question

There are some retention policies for the data that my company uses. For different types of data, we have different rules and these different types of data and files are used in different microservices and databases.

For example :

• file type "A" should get deleted after 30 days,
• file type "B" should get deleted after 100 days,
• and file type "C" should get deleted after 7 days.

We have a job that should delete these files for each microservice.

I want to test this cron job (automated, not manual) and make sure our data would get deleted at the right time. I need to do it with the bare minimum amount of work (kinda in a hurry), and I don't want to use an approach that may cause us liability issues and security issues. What should I do?

Answer 1

Right now, the way the question reads, it sounds like the OP wants one cron job to handle the file deletion.

In this scenario, you're not really testing a cron job, but ensuring the files get deleted. Cron jobs have been around a long time and tend to just work if setup correctly. They run whatever you want at a specified time and interval.

What I'd do is setup multiple cron jobs, one for each type of file that needs to be deleted at a specific time. You can test the setup of the cron job on some random files at a smaller interval, say 10 minutes or 1 hour, to ensure accuracy. If that works, adjust the times accordingly.

I could also argue that this question might be better asked in DevOps or ServerFault.

Answer 2

There are a few different approaches you could take to test your data retention policy and cron job for deleting old data:

Use test data: One option is to create a set of test data that you can use to simulate the different file types and the passage of time. You can then run your cron job and verify that the test data is deleted according to the specified retention policies.

Use real data: Another option is to use real data, but make sure to carefully track which data you are using and when it should be deleted. This can be a bit more time-consuming, but it can give you a more realistic sense of how the cron job will perform with actual data.

Use a testing tool: There are also tools available that can help you automate the testing process. For example, you could use a tool like JUnit or TestNG to create automated tests that verify that the cron job is functioning correctly.

It's important to make sure that you test your data retention policy and cron job thoroughly to ensure that it is working correctly and that you are complying with any relevant regulations. However, you should also be mindful of security and liability issues and take steps to protect sensitive data and avoid any potential legal issues.