High-level help desired in forming a plan of attack for testing of background "cleanup" tasks

Question

I have written and continue to maintain a fairly complex back-end service written in Java that scrapes arbitrary web sites, involves long-running processing jobs, and includes background tasks that run periodically and watch for strange states in the database, process space, and filesystem and fix things up.

I want to Unit- or Module-test more of my system. Most troubling are these background tasks that run periodically, testing for abnormal conditions to arise with the system, like a sub-process not dying when it should, or a database field not getting updated because the parent process dies prematurely.

How do I go about testing this stuff via mocking or other standard testing paradigms? That is, without having to actually cause these strange conditions that the background tasks watch for?

The normal course of things is:

1. Parent process sets job status to “processing”, launches subprocess and starts communicating with it

2. Subprocess does its thing, sends “done” to the parent process, and then dies

3. Parent process gets “done” message from subprocess, updates job status to “complete”

There are try/catches and timeouts in place to catch many things that can go wrong. Two things, however, can go wrong that can’t easily be dealt with via regular serial logic:

1. Parent process dies. Subprocess eventually gives up trying to talk to parent process, and dies. Subprocess has no access to the job status database. Job status is left in orphaned “processing” state.

2. Subprocess sends “done” but then never dies. It holds onto the memory, and possibly file handles, eventually causing problems as multiple such cases accumulate.

You might think that the parent process could always make sure that the child process was dead at the end of a job, but we’ve tried actively killing the process, and this doesn’t seem to work 100% of the time. If it works at all, it takes a long time for the job to finally die, longer than we want to keep the parent process busy waiting for the child process to finally fully go away (the child process is a beast not of our making). We’ve had better luck having an independent background task come along later and periodically check for zombie subprocesses and try to kill them. They eventually can go away. It might be that there’s more to be done here to try to get the parent process to kill the child process, but until then, the question of testing the current mechanism still seems valid.

There are many parent processes running these jobs on separate EC2 instances, so when a parent process starts up, it can’t assume that any job in a “processing” state is an orphan and should be rescheduled. Due to our autoscaling behaviour, we can’t really mark jobs that that particular “worker” had started before, so it can find those and reschedule them when it restarts. That worker may never restart.

We deal with 1) via a heartbeat mechanism. The parent process periodically updates a database table to say that it’s still working on a job, and updates a “last check-in date”. We have a background process that runs and checks for the “last check-in date” to be sufficiently far in the past that it can be assumed that the associated job has been orphaned. The background process then reschedules the job.

We deal with 2) in a similar fashion. The parent process puts the process id of each running subprocess in a table while a job is running, and then removes it after it has tried its best to kill the process. We then have a background task that runs, does a “ps” to collect all of the subprocesses that are still running, looks to the table of running jobs to see if a running subprocess’s id is in the table. If it isn’t, tries to kill that subprocess.

I have really no idea how to test these background tasks. They have to do with the system not behaving in an expected way, or someone coming along and killing a process without letting it do proper cleanup before dying. I’m thinking of various parts of the system I could mock, but nothing seems very right in terms of these background tasks.

Can anyone offer any advice as to how I can test these background mechanisms that only look for strange system behaviour? I currently use Spring, Mockito, JUnit4, H2 and various assertion libraries for unit and module-level tests.

Answer 1

If I understand your question correctly you are looking for a way to test processes that check for an invalid db state(s) and fix them. What I can suggest is to make sure that the cron (or another scheduler that handles job execution) contains only the method invocation, and not its definition, that way you can test it without the necessity of running the cron job itself (so setting the unit test up is easier).

so instead of :

cron.schedule('0 21 6 * * *', () => {
    // ... your business logic
});

Try something like this:

function runDbChecker(timeToRun){
    // ... your business logic
}

cron.schedule('0 21 6 * * *', () => {
    runDbChecker(new Date());
});

For the fixing processes, themselves try to separate the code that identifies the problem and the code that actually fixes it and test them separately by providing specific mocked data that you know will "trigger".

A simple example would be to prepare a "detector" function that takes an entity as an input and returns a true/false value indicating if the entity should be passed to a specific "fixer" function. Since the function itself is not communicating with external services (it is given the entity as input) it should be easier to unit test since it just contains the business logic. If the business logic requires checking different tables in the db you should mock those responses with some predefined data that would confirm that indeed this data is broken.

A similar approach would be to create a specificEntityfixer function that given an input of a broken entity would return a new (fix) entity that should replace the old one. That way you can easily prepare some broken input entities (probably the same set as with the "detecting" function) and expected fixed entities that should be returned as an output.

Your detecting and fixing functions probably rely on more than just the single entity to do their job and will require a lot of mocking queries. I would suggest that you start with a good unit test coverage of these cases and work your way up to integration tests using predefined db dumps with samples of real broken entities that should be fixed automatically.

Regarding testing the processes themselves when spawning a child process that you are unsure of I would add a kill switch that triggers when a process is running for too long. You mentioned long-running scraping tasks, setting up a specific TTL might be hard if you have one then use it, if not start with an arbitrary number (like 1 hour) monitor and modify.

What you might also want to try is to add some kind of log monitor to check if the subprocess is actually working or just hanging there (it could also potentially monitor if the process is not stuck in an infinite loop of repeating operations due to an invalid retry mechanism).