I have written and continue to maintain a fairly complex back-end service written in Java that scrapes arbitrary web sites, involves long-running processing jobs, and includes background tasks that run periodically and watch for strange states in the database, process space, and filesystem and fix things up.
I want to Unit- or Module-test more of my system. Most troubling are these background tasks that run periodically, testing for abnormal conditions to arise with the system, like a sub-process not dying when it should, or a database field not getting updated because the parent process dies prematurely.
How do I go about testing this stuff via mocking or other standard testing paradigms? That is, without having to actually cause these strange conditions that the background tasks watch for?
The normal course of things is:
Parent process sets job status to “processing”, launches subprocess and starts communicating with it
Subprocess does its thing, sends “done” to the parent process, and then dies
Parent process gets “done” message from subprocess, updates job status to “complete”
There are try/catches and timeouts in place to catch many things that can go wrong. Two things, however, can go wrong that can’t easily be dealt with via regular serial logic:
Parent process dies. Subprocess eventually gives up trying to talk to parent process, and dies. Subprocess has no access to the job status database. Job status is left in orphaned “processing” state.
Subprocess sends “done” but then never dies. It holds onto the memory, and possibly file handles, eventually causing problems as multiple such cases accumulate.
You might think that the parent process could always make sure that the child process was dead at the end of a job, but we’ve tried actively killing the process, and this doesn’t seem to work 100% of the time. If it works at all, it takes a long time for the job to finally die, longer than we want to keep the parent process busy waiting for the child process to finally fully go away (the child process is a beast not of our making). We’ve had better luck having an independent background task come along later and periodically check for zombie subprocesses and try to kill them. They eventually can go away. It might be that there’s more to be done here to try to get the parent process to kill the child process, but until then, the question of testing the current mechanism still seems valid.
There are many parent processes running these jobs on separate EC2 instances, so when a parent process starts up, it can’t assume that any job in a “processing” state is an orphan and should be rescheduled. Due to our autoscaling behaviour, we can’t really mark jobs that that particular “worker” had started before, so it can find those and reschedule them when it restarts. That worker may never restart.
We deal with 1) via a heartbeat mechanism. The parent process periodically updates a database table to say that it’s still working on a job, and updates a “last check-in date”. We have a background process that runs and checks for the “last check-in date” to be sufficiently far in the past that it can be assumed that the associated job has been orphaned. The background process then reschedules the job.
We deal with 2) in a similar fashion. The parent process puts the process id of each running subprocess in a table while a job is running, and then removes it after it has tried its best to kill the process. We then have a background task that runs, does a “ps” to collect all of the subprocesses that are still running, looks to the table of running jobs to see if a running subprocess’s id is in the table. If it isn’t, tries to kill that subprocess.
I have really no idea how to test these background tasks. They have to do with the system not behaving in an expected way, or someone coming along and killing a process without letting it do proper cleanup before dying. I’m thinking of various parts of the system I could mock, but nothing seems very right in terms of these background tasks.
Can anyone offer any advice as to how I can test these background mechanisms that only look for strange system behaviour? I currently use Spring, Mockito, JUnit4, H2 and various assertion libraries for unit and module-level tests.