Unit testing for chains of generators in Python

Question

I want to write unit tests for methods that take a generator as input and produce a generator. I am building on this answer.

For this question let's take the method:

def candidates_for_deletion(self, generations_to_keep=2):
    for files in self.group_by_version(self.backups()).values():
        yield from files[:-generations_to_keep]

I wrote the following test which works, but I don't think I can call it a unit test, as it relies on all methods down to mocked S3 bucket to recreate the data:

import boto3
import pytest
from moto import mock_s3
from gitlabbackups import GitlabBackup


S3_OBJECTS_LIST = [
    '1526781621_2018_05_20_10.8.2_gitlab_backup.tar',
    '1530151221_2018_06_28_11.0.0_gitlab_backup.tar',
    '1530237621_2018_06_29_11.0.0_gitlab_backup.tar',
    '1530324021_2018_06_30_11.0.0_gitlab_backup.tar',
    'some_other_file',
    '1525917621_2018_05_10_10.8.2_gitlab_backup.tar',
    '1530410421_2018_07_01_11.0.1_gitlab_backup.tar',
    '1530496821_2018_07_02_11.0.1_gitlab_backup.tar',
    'a file with spaces inside',
    '1530583221_2018_07_03_11.0.1_gitlab_backup.tar',
    '1530669617_2018_07_04_11.0.1_gitlab_backup.tar'
]


def create_s3_bucket_with_sample_data(bucket):
    conn = boto3.resource('s3', region_name='us-east-1')
    conn.create_bucket(Bucket=bucket)
    for i in S3_OBJECTS_LIST:
        conn.Object(bucket, i).put('some content')


@mock_s3
@pytest.mark.parametrize("generations_to_keep,expected_candidates_for_deletion", [
    (1, ['1525917621_2018_05_10_10.8.2_gitlab_backup.tar',
         '1530151221_2018_06_28_11.0.0_gitlab_backup.tar',
         '1530237621_2018_06_29_11.0.0_gitlab_backup.tar',
         '1530410421_2018_07_01_11.0.1_gitlab_backup.tar',
         '1530496821_2018_07_02_11.0.1_gitlab_backup.tar',
         '1530583221_2018_07_03_11.0.1_gitlab_backup.tar']),
    (2, ['1530151221_2018_06_28_11.0.0_gitlab_backup.tar',
         '1530410421_2018_07_01_11.0.1_gitlab_backup.tar',
         '1530496821_2018_07_02_11.0.1_gitlab_backup.tar'])
])
def test_candidates_for_deletion(generations_to_keep, expected_candidates_for_deletion):
    create_s3_bucket_with_sample_data('mocked-bucket')
    gitlab_backups = GitlabBackups('mocked-bucket')
    candidates_for_deletion = gitlab_backups.candidates_for_deletion(generations_to_keep)
    assert expected_candidates_for_deletion == list(candidates_for_deletion)

I believe I should provide the output of backups(), i.e.

 {'10.8.2': ['1525917621_2018_05_10_10.8.2_gitlab_backup.tar',
             '1526781621_2018_05_20_10.8.2_gitlab_backup.tar'],
  '11.0.0': ['1530151221_2018_06_28_11.0.0_gitlab_backup.tar',
             '1530237621_2018_06_29_11.0.0_gitlab_backup.tar',
             '1530324021_2018_06_30_11.0.0_gitlab_backup.tar'],
  '11.0.1': ['1530410421_2018_07_01_11.0.1_gitlab_backup.tar',
             '1530496821_2018_07_02_11.0.1_gitlab_backup.tar',
             '1530583221_2018_07_03_11.0.1_gitlab_backup.tar',
             '1530669617_2018_07_04_11.0.1_gitlab_backup.tar']}

instead of relying on other methods, but then I am not sure if it should be a helper function returning a generator.

Is there a way to write proper unit tests for such methods or should I rewrite the code?

Answer 1

Looks like a usefull test, its possibly an integration test.

That said you should put this in a unitest2.testcase class https://docs.python.org/dev/library/unittest.html

It works nicely with pytest https://docs.pytest.org/en/latest/unittest.html

You can create all your fixtures in the setUpClass method

then all the methods starting with test_ are executed and you can use a nice library of built in assertions.

You you can add your generator generator functions as methods on your TestCase instance then call them from its test_ methods.

Both those links cover the rest of the details.