3

I have a Python package which may contain several modules. Of course, I have unit tests, component tests, and functional tests. All of them are in package/module/test directories, most of them are in files labelled test_ClassUnderTestName.py, and contain classed derived from unittest.TestCase matching class UnitTestClassUnderTestName and so on. I use py.test to run all the tests -- with the coverage plugin.

How can I improve my set up to make it more logical, easier to use, and make the setup have more quality?

If I cannot improve it, what would be the best practices that you use to achieve the best quality of tests organisation within Python packages?

Edit after comment:

  • Tests are name like so: test_parse_data_failure(), test_parse_data_command_X(), etc...
  • I have three classes, one for each type and all in one file.
  • Components would be a full (happy path) test path to parsing some data.
  • Functional tests would be to parse data, and do something with it.
Improve this question
6
  • I hope that my question is clear enough but if it is not, I would really appreciate criticism to make it better. Thanks.
    – Sardathrion - against SE abuse
    Dec 21, 2012 at 9:34
  • You mentioned naming convention for unit tests files. You could also describe: example names for single test cases (methods), example names and organization of functional tests, what exactly components are in your case (i.e., what your component test cover in comparison to unit tests). That would help other answer your question.
    – dzieciou
    Dec 21, 2012 at 14:38
  • 1
    Others will give more complete advice, but I usually write what the test should do e.g. parsing_should_throw_exception_for_invalid_input(). This is in according to given-when-then grammar style.
    – dzieciou
    Dec 21, 2012 at 18:42
  • BTW, I think your question is universal for any language.
    – dzieciou
    Dec 21, 2012 at 18:47
  • @dzieciou: Sure, feel free to re-tag it. I am not sure what tags would be appropriate.
    – Sardathrion - against SE abuse
    Dec 21, 2012 at 18:48

3 Answers 3

Reset to default
3

I do not use Python, so I cannot offer any Python-specific advice. However, here are some observations about writing and using automated tests in general.

Try to give your tests meaningful names. If your test results include the name of the test, it helps for the name to be meaningful -- as opposed to "test1", "test2", and so on. Of course the name may not convey exactly what the test does, so I will still need to look at the source code to understand what went wrong.

Provide a way to run tests individually. A frequent complaint I hear from developers about automated tests is that they take too long to run. Sometimes that's because it is not possible run just the test that failed. When a developer is iterating in a edit/test loop, they want to wait as little time as possible between tests, especially if most of the test results are not relevant to their immediate task. A mature test runner will provide a way to do this, but sometimes test runners only work at the class or package level.

Don't overthink the organization. You do not know how others will want to use your tests. Of course you would like for it to be as easy as possible to run every test, but you should also consider that they might want to run just one test (as I mentioned above) or a subset of the tests.

Improve this answer
3

Although potentially extreme, I frequently find myself using GivenWhenThen syntax for naming tests so it's crystal clear what they do and expect (I can't tell what test_parse_data_failure() or test_parse_data_command_X(), actually do from the function names.

Instead, I prefer names like (guessing at what your functions do) GivenBadData_WhenIParse_ThenFailure, or GivenCommandXData_WhenIParse_ThenExecuteCommandX.

This is the start of good logging, which I believe is critical to test success. Every test failure (IMO) should be fully diagnosable from the log file - and if I know what the test is doing without looking at the code, I have a much better chance of diagnosing the failure quickly.

Improve this answer
2
  • Every failure AND fully diagnosed ? what about the unknown unknowns, or even the known unknowns ?
    – Rsf
    Dec 23, 2012 at 13:42
  • It's a goal, not a requirement. But when a failure is not diagnosable from the log, I/we add additional logging to make diagnosis easier in the future.
    – Alan
    Dec 23, 2012 at 19:03
3

We are currently going through some titling clean up as well. Tests with titles such as "Validate comments on status," or "Verify accounts not created XXX state reject P2" are simply not helpful from a high level without drilling into the test.

We try to simplify the title of a test to a What --> How syntax. In other words, what aspect of the functionality or scenario is being tested and a brief description of how it is being tested. For example, "Primary user accounts are not created when XXX state rejects invalid account."

A clear title is important for managers or others not familiar with the feature area to grasp the purpose of the test without having to drill down into the specific mechanics of how the test actually does the work.

Comments at the header should include a detailed purpose of the test, and a description of the oracle(s).

We use sub-folders/dirs under major feature paths to organize tests into categories or test suites (e.g. unit, scenario, performance, stress, etc) to help devs or new team members locate specific types of tests.

Also, I recommend that you remove redundant information from the title. For example, some tests included the priority of the test in the title. Since the test priority is already tracked via another mechanism including the priority in the title was redundant.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.