Test cases in source code

Question

Will it be a good practice to have test cases stored along with the source code in a file or within the source code comments itself? Few months back, I met a programmer from a big corporation who suggested me that their company stores all the test cases same way.

Answer 1

I wouldn't store written test cases within code comments. It's too scattered and would make reading both the test cases and the source code itself more difficult. Additionally, many written test cases use a richer medium (involving pictures, charts, etc) that isn't well expressed within comments.

As far as storing your written test cases within your source control system, that sometimes makes sense. They can easily be versioned along with your code that way.

But consider who the readers of your test cases will be first. Do they all have access, licenses, etc to your source control system? Is this an efficient mechanism for all your readers, or just for programmers?

At my shop, we store all documents (as well as most other test assets) in SharePoint. For us, that's the "standard" mechanism used throughout the organization, and is well-suited for all the readers of our test documents.

Answer 2

I wonder if the cases were noted that way to be pulled out with a Doc system, like PerlDoc or something, when I was working with a Perl framework some cases were noted that way within the test code. Although with the actual production code? Nah, I don't see that since that means you have code bloat to the production system with your cases as comments...unless they are stripped out before compilation.

I've also seen it where the SCCS has a test folder off the main code tree, pretty much most open source projects I have seen do this as well, so the tests are connected but don't need to be checked out and compiled.

Personally I like to keep my tests separate, my test frameworks have their own location within the SCCS that way they can be kept up to date and don't rely on production builds to be updated. It also keeps me in a safer environment so there is no cross-contamination whenever frameworks that support the production code get updated (like .NET 3.5 to 4.0) if my test framework doesn't support it yet.

Answer 3

One might argue that test cases are more likely to be updated if they are documented in the source code. However, in my experience, comments are often outdated. I suspect test cases would be outdated as well.

When a test case is discovered to be outdated (or missing), who will updated it: a developer or a tester? In my experience, developers are usually uncomfortable with testers making any changes to source code, even if only to comments.

In my organization, test cases are maintained in a Wiki.

Answer 4

Perhaps you misunderstood. Test code/comments should not be included in product source code.

But, at some company's test code is managed the same way as product code via source control, and is checked into the same build branches.

Answer 5

Consider the maintenance of the test cases. If the test cases themselves need to be updated and they are in the comments of the source code, modifying the test cases will force a rebuild of the software. This is a bad practice. The code should only be rebuilt when the code itself is changed.

Managing test cases in a separate file is a different story. Keeping track of test case revisions via source control allows for traceability between test case revisions and software releases.

Answer 6

I'm confused by the question i'm afraid.

We have automated regression tests (checks) which are very much part of the source code. The checks are run every time a programmer commits code (continuous integration).

However, the automated checks are not deployed to live - only the app code is wrapped up in a war file & released.

This way, if a Programmer inadvertently changes the logic of the app when coding, an automated check will fail.

Personally, I wouldn't have test cases as comments in the code - I can't see the benefit in this. Wouldn't they bloat the code & make it more 'noisy'?

I feel user246 has the right idea with having the test cases in a wiki - this can serve as window on all tests (at all levels) which can be read by programmers & business analysts alike (is this right user246?)

Answer 7

Storing testcase along with the source code in a file or within the source code comments itself may work for Unit testing where there is always one to one mapping with code unit. However when the source spans across hundreds of files and millions of lines of code all cases can not be mapped with code block in a single file for system integration testing and subsequent testing phases.

Its good practice to use some sort of version control system for testcases, but mixing it with source code file itself may be not be a good idea (apart from Unit Test cases).

Answer 8

TL;DR answer

Are each of these considered Best practice:

• Storing test cases in comments : Absolutely not, it makes automated tests far too difficult.
• Storing test cases in the same file as the the source code : Not really, it's too distracting.
• Storing test cases in the same repository as the source code : Yes, especially if developers do TDD.
• Storing test cases in a separate test repository : Yes, but only if required by procedure.

Full answer

Firstly, the idea of storing test cases in comments in the code seems like the worst of all worlds. You can't easily use a testing framework to automate your tests, you have to do some magic to extract the tests and run them (so you get no help from your IDE when writing them) and your code is bloated with tests which obscures the purpose of the code.

Storing test cases in the same source file as the production code under test is the simplest option. But without a lot of pre-processor directives or annotations you will end up with your test code bloating your production code unnecessarily, and depending on how you have structured your code, you may end up accidentally exposing internal implementation to users of that code.

The best option is to storing test cases separately from code and structure your code to support that. This has a number of advantages, as described in other answers, but there are also pitfalls. Luckily there are often ways to get around them, which others do not seem to have discussed.

Example

For instance, if you want to unit test some private behaviour of a class, depending on the language/environment, you may have three options:

1. Put the tests in the class you want to test.
2. Put the tests in another class/source file & expose the private methods you want to test as public methods (yeuch).
3. Use a testing environment that allows you you keep test and production code separate, yet still allow testing code access to private methods of the production code.

In the Eclipse world, 3. can be achieved by using fragments. In the C# world, we might use partial classes. Other languages/environments often have similar functionality, you just need to find it.

Blindly assuming 1. or 2. are the only options would be likely to result in production software bloated with test code or nasty class interfaces that wash their dirty linen in public. *8')

Version control

I find it difficult to imagine doing anything other than storing test cases as code under revision control, just like production code. You might prefer test code to be in a different repository, possibly even with different access permissions for test programmers and production programmers (if you separate the two), but it just as important to know which version of the tests you are running as it is to know which version of the code is being tested.

Answer 9

This is common practice in Python for modules and makes good sense.

If you choose to run the module by itself then it runs its own test cases providing example of its own API as well as verification that it works.

For larger codebases these do become pure examples however rather than a full test suite.