How long should test results be kept?

Question

We've got Jenkins jobs running Unit Tests, Integration Tests, JMeter Tests, and Selenium Tests. Given my choice I'ld keep all these test results forever, but maybe that is excessive.

1. How long does your team keep test results?
2. How far back in history have you found useful?
3. Have you run into any performance issues related to keeping too much history?
4. Do you archive test results outside of Jenkins? How?

Answer 1

As always, it depends. If there are regulatory requirements, results need to be kept for as long as the regulations state. Otherwise, it's a business decision.

1. How long does your team keep test results? In my experience this depends on a number of factors. I've been in places where test results for new development would be kept indefinitely, but test results for regression were regarded as ephemeral. I've also been in places where the full logs were discarded or archived, but a summary was retained indefinitely.

2. How far back in history have you found useful? Again, it depends. Mostly I've found that if something goes back more than 3 major versions, looking for more resolution is a waste of time. Occasionally it's been necessary to go back to the test case documents from the original implementation to determine whether something was covered initially. If regressions are reported as soon as they're observed, the reporting should include the first known failing build and the last known good build. Ideally these are sequential, but this isn't necessarily the case.

3. Have you run into any performance issues related to keeping too much history? This is one of the few that doesn't get "it depends" - disk space is limited, and logs accumulate. The process I prefer to use is to automatically archive logs off to a pre-defined location at least once a day, including application logs where appropriate (at one employer this was essential if you didn't want to be dealing with trying to open 4GB text files - I created the daily archive task because I was sick of trying to access immense log files). The archive location stores the logs without compression for a period of time, then zips logs older than the agreed time-frame. We still needed to clean out the oldest logs periodically and ultimately decided that if it was older than a year, we didn't need to keep it. We versioned our test script code, so it was possible to rerun old code against old versions of the application if necessary.

4. Do you archive test results outside of Jenkins? How? I've used a task that would export the logs to a datestamped directory on a daily basis. It's also possible to do it per build with a datetime stamp on the directory. Zipping via command line is easy to do, so that was also part of the task. The naming convention was usually %testsuitename%%YYYY%%MM%%DD%[%hh%%mm%%ss%] for the directory or zip file.

Answer 2

Despite the fact the question is about Jenkins results - I'll add my own 5 cents, but regarding results of manual testing - some of the following points may be as well useful:

1. As stated in other answers, results of intermediate builds are valuable within several months OR current project stage, therefore our usual practice is to keep these within current stage. However, there's an exception regarding results of the major / Release builds - these are usually kept longer, most likely for the whole project time-frame and some time later (see more details about that aspect below).
2. Actually, I had cases from my own experience when testing results about half year old or even older were useful - for next stages planning, retrospective, contract negotiations and other similar cases. I don't think something older than 2 years was ever actually demanded - at least within my own 5+ years QA experience.
3. At the same time we have internal DB called "Project DataCenter", where project managers briefly log details regarding current projects state (on monthly basis): quality, milestones, communication, etc. These are kept for much longer terms - I'd say within 5 years after project end or even more. These records may as well include testing artifacts like reports, scenarios, results, etc.
4. Do not neglect contract agreements - sometimes results / documentation keeping is stated among other terms and conditions. These may vary from 6 months to 10+ years - I faced such cases on my own.
5. Perhaps in my case performance / storage question is not so important, but we do have automated archiving / backup system for our doc storage. Sadly I don't know exact details (this is regulated by our Internal Support / IT departments), but something like projects data older than 5 years is removed to archive and no longer available for direct access.

Hope that was somehow useful!)

Answer 3

I feel like this is a polling question rather than a SE-type question, but if not....

1) We keep them for two weeks at my current role, but at other companies several years.

2) Rarely have I needed something more than a couple of days really as I run daily automation tests, but I have found it useful a couple of times when a test has been failing for ages, to go back a couple of years and see when / why it was broken.

3) Yes, run out of disk space on the servers, test logs can be surprisingly big sometimes.

4) Often just zipped up by date and testsuite name on the servers.

Answer 4

The real answer is that you need to keep test results for as long as they are useful. Storage is cheap, so don't let that be the driver behind how long you keep any test artifact.

But determining how long results can be of use depends on a lot of company and project-specific conditions.

We tend to keep functional results only for the duration of the project, but keep performance test results much longer.