Handling of Test Documents throughout Retests

Question

Community,

I'm currently doing development and tests for a Windows Desktop Application that plays together with some PLCs. One of my main tasks is to perform Functional Tests on a System Level for each Alpha/Beta version.

There's a Test Document (MS Word) that defines the test cases and has an Accpeted/Not Accepted mark for each test case. When encountering errors, a screenshot is taken and put together with some short description into the Test Document.

All other deviations/observations that are not related to the Functional Tests are put into a separate Word Doc.

At the end of the tests, I'm creating some overview spreadsheet that lists all results and is a base for discussion.

Of course, every now and then some tests will fail. Some bugs are so huge that they'd be blocking the release and require a bugfix that results in a new delivery that results in a new build for the possible release. This build will then be retested. The failed test cases are to be repeated and possible other test cases also, depending on the impact of the change.

The common procedure results in creating a new copy of the (empty) test document and executing the test cases. Of course, most of the other cases won't be executed and will have no result in their checkbox. This doesn't look nice and is also not easy to get an overview (e.g. if you have two or more retest documents).

Sometimes we're also creating a Test Plan (Word) that defines a subset of the Test Cases to be executed.

Handling these documents is a bit tricky:

• no linkage from overview to failed testcase (manual search)

• test case document rather long (~60 pages), keeping an overview hard

• for each retest of a failed build, a new copy of the word document is created

• Recording test results and reviewing them is hard for me

Do you have any recommendations for handling the documents, recording results and decisions and versioning the documents on a per release/build base? How are you doing it? Examples welcome!

BTW: The toolset (MS Office) can't be changed (boss sez so).

Thanks a lot, Alex

Answer 1

Given the mandate to use Microsoft Office, I'd look at using spreadsheets to manage the test execution, working something like this:

• The master list of all test cases for the project is either a Word document or an Excel spreadsheet, linked from the Test Plan.
• Each test run can be either a separate spreadsheet or a worksheet within a single spreadsheet, depending on how complex the system is. For a highly complex system, I'd use one spreadsheet with worksheets set up for each logical grouping of test cases (e.g. configuration test cases would be one worksheet)
• The first worksheet on any test run would contain a summary of test results, including a list of failed test cases (create these as internal links to the actual test case and result by doing a copy - paste as link).
• The summary would also include pass/fail percentage possibly itemized by test case priority.
• If a defect is severe enough to block further testing, this would also be noted on the summary
• If you have no other means to track defects, I'd use a separate spreadsheet and link to it so that you can number and link any defects you find.
• Your spreadsheets would live in a network share where every member of the team had access to them. For multiple testers, you might consider sharing them.

An alternative, if by "Microsoft Office" your boss includes Access, would be to build an Access database with all this information and overlay it with some simple forms to make data input, searching, and retrieval easier (this assumes you have the SQL knowledge to do this). It would take a while to set up but would take a lot of the manual effort out of your test recording.

Answer 2

This is a huge question!

I've worked at similar places where they want a QA team to work without the QA tools like bug-tracking software, for example.

If you're being forced to use Microsoft Office, then it might be worth splitting up the document, so rather than tracking the defects in one test document per iteration, you could literally have one page per defect (Description, Steps to Reproduce, Branch/Revision Info, Test Evidence), printed out, and keep physical copies - which sounds like it'll be easier to maintain.

Imagine like when companies went from a physical kanban and using post-its on a board to a digital kanban, you'll be doing it the other way round.

The other option is (and we've used this before) an Excel sheet. Be warned, it takes up a huge amount of time... but it's quite easy to copy and paste the data between sheets (one new sheet per iteration) and with its graph features it's quite easy to create metrics for your stakeholders. It's also easy to duplicate, edit and share Excel documents and keep on top of version control.

I might update this at some point. Just wanted to throw in my two cents :)

Hope this helps, Dan