Tell me more ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.
up vote 11 down vote favorite
2

In your test strategy/testing where would the installation testing (set-up/deployment with or without installer) belong? Is it a part of functional testing or non-functional testing?

share|improve this question
2  
I am not sure what functional vs. non-functional means in your context, but installer testing is its own beast. – user246 Jul 18 '11 at 13:17
@user246: I derive the functional/non-functional distinction from Brian Marick's testing quadrants which has been eloborated in the book Agile Testing by Lisa Crispin and Janet Gregory as well..for me business facing tests are mostly functional and technology facing tests make up for the non-functional requirements...although none of these talks specifically about installation testing and hence I largely agree with you that it is its own beast (depending on the context) – Rajneesh Jul 19 '11 at 7:45

6 Answers

up vote 4 down vote accepted

As I indicated in this reply Accessibility Testing - Should it be considered functional testing or non functional testing? I like the following definitions from http://www.lessons-from-history.com/node/83 "a functional requirement specifies what the system should do" "a non-functional requirement specifies how the system should behave"

We have always treated setup/upgrade testing as functional testing because we are essentially testing what the the installer engine should do. Of course, with complex installer packages with multiple configuration dialogs you also have a behavior element as well.

You might also want to check out http://www.testingmentor.com/imtesty/2011/02/21/state-transition-testing-thinking-in-models/ for an example of testing flow, and also combinatioral testing for testing complex configuration matrices for setup/upgrade.

share|improve this answer
up vote 10 down vote

For me it's a non-functional requirement, even if the key users have some requests regarding the instalation folder location (for better integration with other software packages).

UPDATE:

The tests for the available features detection are still placed in the functional requirements area, given that:

  • the software can be bin-deployed in some cases, and some optimized libraries can be made available after the manual installation
  • some libraries are available only for some platforms (like .NET 4.0 only, or Python 2.x only)

A small example: the deployed software should detect and use a CUDA-enabled GPU. There will be 3 sets of tests:

  1. for the normal code path
  2. for the CUDA code path
  3. for the correct CUDA detection (the GPU can be upgraded/downgraded after the installation)
share|improve this answer
2  
I like the way that you flushed this out to make your answer more specific. I'd give another +1 if I could. – Lyndon Vrooman Jul 18 '11 at 10:39
up vote 9 down vote

I would agree with alexandrul's answer, with a few small caveats.

First of all, it depends on what the installer does. If it's just a plain normal install, it would fall under non-functional. If there are options in the installer that greatly affect the functionality of the application (ie: add/don't add specific features), I would normally put it in my functional testing (back when I still considered the seperation of the two important).

share|improve this answer
+1 good point, I have to add more details to my answer – alexandrul Jul 18 '11 at 10:21
up vote 5 down vote

Lyndon makes some good points, it does depend on what your installer does. When I have worked with them in the past, mostly I deal with SaaS and not so much with client apps anymore, but there were a few in my past jobs and we mostly had functional tests for them. The rationale was we were not just dealing with installs, although that was there, but we needed to deal with uninstalls - which is something most people forget. The expectation I have is when I uninstall a piece of software I expect my environment to reflect that software being completely gone, because if I reinstall it should be as if the software had never been there. If that is your requirements, some applications do want to leave something in place because of trial/buy versions and want to be sure that you are not reinstalling the trial version over and over again to avoid buying it. So in addition to what alexandrul notes I would also check:

  • Uninstall, all expected items removed (code, directories and/or registry entries)
  • Reinstall
  • Maintenance (Repair install), usually the one that fixes an install of the software in case of issues
share|improve this answer
+1 for reminding me of a scenario that I had forgotten about and for a really good answer. – Lyndon Vrooman Jul 18 '11 at 12:32
2  
Yes. Moreover, the installer may need to behave differently depending on which previous version of the software was installed. Upgrades are particularly treacherous. – user246 Jul 18 '11 at 13:15
+1 I have a tendency to trust the installer and focus on feature-detection parts, but that's a risk that might not be acceptable for most applications. – alexandrul Jul 18 '11 at 13:16
Yes, and upgrades with different versions also just adds to the test matrix. +1 for treacherous, I've seen that in play just for stuff I had bought. – MichaelF Jul 18 '11 at 14:58
+1 for the detailed explanation of the test responsibilities – Aruna Jul 18 '11 at 17:08
up vote 0 down vote

I found it interesting the timing of your question with this well written blog on Installation Testing: http://lauralistar.wordpress.com/2011/07/18/software-installation-uninstallation-testing/

share|improve this answer
up vote 0 down vote

Seems like if there are requirements around the installation, then it's a functional test. If there's a user story which includes the installation, then it's a functional test. Lacking those, it's a non-functional test which needs to be covered when considering all of those other non-functional aspects of the application under test.

Also, Squirrel makes a good point in that it really doesn't matter as long as it's done.

share|improve this answer
What do you mean by "there are requirements"? That there is a requirements specification document? Also, note requirements can both describe what the system should do (functional part, according to Bj Rollison's answer) and how it should behave, e.g., is it fast enough (non-functional parameter). Though, the latter is less commonly documented. – dzieciou Nov 3 '12 at 7:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.