14

In your test strategy/testing where would the installation testing (set-up/deployment with or without installer) belong? Is it a part of functional testing or non-functional testing?

  • 2
    I am not sure what functional vs. non-functional means in your context, but installer testing is its own beast. – user246 Jul 18 '11 at 13:17
  • @user246: I derive the functional/non-functional distinction from Brian Marick's testing quadrants which has been eloborated in the book Agile Testing by Lisa Crispin and Janet Gregory as well..for me business facing tests are mostly functional and technology facing tests make up for the non-functional requirements...although none of these talks specifically about installation testing and hence I largely agree with you that it is its own beast (depending on the context) – Rajneesh Jul 19 '11 at 7:45
5

As I indicated in this reply Accessibility Testing - Should it be considered functional testing or non functional testing? I like the following definitions from http://www.lessons-from-history.com/node/83 "a functional requirement specifies what the system should do" "a non-functional requirement specifies how the system should behave"

We have always treated setup/upgrade testing as functional testing because we are essentially testing what the the installer engine should do. Of course, with complex installer packages with multiple configuration dialogs you also have a behavior element as well.

You might also want to check out http://www.testingmentor.com/imtesty/2011/02/21/state-transition-testing-thinking-in-models/ for an example of testing flow, and also combinatioral testing for testing complex configuration matrices for setup/upgrade.

11

For me it's a non-functional requirement, even if the key users have some requests regarding the instalation folder location (for better integration with other software packages).

UPDATE:

The tests for the available features detection are still placed in the functional requirements area, given that:

  • the software can be bin-deployed in some cases, and some optimized libraries can be made available after the manual installation
  • some libraries are available only for some platforms (like .NET 4.0 only, or Python 2.x only)

A small example: the deployed software should detect and use a CUDA-enabled GPU. There will be 3 sets of tests:

  1. for the normal code path
  2. for the CUDA code path
  3. for the correct CUDA detection (the GPU can be upgraded/downgraded after the installation)
  • 2
    I like the way that you flushed this out to make your answer more specific. I'd give another +1 if I could. – Lyndon Vrooman Jul 18 '11 at 10:39
10

I would agree with alexandrul's answer, with a few small caveats.

First of all, it depends on what the installer does. If it's just a plain normal install, it would fall under non-functional. If there are options in the installer that greatly affect the functionality of the application (ie: add/don't add specific features), I would normally put it in my functional testing (back when I still considered the seperation of the two important).

5

Lyndon makes some good points, it does depend on what your installer does. When I have worked with them in the past, mostly I deal with SaaS and not so much with client apps anymore, but there were a few in my past jobs and we mostly had functional tests for them. The rationale was we were not just dealing with installs, although that was there, but we needed to deal with uninstalls - which is something most people forget. The expectation I have is when I uninstall a piece of software I expect my environment to reflect that software being completely gone, because if I reinstall it should be as if the software had never been there. If that is your requirements, some applications do want to leave something in place because of trial/buy versions and want to be sure that you are not reinstalling the trial version over and over again to avoid buying it. So in addition to what alexandrul notes I would also check:

  • Uninstall, all expected items removed (code, directories and/or registry entries)
  • Reinstall
  • Maintenance (Repair install), usually the one that fixes an install of the software in case of issues
  • 2
    Yes. Moreover, the installer may need to behave differently depending on which previous version of the software was installed. Upgrades are particularly treacherous. – user246 Jul 18 '11 at 13:15
  • +1 I have a tendency to trust the installer and focus on feature-detection parts, but that's a risk that might not be acceptable for most applications. – alexandrul Jul 18 '11 at 13:16
  • Yes, and upgrades with different versions also just adds to the test matrix. +1 for treacherous, I've seen that in play just for stuff I had bought. – MichaelF Jul 18 '11 at 14:58
1

I don't agree with the idea that "if there is a refined specification of installation it's functional otherwise not".

Thinking in that way is like "do not describe in detail your non-functional requirements or they become functional".

I think that non-functional requirements need to be well specified. And they continue to be non-functional.

Apologies for my bad english. I'm improving it

1

Functional testing deals with checking the functionality of a system or application works as expected. You give certain inputs and check whether you get the expected output. You will usually use some sort of heuristic to tell whether result is expected or not.

Non Functional testing deals with checking the non-functional aspects of systems such as performance, reliability, scalability, usability etc.

Keeping the above definitions in mind, I think installation testing could be function/non-functional testing based on the context. I am saying this because when I do installation testing I-

  • Delete the app and install the new app
  • Delete cache, uninstall app and install new app
  • Install app over existing app
  • Install app over Wi-Fi/4G/Bluetooth/USB

and do more scenarios

All the above scenarios are trying to see how the app reacts/functions based on different actions. Some of these could be mentioned as part of the acceptance criteria then it becomes functional testing but we could also try the above scenarios to see how the app performs and reacts to different types of common user actions, then it could be non-functional testing. So, it is hard to categorize them into one single bucket of functional or non-functional testing. This is especially true when testing mobile applications. These are just my point of view.

-Raj

0

I found it interesting the timing of your question with this well written blog on Installation Testing: http://lauralistar.wordpress.com/2011/07/18/software-installation-uninstallation-testing/

0

Seems like if there are requirements around the installation, then it's a functional test. If there's a user story which includes the installation, then it's a functional test. Lacking those, it's a non-functional test which needs to be covered when considering all of those other non-functional aspects of the application under test.

Also, Squirrel makes a good point in that it really doesn't matter as long as it's done.

  • What do you mean by "there are requirements"? That there is a requirements specification document? Also, note requirements can both describe what the system should do (functional part, according to Bj Rollison's answer) and how it should behave, e.g., is it fast enough (non-functional parameter). Though, the latter is less commonly documented. – dzieciou Nov 3 '12 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.