In your test strategy/testing where would the installation testing (set-up/deployment with or without installer) belong? Is it a part of functional testing or non-functional testing?
As I indicated in this reply Accessibility Testing - Should it be considered functional testing or non functional testing? I like the following definitions from http://www.lessons-from-history.com/node/83 "a functional requirement specifies what the system should do" "a non-functional requirement specifies how the system should behave"
We have always treated setup/upgrade testing as functional testing because we are essentially testing what the the installer engine should do. Of course, with complex installer packages with multiple configuration dialogs you also have a behavior element as well.
You might also want to check out http://www.testingmentor.com/imtesty/2011/02/21/state-transition-testing-thinking-in-models/ for an example of testing flow, and also combinatioral testing for testing complex configuration matrices for setup/upgrade.
For me it's a non-functional requirement, even if the key users have some requests regarding the instalation folder location (for better integration with other software packages).
The tests for the available features detection are still placed in the functional requirements area, given that:
- the software can be bin-deployed in some cases, and some optimized libraries can be made available after the manual installation
- some libraries are available only for some platforms (like .NET 4.0 only, or Python 2.x only)
A small example: the deployed software should detect and use a CUDA-enabled GPU. There will be 3 sets of tests:
- for the normal code path
- for the CUDA code path
- for the correct CUDA detection (the GPU can be upgraded/downgraded after the installation)
I would agree with alexandrul's answer, with a few small caveats.
First of all, it depends on what the installer does. If it's just a plain normal install, it would fall under non-functional. If there are options in the installer that greatly affect the functionality of the application (ie: add/don't add specific features), I would normally put it in my functional testing (back when I still considered the seperation of the two important).
Lyndon makes some good points, it does depend on what your installer does. When I have worked with them in the past, mostly I deal with SaaS and not so much with client apps anymore, but there were a few in my past jobs and we mostly had functional tests for them. The rationale was we were not just dealing with installs, although that was there, but we needed to deal with uninstalls - which is something most people forget. The expectation I have is when I uninstall a piece of software I expect my environment to reflect that software being completely gone, because if I reinstall it should be as if the software had never been there. If that is your requirements, some applications do want to leave something in place because of trial/buy versions and want to be sure that you are not reinstalling the trial version over and over again to avoid buying it. So in addition to what alexandrul notes I would also check:
- Uninstall, all expected items removed (code, directories and/or registry entries)
- Maintenance (Repair install), usually the one that fixes an install of the software in case of issues
I don't agree with the idea that "if there is a refined specification of installation it's functional otherwise not".
Thinking in that way is like "do not describe in detail your non-functional requirements or they become functional".
I think that non-functional requirements need to be well specified. And they continue to be non-functional.
Apologies for my bad english. I'm improving it
Functional testing deals with checking the functionality of a system or application works as expected. You give certain inputs and check whether you get the expected output. You will usually use some sort of heuristic to tell whether result is expected or not.
Non Functional testing deals with checking the non-functional aspects of systems such as performance, reliability, scalability, usability etc.
Keeping the above definitions in mind, I think installation testing could be function/non-functional testing based on the context. I am saying this because when I do installation testing I-
- Delete the app and install the new app
- Delete cache, uninstall app and install new app
- Install app over existing app
- Install app over Wi-Fi/4G/Bluetooth/USB
and do more scenarios
All the above scenarios are trying to see how the app reacts/functions based on different actions. Some of these could be mentioned as part of the acceptance criteria then it becomes functional testing but we could also try the above scenarios to see how the app performs and reacts to different types of common user actions, then it could be non-functional testing. So, it is hard to categorize them into one single bucket of functional or non-functional testing. This is especially true when testing mobile applications. These are just my point of view.
I found it interesting the timing of your question with this well written blog on Installation Testing: http://lauralistar.wordpress.com/2011/07/18/software-installation-uninstallation-testing/
Seems like if there are requirements around the installation, then it's a functional test. If there's a user story which includes the installation, then it's a functional test. Lacking those, it's a non-functional test which needs to be covered when considering all of those other non-functional aspects of the application under test.
Also, Squirrel makes a good point in that it really doesn't matter as long as it's done.