Deployment Testing

Question

What kind of things should be looked at after you have deployed a new system? So far I can think of the following:

Things to check:

a) Check it deploys to right folder: Open machine where build is being deployed too check it has gone to specified folder and if necessary check that the correct files have been overwritten.
b) Check deploying correct SVN : Make sure that the latest svn revision is the correct one found on the deployed machine.
c) Application (that it actually works)
d) Check that the changes made are present and test for any bugs.
e) Performance Issues (slow, crashes etc)

Answer 1

I just wrote a deployment validation tool with the ability to validate the following things:

1. IIS

   1. Application Pool
      1. Exists
      2. State (started)
      3. Worker Processes
      4. Exists
      5. State (running)
   2. Sites
      1. Name
      2. State (started)
      3. Applications
         1. ApplicationPoolName
         2. Path
         3. PhysicalPath
         4. Virtual Directories
            1. LogonMethod
            2. Path
            3. PhysicalPath
            4. User
   3. Url post/get Validation
      • Response Code
      • Response contains or not contains string.

2. Registry

   1. Key value
   2. Type

3. Database

   1. Table
      1. Name
         1. Column
         2. Name
         3. Type
         4. MaximumLength
      2. Version
   2. View
      1. Name
         1. Column
         2. Name
         3. Type
         4. MaximumLength
      2. Version
   3. StoredProcedure
      1. Exists
      2. Version
   4. Permissions
      1. Server (Whitelist to test least privlege and optional non-whitelist to verify functionality)
      2. Database (Whitelist to test least privlege and optional non-whitelist to verify functionality)
   5. Query
      1. Expected Response (contains list of values, optionally in a specific order, optionally contains only that list or the list plus additional values).

4. Windows Service

   1. Name
   2. Status
   3. StartupType
   4. LogOnAs

5. DeploymentFiles (dlls, exe's, any files copied to the server)

   1. Product Version
   2. File Version
   3. Exists
   4. UserPermissions
      1. User
      2. AccessControlType
      3. FileSystemRight

6. TextFiles (log files, config files, human readable files)

   1. Location
   2. Text file contains or not contains strings

In addition to this tool, many of our deployment validations will include a subset of our end to end functional tests, whether UI automation, automation of web requests, or other functional automation.

Thanks to Kunal for the starting point: http://www.testingwithkunal.com/category/deployment-testing/

Answer 2

Good to hear a deployment testing question. The deployment process itself seems to be often ignored. Is your deployment process automated? If it isn't quality starts there. A consistent repeatable process will do wonders for successful deployments.

After that is complete you can look into adding quality checks into your automated deployments like adding a txt file to your deployment so you know which build/revision number, Validate the service is up, confirm old files are removed, Automate a smoke test of the application, etc...

If your talking about testing a newly automated deployment, I would focus on testing things that could fail and validate that the error handling is understandable. More importantly look for items that could fail but would not fail the deployment. The worse kind of failed deploy is one that you didn't even know failed and of course add quality testing steps into the deployment process (like I mentioned above).

Answer 3

After you deploy to the system, the first things I look for:

Deployment specific things?

• Deployed the right files/folders
  • Ensure that the assemblies that should be signed, in fact, are signed correctly
  • Assembly versions are correct
  • GAC'd assemblies are GAC'd
  • Check desktop icons/start menu icons if necessary (assuming Windows)
  • Check that services are installed/running (if there are any)
  • Nav to any web services that are supposed to be installed/running
• Correct registry entries are made
• On a test VM/Machine I scrub all event logs before I deploy and take a look at all of the event logs after deployment
• Check any app-specific logs for errors/warnings
• Verify bug-fix (I'm assuming this is the scenario from your post)
• Verify docs have correct steps for the deployment scenario (if it's a customer activity)

I do a small sanity check that the app works, but I don't check application performance at that stage. That's another scenario entirely.

That's all I can think of at the moment. Generally, I make a checklist "big" scenarios that have a lot of product dependant things.

Now this is all verification. If I were to do some exploratory testing of the deployment/installation, it would be different. I'd be trying to bugger things up and break the deployment.

Answer 4

We do something similar that we call Release Testing. Basically, we perform just enough testing to confirm that the system has been deployed and configured correctly, that it contains the expected enhancements/fixes, and that it hasn't broken something else along the way.

What we actually do during Release Testing varies from project to project, and release to release. We always try to contain the time needed for a release, since it sometimes involves downtime for our customers, and often is performed off-hours (when nobody wants to waste time).

Sometimes, there's a conversion associated with a release - often this involves a significant amount of testing to ensure that the conversion behaved properly.

Somethings, it's a simple release and might just involve a quick smoke test, followed by some targeted testing to ensure the desired few fixes were properly included in the release.

Answer 5

Agree with Lyndon, but why just for the release?

how about heartbeat tests that run periodically checking that everything is as it should be. This way not only the upgrade will get tested, but if someone acidently takes out a cable you will know about it (hopefully sooner than your clients).

Be pro-active about testing - production monitoring is continuous testing, and doesn't have to be limited to checking the simple stuff (like have you got enough disk space).

Answer 6

I think that I may kind of understand the question, and, here goes my answer for what I believe that I understand.

Point C is probably the most important and essentially covers points a, b, and d.

To append,and add a point f, some form of vulnerability testing should be performed on the new environment. All to often, moving machines, even if it's an upgrade, can open holes where information can pass through that you didn't mean to. This also counts for application upgrades.

To append to point C, yes, you should test that the application works. Make sure to also check some sort of error logging on the machine that it's being deployed to to ensure that there are no new errors being created. Ensure that other applications on the same machine still work as they previously did as well. This is one that far too many people miss.

If you are indeed moving machines, ensure that you have the same prerequisites ie: COM's, pre-req application versions, etc.

With a little bit more context into what type of deployment you are doing (migration, upgrade, etc), I'd be happy to elaborate a bit more.

Answer 7

What tools did you use, @sam-woods for your deployment verification?