10

I'm thinking of live production sites here, where there is a continuous community or much content being updated, sort of like a newspaper site or community site (the one I work at is a healthcare improvement company) and we have news articles, programs and so on. While the underlying code is tested and released there are often production issues that come up, maybe certain pages stop working, style sheets suddenly don't appear like they did due to a service patch or some tweak done to improve production security or performance.

There is Acceptance Testing for when a deployment goes out to make sure things work right, Regression Testing to make sure nothing has been broken since previous releases. Considering we've done these types of tests and things are working fine what would you call testing, or checking, the site to make sure things still work fine? To be sure new pages are being crawled for search indexing? To check that newly added pages are using plug-ins correctly? Or so on.

My manager and I were discussing some issues that have cropped up and decided we needed to do something like this, but then to sort of note it in our status reports we didn't really have a name for it. Anyone done this and have a term that might be good? Or would you consider it covered by something like Continuous Acceptance Testing or some such term?

Improve this question
2
  • In your circumstances, I would call it production testing.
    – user246
    Commented Oct 12, 2011 at 13:05
  • Another possibility: "production monitoring".
    – user246
    Commented Oct 12, 2011 at 13:24

8 Answers 8

Reset to default
3

When I was at Netflix, I called the team Operational Quality. The focus was on insuring that the code that was built & validated during the dev & test process actually worked the way it was supposed to in the production environment.

Some of the tools included:

  • continuous measurement of the performance of the different applications (down to the method, service and database call levels)
  • continuous monitoring of the rate of different errors/exceptions getting thrown
  • a close partnership with the NOC in responding to decreases in throughput (requests per minute per server)

So, while that was the team name, what was this process called? Never really had a name for it. Perhaps "Application Assessment"? "Operational Assessment"? "Live Site Health Checking"? "Production Fault Detection"? "Operations Sanity Checking"?

Not sure what I'd call it, but I'd look for the verb first. Are you 'detecting', 'assessing', 'checking', 'monitoring'? Then find the noun to put in front. This is clearly 'production', but it's also a 'site', or maybe you're concerned about 'experience'?

Maybe "Customer Experience Assurance"?

Improve this answer
1
  • Personally I like the term Operational Quality, it means a little bit more in our space. Since we are a Healthcare Improvement Non-Profit terms have special significance to be understood across departments. Oddly, typical Engineering terms I used to use we don't to avoid confusion.
    – MichaelF
    Commented Feb 2, 2012 at 13:59
4

I'd call it Production Monitoring as well. In my company we have "Production Engineers" performing this sort of task, not QA folks.

Improve this answer
1
  • Two votes for Production Monitoring, although I think of this more along the lines of the NOC checking on the health of the system. That's how I've used the term in the past.
    – MichaelF
    Commented Oct 13, 2011 at 16:28
2

I'm not sure there is a specific term for this. Anyways you can always come up with a term of your own and consult with your team whether that sounds appropriate for what you are doing.

I guess you can call it "Maintenance Testing" where you have a team continuously testing the live site for maintaining the proper execution of the site along with the design.

You also mention that you do check to make sure that new updates and releases are not causing any issue in the existing system. This you can call "Regression Testing".

OR

You can come up with another term of your own that sounds more suitable for your purpose.

Improve this answer
1

You could call this testing in production. There's been a lot of discussion of that - see for example:

Testing in Production: How we combined tests with monitoring, Jonathan Hare-Winton & Sam Cutler, The Guardian

Improve this answer
1
0

Similar to Joe Strazzere's answer, at our place we call them Production Support.

Dev and QA work on major releases and enhancements to the site, but if there are any client reported issues which would take less than 40 hours to fix, it is all handled by Production Support. Everything, from coding and debugging to testing.

Sometimes just so they feel more confident, they would have the QA folks test their fixes before pushing to prod though!

Improve this answer
1
  • We are a small company, so if there are issues we as QA deal with them, there really is no outside group to hand off to.
    – MichaelF
    Commented Oct 14, 2011 at 19:59
0

We refer to this as sustained engineering.

Our sustained engineering teams fix product bugs identified by customers post release; they also make sure changes by upstream dependencies don't introduce issues, and also release period service packs with updates as required.

Depending on the size of your org. the sustained engineering teams can be separate teams or another aspect of the SDLC for your business.

Improve this answer
0

Don't know anything about costs or implementation, but our organisation uses Site Confidence

It executes automated checks (based on typical user journeys) every couple of minutes & reports back to us. Fast feedback & all that jazz.

Improve this answer
0

Since it is a part of the Quality Assessment Process, I think naming it as OQA or Operational Quality Assessment would be great for your QA company. You can further add all the monitoring and quality control operations into the stem procedures while the overall process could be defined as OQA.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.