Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I'm thinking of live production sites here, where there is a continuous community or much content being updated, sort of like a newspaper site or community site (the one I work at is a healthcare improvement company) and we have news articles, programs and so on. While the underlying code is tested and released there are often production issues that come up, maybe certain pages stop working, style sheets suddenly don't appear like they did due to a service patch or some tweak done to improve production security or performance.

There is Acceptance Testing for when a deployment goes out to make sure things work right, Regression Testing to make sure nothing has been broken since previous releases. Considering we've done these types of tests and things are working fine what would you call testing, or checking, the site to make sure things still work fine? To be sure new pages are being crawled for search indexing? To check that newly added pages are using plug-ins correctly? Or so on.

My manager and I were discussing some issues that have cropped up and decided we needed to do something like this, but then to sort of note it in our status reports we didn't really have a name for it. Anyone done this and have a term that might be good? Or would you consider it covered by something like Continuous Acceptance Testing or some such term?

share|improve this question
    
In your circumstances, I would call it production testing. –  user246 Oct 12 '11 at 13:05
    
Another possibility: "production monitoring". –  user246 Oct 12 '11 at 13:24

5 Answers 5

up vote 2 down vote accepted

When I was at Netflix, I called the team Operational Quality. The focus was on insuring that the code that was built & validated during the dev & test process actually worked the way it was supposed to in the production environment.

Some of the tools included:

  • continuous measurement of the performance of the different applications (down to the method, service and database call levels)
  • continuous monitoring of the rate of different errors/exceptions getting thrown
  • a close partnership with the NOC in responding to decreases in throughput (requests per minute per server)

So, while that was the team name, what was this process called? Never really had a name for it. Perhaps "Application Assessment"? "Operational Assessment"? "Live Site Health Checking"? "Production Fault Detection"? "Operations Sanity Checking"?

Not sure what I'd call it, but I'd look for the verb first. Are you 'detecting', 'assessing', 'checking', 'monitoring'? Then find the noun to put in front. This is clearly 'production', but it's also a 'site', or maybe you're concerned about 'experience'?

Maybe "Customer Experience Assurance"?

share|improve this answer
    
Personally I like the term Operational Quality, it means a little bit more in our space. Since we are a Healthcare Improvement Non-Profit terms have special significance to be understood across departments. Oddly, typical Engineering terms I used to use we don't to avoid confusion. –  MichaelF Feb 2 '12 at 13:59

I'd call it Production Monitoring as well. In my company we have "Production Engineers" performing this sort of task, not QA folks.

share|improve this answer
    
Two votes for Production Monitoring, although I think of this more along the lines of the NOC checking on the health of the system. That's how I've used the term in the past. –  MichaelF Oct 13 '11 at 16:28

Similar to Joe Strazzere's answer, at our place we call them Production Support.

Dev and QA work on major releases and enhancements to the site, but if there are any client reported issues which would take less than 40 hours to fix, it is all handled by Production Support. Everything, from coding and debugging to testing.

Sometimes just so they feel more confident, they would have the QA folks test their fixes before pushing to prod though!

share|improve this answer
    
We are a small company, so if there are issues we as QA deal with them, there really is no outside group to hand off to. –  MichaelF Oct 14 '11 at 19:59

We refer to this as sustained engineering.

Our sustained engineering teams fix product bugs identified by customers post release; they also make sure changes by upstream dependencies don't introduce issues, and also release period service packs with updates as required.

Depending on the size of your org. the sustained engineering teams can be separate teams or another aspect of the SDLC for your business.

share|improve this answer

Don't know anything about costs or implementation, but our organisation uses Site Confidence

It executes automated checks (based on typical user journeys) every couple of minutes & reports back to us. Fast feedback & all that jazz.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.