Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Our end clients are requesting the MTBF (Mean Time Between Failures) estimation for our software. I know that MTBF relies on a model not suitable for software, it uses a mechanical model where parts have an inherent physical life span, and are being replaced when broken. Using MTTF (Mean Time To Failure) is somewhat better, but still has its drawbacks.

Anybody reported a reliability estimation for software or have pointers to how it should be done ?

share|improve this question
1  
Manufacturing types are tough to work with as a technological type. They try to apply all the metrics from their factories to software (as if we're "stamping out widgets" or something) and yet don't accept analogies when they apply. For example, I hear every week "We're not a software shop." Yet, when they hear "Would you use bad mechanics practices just because you don't make machinery?" they'll say "well that's completely different..." Oh, oh I see. The bottom line is, they understand manufacturing. They don't understand software. They'll misread any metrics you give them. –  corsiKa Jun 2 '11 at 22:24

3 Answers 3

Wikipedia has the best answer for this one.

Reliability engineers / design engineers, often utilize Reliability Software to calculate products' MTBF according to various methods/standards (MIL-HDBK-217F, Telcordia SR332, Siemens Norm, FIDES,UTE 80-810 (RDF2000), etc.). However, these "prediction" methods are not intended to reflect fielded MTBF as is commonly believed. The intent of these tools is to focus design efforts on the weak links in the design.

So in software MTBF, is normally used as a service reliability metric, not an engineering goal.

You can calculate MTBF with a physical product, such as a car part, or a hard drive, you can physically test until failure, and do it enough times to statistically derive the MTBF.

The wikipedia page has that formula as well.

MTBF = sum of (start of downtime - start of uptime) from several tests, divided by the number of failures.

share|improve this answer
    
Wikipedia was the first place I read. Although their definition is probably correct it has holes in it- - Do I calculate up time on a single version or since the beginning of time? - Is number of failures equals number of bugs or should take into consideration the expected frequency of the bug ? - What if my up time is a collection of short runs ? –  Rsf Jun 1 '11 at 10:32
    
I am not sure what you are trying to use the metric for, normally you would collect this data from the support function for server reliability or customer hardware failures. You would calculate from date usage started. Number of bugs is not the number of failures as one design issue could cause hundreds or thousands of physical devices to fail, like if say, it was a firmware bug that only occurred in rare situations. –  Bruce McLeod Jun 1 '11 at 10:36
    
Ah ah... that's the problem. We got a request from a client to supply MTBF numbers, the request went through our long chain of command and was broken to pieces (many teams, each gives their number) so asking back what did they mean is useless. I also know that it's not uncommon for marketing people not to know what they are asking for... –  Rsf Jun 2 '11 at 8:40

If you are running a service or a platform (e.g. operating system), then MTTF and MTBF are very important factors in software and have been used for quite some time to assess reliability usually via some form of stress testing.

One of the primary purposes of stress testing is to find the MTTF (mean time time failure) which generally occurs due to memory leaks, resource capacity, bandwidth, etc. Simply stated, stress testing can help us determine the average amount of time our software is used before it crashes or hangs.

MTBF is esp. important for services and servers, so companies know when a server goes off line how many seconds, minutes it will take to 'reboot' the software (assuming no hardware failure).

Being able to predict MTTF and MTBF helps services load balance and also perform scheduled maintenance.

That being said, many people often confuse MTBF and MTTF. So, if you don't understand what your clients want you shoud ask them to clarify how they will use that information. I suspect they are likely looking for reliabilty data, of which MTBF, MTTF are just 2.

share|improve this answer

Bishop and Bloomfield have proposed a method of predicting long term reliability growth in software. It is based on an initial assessment of faults. Their work is very easy to understand and follow and later works look at estimating software defects by various methods.

A large set of their papers can be found here.

http://www.adelard.com/resources/reports/papers.html#P12

Papers that may be helpful, or provide a good starting point are listed here:

A Conservative Theory for Long-Term Reliability Growth Prediction P G Bishop and R E Bloomfield, 1996 http://www.adelard.com/papers/issre96m.pdf

Estimating Residual Faults from Code Coverage P.G. Bishop http://www.adelard.com/papers/SCOMP02_faults_vs_coverage10.pdf

Worst Case Reliability Prediction Based on a Prior Estimate of Residual Defects P.G. Bishop, R.E. Bloomfield http://www.adelard.com/papers/issre02_34_bishop.pdf

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.