Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Only 1% of our clients use particular features or scenarios. Do you think we should spend as much time testing these features and scenarios as we spend on our central features and scenarios, or we should test mostly the "happy path" for these features?

Thank you.

share|improve this question
2  
Well, as always it depends. Maybe those 1% of your clients are paying 99% of your income. Maybe those are quite important customers: if you lose them for not working features, maybe other customers will follow them. So the question is (a) how important are those clients for you and (b) how those features are important for them? –  dzieciou Mar 16 at 9:49
    
A lot of the work of a tester is about analysing risk. What is the likelihood of part of the system failing and what is the risk to the business if that part of the system fails. –  AdrianHHH Mar 17 at 10:02

5 Answers 5

As already stated in a comment, it depends.

You have to consider a lot of things in order to decide your test strategy. Usually you have limited resources for testing (time, budget, testers, infrastructure etc.), so you cannot test everything equally thorough. A risk-based approach helps you decide where to concentrate your testing efforts and how thorough to test.

For this, you should first identify all relevant risks, and assess them. The assessment includes categorizing each risk and determining the likelihood of occurrence and impact upon occurrency for each risk. The impact should be assessed by all stakeholders, that is, you should involve your customer(s)/clients.

This helps you deciding which test techniques to use and the priorities of each test case or test object of your system. The effort associated with developing and executing a test is proportional to the extent of the risk.

For the particular features you asked about, some questions, which you should answer yourself, are:

  • Would it be a showstopper for your clients if these features would not work as expected (business risk)?
  • How important are those clients to you? If you lose a client, this could have a very negative impact to your image (business risk).
  • Is the implementation of those features very complex or has many interfaces to other parts of the system? Or is it quite simple (technical risk)? If it is complex, you should test it more thorough than if it's simple.

Of course, there are many other questions you have to answer for yourself before you can decide your approach.

share|improve this answer

As always, it depends.

At my previous employer, we had some 400-odd customers (business-to-business software). The one customer who was the only user of certain features was one of the largest customers so it was absolutely critical to make sure those features worked well.

In anything business-to-business that's pretty common: the biggest customers will typically also use the software to its limits and will be the only users of some of the more obscure features. It's quite possible those features were developed specifically for those customers.

I typically look at features used by a small subset of customers this way:

  • Which customers use the feature? - If the customer(s) using the feature are among my employer's biggest/best customers, it gets more testing than a feature used only by the smallest customers (because the big customers will cause more financial problems for my employer if they leave than the small customers will, and I don't have infinite time or resources).
  • How critical is the feature to its users? - If the feature involves financial data, tax data, or other mission-critical data, it gets more testing than a feature that is less mission-critical.
  • How stable is the feature? - Something that's been built, tested, deployed, and hasn't changed since then is going to get less attention than something that's constantly changing or getting a lot of bugs reported against it.
  • Can the feature impact core function? - A feature that interacts with core functionality and has the potential to break core features is going to get more attention than one that doesn't.
  • Will the only users of this feature be upgrading to this version? - Although it's a long way from ideal, sometimes it's necessary to accept that a particular version of the software isn't going to work for that 1% of customers. One customer at my last employer had a six month upgrade process, so there were sometimes versions of the software that customer simply could not use (usually because a feature developed for that customer was being adopted by a different customer, but the changes for the second customer were so extensive they broke the first customer's feature and required an extensive unification project which couldn't be done in the time frame available).
share|improve this answer

Spending a lot of time for testing these features is not good. But also it's not good to close your eyes to them.

I think the best practice is to mark out specific day(s) (for example, once a month) and create tests just for these features.

So, you won't spend much time and that 1% of customers who use those features will be happy and appreciate your work.

share|improve this answer

I think that risk analysis should be done. If the feature, although has low probability of being used, has a great impact, in case it is disfunctional, then it should be included in the test plan, listed as per other test cases' impact and probability analysis.

share|improve this answer

It depends from your management actually. What is the propose of your product? Based on your question I would like to execute testing of particular features just once during iteration if you have some Agile. You should not spend all your time for testing this features but still need to test basically, do not ignore them and please execute very basic smoke test.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.