Let's say you are in charge of an SQA & Testing company that offers outsourced testing services to other companies that develop software. Let's say you offer 4 general types of testing:

  1. Functional Testing
  2. Performance Testing
  3. Security Testing
  4. Usability Testing

Let's assume you use a fixed price model: your client comes with a project with pre-defined requirements and scope, and a fixed duration, which are unlikely to change. You estimate how much it will cost you to provide testing for that project, and then you offer your client a fixed price that hopefully ensures you will gain some profit above your costs.

My question is about how to do the estimation above. How would you go about estimating the costs of doing testing to a given project X for your client? What information about the project would you look at in order to make a good estimation?

To formalize my question a little bit more, let's say you have to define 4 functions:

  • FunctionalCost(X): estimated cost of providing functional testing for project X
  • PerformanceCost(X): estimated cost of providing performance testing for project X
  • SecurityCost(X): estimated cost of providing security testing for project X
  • UsabilityCost(X): estimated cost of providing usability testing for project X

How would you define these functions?

  • It depends ON A LOT OF FACTORS! Jun 30 '17 at 3:20
  • but is there a way you could formalize how you analyze those factors in order to come up with a cost estimation? Jun 30 '17 at 3:30
  • All that depends on the requirements of the project. I believe each project is different. Hence, if you analyze the requirements for each you would get different numbers and other parameters. Can you formalize that? Can you formalize and come up with a singleton requirement for every project? If yes then you can do the same for testing of it! Or else, you just have to figure it out per on project basis. Jun 30 '17 at 3:34
  • But I mean, if you were to estimate how many man-hours you will need to put in to test the project, are there any metrics about the project that would help you in making that assessment? Jun 30 '17 at 3:40

I'd be incredibly reluctant to do this, at least if I wanted any repeat business/didn't want to go broke.

At some level, you're trying to do the same thing you try to do when estimating test time for an internal project; the things I look at include code complexity, developer experience in general, developer experience in the area being changed, tester experience in general, tester experience in the thing being tested, interaction with other pieces of software, impact of misses, types of problems likely to be found, ease of finding problems, ease of noticing problems, and so on.

Some of those things I can get from a design document, but a lot of it requires historical knowledge. And as a third-party provider, you're going to have no way of knowing most of that historical knowledge. Even if you're repeating work for the same company, you aren't going to know if the reason the last project went so well is because they had their two best developers on it, but one of them has retired, and the other one is only working 1/2 time on the next version. Or they've moved to a new framework they aren't familiar with. Or a new language.

There's also the question of what you're going to promise, on your side of the contract. x number of defects found? y number of defects found in production? What if you think something is a defect and they don't? Or you find the number of defects you said you would, but a big one slips by?

Basically, you'd be promising to absorb an unknown amount of risk for a fixed fee, and at some level, be giving your employers a reason to give you horrible code that they want you to test quality into.

  • What if part of the requirements is to only perform black-box testing (no source code inspection)? would that change your answer in any way? Jun 30 '17 at 15:37
  • No. I'd actually be much more willing to do white box testing for a fixed fee than black-box testing, because if I can see code I'm estimating the cost of testing, it would be a lot easier to estimate the number of variations that would be required, and the source code quality in general (assuming I was familiar with the language.) But the contract would have to be very well written. Jun 30 '17 at 15:43

If you or your team have accomplished such projects before, your estimations can be based on the previous experience. If this type of project is new for you, the estimation will be less precise, but you can try defragment the functionality into basic elements and then it will be easier to estimate each one of them separately (e.g. a simple authorization form - it is more or less clear how many cases you need here and how much time does it take).

Once you have estimation of working hours, you will obtain the cost value after multiplying it by your rate. For a team with different rates it would be more complicated of course.

So I would define the first function as:

FunctionalCost(X) = Rate * (T(el 1) + T(el 2) + ... + T(el N) + T(optional: risks etc.))

Not so sure about performance, security and usability. These areas require additional assumptions.

  • That's assuming you can be confident about the code quality, which I don't think you can. Jun 30 '17 at 14:54

If it is routine project, you have previous experience and use that.

Diving into a new kind project, you have no idea what is lurking inside in dark corners. So it is considerable risk. Question is, who will be shouldering the burden of that risk. There are many options how to deal with the uncertainty:

  • Customer holds all the risk (testers are paid for billable hours). This leaves customer flexibility to change priorities.
  • Test team holds all the risk (fixed price contract). So you have to pad estimates heavily, and fight against any changes, and make changes very expensive for customer.
  • Shared risk (Milestones). Everybody makes best estimate, and after each milestone estimates are re-evaluated and each party can continue project or end it.
  • In the case of Customer holds all the risk, how can the customer make sure you are not lying when you tell them you've been working for X hours? what kind of tracking system should be used? Jul 1 '17 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.