The startup I work for is looking into outsourcing our manual testing to an unaffiliated testing lab called Applause (formerly uTest). Right now the Operations team is doing all the manual labor, and none of them have training as testers.

Applause offered us a one-month trial period, after which we would enter into a 12-month contract provided they meet a set of conditions agreed upon by both parties. What criteria should I ask for that would be fair to both parties? I feel as though I cannot ask for measurements like "x amount of bugs".

Here are some example criteria that Applause has used in the past:

  • Correctly identify the severity of defects (to within 80% accuracy) which they provided
  • Issues logged as always occurring are in fact reproducible using the steps, and environment provided 80+ %
  • Issues contain appropriate level of detail and information (i.e. log files, screen shots, etc) required 80+ %
  • Test cases and environments covered and completed within the time requested
  • Is mentioning the firm necessary? It would seem information that could potentially taint the response. "Oh don't use them I've had trouble with them" as opposed to "You need ____ criteria so you don't get bit by ____ problem" which is what the question should be.
    – corsiKa
    Commented May 27, 2015 at 17:17

4 Answers 4


I would suggest you look at their

  • Device Fragmentation
  • Scope of testing, (white box / black box) test aspects, App store testing, versioning, Security testing, memory usage and usage of best practices (mind maps, automated tools, carriers etc..)
  • Tools used / Automation Efforts
  • Previous projects in similar domain / expertise
  • Meaningful Measure of Number of priority issues identified at early cycle of projects / Test efforts
  • Production issues support / fixes / troubleshooting
  • Skillset of resources / Knowledge Transfer after testing in terms of tools, processes/ Support for Reproducing issues after release to production

There are multiple companies. You can invite them with a summary of above info and their charges. After a detailed discussion you can arrive at pros / cons of each provider.

  • 2
    You could even use the free trial to have them test something you have already tested (but not fixed) to see if they find what you found and hopefully more than that. I have used this approach in hiring testers. I made a web app with a laundry list of issues I purposefully put in to see what each candidate would catch.
    – kirbycope
    Commented May 27, 2015 at 16:08
  • 1
    siva- Good answer with lots of ideas. I already upvoted, and will accept by the end of the day. @kirbycope- we thought about providing applause with an old APK to see if they catch everything. Great suggestion. Commented May 27, 2015 at 16:23

You cannot outsource understanding of your system - and you cannot test what you do not understand how it works (or hire someone who does not have time to learn your system to test it). So i would expect that it will be your company responsibility to develop test scripts, expected answers, and what service automate for you is clicking and checking results.

Also, having adversarial relations with QA is IMHO a recipe for disaster which will end up in blame-shifting and finger-pointing. maybe that's why they changed name from uTest to Applause?

About my experience with a bit similar service (testing by untrained users) see this my answer

  • 1
    I would argue that if the system in test is intended for untrained users (most web apps) then you do not need an intimate knowledge of the systems (white box).
    – kirbycope
    Commented May 27, 2015 at 16:04
  • 1
    And I would agree :-) but you still need to define what are expected goals of the users. And if you cannot observe your users and find out where they were derailed/confused, even if ultimately succeeded reaching the goal, experience was not as smooth as you would like to, value of such 3rd party testing is rather low. Commented May 27, 2015 at 16:48

I heard good things about uTest, but I am not sure they are a good choice for every product.

When I tested a simple application for mobile Android devices their services were priceless- a wide range of devices, geographic locations and field tests all defined in the contract to some degree for example- test in X Y Z cities or continents, using 10 out of the 20 most used devices.

I am not sure they are a good fit when the product becomes complicated and a lot of resources are needed, either time, hardware, installation or configuration since this will dull their advantage of agility.

The bottom line is the same as for any other employee or contract- you must have trust. An item like "Correctly identify the severity of defects" is a source for endless argues is you don't like the services you get and looking for a way out.

  • 1
    Thanks for that. We have a very simple product, but customize it for different cities. With a portfolio of 15 white labels, it sounds like Applause may be a good fit. Commented May 27, 2015 at 14:46

I would suggest that you rather pay for them to work for two months with option to extend that to 12 months or whatever you need. That way you don't have to argue if some criteria has been fulfilled, you can get out of the deal if you are not satisfied with them, and ultimately only lose much smaller amount if they fulfill things to the letter, but are still not what you are looking for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.