2

I just asked a question about best practice for providing feedback to a 3rd party dev shop, and this question's more about what to expect from them, and basically how to review an iPhone app.

Right now we get the app without any QA by the dev shop, which is fine. I'm finding lots of bugs, which is natural, but I wonder what's the best practice: should I as the client be testing this for every bug I can find, or should I just be trying to see if the functionality works and when it doesn't work, indicate such in my feedback to them?

I'm finding that what I've been doing (trying to squash every possible bug I can) is becoming extremely time consuming but it seems unavoidable if I'm trying to test the functionality of the app and keep coming across bugs.

I want to feel like I should be providing more feedback on look and feel, and less on the little things like "phone number formatting isn't working right".

Are my expectations off?

Improve this question
4
  • Are you locked into a contract with this 3rd party shop ? If they are delivering such poor quality, can you change supplier ?
    – Phil Kirkham
    Commented Jun 29, 2012 at 17:27
  • 4
    "I'm finding lots of bugs, which is natural" Hmm, I'm not sure I agree. Perhaps you aren't holding them to a high enough standard?
    – Joe Strazzere
    Commented Jun 29, 2012 at 19:33
  • 4
    There are no "best practices", there are only good practices in context.
    – Bruce McLeod
    Commented Jul 1, 2012 at 22:45
  • Updated title. q starts as from them but rest of q and summary is about feedback to them.
    – Michael Durrant
    Commented Feb 2, 2017 at 13:28

5 Answers 5

Reset to default
4

Was there any requirement for formal QA to be performed on there side. Many small shops don't have dedicated testers. It's just the developers who are writing the code who are testing it and the customer performs a testing/UAT role. The client should be aware of this up front.

I'm finding that what I've been doing (trying to squash every possible bug I can) is becoming extremely time consuming but it seems unavoidable if I'm trying to test the functionality of the app and keep coming across bugs.

Yep, welcome to the life of a tester.

I think that you're expectations may be off if the functionality hadn't defined already as to how it was to function. If this had been clearly defined however, and it happens often (multiple times per requirement/iteration), you may want to look for a different shop in the future.

Improve this answer
3
  • Thanks for the answer. They do have QA in house, but I we both agreed that it didn't make sense to test the app out until a section of it was considered done from a development perspective; once it had all the moving parts, then it'd go through testing to find all the bugs and work them out. I'm new to being a client (and development of any kind in general) so these answers are helpful to figure things out. Thanks.
    – Michael Paul
    Commented Jun 29, 2012 at 17:31
  • 3
    @Michael what is your definition of 'done' ? Are you more concerned with look and feel or functionality or speed to market ? Have they asked you what your quality concerns are and what is important to you ?
    – Phil Kirkham
    Commented Jun 29, 2012 at 17:34
  • In this context, "done" means the screen's developed, looks great, and has the functionality to complete the task. (For example, you're looking at a screen with a table on it, and we have the ability to tap "add" and create a new record on that table.) Mostly concerned with look, feel and functionality. Speed to market is not as important as making sure everything's as quality as it can be, but it's important nevertheless. We've indicated previously that quality is kind of lacking and are evaluating the increase/decrease in quality with every iteration. Good learning experience.
    – Michael Paul
    Commented Jun 29, 2012 at 20:15
3

Find a different dev shop where they undertand and DO testing so that they deliver an app with very few bugs ?

( yes there are places like that, I'm working at one )

Improve this answer
2
  • They do testing in-house, but we agreed to save the testing until each portion of the app is complete... perhaps we should revisit that agreement in the future.
    – Michael Paul
    Commented Jun 29, 2012 at 17:34
  • 1
    No reason that each part cant be tested...
    – Phil Kirkham
    Commented Jun 29, 2012 at 17:39
2

Addressing your question specifically, it is your job to report every bug you find. When you start making yourself ignore the smaller details, you do the customer and yourself a dis-service. Down the road the might notice these smaller bugs and wonder why you didn't find it. Or they may fix all the big functionality issues then want you to catch all the smaller bugs, but maybe you have a hard time seeing them because you made your brain start ignoring them.

In the future you should probably try to find a client whose developers don't write such buggy code and maybe talk with them ahead of time about what your expectations are of them, and what they're expecting you to report back with.

Improve this answer
1
  • Good points. We just asked them to involve their testing team. (Up until this point no testing team had been involved--it was just the developer working on the front-end) so we'll see if that makes a difference in bug squashing before we see the next build of the product. I'm hoping it does, because our time is much better spent providing feedback on look and feel rather than reporting bugs. Costs more to have QA, but speed to market it faster and it means less time spent documenting bugs and more time thinking about the product.
    – Michael Paul
    Commented Jul 5, 2012 at 14:11
2

Michael, the solution is depends on your plans, budget, etc.

If you plan to continue working with this dev shop for a long time and you have enough budget – then the best decision (you know it without this words) – is hire a tester for the team.

But it will not fix all the issues at a moment. As I understand, you are the Domain Expert and only you know how the application should behave and what problems it should solve.

A new tester, first of all have to learn the domain. During this time he/she will make a lot of mistakes but eventually you will get your “advocate” on the dev team side. If you have enough time and budget – then it would be the best solution.

Otherwise – you are the only tester.

Michael, I would like to highlight that the developers (almost all) are focused on technology and implementation. They will implement everything as they’ve understood you said. You and the dev team have different mindsets.

In the parallel question about the tool, you have chosen the Pivot Tracker as a collaboration tool.

I would recommend you to look at Bug Trackers also, for instance, Mantis: http://www.mantisbt.org/

Bug tracking software – is designed to create/triage/prioritize the software bugs. It is just an advice; maybe you will find that Pivot Tracker is enough for dev management.

And maybe the Specification by example could help you? http://en.wikipedia.org/wiki/Specification_by_example

The key idea of this practice is specifying with precise examples. So, the software specification made up with examples. For instance:

  • The phone numbers on the search results page should be formatted like (888) 999-222-2
  • The user search page should correctly display names in any encoding like Dmitry Zhariy or Дмитрий Жарий

Using this technique you could create a requirements for most important application units and give the examples for better understanding.

Improve this answer
1

It's kind of concerning that you say they have testers, but you are finding loads of bugs.

It might be worth finding out what the testers are actually testing(!) — not with the intention of giving them a hard time, but if you are finding stuff that shouldn't be there, then it's not really right.

Or are you getting it at the same time as the testers? Would it be worth allowing it to go through some of their tests before you spend a lot of your valuable time pointing out things?

If they still aren't testing enough, then it may be worth hiring a tester to help you test it. Of course it depends on the app, but it doesn't necessarily need to be a fulltime role.

You may find this MindMap/Checklist on Mobile Testing useful.

Improve this answer
1
  • Thanks for your answer. To clarify: up to this point, we've gone without their QA team getting involved but we have now invoked the QA team and expect to see much better results going forward. Interesting mind map--definitely broadened my horizons. The stuff I'm finding as I evaluate the app has been all around the "Function" area of that mind map. So many other things to consider! :)
    – Michael Paul
    Commented Jul 5, 2012 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.