28

We are four weeks in testing a new application. In summary

  • There is only one developer assigned to this project, basically, 8 man hour is available for this project.
  • The rate of bug discovery is uneven, but the senior management is questioning my team competence because what they want to see is: enter image description here

But the actual bug discover is this:enter image description here

What should I tell them?

  • 31
    tell them: In theory, there is no difference between the theory and the practice. In practice, there is. Such curve might be observed as average over hundreds of projects. In a single project, not a chance. Valid question BTW, +1. – Peter M. Jun 21 '17 at 21:55
  • 5
    It's way too early to know how well those two curves match. If the next month you find twice as many bugs and then the month after that is just like this month, you'll be fitting the left hand half of that chart perfectly. – David Schwartz Jun 22 '17 at 4:07
  • 6
    You tell them to read this question so we can tell them how idiotic they are. Why is management so abysmally inept just about everywhere? – Davor Jun 22 '17 at 9:53
  • 2
    It's not obvious to me that the actual bug discovery curve is really that different from the theoretical one. You would need to average over long periods of time to see anything but that's just an artefact of the low number of bugs and completely normal with this type of data. – Relaxed Jun 22 '17 at 11:14
  • 2
    The first graph is made from hundreds of testers working for tens of thousands of man hours. You give me that, and I'll give you that graph as the second one. Don't expect it to come from anything less. – Kevin Fee Jun 22 '17 at 20:00

10 Answers 10

11

There's not enough data for any trend to be clear.

You found ~26 bugs in the last half of May. You found ~27 bugs in the first half of June. If the bug discovery rate keeps going like this, you're projected to maybe find 40 bugs in the month of June (40 being somewhat conservative).

So you have potentially two data points: 26 and 40.

Consider this hypothetical situation:

  • July: 37 bugs

  • August: 46 bugs

  • September: 33 bugs

  • October: 33 bugs

  • November: 29 bugs

  • December: 17 bugs

  • January: 25 bugs

  • February: 24 bugs

  • March: 19 bugs

  • April: 18 bugs

  • May: 14 bugs

Put that in your favorite spreadsheet app and plot it as a bar graph. Look familiar? If your upper management is trying to find some significant progress on the order of 160 hours of work by one tester, they're sorely mistaken. What you need to show management is that you've already identified 43 bugs in the code while implementing this new workflow/tool/testing technique.

It may be that you're in the hump on the left of their 'expected' outcome. It could also be that 1 tester on 1 project isn't enough to generate that kind of pattern at all. In fact, if bug finding remains constant, that means you don't have enough people testing in order for them to find the bugs fast enough; eg by the time they find 20 bugs, another 20 have been created and are waiting to be found. The only way to see that downward trend as time goes on is if your testers are able to find bugs faster than they're created.

Just try to emphasize that this is a problem whether it's recorded or not, and that there isn't enough data to come to any sort of conclusion yet. If they seem like they want to get rid of that position or stop testing altogether, try to get them to see this as a probationary period and try to convince them to give your team 6 months to try it out, and even go so far to schedule a date/meeting where the results of the probationary period are presented.

So to summarize:

  • Emphasize the small data set.

  • Try to get testing extended to at least 6 months

  • Present results at the end of the 6 months, and try to justify the benefits received from testing

  • Emphasize these bugs exist whether they're recorded or not, and bugs in code are potentially very expensive risks.

7

There is a gap in communication between you and your senior management.

Personally, I do not think senior management take the first bug discovery diagram too seriously. It is merely an idealistic guideline which perhaps never happens in real life.

I speculate that the reason your senior management questioned your competence or commitment is perhaps you had provided no explanation on those days shown below.

enter image description here

My personal recommendation is:

  • Given they had had question over your competence or commitment, you should provide a detailed report regarding why there was no bug discovered on several days (ignore weekends and public holidays of course)
  • Even when the senior management has not asked, provide a daily update regardlessly. Whenever there is a project status update, cc them to keep your working progress transparent.
  • 5
    Since OP said that there is only 1 developer on this project and those gaps reflect a weekend, an entire week and a weekend, respectively, I would think that the developer was on vacation for the week and not working on the weekends. Still, good insight. Particularly the bolded bit. – user26379 Jun 22 '17 at 15:28
7

Some bugs may take days to track down and report correctly (to be reproducible). And if bug is not reproducible, bug report is worthless.

Tell your management: "be careful what you ask for: you will get it". If they want to count the bugs, testers will enter big number of trivial bugs but would tend to ignore bugs which take long time to report properly.

Workplace exchange has many questions related to how to deal with management which has unreasonable metrics.

In one of my previous jobs, that's what happened: management hired bunch of unskilled summer interns who were tasked to find highest count of bugs. Received minimal training, so reported even correct system behavior as a bug.

They reported so many "issues" (and many duplicates, because they were not even trained and required to check it "bug" was already reported) that several developers spend full time just triaging (and rejecting) most of the "bug" reports. And of course they did not detected any serious bugs.

Experiment was such a disaster that it was abandoned in just few short weeks.

4

There are a number of things to find out in order to answer that.

What you should tell them ?

I would say I need more info and I would get back to them.

Then I would try and find out the following:

  • was the application still being changed or was it frozen in time ?
  • what was the change in the number of features from start to end
  • what was the severity of the early vs late bugs ?

I would collect the above and try to see what patterns it shows.

Also - 2 bugs a day (most days) over 1 month is a very small sample for this sort of analysis. I would prefer to compare month-to-month for such a small sample, not day-to-day, in order to fit that curve. Month-to-month it might fit that curve which would just mean the time frame needs to be adjusted to see that pattern. This is the office politic way of saying you need more time and resources.

Also what year is that? what about weekends?

4

This won't help in particular, but I am severely tempted to tell your management to retake statistics if they adore it that much. With small numbers like this, all bets are off regarding statistical models, and when you model stuff like that, it's basically how you fare when a random person comes by in random intervals and randomly discovers some bug anyhow. In reality, testing is not random but you'll expand and develop suitable procedures and areas to research as bugs are getting discovered.

They want to see reality adapted to models. That's not what models are for.

2

Many things can affect number of bugs discovered.

  • Methodology. If you're fuzzing there will certainly be random variations.
  • Changes. If software was being changed while testing new bugs could easily be introduced and increase number found in the following days.
  • Bug complexity. Encountering a bug is only the first step in reporting it. Finding enough information to make a valuable report may take a long time for complex bugs.
  • Focus. If you're targeting different areas of functionality on different days, yesterday's number will have no impact on today's number. You're exploring different terrain. Specific things that might differ:
    • Previous testing. An area that has already had heavy testing will likely have fewer bugs to find than another area with no previous testing.
    • Feature age. A feature that has been around longer will likely have had its major bugs discovered simply through use.
    • Developer. Code written by an experienced developer will likely have fewer bugs than code written by a new developer.

Depending on size of the software + time devoted to manual testing and/or computer power devoted to automated testing, you might still be at the beginning of testing. If your current chart is short term it doesn't make sense to evaluate it against long term trends.

You could write an explanation of testing that discusses whichever of these is relevant to your situation. Discovering bugs is a discovery activity. You investigate because you don't know what you'll find, where, or when. It's not predictable or consistent. Seeing fluctuations in the number of bugs discovered is the normal, expected situation.

Personally I'd be concerned if I saw a chart that was too consistent. That would speak to massaging the chart by reporting spurious bugs.

  • Another thing that can differ for different areas of functionality: difficulty of testing. Especially with manual testing, the amount of time required to test a piece of functionality X times can vary wildly. Click 1 button vs traverse 5 form pages and fill everything out each time. – Lycan Jun 22 '17 at 14:57
1

Perhaps ask him where he got that model. The most important thing about data collection is understanding what systems and processes were used to collect that data. If you don't understand what that data is based on, then you will not be able to respond effectively to his claim.

The graph he presented you is like saying "The following graph presents, on average, the temperature of food during cooking and resting periods." It depends on what food was tested, whether they covered the food while resting, etc. If you take that graph at face value, you're making a lot of assumptions.

Since your boss is questioning the competency of your team, take care to ask thoughtful questions to understand why he thinks your bug detection rates should follow that graph. Competency and credibility are established by the quality of the questions you ask.

If you ask the right questions, your boss will end up convincing himself as to why the graph should not represent your project. Even better, through the right questions, you can resolve the problem with a new model or solution to use.

1

Reality versus made-up theorie. It so depends on a lot of factors like: the quality of the developers, the quality of the testers, the complexity of the domain and maybe even how good the requirements are. Building the right product the right way is pretty hard.

Robert C. Martin repentantly says that a great goal is that the QA department should never find bugs

http://blog.christoffer.me/9-things-i-learned-from-reading-the-clean-coder-by-robert-c-martin-on-how-professional-developers-conduct-themselves/

Just like UncleBob I would expect developers to handover NO (nearly none) defects to a QA team. In a recent 4 developer project (running for a half year) we only found 2-3 defects in production, nothing serious though. Also we have no QA person or team. Still we process and do complex analysis on big-data from multiple sources, on top of some pretty complex dash-boarding and filtering stuff. We must suck in testing according to your management, because we create and find nearly no defects. Good developer practises go a long way.

In a single developer project defects should be even lower, because it is so much easier to understand your own code-base. Keeping dependencies to the minimum, no other developers to break your code must also be a great feat.

Looking at your stats and the number of defects you find a project developed by a single developer I would seriously say it looks like a sloppy developer. Time for him/her to read something about CleanCode and technical excellence.

1

The miscommunication is about smoothness of "The Curve". I see a huge possibility the senior management wanted to actually say something entirely different: they planned to test for four weeks max and they expected to have stable practically bug-free app by now. (Clear message when I look at their graph - right side is a stable bug-free app.)

I will be devil's advocate and assume that this schedule was communicated to the team. Normally the horizontal axis of this graph (the time to a stable program for which no new bugs are ever reported) is supposed to last years or decades. It can only be shortened to four weeks for a really really simple small program. I also assume that you didn't challenge this big assumption. You say nothing about it in your question, like you take for granted doing in weeks what is done in decades for big applications.

If senior manager failed to communicate the "four weeks" schedule earlier, or failed to say what was their goal, that is their big fault. It's their responsibility to communicate clearly and effectively and to know they expect something next to impossible. So I think you were communicated that unearthly plan.

Into fourth week and you're routinely able to pull 3 or 4 bugs a day. Whoa, that's still a lot, isn't it? They expected to get way more testing done in week one and two. I know I would, under such schedule.

Particularly week one doesn't look like an honest effort at all if you have only four weeks to go. Did the testers persistently "called it a day" after registering just two bugs? Maybe registering a bug is some major beaurocratic chore? (That's up to direct manager to analyze/fix, not up to the senior manager.) Maybe insufficient resources were assigned to testing? (Again, that's up to direct manager, not to senior manager). Maybe a programmer delivered new functionality slowly troughout the testing, so there was almost nothing to test in the first week? (a management/scheduling/programmer problem - your ambitious schedule should be obviously extended).

  • -1. Expecting to find a consistent number of bugs from one day to the next is unreasonable. Fewer bugs found =/= lack of honest effort. – Lycan Jun 22 '17 at 14:48
  • @Lycan I'm not saying the senior manager is reasonable. I'm not saying that they communicate clearly. I'm not saying they actually wanted day-to-day smoothness. But I'm saying that they could be rightly surprised to have more than 25% of bugs found in the week fourth of four. – kubanczyk Jun 22 '17 at 15:45
  • 3
    Not really. Without knowing the background to those numbers, they're meaningless. If the tester(s) had no exposure to the code before, week 1 may well have been learning. Or setup. Or a host of other things. Maybe the bugs they were finding were so severe they couldn't progress, and once they showstoppers were fixed, you had a host of bugs found on May 29th. – Kevin McKenzie Jun 22 '17 at 15:55
  • 1
    You also can't find bugs that aren't there. What if it just so happened that the stuff scheduled to be tested in week 4 had a larger number of actual defects present? Being surprised by things like "more than 25% of bugs were found in week 4 of 4" is also implicitly expecting your developers to evenly distribute their mistakes. Number of bugs found simply isn't a measure of testing work done, so saying things like "week one doesn't look like an honest effort at all" based on bug count stats alone is absurd and rather insulting to the team in question. – Ben Jun 25 '17 at 4:41
0

You need to look at this from your senior managers' point of view. They don't have the time or the technical expertise to go into detail on your project, so they want a graph or number that they can look at and instantly see if the project is going OK, or if there's a problem and they need to give it some attention. They've picked one that doesn't work. Give them an alternative that does let them see the project status and progress at a glance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.