Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
We are four weeks in testing a new application. In summary
But the actual bug discover is this:
What should I tell them?
There's not enough data for any trend to be clear.
You found ~26 bugs in the last half of May. You found ~27 bugs in the first half of June. If the bug discovery rate keeps going like this, you're projected to maybe find 40 bugs in the month of June (40 being somewhat conservative).
So you have potentially two data points: 26 and 40.
Consider this hypothetical situation:
July: 37 bugs
August: 46 bugs
September: 33 bugs
October: 33 bugs
November: 29 bugs
December: 17 bugs
January: 25 bugs
February: 24 bugs
March: 19 bugs
April: 18 bugs
May: 14 bugs
Put that in your favorite spreadsheet app and plot it as a bar graph. Look familiar? If your upper management is trying to find some significant progress on the order of 160 hours of work by one tester, they're sorely mistaken. What you need to show management is that you've already identified 43 bugs in the code while implementing this new workflow/tool/testing technique.
It may be that you're in the hump on the left of their 'expected' outcome. It could also be that 1 tester on 1 project isn't enough to generate that kind of pattern at all. In fact, if bug finding remains constant, that means you don't have enough people testing in order for them to find the bugs fast enough; eg by the time they find 20 bugs, another 20 have been created and are waiting to be found. The only way to see that downward trend as time goes on is if your testers are able to find bugs faster than they're created.
Just try to emphasize that this is a problem whether it's recorded or not, and that there isn't enough data to come to any sort of conclusion yet. If they seem like they want to get rid of that position or stop testing altogether, try to get them to see this as a probationary period and try to convince them to give your team 6 months to try it out, and even go so far to schedule a date/meeting where the results of the probationary period are presented.
Emphasize the small data set.
Try to get testing extended to at least 6 months
Present results at the end of the 6 months, and try to justify the benefits received from testing
Emphasize these bugs exist whether they're recorded or not, and bugs in code are potentially very expensive risks.
There is a gap in communication between you and your senior management.
Personally, I do not think senior management take the first bug discovery diagram too seriously. It is merely an idealistic guideline which perhaps never happens in real life.
I speculate that the reason your senior management questioned your competence or commitment is perhaps you had provided no explanation on those days shown below.
My personal recommendation is:
Some bugs may take days to track down and report correctly (to be reproducible). And if bug is not reproducible, bug report is worthless.
Tell your management: "be careful what you ask for: you will get it". If they want to count the bugs, testers will enter big number of trivial bugs but would tend to ignore bugs which take long time to report properly.
Workplace exchange has many questions related to how to deal with management which has unreasonable metrics.
In one of my previous jobs, that's what happened: management hired bunch of unskilled summer interns who were tasked to find highest count of bugs. Received minimal training, so reported even correct system behavior as a bug.
They reported so many "issues" (and many duplicates, because they were not even trained and required to check it "bug" was already reported) that several developers spend full time just triaging (and rejecting) most of the "bug" reports. And of course they did not detected any serious bugs.
Experiment was such a disaster that it was abandoned in just few short weeks.
There are a number of things to find out in order to answer that.
What you should tell them ?
I would say I need more info and I would get back to them.
Then I would try and find out the following:
I would collect the above and try to see what patterns it shows.
Also - 2 bugs a day (most days) over 1 month is a very small sample for this sort of analysis. I would prefer to compare month-to-month for such a small sample, not day-to-day, in order to fit that curve. Month-to-month it might fit that curve which would just mean the time frame needs to be adjusted to see that pattern. This is the office politic way of saying you need more time and resources.
Also what year is that? what about weekends?
This won't help in particular, but I am severely tempted to tell your management to retake statistics if they adore it that much. With small numbers like this, all bets are off regarding statistical models, and when you model stuff like that, it's basically how you fare when a random person comes by in random intervals and randomly discovers some bug anyhow. In reality, testing is not random but you'll expand and develop suitable procedures and areas to research as bugs are getting discovered.
They want to see reality adapted to models. That's not what models are for.
Many things can affect number of bugs discovered.
Depending on size of the software + time devoted to manual testing and/or computer power devoted to automated testing, you might still be at the beginning of testing. If your current chart is short term it doesn't make sense to evaluate it against long term trends.
You could write an explanation of testing that discusses whichever of these is relevant to your situation. Discovering bugs is a discovery activity. You investigate because you don't know what you'll find, where, or when. It's not predictable or consistent. Seeing fluctuations in the number of bugs discovered is the normal, expected situation.
Personally I'd be concerned if I saw a chart that was too consistent. That would speak to massaging the chart by reporting spurious bugs.
Perhaps ask him where he got that model. The most important thing about data collection is understanding what systems and processes were used to collect that data. If you don't understand what that data is based on, then you will not be able to respond effectively to his claim.
The graph he presented you is like saying "The following graph presents, on average, the temperature of food during cooking and resting periods." It depends on what food was tested, whether they covered the food while resting, etc. If you take that graph at face value, you're making a lot of assumptions.
Since your boss is questioning the competency of your team, take care to ask thoughtful questions to understand why he thinks your bug detection rates should follow that graph. Competency and credibility are established by the quality of the questions you ask.
If you ask the right questions, your boss will end up convincing himself as to why the graph should not represent your project. Even better, through the right questions, you can resolve the problem with a new model or solution to use.
Reality versus made-up theorie. It so depends on a lot of factors like: the quality of the developers, the quality of the testers, the complexity of the domain and maybe even how good the requirements are. Building the right product the right way is pretty hard.
Robert C. Martin repentantly says that a great goal is that the QA department should never find bugs
Just like UncleBob I would expect developers to handover NO (nearly none) defects to a QA team. In a recent 4 developer project (running for a half year) we only found 2-3 defects in production, nothing serious though. Also we have no QA person or team. Still we process and do complex analysis on big-data from multiple sources, on top of some pretty complex dash-boarding and filtering stuff. We must suck in testing according to your management, because we create and find nearly no defects. Good developer practises go a long way.
In a single developer project defects should be even lower, because it is so much easier to understand your own code-base. Keeping dependencies to the minimum, no other developers to break your code must also be a great feat.
Looking at your stats and the number of defects you find a project developed by a single developer I would seriously say it looks like a sloppy developer. Time for him/her to read something about CleanCode and technical excellence.
The miscommunication is about smoothness of "The Curve". I see a huge possibility the senior management wanted to actually say something entirely different: they planned to test for four weeks max and they expected to have stable practically bug-free app by now. (Clear message when I look at their graph - right side is a stable bug-free app.)
I will be devil's advocate and assume that this schedule was communicated to the team. Normally the horizontal axis of this graph (the time to a stable program for which no new bugs are ever reported) is supposed to last years or decades. It can only be shortened to four weeks for a really really simple small program. I also assume that you didn't challenge this big assumption. You say nothing about it in your question, like you take for granted doing in weeks what is done in decades for big applications.
If senior manager failed to communicate the "four weeks" schedule earlier, or failed to say what was their goal, that is their big fault. It's their responsibility to communicate clearly and effectively and to know they expect something next to impossible. So I think you were communicated that unearthly plan.
Into fourth week and you're routinely able to pull 3 or 4 bugs a day. Whoa, that's still a lot, isn't it? They expected to get way more testing done in week one and two. I know I would, under such schedule.
Particularly week one doesn't look like an honest effort at all if you have only four weeks to go. Did the testers persistently "called it a day" after registering just two bugs? Maybe registering a bug is some major beaurocratic chore? (That's up to direct manager to analyze/fix, not up to the senior manager.) Maybe insufficient resources were assigned to testing? (Again, that's up to direct manager, not to senior manager). Maybe a programmer delivered new functionality slowly troughout the testing, so there was almost nothing to test in the first week? (a management/scheduling/programmer problem - your ambitious schedule should be obviously extended).
You need to look at this from your senior managers' point of view. They don't have the time or the technical expertise to go into detail on your project, so they want a graph or number that they can look at and instantly see if the project is going OK, or if there's a problem and they need to give it some attention. They've picked one that doesn't work. Give them an alternative that does let them see the project status and progress at a glance.
