Can a performance issue be defined as blocking bug?

Question

I am the QA Manager of my organization.

We have a performance issue, where a process takes 6X times rather then X. I defined the issue as blocking since there is no workaround on eliminating the performance issue, and while the scenario performs badly, the system is busy (which is basically unaccessible). The product manager mentions that this issue is critical, and in general, performance issues will never be blockers. I took it to the extreme by asking: "and what the 6x seconds were 10 hours? would it still not be a blocker?" and still got the same response.

Two questions:

1. Am I wrong?
2. How can I prove my point?

Answer 1

This is not QA decision but business decision. If performance is satisfactory for the customer - it is good enough.

Also remember that the most important speed is speed to the market - deliver most value for your customers to make money to support additional development of the product. If product is late to the market, and is beaten by inferior competition, how much matter how perfect it is?

Check requirements - do you have performance requirements?

But even if performance requirements are not fulfilled, it is business decision to release (or not) a product. Get the best stats you can get, and present them to the deciders.

Can you release now (it works fast enough for big enough share of your customers), and work on improving performance later? Again, business decision, you responsibility is to provide info for management to make the decision.

Answer 2

The question of whether a performance bug is a blocker depends entirely on whether it stops a user from doing whatever they are trying to do with the software.

I have worked on software that took over 24 hours to run, during which the machine couldn't be used for anything else; that wasn't considered a bug at all, because the user workflow was to start that process before leaving Friday, and expect the result to be available on Monday. Anything that might cause the process to abort partway through was a blocker, because the users wouldn't want to wait another week, but if the runtime doubled to 48 hours that was fine, as the user's workflow wouldn't be impacted. I've known people who were working on academic research for which they'd start a program analyzing a huge dataset, and expect that it would take weeks -- that was fine, as they just needed a result by the end of the term.

Conversely, on a real-time video game, if the time it takes to render a frame of graphics exceeds .033 seconds that is likely to be visible to the players, so anything that impacts performance at all will be important (might be critical or blocker depending on how bad it gets; 30 fps already annoys some people, if the game drops to 10 fps even for short periods nobody is likely to play).

Someone in your organization should be in charge of knowing what the users need and what will be a problem for the users -- depending on how your team is organized, this might be a "producer", a "project owner", a "customer representative", a "team lead", or a "manager", but whatever the title, that person is the only one who can answer the question of whether your software is too slow when it runs for one second, or too slow if it runs for an hour, or too slow if it runs for a day. QA's task is to measure the software and let the team know how slow it can get on the expected workloads.

Once you have the information about how fast the software is and the needs and wants of customers, the product manager can make the call as to how important it is to improve performance. If the software is slower than the minimum that users need, that is likely a blocker. If it is slower than they want but meets their minimum need, that may be anything from minor to critical depending on how much they want it to be faster. If it is as fast as the users want, it doesn't matter if it could be made faster, it isn't worth spending time on improving performance (for example a game that runs its graphics at 400 frames per second; that is well beyond the point where anyone would notice if you improved it to 500 frames per second, so it isn't worth spending any engineering effort even for a dramatic improvement in speed).

Remember, every hour that an engineer works on improving performance is an hour that they could have used to fix some other bug -- the important thing is to determine whether the users want more performance more than they want other things fixed; if to a user the current speed is so slow that the program might as well have crashed (because the speed causes them to miss an important deadline, for example), that may be a blocker just like an unavoidable crash bug. But if the user is just mildly annoyed by the current speed, that may well be the same importance as fixing typos in user-visible text that will annoy users that like proper spelling and grammar.

Answer 3

It depends.

• If your application under test is internally developed and internally used in your organisation and it doesn't have to be realtime it might be fine to have 60 seconds response time
• If you are running a e-commerce portal and it fails during Black Friday or works very slowly - most likely you will be losing thousands of bucks each second

As a QA you should be raising any suspicious issue and providing as much related information as possible as "this action is slow" sucks and "this action is slow due to XXX module cannot process more than YYY files per second" or "this action is slow due to lack of free RAM and swapping" is much better. So instead of asking questions on SE take profiling tools and operating system metrics like CPU, RAM, Disk, Network usage, etc. and find out why it takes that long.

Answer 4

We should treat performance issues just exactly like we treat software bugs. Performance issues are just bugs, and they deserve the same type of attention to detail that we give to bugs in a line of code. After all, performance issues usually happen much earlier in the user experience, driving potential users, customers, and subscribers away long before they ever get to our carefully crafted and maintained software. Performance issues are introduced by mistake and “unintended behavior”, the very definition of a bug. If the product is a performance based one and it's having performance requirements, the performance issues should be treat like blocker bugs. Otherwise it will affect the business very badly.

Answer 5

if you have a performance requirement, then slow performance is the blocker issue. So QA can raise the bug and update to manager.

IF customer not specified any performance requirement(specification) then its a business decision. But QA can raise performance improvement enhancement and suggest for performance improvement.

Answer 6

Performance issues could be an indirect reference to a bug somewhere else in the program where you are not looking which may/may not be obvious now but definitely you have to address it later. For example: Memory Leak issue, GC problem, etc Not to miss, there could be architectural issues with the server layout or load Balancer as well.

Answer 7

A bug is where the behavior doesn’t satisfy the requirements or match the specification. So performance can certainly be a bug (in some game scenarios something happening too fast can be a bug).

Whether or not a bug is a blocking bug or not is a decision you make, not a fact you observe. If the product manager finds it acceptable to release a product with a given performance, that performance level is evidently acceptable.

Instead of trying to prove a point, you should be getting an official statement defining the minimum acceptable performance criteria for this action and determine if your application meets that criteria.

Answer 8

I fully agree with Peter Masiar's answer. As QA has a signalling role, not a deciding, but I do want to present a counter perspective.

If you rapidly loose users, would it be a blocker than? I would think so. It's possible all your users would jump ship to a competitor application. For example: Lets say a major chat application takes hours to send messages instead of seconds, this issues stays on pfor some days. Probably me and my friends would switch chat-app with ease and never look back.

Monitoring user behaviour with tracking systems is a good way to prove that this could be a crisis issue. Set some key metrics like number of logins, conversions and feature usage statistics. Create benchmarks and compare releases to this. Monitor them automatically and get warnings.

Actually in a continuous delivery environment I might even expect an automated rollback if such a performance degrading feature would reach production.

Companies who do extensive A/B testing based on performance indicators like conversions might also think this is blocking and do a rollback when they see they are loosing money and or users.

It just all depends if you have enough statistics to make a good call.