The question of whether a performance bug is a blocker depends entirely on whether it stops a user from doing whatever they are trying to do with the software.
I have worked on software that took over 24 hours to run, during which the machine couldn't be used for anything else; that wasn't considered a bug at all, because the user workflow was to start that process before leaving Friday, and expect the result to be available on Monday. Anything that might cause the process to abort partway through was a blocker, because the users wouldn't want to wait another week, but if the runtime doubled to 48 hours that was fine, as the user's workflow wouldn't be impacted. I've known people who were working on academic research for which they'd start a program analyzing a huge dataset, and expect that it would take weeks -- that was fine, as they just needed a result by the end of the term.
Conversely, on a real-time video game, if the time it takes to render a frame of graphics exceeds .033 seconds that is likely to be visible to the players, so anything that impacts performance at all will be important (might be critical or blocker depending on how bad it gets; 30 fps already annoys some people, if the game drops to 10 fps even for short periods nobody is likely to play).
Someone in your organization should be in charge of knowing what the users need and what will be a problem for the users -- depending on how your team is organized, this might be a "producer", a "project owner", a "customer representative", a "team lead", or a "manager", but whatever the title, that person is the only one who can answer the question of whether your software is too slow when it runs for one second, or too slow if it runs for an hour, or too slow if it runs for a day. QA's task is to measure the software and let the team know how slow it can get on the expected workloads.
Once you have the information about how fast the software is and the needs and wants of customers, the product manager can make the call as to how important it is to improve performance. If the software is slower than the minimum that users need, that is likely a blocker. If it is slower than they want but meets their minimum need, that may be anything from minor to critical depending on how much they want it to be faster. If it is as fast as the users want, it doesn't matter if it could be made faster, it isn't worth spending time on improving performance (for example a game that runs its graphics at 400 frames per second; that is well beyond the point where anyone would notice if you improved it to 500 frames per second, so it isn't worth spending any engineering effort even for a dramatic improvement in speed).
Remember, every hour that an engineer works on improving performance is an hour that they could have used to fix some other bug -- the important thing is to determine whether the users want more performance more than they want other things fixed; if to a user the current speed is so slow that the program might as well have crashed (because the speed causes them to miss an important deadline, for example), that may be a blocker just like an unavoidable crash bug. But if the user is just mildly annoyed by the current speed, that may well be the same importance as fixing typos in user-visible text that will annoy users that like proper spelling and grammar.