15

I have been faced with a QA test case where refreshing a page multiple times almost instantly (6x, 7x, 8x) creates a bug. This is totally random and every time the number of refreshes require to reproduce changes.

So my question is that should this be even a test case?

Edited It is caused due to uninterrupted page load. Further it is suspected due to DOM elements not rendered fully

  • 3
    "This is totally random" how do you know it? – João Farias Aug 9 at 14:10
  • 3
    Are you waiting until the page is fully loaded in between refreshes? Or, are you interrupting the page load with a refresh? – Lee Jensen Aug 9 at 15:26
  • Yes Lee, the page load is interrupted. – husaynali Aug 11 at 0:54
  • 4
    You're going to fix the bug, right? You're not just looking for an excuse to not fix it? – user253751 Aug 11 at 6:36
  • 3
    Uninterrupted page load is what happens when a page load and the user does not do anything (does not interrupt the page load like click a link, press button, refresh etc.). Do you mean interrupted page load? – slebetman Aug 11 at 17:54
23

Yes, while rather crude and possibly not the most graceful of test cases, such 'abusive' test cases are an important tool in stepping off 'happy path' thinking and testing.

  • Effectively a test like this is a crude load/response test.

While in theory it shouldn't happen in the real world environment, the fact that a bug was discovered with such abusive input should be raising red flags for investigation even if ultimately no resources ever get devoted to fixing the issue. ["Known issue, won't fix" is totally a legit 'solution' to some problems. I guess...]

  • Why does this non-standard input/interaction cause an unexpected state/bug to be entered?
  • Does the above answer expose potential security risks [Such as the system grabbing incorrect, possibly sensitive, data for the wrong user?] or performance spirals? [Such as momentary excessive loads corrupting sessions for some users, who then all end up running multiple hard-refreshes to clear the issue, which in turn creates more server load and causes more user sessions to corrupt?*]

Tests like this go right along with things like navigating back and forth between two screens/states over and over again. The software isn't supposed to be able to fail on things like that, and the low level unit tests should catch it, but at the end of the day what something should do isn't nearly as important to QA as what it actually is doing.

"Something goes wrong when..." are important points to find in the software, and while we may prefer to find more graceful ways to discover such issues the truth is that at times the easiest way is to simply hit things 'at random' with a metaphorical hammer to see if anything breaks.

As a project matures and testing methodologies improve, then such a case may become defunct in favour of more targeted and graceful solutions.


It is also important to remember the dangers of thinking "the user will never do something like that in the real world..." because users are mythical creatures and will seemingly do everything in their power to do things no one on a dev team could ever imagine them doing...

  • Are they 'aggressively impatient', and hammering on the refresh button out of anger if something goes wrong?
  • Do they have a mobility issue or faulty hardware that results in a surge of unexpected or unintended clicks?
  • Does the 'rapid refresh' bug kick in if the server is under load, and the user is actually refreshing fairly casually once a minute or so trying to get your site to respond?

At the same time realities of resource management need to be considered. Test cases like this may not be well suited to being constant 'front and centre' operations that all get run day to day.

  • Maybe group them as a pool, then periodically run a subset of them against a given build rather than the entire pool.
  • Possibly useful as part of morale events: Pull the team together with their food and drinks of choice, and make a team game of 'who can break things in the weirdest way'.

* I have seen one client reduce their server requirements to nearly an 8th of what they had been using after resolving the software issue behind a problem like this, rather than continue to throw hardware money at it.

  • This is a great answer. I paused reading after the first section to upvote it, and it only got better from there. "at the end of the day what something should do isn't nearly as important to QA as what it actually is doing", "users...will seemingly do everything in their power to do things no one on a dev team could ever imagine them doing", and the point that tests like this are useful but don't necessarily need to be encoded in a test case/regularly run format... good stuff. – c32hedge Aug 12 at 17:09
5

Probably not. I like to challenge defects like this with a simple question: Would an actual user execute this behavior?

In most situation I would not expect users to refresh a regular page ten times in a row. Some pages might only update the page content on a refresh, forcing the users to refresh often to get the latest data. I think that is more a UX-issues instead.

So talk to the users. Do they refresh often?

  • Yes. It is testcase you should run every time.
  • No. Ignore the issue. Why swamp developers with defects that users never run into?

I do like to add a test like this to a set of heuristics. A list you sometimes mix during an explorary testing session.

update: In this case I do wonder if the bug really only happens on quick refreshes, or might it also happen the first time, but only 1 out of 10 times. Is it the load on the server? Is it database locking? What happens with 10 users at the same time.. Ohnoo my brain is on overdrive. Maybe you are on to a real issue. Do research it.

  • It is more of an uninterrupted refresh and bug is more of a UI side rendering. – husaynali Aug 11 at 0:57
  • 4
    @husaynali Are you 100% sure that it's a UI only bug? Are you sure the UI issues was not caused by the backend sending weird data or errors? Have you done investigations? – slebetman Aug 11 at 17:56
  • @slebetman Yes, I have confirmed on this that it is not due to any backend errors but due to unavailable DOM element which occurs if we do keep on clicking F5 multiple times until it occurs. – husaynali Aug 12 at 14:25
5

Probably, refresh is not a simple operation as it seems so that's a good system test.

Having said that I would give it low priority and execute it only once in a while since it would significantly slow down the rest of the tests

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.