How can a Software Tester use "Out of the Box" thinking approach to find more bugs?

Question

Well, I am a black-box tester and I don't think out of the box.

I just think about happy paths, thus I want to improve.

I always read that testers should have great critical, analytical skills and should think out of the box.

Well, what could be some real-life examples/scenario's where one could use this way of thinking?

For instance I am testing a web based EIS application.

Answer 1

Some suggestions in addition to the good ones already listed:

• Look for things the software shouldn't do. Make it do them.
• It's web-based, so what happens if it loses the network/internet partway through a multiple-screen operation?
• Look for words in the specifications/user stories like "should", "always", "never", and so forth. These indicate conditions involving rules. Now look at what happens at the edges of those rules. What happens just inside a limit? Exactly on the limit? Just outside it?
• With something like an EIS, you're dealing with a lot of complex variants that combine in rules. Set up the most absurd combinations of the rules you can think of, the things you doubt anyone would actually do. What happens?
• Play with personas. Pretend you're the CEO of Santa Claus, Inc. and you want to know whether to get extra chimney breakage insurance this year. Or pretend to be a disgruntled accountant wanting to sabotage the CFO's financial forecasting for some reason. Or pretend to be the CEO's 5 year old daughter who wants to see the pretty flashy things show up on the screen because Daddy forgot to log off. And so on...
• Be destructive. Kill the system mid-transaction. What happens? (I'm actually serious here. At one place I worked, we had to reconstruct data and clean out corrupted data because the client's users would simply kill a process intended to run permanently when they wanted to use that machine for something else.
• Learn Murphy's Laws of software. Then use them for good.
• Paste the entire text of Hamlet into memo fields and save. (Hat tip to QA Hates You for this one)
• Actually, read all of QA Hates You's dirty tricks, clean tricks and other posts. He has some lovely suggestions for devious testers to work with.
• If the system lets you do something, do it, whether you should do it or not.
• That includes running really massive data crawls.
• And SQL injection.

You should find with a bit of practice looking for the gaps in assumptions (including yours) becomes easier.

Answer 2

Welcome to SQA, Eddy.

I have no experience in testing your software, so I have no real-world scenarios that may suit your application. And scenarios I know are typical for software that I tested, so they might be useless for you anyway. However, I have a number of general tricks or techniques that opened my mind:

• Learn from existing bug reports. If your customers, other testers, and stakeholders are reporting bugs found on production, you may learn a lot about what can go wrong in your application. Do not just read them. Try to reproduce them. Involve yourself in verifying fixes for them, or helping developers to isolate root causes.

• Learn from checklists. Internet if full of checklists with examples of invalid input tests. Do not learn them by heart. Pick some of them and try on your application. Think of other possible invalid input cases.

• Learn about ins and outs of the app. On one side, try to understand how your application is used by your customers, what business scenarios stand behind their requirements. On the other side try to understand what is the architecture of your system and how data flow from an end-user through all modules of your system. This is to learn thinking about a user as part of a system, so you will be able to answer how much a user will be impacted by a failure in a particular module.

• Read good books. I highly recommend reading the chapter "Thinking Like a Tester" in "Lessons Learned in Software Testing". It will tell you how to explore an application, how to ask questions about your application, find logical inconsistencies in your application and requirements, etc.

Answer 3

There's a guy called Michael Hunter who goes by the name 'The Braidy Tester', among others. He worked (or maybe still works) at Microsoft as a tester. He's written a great guide to testing the unhappy path - it's called You Are Not Done Yet (pdf), and while its main focus is desktop apps, it's also more widely applicable. I excerpt just some of the section headings here:

Input Methods

You are not done testing yet unless...you have tested the following input methods:

(8 bullet points)

Alerts

You are not done testing yet unless…you have searched out every alert, warning, and error message and dialog box your application can display and checked the following:

(15 bullet points)

Text Entry Fields

You are not done testing yet unless...you have covered the following boundary conditions for every text entry field in your application.

(20 bullet points)

Platform

You are not done testing unless...you have considered which platforms to include in and which to omit from your test matrix.

(talks about desktop platforms, but the browser is definitely a consideration for web apps!)

Performance and Stress

You are not done testing unless...you understand the performance characteristics of your application and the manner in which your product deforms under stress.

(18 bullet points)

Documentation

You are not done testing unless...you have reviewed all documentation, a) to ensure that it is correct, and b) to help generate test cases.

(19 bullet points)

It's a really great resource. If you show it to a developer, they may well conclude it's impossible to ever write working software, which should be a nicely humbling experience.

Answer 4

There are a lot of great answers here - especially specific examples, so I'd like to add a note about becoming a more "out of the box" tester in general. If you are a tester, hopefully you already have that innate curiosity, a need to know why, and to push things. Even if you don't spend that much time on exploratory testing, I think it's important to develop the skill of being able to break things in unique ways, finding the most remote of corner cases. There are two ways to do this:

1. Learn From Others. Learn from lists like the ones provided in responses on this page. Use already existing documentation and regression steps (and make sure you update and maintain those resources so they remain useful). Research and apply both the broad (like boundary testing) and the specific (for example, there are a lot of examples of all the ways you can test a login or signup page).

2. Use Your Experience. Research will get you far. But one of the best things about this career is, the longer you do it, the better you will get in regards to this particular skill, as long as you remember to use what you've seen along the way. After I'm done with my initial run of trying to break something, I do a mental inventory of past bugs I've found that could be possible here.

Related Protip: Chances are, you don't have perfect memory, so make sure you are filing good bug reports, so you can easily search your past findings and apply them to current situations. I can't tell you how many times something seemed similar to a project I worked on two years prior, and doing a quick Jira search ended up giving me great ideas of what to test - and in a lot of cases, the same things end up being broken. I also like to keep track of the "really cool" bugs I find, and encourage my team to do the same. Occasionally reviewing these, and sharing them with other quality assurance members helps everyone grow their "out of the box" muscles.

Answer 5

One thing you can do is to start "breaking things", and test the application's ability to handle failure modes. For example, you can edit or delete cookies during a session.

To guide your failure mode testing, you could build a "Failure Mode Effect Analysis", which is essentially a spreadsheet that lists the various things that could fail - the probably and the severity if it fails.

Good luck

Answer 6

To teach someone to think outside of the box is well, impossible. The thing you have to start doing is think about things like the following:

• What would happen if I put this into the hands of a 2 year old
• What would happen if someone used the software with malicious intents
• What would happen if someone who knows absolutely nothing about the software used it

These should be your starting points. A good example for each:

• On a previous project, clicking items multiple times submitted them multiple times. No good.
• The project had admin rights and you could open IE with the project. This gave the ability to access anything (including cmd.exe) with admin rights, again, no good.
• The software was difficult to learn and used uncommon acronyms which allowed the user to make extremely 'fatal' mistakes. This increased risks within the software. While not a typical defect it is an area that could be improved.

Answer 7

The key question here is the allocation of time and effort. For anything other than a very tiny, trivial embedded controller, the things you COULD test go quickly into the billions and trillions if you start trying to enumerate them. Randomly thinking up cool and clever obscure things to test might be emotionally satisfying, but you may not be helping anyone by doing so. In fact, given your limited time budget, you might actually be hurting.

The better approach is:

1) How much time to I have available and how many tests can I perform in that time?

2) Given the extremely tiny percentage of "total coverage" involved, what are the most valuable things I can test with that time?

This second one is very difficult and will change depending on where you are in the development path. For example, early in the program you will want to identify particular tests which if failed eliminate large parts of the functionality of the product (a whole subsystem won't come to life at all) You certainly don't want to be fiddling around with some clever "out of the box" test in some obscure corner of the product if you have not yet verified that all the major subsystems are up and running.

If and when you get to a stage in the program at which all the predefined tests of major functions pass and you have extra time on your hands (very few projects ever ewch this stage) then you might to start to be creative. Even then, however, the focus should be on some careful thought about what matters to the customer and attack that in some sort of priority order.

The FMEA framework mentioned above is an excellent place to start. Although most FMEA material is hardware focused, you can certainly produce a sensible, simple framework for examining your software system as well. I usually recommend something like this:

A) list all inputs at a high, functinal level (get ocean water temperature) B) list all outputs (command rudder to turn, print weather report, sound fire alarm) C) list transformations ie: big blocks of function that turn inputs into outputs D) list underlying platform services your software is dependent on.

Then go throught this list and ask "what would happen to my customer if this item was totally missing" or (usually worse!) if the item was there, but was erroneous.

Go through that list and get some sense of how severly the customer would be hurt. Focus the little bit of extra time you might have on the top few most severe impact items on the list. For these items, do all the things that everyone else mentioned. Try to make it fail. Stress test it. Use yiur creativity..

Hope this helps,

David Hetherington

Answer 8

Think of the daft/cleaver things that users do then over do them, think of what happens in the real world, with real people:

1. Power cuts, signal loss, phone calls, door bells, meetings can all interrupt the filling in of a form - what happens if you half fill the form in before leaving for the night then try to finish in the morning?
2. My favourite for web tests that developers hate is log on as one user, start filling the form, in a separate window log in as a different user, run a query or call up a form, return to original window and click submit, ditto with a different browser. - People do sometimes do exactly this if they need to check something from the previous pages - what should the behavior be, is it specified?
3. Also elderly or experimental browsers are usually able to produce interesting results as are slow or shared web connections, firewalls, etc. Once had a web site demonstrated that looked grate on a Mac with 19" XGA display and a T1 connection - I made them try it on PC with VGA on a 9600 baud connection.
4. To make yourself really popular install a screen reader, turn off the monitor and try using the system with the keyboard only - the developers may curse you but the estimated 6 million plus blind or visually impaired adults in the USA alone will not.
5. Try turning the colour down on the monitor or changing the display settings to simulate colour blindness - does that make the forms unusable? (About 8 percent of males, but only 0.5 percent of females, are color blind to some degree or another).
6. Try not answering binary questions - even Male/Female may be a problem if someone is unwilling to answer or is in transition - it also may have been different at birth to what it is now my favorite answer to a questionnaire that asked Sex? was "as often as possible".

Answer 9

I believe that for finding good or more bugs you:

1. need to understand the functionality in-depth, in a way you should be aware of all the ins and outs of the application.
2. domain knowledge is also of utmost importance.
3. try attacking the application with incorrect data.
4. stress test the application.
5. run negative tests.

For more examples you can go through Finding Bugs.