6

This came up while talking to another developer. He recalled some product that was in development like 15 years ago (but didn't remember the name or details, or whether it went to production) that would:

  • For a given page/screen of an application, automatically identify all the interactive elements (text boxes, dropdowns, buttons, etc.)
  • For each type of element, have some concept of providing random input (clicking a button; providing different kinds of text input such as long strings, special characters, etc.; single or multi-selection in a list; etc.)
  • Randomly choose an interactive element on the page, apply random input, and repeat with another randomly selected element.

Questions:

  1. What would you call this testing technique? It's similar to fuzz testing or combinatorial testing but those don't seem to quite capture it.
  2. Are there any existing tools that can be used for this?

I'm not too concerned about targeting web vs. desktop applications--mostly just interested in knowing if something like this is out there. Open-source would be ideal for learning from the actual code regardless of whether we could directly use the tool on our own applications.

EDIT: Peter raised a good point that merits a clarification. By "providing random input", I really meant "randomly select from a well-defined list of 'interesting' inputs". For example, for a text field, a subset of this list might be:

  • Empty string
  • Extremely long string
  • All numbers
  • Mix of letters and numbers
  • Special characters
  • Entry that would match backing language's implementation of binary, octal, or hex notation (we once found a bug in our application where a "numeric" text field validated successfully with input like 0xaf38)
  • Unicode characters
  • Uppercase
  • The word "Null"
  • etc.

The idea of randomly choosing elements to apply such input to would be to catch odd interactions between input fields, but you could also have such a system that would just loop through each field and test it in isolation, as Cherree's comment touched on.

  • So pure speculation... were I to try something in this area, I think I'd come up with a finite list of "rules" to test for each interactive element and either for-each them or do a combinatorial/matrix approach. The fully random bit strikes me as a little too heavy-handed of a tree shaking approach . I don't think the results (signal vs noise) would be a good use of time in most teams. – Cherree Nov 2 '17 at 18:44
  • @Cherree you commented while I was adding my edit :) – c32hedge Nov 2 '17 at 19:00
  • surrounded by great minds in this place ;) – Cherree Nov 2 '17 at 19:02
4

Sounds to me like (smart) monkey testing. Definition from the ISTQB glossary (v3.1):

Testing by means of a random selection from a large range of inputs and by randomly pushing buttons, ignorant of how the product is being used.

Quite often fuzz/monkey/random/… testing refer to the same thing, but I like this distinction:

[…] Fuzz testing is more about using random data, Monkey testing is about random actions.

A nice example is gremlin.js, which can be used "[…] to check the robustness of web applications by unleashing a horde of undisciplined gremlins."

However, with the aid of AI/ML, you can make such a monkey (or gremlin) way smarter. For instance, ReTest (disclaimer: I'm a software engineer at ReTest) uses a genetic algorithm to optimize towards code coverage. We are currently also working on the integration of a neural network that is used to mimic human behavior based on existing tests.

If you want to get started with an OS solution such as gremlin.js, you can also adapt its sources (or open a pull request) to use more interesting inputs. Something we do at ReTest is to use values from the big list of naughty strings.

The cool thing with monkey testing is that you can easily set up an old machine and let it test your SUT 24/7. But as already pointed out by Peter Masiar, looking at the results may become expensive. How much work it will be and if it is worth the effort depends on your particular SUT and the monkey testing tool you choose.

  • Cool, thanks. These are some great references. – c32hedge Nov 2 '17 at 21:51
2

OP substantially changed the question. See edit at the end.

I would call it "mad monkeys from hell bashing keyboards at random" :-) and I doubt it will give you any valuable input.

In one of my previous position, someone decided it could be a good idea to unleash a group of interns to use our application (for mortgage loan servicing) with no training, just to see if they uncover some bugs. Like you suggest, but with no automation (cheaper to start).

So these interns started clicking around and reporting if they considered something is not working (according their very limited understanding of "working").

We found out that even they were working for cheap, and they are able to report significant number of "issues". So far so good.

We found that triage of the reported issues consumed significant time of more experienced people (so it was not free, but rather expensive), and no issues reported this way were of any real concern.

Exploratory testing is endeavor best performed by sapient humans, not "mad monkeys from hell".

This "experiment" was abandoned after just few days, and all issues reported by them just dropped, was total waste of time.


Edit: After your change, you describe standard testing for corner cases. You want to add interesting unicode strings (users will input them).

  • 1
    I've actually used your approach both with interns and new hires at a couple ecomm focused companies. While it usually doesn't shake out many business requirements misses it's can be VERY interesting for UAT style feedback and opinionated feedback about UX, design, and lack of information to complete workflows. I could definitely see where your mileage may vary depending on the focus of the software and the amount of pre-existing knowledge you expect your users to have. – Cherree Nov 2 '17 at 18:48
  • 1
    You have a way of finding holes in reasoning :) See my update to clarify the "random input" bit. – c32hedge Nov 2 '17 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.