3

How important is it for manual testers to know what technologies/frameworks were used to build an application under test?

To be specific: imagine there is a web application written in AngularJS v2 which is using Bootstrap v3 for styling. How would this information help someone who is testing the application?

2

Seemingly it does not help with Blackbox testing, but for other types of testing it could help a lot. Think of security-testing, load-testing and test-automation.

Knowing the technology stack could help or impact with testing things like:

  • Known defects in the stack, with info from the suppliers.
  • Common mistakes that can be made with the frameworks might need extra attention
  • Monitoring change-logs when you update the tech-stack. This could also trigger extra test work if for example the UI-framework had (a lot of) changes in important elements used.
  • Choose testing frameworks that work better with the stack, for example protractor with AngularJS instead of plain Selenium.

It is extra information that you can use to estimate the risks. From my perspective the more information you have the better you can act.

  • Hmmm, I feel that I do blackbox testing with my selenium automation - no internal access, just using the site as a user would. Whitebox UI testing feels more like looking behind the scenes at what's being logged and recorded and communicated with other systems. – Michael Durrant Aug 4 '17 at 0:47
5

There's two types of testing in the context of your question: black box testing and white box testing.

General issue

A black box tester doesn't know anything about the system. This allows the tester to uncover bugs that someone more familiar with the system might overlook. A white box tester has more familiarity with the system. Based on knowledge of the application's internals, that a white box tester may always perform task X before performing task Y, whereas someone without familiarity might go straight to performing task Y in a system where task X must be completed before task Y, or both must be completed before the next task.

Having both types of testers helps to uncover bugs where there are user interface issues, documentations issues, or missing validations.

Black box tester

Let's use an e-commerce website for an example. The order form indicates which fields are required but also allows the user to skip around and leave some of them empty to come back to later. A white box tester might just omit the required shipping information, and just enter billing information followed by submit. If they're then directed back to the missing fields for completion, then the design is likely fine. If not, and they land on a success page, they might not know that there's an issue.

White box tester

With a white box tester performing the same test, they'll notice something is wrong after landing on a confirmation page.

The black box tester used the system and everything they saw indicated a success. The white box tester, on the other hand, knows the internals of the backend and knows that, without the required shipping fields, the application should not be accepting the submission. They'll know something is wrong despite no explicit error.

Conclusion

This is why it makes sense to have a balance of both types. A black box tester may be less likely to test this scenario. Maybe they just assume people know they need to put their shipping information into the form before hitting submit, so the test case never occurs to them. A white box tester is more likely to test this case, but less likely to realize that there's an issue. When you have both doing quality assurance on your software, you're more likely to uncover the issue, either through coordination/communication with the two types of testers, or by an engineer seeing a backend failure triggered by the test that the white box tester may have never tested.

Anecdote

This happened frequently in the automated customer service speech recognition industry. People familiar with a system (a white box tester) may be mis-recognized, try again, succeed, and chalk it up to a misrecognition. Someone less familiar with the system (a black box tester) would be more likely to report an issue. Usually it was just a misrecognition, but other times, it helped to uncover issues. It sometimes turned out that the white box tester's misrecognition was actually caused by some bug in the system. If it weren't for the black box tester reporting the issue to be looked into, the bug might not be found.

Alternatively, a black box tester might complete the speech recognition transaction with no issues, whereas a white box tester might realize that the transaction should have failed because a particular piece of information critical to the transaction was never collected.

Again, this is why it can be beneficial to have both types of quality assurance on a team.

Edit: Apologies if this is confusing. I wrote this at around 3AM local time when I should have been working to meet a deadline.

2

To add to the other answers here, the one time when it helps to know 'the technology behind the system under test' is when you're using recording tools to record automation tests. For instance, I once attempted to use Ranorex on an old .NET WinForms application, which Ranorex handled badly. It just could not identify any elements, relying entirely on the objects physical location on the screen, and therefore creating unreliable flaky tests. Had the application been a recent version of WPF and we wouldn't have seen the same issues.

Always check your automation software is compatible with the technology behind an application prior to buying the licences - one size does not fit all.

1

My personal opinion is, it is desirable but not always necessary.

  1. What kind of manual testing is to be executed? If it is more of a lower level of manual testing, e.g. manual testing on a component level, it definitely helps to know the technology behind the system under test.
  2. When it comes to a higher level of manual testing, e.g. system testing or user experience testing. Knowing the technology behind it may backfire, as knowing the ins and outs of a system may hinder a tester's ability from thinking or behaving as a customer who does not know anything.
1

Knowing the technology can help you identify potentially interesting variables to explore with testing.

Knowing the technology can help you identify limitations and vulnerabilities of the technology. Knowing those things, you can write tests to explore how the system deals with those limitations and vulnerabilities.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.