Level of testing intrusion and false alarms

Question

I came across this idiom "level of testing intrusion". Its definition is:

• the greater the number of changes that are required to be made to the system under test specifically for automated testing, the higher the level of testing intrusion.

There is also a statement says:

• The higher levels of testing intrusion the higher risk for false alarms.

I do not understand the correlation between levels of testing intrusion and the higher chance for false alarms. Can anyone please use examples to explain it?

Answer 1

One example I could think of would be for application performance testing. If you are closely monitoring memory/CPU utilization, then any additional programs you have running that would not be typically running on your end users' systems (such as an automated test product) would adversely impact your test results.

Also if you have a situation where you have a lot of specialized software on your end users' systems, it is possible that installing specific versions of .NET, Java, or other required frameworks for your automated test tools could cause interoperability issues with your other software. Hence you could put a test system in a state where it is not functioning properly when in actuality, your end users' systems will not have any issues.

Answer 2

Consider SUT you have supports GUI interface (main interface customer/end user) which underneath is using private API to interface with SUT internals,.While doing Test Automation a situation may come when you need to test the SUT via its private API, say because of GUI element detection via automation tool is unstable. Because you are working on custom private API directly, skipping the standard user-facing GUI interface, this is case of high level of intrusion.You might see new issues with these higher level of intrusion which customer might not see ever.

Answer 3

If you are not directly interfacing with the same elements that a normal user will interface with in actual scenario then there is a probability that results may be different, this probability is directly proportional with the number and complexity of your extra test hooks.