How does TestNG influence on project performance?

Question

The task is to measure changes in performance of a system during daytime execution.

Test system description

Tests are performed on java desktop application which is started with cmd command:

java -cp [AUT dependencies, .jars] [main app class]

Interaction with application GUI is done with jemmy library so test project is started like:

java -cp [AUT & testng dependencies/jars] org.testng.TestNG testng.xml

And application is launched in the code of tests:

new ClassReference("[main app class]").startApplication();

So test project is a part of application and the question is how much test project influence performance of AUT?

Test execution

Tests are executed every 2 minutes within a single run of testng command. In testng.xml file <test> blocks a repeated so that execution would not restart test (as well as AUT). But after ~40 minutes of execution testing fails because of wait timeout expiration. Here is a processor & RAM load graph:

And heap usage dynamics:

------------------------------------------------------------------------------------
|       ||     Exec time [R# - Run number] [S/F/C - at run start/finish/crash]     |
------------------------------------------------------------------------------------
| [MB]  || R#2 S | R#2 F || R#10 S | R#10 F || R#21 S | R#21 F || R#22 S | R#22 C ||
------------------------------------------------------------------------------------
| Max   ||  455  |  455  ||   455  |   455  ||   455  |   455  ||   455  |   455  ||
------------------------------------------------------------------------------------
| Total ||  286  |  374  ||   373  |   374  ||   454  |   454  ||   454  |   454  ||
------------------------------------------------------------------------------------
| Free  ||  123  |  240  ||    86  |    94  ||    67  |    51  ||    70  |    50  ||
------------------------------------------------------------------------------------
R#2 - 04:00
R#10 - 20:00
R#21 - 42:00
R#22 - 44:00

Update:

Started testing with minimal number of actions in AUT (just clicking menu buttons). After an hour of execution heap usage was:

• maximum - 455
• total ~430
• free ~200

After 02h:00m:00-02h:20m of execution heap usage was:

• maximum - 455
• total - 455
• free ~35-70

After 02h:32m test failed:

• maximum - 455
• total - 455
• free - 53

Answer 1

Testing with minimal number of actions in AUT (just clicking menu buttons) performed under UFT 12.52 did not show any slowing down in AUT after 2 hours and 30 minutes of execution, but after that time 2:30 - 3:00 application started to work slowly and finally stopped responding.

Heap analysis shows that testng does not occupy more than 2,2 MB.

And time of operations is of a same order (unit: seconds) under jemmy&testng and uft-12.52

So in my case accuracy is sufficient.

Answer 2

I don't know the answer, but I know how I would try to find out. I would try two experiments:

1. Your test is making the AUT do a lot of some kind of work. Try to find a way to run that part of the AUT without using TestNG, e.g. by modifying the AUT to perform those operations for some number of iterations. Watch for whether it crashes, and whether it uses more memory over time.
2. Replace your AUT without some trivial program that performs similar memory operations, and then run that with TestNG. Watch for whether it crashes, etc.

I would also try attaching a Java profiler such as YourKit to the AUT, or to the combined TestNG/AUT process. I would check GC times and examine a heap dump to look for container (list/set/map) data structures that accumulate from one run to the next.