The task is to measure changes in performance of a system during daytime execution.

Test system description

Tests are performed on java desktop application which is started with cmd command:

java -cp [AUT dependencies, .jars] [main app class]

Interaction with application GUI is done with library so test project is started like:

java -cp [AUT & testng dependencies/jars] org.testng.TestNG testng.xml

And application is launched in the code of tests:

new ClassReference("[main app class]").startApplication();

So test project is a part of application and the question is how much test project influence performance of AUT?

Test execution

Tests are executed every 2 minutes within a single run of testng command. In testng.xml file <test> blocks a repeated so that execution would not restart test (as well as AUT). But after ~40 minutes of execution testing fails because of wait timeout expiration. Here is a processor & RAM load graph:

enter image description here

And heap usage dynamics:

|       ||     Exec time [R# - Run number] [S/F/C - at run start/finish/crash]     |
| [MB]  || R#2 S | R#2 F || R#10 S | R#10 F || R#21 S | R#21 F || R#22 S | R#22 C ||
| Max   ||  455  |  455  ||   455  |   455  ||   455  |   455  ||   455  |   455  ||
| Total ||  286  |  374  ||   373  |   374  ||   454  |   454  ||   454  |   454  ||
| Free  ||  123  |  240  ||    86  |    94  ||    67  |    51  ||    70  |    50  ||
R#2 - 04:00
R#10 - 20:00
R#21 - 42:00
R#22 - 44:00


Started testing with minimal number of actions in AUT (just clicking menu buttons). After an hour of execution heap usage was:

  • maximum - 455
  • total ~430
  • free ~200

After 02h:00m:00-02h:20m of execution heap usage was:

  • maximum - 455
  • total - 455
  • free ~35-70

After 02h:32m test failed:

  • maximum - 455
  • total - 455
  • free - 53
  • I think there are two questions here. You talk about performance in the sense of execution times, but then you also mentioned an application crash. Are you asking about both?
    – user246
    Jul 8, 2016 at 11:07
  • @user246, I'd like to figure it out. If TestNG loads system greater and greater during execution then performance testing is not possible with TestNG, and even leads to test crash after some time. I've got an idea to run empty tests, and check whether they would crash. Jul 8, 2016 at 11:12
  • In what way does it crash? I see it tends to use more memory over time. Does it run out of heap? Or does it crash in the sense of something like a JVM segfault? Also, what Java version are you using?
    – user246
    Jul 8, 2016 at 12:43
  • @user246, it is not crash of the application it is test failing because of test timeout expiration Jul 8, 2016 at 12:56
  • What Java version are you using?
    – user246
    Jul 8, 2016 at 13:22

2 Answers 2


Testing with minimal number of actions in AUT (just clicking menu buttons) performed under UFT 12.52 did not show any slowing down in AUT after 2 hours and 30 minutes of execution, but after that time 2:30 - 3:00 application started to work slowly and finally stopped responding.

Heap analysis shows that does not occupy more than 2,2 MB.

And time of operations is of a same order (unit: seconds) under & and

So in my case accuracy is sufficient.


I don't know the answer, but I know how I would try to find out. I would try two experiments:

  1. Your test is making the AUT do a lot of some kind of work. Try to find a way to run that part of the AUT without using TestNG, e.g. by modifying the AUT to perform those operations for some number of iterations. Watch for whether it crashes, and whether it uses more memory over time.
  2. Replace your AUT without some trivial program that performs similar memory operations, and then run that with TestNG. Watch for whether it crashes, etc.

I would also try attaching a Java profiler such as YourKit to the AUT, or to the combined TestNG/AUT process. I would check GC times and examine a heap dump to look for container (list/set/map) data structures that accumulate from one run to the next.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.