1

As a part of our quality metrics we need to calculate the time the application under test (a client - server desktop application) needs to do several common tasks like time to open a window, time to save a document, etc.

The common use cases include the maximum time allowed to complete the use case in a specified machine and we need to validate these times every sprint.

We use UFT (HP Unified Functional Testing (UFT)) as our testing tool but the times we get on a sample application have big variations. We have disabled everything on the computer, the CPU is near 0% and we have variations of 15% on the measures we do.

Have you done this kind of measurements in the past? And what tool did you use?

Thanks!

2

Since you're dealing with client-server architecture, you're also dealing with network latency, spin-up times, disk latency on the server and the client system, potential I/O bottlenecks (and the testing tool itself is not exactly lightweight in I/O terms), network traffic, and numerous other factors.

Your first priority should be to limit/minimize as many variables as possible. That means isolating your application's network into its own limited-access subnet so that the application client and server systems aren't competing with the rest of your organization network.

For this kind of testing, you also want the systems on dedicated hardware if at all possible. If you're running the client or server on a virtual machine that shares resources with other VMs on the host, you're going to see a wide variation in performance times.

If you can, the system(s) should be on as close to the expected production environment as possible. If the server is expected to be load-balanced, then your test should use load balancing. If there's a specific operating system targeted, use that OS.

If, once your environment is as clean as possible, you're still seeing large variations, you could see if it's possible for the developers to add logging to capture start/end time for your targeted tasks. You may find that the test tool is adding variability to the recorded times (this isn't uncommon: all of the big box test tools I've worked with are extremely I/O heavy and can introduce significant latency issues. The OS you're using is also a factor - different operating systems have different I/O profiles).

If you absolutely can't eliminate the variability, your last option is to run your tests multiple times and report averages and modal values.

  • @user246 - thanks! These are all factors I've had to deal with in the past. – Kate Paulk Apr 12 '16 at 14:04
  • Thanks Kate. We have coded a very simple app with 4 buttons with 4 different sleeps and with this extremely simple tool we still have variations up to 15%. All the other factors we already considered and the performance machines are isolated with dedicated hardware and network. We have open a ticket with HP. – Ignacio Soler Garcia Apr 12 '16 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.