How to reduce the time of our regression tests - interview question

Question

I recently attended an interview and they asked me:

How do you reduce the time of your regression suite?

I answered that we trigger only the high level jobs and rest of of our regression we do manually but the interviewer was not happy with my answer.

After that the interviewer asked about the architecture of my project. I don't understand why they asked me this after asking how we reduce the time the regression suite takes.

Answer 1

Starting from your first question:

Your answer did not give the interviewers any indication that you understand the technology challenges behind your automation suite. They were expecting answers that would cover possible technical solutions like:

• Changing from a full browser to a headless browser stack
• Optimizing code to reduce inefficiencies
• Removing static waits from code (e.g replacing Thread.Sleep with a dynamic wait)
• Running multiple tests concurrently instead of a single-threaded run of all tests
• using distributed systems to farm and manage your test runs, up to and including container farms over the cloud
• Minimizing the number of times lengthy setup and tear down routines have to run in a suite (while keeping your tests independent)
• Regular refactoring and deprecating obsolete or never-failing tests
• Adjusting scheduling to run a light suite of must-have tests on a per-build basis and longer test suites overnight or on some other schedule

Any of these answers would have been more satisfactory to the interviewers than the one you gave. Your answer would have left the interviewers with the impression that you did not know how to improve your test suite's performance or run time.

Which leads into the second question: the interviewers wanted to know more about the architecture of your regression system. You mentioned in a comment on one of the answers that they wanted a drawing. This is because the simplest way to illustrate any kind of architecture is with a diagram.

They were expecting you to know such things as:

• Whether your build server worked on a continuous build, continuous deploy, scheduled, or on demand basis
• Whether your regression suite used distributed on local resources
• How individual test allocation within the suite was handled (e.g. all tests run on one client system, all tests run on the server, a random selection of tests run on each server)
• What design patterns you used in building your tests, such as page object patterns, and whether you have it all in one big project or you separate your page objects so they can be called from multiple separate test suites

There are many other things you could have covered, including the advantages and disadvantages of the software and hardware architecture your tests are using and the reasons for the decisions. That kind of answer would have indicated to the interviewers that you understood what you were doing with your automation.

The answer you did give to their first question implied that you didn't understand what you were doing, and your inability to answer or understand their second question probably confirmed that impression.

This might seem harsh, but this is how interviews work. The interviewers are looking for reasons not to hire you. In this interview, you gave them reasons.

Answer 2

How do you reduce the time of your regression suite?

• This may depend on what technologies the regression suite is based on. There can be technology/tool specific optimizations, like using faster database drivers, test runners, language/interpreter optimizations, or, as @siutex mentioned use Explicit Waits instead of hardcoded time delays, which tend to wait more than needed. Let's not go this way further and stay on a high-level

• The first thing to mention would probably be to "scale" your regression test suite - divide the regression test suite between CPUs or separate machines. For instance, in Python both nose and pytest test runners support parallelization between multiple CPUs or hosts (pytest-xdist)

• Reduce the effect of a network - decrease the negative performance effect of the network traffic between the components of your system. For instance, let your test database be on the same server as your web application under test. Look into using containers (thinking docker)

• Find and remove repetitive/redundant test "set up" and "tear down" procedures. As a practical example that I stumble upon in our Python regression test suite - oftentimes things that were executed in every test method of a class (setUp and tearDown methods) did not need to be executed for every method and could be applied once per test class only (setUpClass and tearDownClass methods)

• And, of course, profile and measure to understand the bottlenecks - detect slowest parts of your test suite and figure out why are they slow and see if you can apply optimizations.

I would also note that you never-ever want your test suite optimizations to effect the quality and reliability of the tests. For instance, sqlite3 might be easier and faster than spinning up a full-fledged MySQL or PostgreSQL for test purposes, but you might not catch database or database-driver related issue in your regression test suite which may potentially lead to hair-pulling.

Answer 3

Question 2:

Architecture means:

-How does look your automated test project. Do you trigger test remote or locally, Do you use CI system, how many servers etc.

Question 1:

- You can use webdriver wait instead of Thread.Sleep

- You can run your test headless (phantomjs or firefox xvfb)