2

I recently attended an interview and they asked me:

How do you reduce the time of your regression suite?

I answered that we trigger only the high level jobs and rest of of our regression we do manually but the interviewer was not happy with my answer.

After that the interviewer asked about the architecture of my project. I don't understand why they asked me this after asking how we reduce the time the regression suite takes.

  • 3
    They asked about architecture because your answer for first question was rather weak. You need to learn, and learn a lot. Attitude is NOT a valid replacement for experience. And we here will not write you answers to what is explained in college-level courses, like system architecture. – Peter M. - stands for Monica Jun 7 '17 at 13:51
  • 1
    @alecxe - in this case, because there is already an answer about the second question, I'm going to adjust your edit to leave the second question in place but modify to better focus on the implied problem. – Kate Paulk Jun 7 '17 at 14:00
  • 4
    See, both questions have different answers & I think you have to understand lots of basic concepts about the Selenium, Framework-design, Project Architecture: How you have designed your current project in your company & generally this type of questions are asked to check whether the candidates really worked or not on the project. – Bharat Mane Jun 7 '17 at 14:09
  • @BharatMane - I'm taking the perspective here that the interviewer was looking for evidence of deep understanding about automated regression and architecting regression suites. That's why I've kept this open: I think it's a valuable question that illustrates the sort of information interviewers look for and why they would ask specific follow-ups. – Kate Paulk Jun 9 '17 at 12:30
6

Starting from your first question:

Your answer did not give the interviewers any indication that you understand the technology challenges behind your automation suite. They were expecting answers that would cover possible technical solutions like:

  • Changing from a full browser to a headless browser stack
  • Optimizing code to reduce inefficiencies
  • Removing static waits from code (e.g replacing Thread.Sleep with a dynamic wait)
  • Running multiple tests concurrently instead of a single-threaded run of all tests
  • using distributed systems to farm and manage your test runs, up to and including container farms over the cloud
  • Minimizing the number of times lengthy setup and tear down routines have to run in a suite (while keeping your tests independent)
  • Regular refactoring and deprecating obsolete or never-failing tests
  • Adjusting scheduling to run a light suite of must-have tests on a per-build basis and longer test suites overnight or on some other schedule

Any of these answers would have been more satisfactory to the interviewers than the one you gave. Your answer would have left the interviewers with the impression that you did not know how to improve your test suite's performance or run time.

Which leads into the second question: the interviewers wanted to know more about the architecture of your regression system. You mentioned in a comment on one of the answers that they wanted a drawing. This is because the simplest way to illustrate any kind of architecture is with a diagram.

They were expecting you to know such things as:

  • Whether your build server worked on a continuous build, continuous deploy, scheduled, or on demand basis
  • Whether your regression suite used distributed on local resources
  • How individual test allocation within the suite was handled (e.g. all tests run on one client system, all tests run on the server, a random selection of tests run on each server)
  • What design patterns you used in building your tests, such as page object patterns, and whether you have it all in one big project or you separate your page objects so they can be called from multiple separate test suites

There are many other things you could have covered, including the advantages and disadvantages of the software and hardware architecture your tests are using and the reasons for the decisions. That kind of answer would have indicated to the interviewers that you understood what you were doing with your automation.

The answer you did give to their first question implied that you didn't understand what you were doing, and your inability to answer or understand their second question probably confirmed that impression.

This might seem harsh, but this is how interviews work. The interviewers are looking for reasons not to hire you. In this interview, you gave them reasons.

| improve this answer | |
  • 1
    Thanks for your answer, really helped me .. :) i have 1year experience in automation so way to go, i am very happy to know flaws in me and correct from my mistakes. thanks for your post – LaxmiMaddali Jun 7 '17 at 14:44
  • Very comprehensive answer, just an observation, a never-failing test may be a test of no value or a test that does test a stable feature. How do we tell the difference? – Yu Zhang Jun 8 '17 at 0:09
  • 1
    @YuZhang "Mutation testing" may help to reveal tests that never fail. – alecxe Jun 8 '17 at 1:57
4

How do you reduce the time of your regression suite?

  • This may depend on what technologies the regression suite is based on. There can be technology/tool specific optimizations, like using faster database drivers, test runners, language/interpreter optimizations, or, as @siutex mentioned use Explicit Waits instead of hardcoded time delays, which tend to wait more than needed. Let's not go this way further and stay on a high-level

  • The first thing to mention would probably be to "scale" your regression test suite - divide the regression test suite between CPUs or separate machines. For instance, in Python both nose and pytest test runners support parallelization between multiple CPUs or hosts (pytest-xdist)

  • Reduce the effect of a network - decrease the negative performance effect of the network traffic between the components of your system. For instance, let your test database be on the same server as your web application under test. Look into using containers (thinking docker)

  • Find and remove repetitive/redundant test "set up" and "tear down" procedures. As a practical example that I stumble upon in our Python regression test suite - oftentimes things that were executed in every test method of a class (setUp and tearDown methods) did not need to be executed for every method and could be applied once per test class only (setUpClass and tearDownClass methods)

  • And, of course, profile and measure to understand the bottlenecks - detect slowest parts of your test suite and figure out why are they slow and see if you can apply optimizations.

I would also note that you never-ever want your test suite optimizations to effect the quality and reliability of the tests. For instance, sqlite3 might be easier and faster than spinning up a full-fledged MySQL or PostgreSQL for test purposes, but you might not catch database or database-driver related issue in your regression test suite which may potentially lead to hair-pulling.

| improve this answer | |
-3

Question 2:

Architecture means:

-How does look your automated test project. Do you trigger test remote or locally, Do you use CI system, how many servers etc.

Question 1:

- You can use webdriver wait instead of Thread.Sleep

- You can run your test headless (phantomjs or firefox xvfb)
| improve this answer | |
  • Actually they asked me to draw the architechture ? – LaxmiMaddali Jun 7 '17 at 14:02
  • 3
    Explaining Project architecture does not mean to answers like use Thread.Sleep. They want detail design pattern -How you have actually implemented your project. So, Generally, you have to explain your framework, which design pattern you hv used? How you hv implemented, about reporting tools...etc Its all about your framework, not which waits are used.. – Bharat Mane Jun 7 '17 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.