2

There is a situation where our automation test suite is taking nearly 8 hours to execute 150 test cases. Because there are lengthy steps which cannot be reduced or remove.

And my manager ask me to reduce the execution time to 2 hours. Technically this is impossible even I carefully remove the waits and sleeps.

But how should I convince my manager on this matter?

5
  • Prome has some good suggestions below. You could also suggest removing 113 lower priority tests to reduce the execution time to 2 hours (keeping 37 high priority tests). Seems weird to me that a test manager is concerned about execution time more than test coverage. Can you run them overnight? Can you meet in the middle (with four hours)? Why is time so important here?
    – trashpanda
    Jul 13 at 10:20
  • At the moment we are running at night, But the manager still need to reduce the execution time.
    – ChathuD
    Jul 13 at 13:09
  • 1
    why is 150 tests taking 8 hrs ? whats the product which tests are causing the time . could you add more details ?
    – PDHide
    Jul 17 at 11:59
  • 1
    It's more than 3 min per test roughly. How much of that 3 mins is in waits/sleeps on average? Jul 19 at 5:20
  • Short answer: you can. you provide multiple approaches with pros and cons. As uusal, the factors are time, cost and quality, pick two if you're lucky. Jul 28 at 14:43
6

A strategic approach

If you cut a couple of hours and don't make other changes than next week more tests will be added... and the time will increase again.
So this is an excellent opportunity for a discussion on strategy.
Not to seek immediate answers (hence I'm not providing them), but to start the conversation for your given situation to see what will work for you.

Items worth discussing when focusing on value:

  • How to know the value of testing a particular device / browser / version? 8 hours is already a subset of the hundreds of hours you could take to test more devices and browsers and versions.
    Don't spend $500 fixing a device that one user, revenue $3, uses. Always talk % of users, not IE version for 0.0001% of users when choosing browser support.
  • How to choose unit vs. integration vs. UI testing? Time to talk test pyramid and agile testing quadrants
  • How to choose happy vs sad paths? There are infinite number of sad paths. How do you decide which ones to test?
  • How to set up parallel testing?
  • What are the current bugs being reported when changes are made?
    Creating tests for real pain points can often add the most value
  • What breaks the most when developers make changes, either for functionality or in the UI. Create automated tests for those breaks.
  • How to 'stub' UI dependencies. This can be the 'big' slow down in calling network services (APIs etc). Set up mock data or a local proxy server with canned responses. This can be dramatic.
  • Use production monitoring of dependent services availability without going through all the UI. Consider monitoring services and using Postman (for example) for checks for API endpoints. This way you'll still know if real production breaks but you can mock and stub dependencies for your tests. Test endpoint format to check breaking API changes.

However... this is what your manager themselves should be considering. If they are not... and you are... you can end up in a very awkward place. I've been there myself. In such a situation you should prepare a long list of questions that start to prompt your managers thinking to change. This change however has to come from them, You can only provide good leadership to them. Leadership is not always managment though we initially think it is.

In this light maybe try to gently start conversations about what is most important to test ('everything' is not a mature answer but bite your tongue if told this). People don't always behave logically (in the best interest of the company and customers that is) in 'management' situations where the style is authoritarian ('you must do x') as opposed to collegial 'How can we work together to meet this objective'.

Focus more on asking additional questions that will guide your work instead of focusing on statements such as 'technically cannot be reduced or removed', 'Technically is impossible'. When you use such absolutist terms to stress your argument you actually lose faith because programming in the real world is a pragmatic activity not an exercise in logic.
Save impossible for anti-gravity machines. At least for now ;)

Short version:

  • layout the factors affecting the choices to take (cost, coverage, people, etc).
  • provide various alternatives with pros and cons for each
  • let management pick the approach. Be sure you have covered the cons well.
  • the options are essentially always: more computing power, more people power and reduced ('better targeted') scope.

Shorter version.

The factors to juggle and balance are money, people and time.

1
  • thanks for the lengthy and more detailed answer.
    – ChathuD
    Jul 14 at 3:01
2

Management usually likes to have reasoning in terms of time/money, so I'd go with the following approach:

  • Take last X (e.g. 30) executions and get average execution time for the whole suite and each test scenario
  • Select top 10-25% scenarios which take the most time on average to execute
  • Create a nice presentation view (e.g. table) with current execution time, estimated execution time after refactoring and estimated developer/QA time to refactor test case + some buffer for maintenance

Now they have an easier way to comprehend how much would it 'cost' to implement said changes and how would they affect execution time of automation test suite.

2

3 min per test is actually huge execution time and can be reduced drastically as below:

As my rule of thumb: UI test should never go over 1 minute, if it goes , find a way to reduce it or break it up in a smaller test. I found 9 out of 10 times, there is an way to reduce execution time.

By following below steps , I was able to reduce protractor(selenium) test execution time by 60-70% :

  • Remove redundant test steps in between tests(this is the biggest time saver)

  • Never use UI to create test data by taking long navigation steps instead use it for verifications as UAT.

  • Never use UI for data combinatorial testing.( Same long UI steps with different sets of data aka data driven testing).Use API tests to cover low level data combinations testing.

  • Don't log off & reuse user session between tests with the same base state(Home Page) wherever possible.

  • Run tests in parallel(using shardTestFiles/maxInstances)

  • Disable angular/CSS animations, it will help in faster page loads $('body').allowAnimations(false);

  • Run tests in a headless browser(from version 59, chrome can be run headless natively)

  • Run tests as direct connection(directConnect: true)

  • Make short & simple to the point tests with one and only one assertion only.

  • remove all explicit waits & reduce implicit waits to a minimum and supplement it with conditional waits(using expected conditions)/fluent waits(with short polling intervals) carefully

Custom Fluent wait Code Example:

   `browser.manage().timeouts().implicitlyWait(0); 
browser.wait(function () { 
browser.sleep(1000); return element(by.id('id1')).isDisplayed() .then( function (isDisplayed) { return isDisplayed; }, function (error) { return false }); }, 20 * 1000);` 
  • Don't use UI tests to create/setup test data(use APIs to seed data), use UI tests to validate end-user business scenarios only.

  • Don't use shared(like with developers/business analysts/not even with manual testers)/or production replica(heavy) database .Use dedicated light QA automation database with carefully designed baseline data only.

  • Restore/refresh database before runs to keep it clean & light.

  • While designing test cases, eliminate unnecessary long UI steps and let the tests focus only on test validations with least page navigations.

1

If this were my dilemma, I would start here:

  1. Prioritize the tests (High/Medium/Low) then decide if you really need to run all
  2. Distribute the test execution (between multiple PC's or VM's)
  3. Avoid sleeps whenever possible. I would try to implement a reusable polling wait method that escapes and continues the test as soon as an element is found and interacted with
3
1

Since testing is a significant component of app development practices that defines the end performance of the product, in my opinion, not even a single test case should be skipped. However, if you want to cut some execution time, consider the code structure and functionalities that are well-defined and you can skip testing them for the QA goals.

Besides, such decisions should only be made after a thorough discussion with your team. You must explain all the likely consequences that may appear if skipped on any particular part of the test action plan.

I just wonder how awkward the situation is since your manager must understand the importance of QA and detailed testing for all the deliveries. Saving time, skipping tests, and rapid deliveries may temporarily help you satisfy your client requirement but eventually it can damage the reputation of software testing companies.

I think you must reconsider talking to your manager about completing the entire loop of tests to ensure refined and valid deliveries.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.