Why do medium to large companies not adopt test automation?

Question

Is test automation simply not a right fit for some medium/large organisations, regardless of whether they "care" or not and assuming it can be applied there? What are the most common reasons for this?

My perspective:

In my experience, there still seems to be a fairly clear separation in medium to large organisations who care about test automation (if only the possibility) and those who don't. Those who care:

• Understand the benefit but also the resource/skill requirements
• Invest time and effort in upskilling manual test resources into automation where possible, and focus on the high-priority features/applications
• Understand what should be automated and what shouldn't, and invest their time and money accordingly

However, those who don't either:

• Don't have the development resource to fix all the bugs!
• Don't understand the value, or appreciate the cost-to-fix ratio
• Don't have dedicated test resource (per project, or as FTE)
• Don't care about test automation (even unit testing)

I'm beginning to see more and more in the "don't care about test automation" bracket. I struggle to believe that any organisation can be so myopic in their reasoning when they weigh up the costs/benefits in pretty-print, but would appreciate anyone's thoughts/experience on why this may be. And whether this is doomed to continue.

Software is an asset to the organisation and if key stakeholders don't appreciate high quality as a reflection on the organisation, they shouldn't be releasing any software in the first place.

I understand that small startups, say, care less about test automation and more about getting their product into the market, but there is still an alarmingly large number of organisations who have obvious bugs/issues and (seemingly) simply don't care. If something critical is broken, it will be fixed. If something less critical is broken but drives away X number of customers...if a tree falls in the woods.

Answer 1

I think the main reason why companies are not doing automated testing is because the "Return on Investment" is hard to prove. Also its hard to prove that you will have less defects in production due to these tests, since they will catch the obvious issues, but not the complex dependency issues that actual users might run into.

The automated testing mindset often comes bottom up, from testers or frustrated developers who know better. Adding automated testing to your process workflow will most certainly slow down the development cycle, unless you already do very rigid manual testing, but this is also something that a lot of companies just do not do.

The main idea of automated testing is that you keep going a steady pace in the long run. Without long manual testing sessions. But this is irrelevant when you never reach the long run because you are going to slow in the beginning. Most new software products start as a proof of concept that becomes a production system and when the product becomes a success it wasn't build with test automation in mind. Now making a legacy system testable is really hard and very time consuming.

So in the end it is a balance. What I noticed is that within medium/larger companies is that automated testing becomes more important after their first main product comes to a grinding halt because it is unmaintainable and any small change has a high risk of breaking something somewhere.

So how do you solve this and introduce automated testing? Do continuous deployments. When any code check-in goes to the production environment automatically it makes continuous automated testing very important and probably mandatory. Some pretty large companies are doing this very successful like Etsy, LinkedIn, Spotify and others.

Answer 2

There are many reasons why companies choose not to invest in test automation. Some of the ones I've encountered include:

• The state of the software makes test automation non-viable. This is particularly common where an organization's flagship software is built in older code and technologies with massively intertwined GUI and logic (classic ASP comes to mind here). If the flagship software is relatively stable and the development team is also relatively stable - and therefore familiar with the application code - it remains cheaper to leave it where it is and add features where necessary than it is to refactor into a cleaner, unit-testable form.
• The organization environment does not allow for more time to develop. It's very common, especially with developers who didn't start with a test-first philosophy, to find resistance at all levels to "taking extra time" to build in unit testing. Without unit testing, higher level automation is something of a losing game, because the only thing less efficient than testing business logic through automated interaction with the application GUI is performing those tests manually. The idea that if you don't spend the time to do it right you will have to spend the time to do it over doesn't make much headway, perhaps because these organizations tend to have the same need to release and upgrade often.
• The organization/developers subscribe to the "testing can be done by any warm body" fallacy. This is endemic to software development: the idea that manual testing can be done by anyone capable of using the software.
• The organization/developers believe that they test their work sufficiently. They might, too. My experience is that dedicated test resources tend to lag the development resources by about ten years. That's a lot of time to build up a resistance to the cost and effort required to build a high quality test automation framework.
• The organization prefers to use development resources to write code, not write test automation. This is another very common situation - and often goes along with resistance to allowing developers the time to design better solutions or to improve their skills.
• The organization does not 'see' ROI of intangibles. This is another common one: accountants tend to be more concerned with tangible income and expenses. Lost, dissatisfied or disengaged customers aren't easily measurable compared to new sales, so they tend to be given less importance. Where this happens, it's a sign that the decision-makers are accountants of the bean-counting variety. If you can't quantify it, they won't measure it, and if they can't measure it they won't allow it. If you've ever tried to convince an accountant like this that there is a positive ROI on bug that don't happen, you know the problem.
• It's more important to release the software. Again, very common. Small to medium, and medium to large organizations often have quite narrow margins. Not releasing new features means falling behind the competition, which in turn means potentially going out of business. They can't afford to invest the extra time needed to start a test automation effort.

Ultimately, many of these reasons come down to a classic no-win situation, where there's no way for the organization to afford the time and effort needed to introduce test automation, but the longer they continue as they are the more they need test automation. Typically, that cycle won't change until either the organizational culture changes or there is a sufficiently powerful and/or charismatic test automation evangelist hired in.

Answer 3

The flaw of your analysis is that you are thinking like an engineer. You are asserting that the cost-benefit analysis shows that automated software testing is superior. But you forgot to specifiy which goals you are optimizing. The automated testing is only superior when you are looking at things like the quality of the end product, or the efficiency of the developers who work on the software, or their job satisfaction.

The goal of a company is, however, to make profit. Period. When another goal like "make a quality product" is aligned with the goal "profit", great, the company makes a quality product. When it is not, there are people who will prefer the goal "profit" and ones who will prefer the goal "quality" - and the second ones generally don't manage large to medium companies. Maybe they try it, but the stakes are against them.

So the reason can be summed up in one sentence:

The CEO (or similarly high-up decision maker) does not believe that it will increase profits as much as investing in something else would.

Is that CEO mistaken? Both your question and the other answers assert he is. In reality, he sometimes is and sometimes isn't. Imagine a monopoly in a market with high entry costs. Investing money to increase product quality would be a mistake for them, unless they have some free cash in need of an investment idea. There are many other constellations - abundant in real life - where other investments have a higher priority.

In the end, we cannot tell you why a given company does not have test automation. The reasons are different on a case-by-case basis. But the more important thing to note is, regardless of how the engineers feel about it, that the question "is this right or wrong" also has to be answered on a case-by-case basis.

Answer 4

Some large organizations don't eschew functional tests because they "can't be bothered with them", or "don't know better", but because they see them as a strategic liability. I work at The Guardian, and we rely on neither automated nor manual regression tests.

Why?

• Automated tests have a long term cost. Anybody who has maintained a regression suite in a large-scale project will have suffered this: brittle end-to-end tests that require continuous maintenance, convoluted database setup, and awkward configuration. Automated tests are software like any other, and subject to their own buggy and unexpected behaviour.
• Automated tests slow down releases. I've worked in environments where each CI build kicks off a functional suite that will run for upwards of an hour. I've even seen automated regression suites that take twelve hours! This is impossibly long for any organization committed to continuous deployment.
• Continuous deployment mitigates the effects of bugs. Most organizations adopt CD under the premise of being able to react quickly to business needs, but one of its real promises is to minimize the impact of quality issues. If a change introduces an issue, our CD pipeline allows us to fix in production within a seven-minute cycle.
• Swimlaning, SOA and caching make the site tough to break. All of our article responses are held in CDN caches, so we can potentially break the entire website and still serve content around the world with no spike in latency. And because our application comprises isolated microservices, even a critical issue to - let's say commenting - cannot and will not ever break article or membership functionality, because these features are implemented on completely different applications running on completely different servers.
• Unit tests are typically preferable to functional tests. They are easier to write, faster to run, less brittle, isolate their concerns, have few bugs of their own and provide great documentation for the code itself.
• Feature switches and A/B testing allow us to try things out on a small audience. We can test a new feature on 10% of readers in a timezone that's currently low-demand. We can use feature switches to turn code on and off instantly without any need to redeploy.
• RUM provides us outsourced eyeballs. Real-User-Metrics allow us to capture error, performance and human interaction bugs on the fly with real users in real time.

If you're interested further in the Guardian's QA strategy, I would suggest taking a read of this blog post and watching a recent talk from one of my colleagues, Sally Goble, the Guardian's Head of Quality (deck here).