How to explain the concept of test automation to a team that only knows manual testing?

Question

I have been working as a QA in our 9 developer team for about a year. Had lots of 'firsts' within the year between personal and professional life. I replaced a QA who only did manual testing and unfortunately, no test cases were left for me to study, and no requirements either. All in all, I had to lean on exploratory testing.

So I have started a QA process (for me and for the team as well), built the bug process as well (they didn't have this process in place), and started to create test cases according to the developers' stories/users' stories and the product backlog items that were recorded in TFS.

They hired me to eventually help them do the automation. And now, the time has come. I have built an automation library of classes using C# and Selenium Webdriver. I have designed my tests using Page Object Model. Then I presented the sample tests to my boss. My boss asked me:

"What is the concept? If this may take time for you to create, I guess we will just move on to manual testing. I don't really get the concept."

Though he said I can continue doing my scripting while I am on my downtime since I don't have anything to test. It took me about 3 months (within those months I was also doing manual testing and at the same time learning C# all over again since I haven't done programming in one decade). I told him this is very helpful with my regression/cross browser testing since we need to test it in Safari. But he still won't get it.

I have about 15 apps to test and counting. Since I have these apps to test one at a time, I need to execute automation testing on the side while manually testing the new features of the other app. He said, "I still don't get it". I felt disappointed and somewhat demotivated with the automation testing I should be doing but at the same time, I need to move on.

My question is how do I convince someone who doesn't have any idea of how automation tests work and how helpful this is to an organization?

Answer 1

Measure it.

Pick a test area, measure how long it takes to test manually, then measure how long it takes to test with automation and how long you spend maintaining your automation. (There are other things you could try to measure too, e.g. bugs found manually vs with automation, but they can be hard to quantify.)

If automation is better, you'll have numbers to prove it.

Answer 2

Seems that your boss (correctly) understands that to write automated test takes longer that perform the same test manually once, and is concerned that you will spend so much time automating you will not have time to do manual/exploratory testing. Tests are code too and needs to be maintained too, adapting them to the changes in the application.

Value in automated test is NOT in the testing the app first time around, but in cheap regression testing. Unlikely you will have resources to automate 100% of your testing. But you could automate 10-20% of the most common functionality, and keep manual testing the rest. And if you have resources, add more. Of course, some functionality has to be retested many times even during developing the first deployable version, focus automation on that.

Measure how much time you would spend to manually test the parts what you already automated. This is time saved. Time is money. For as many weeks as your application will be continued to be developed, your automated test will be saving this money.

Another benefit of having the automated tests is that humans can focus on more interesting manual exploratory testing. It is hard to hire people repeating the same boring manual test week after week. And such skull-numbing drudgery will not be high quality.

Other side of this coin is: if you are creating app for someone else (your company will NOT be paid to maintain it long-term), and your contract does not require a battery of automated tests, you can still deliver tests what you have (showing your professionalism and competence), but your boss will not like if you spend too much time on something customer does not value. Minimal automated tests to allow pass the product to the customer should be enough. Then, it is up to the customer who will be maintaining the code to decide if they want to make changes in the code and how to deal with regression testing - like expanding on your tests. All depends on what deliverable is specified in the contract.

Do not mention it to your boss, but there is another benefit to automation: you are learning new skill and making yourself more valuable.

Answer 3

Proving the ROI of automated testing is hard. Teams that do refactoring to keep code lean and mean do have a high demand for test automation.

Watch the first episode of Clean Code, it will give you and your boss insights about why projects slow down and how bad code can even kill companies. His solution, CleanCode! and clean code is well written code with automated test coverage. (Uncle Bob is a bit of weird guy, but take the time to get used to his act, i think its worth it.)

To be most effective as a test engineer you either focus on automated or manual testing, not both. I have tried and failed...as have others (some guy at Groupon admitted this at an event). In your case 9 developers and 15 applications, you will never keep up.

Instead I would focus on teaching developers how-to test automated. For them its just more code. :) The goal is to go faster on the long-run. To achieve this you need to be able to refactor the code to be understandable, maintainable and testable. To be able to refactor you need lots of automated tests. Most of these tests should be unit-tests and not just functional tests as you are focusing on.

Have a look to run Coding Dojo's to introduce developers to automated testing of their code:

• https://pragprog.com/book/ebdojo/the-coding-dojo-handbook
• https://www.pluralsight.com/courses/the-coding-dojo

Find a good balance between where you put your automated testing efforts, focusing only on Selenium tests will certainly be a waste of time. They are expensive to develop, slow to run and they will break often (high maintenance costs). Read this article about the Test Pyramid, don't just rely on end-to-end tests.

Answer 4

I have this feeling your boss is not into programming, so it is kind of hard to pass on an abstract concept.

One key argument you can pull off from your situation is:

• I replaced a QA who only did manual testing and unfortunately, no test cases were left for me to study, and no requirements as well. All in all I had to lean on to exploratory testing. When a tester leaves, all of his / her knowledge leaves with them. Have someone start fresh consumes a company money and time. If someone had left automated regression tests, knowledge transfer would have been easier and time and money would have been saved.

Occasionally, you can not pass onto ideas to others. I remember once I presented a tool to a manager and he brushed it off without even looking at the results. If you feel like you have tried everything but your boss still does not listen, there is only so much you can do.

Answer 5

Let me preface it that I am a huge fan of automatic testing, or rather BDD/FDD, which means that writing the tests is integral part of the development. So I feel your pain.

You better make sure that what you are intending to implement is really appropriate for this particular company. You say there are 15 apps to test, and additional ones are coming in. You don't tell us how much change there is in the old apps, how complex they are, how critical they are, and all that.

My question is how do I convince someone who doesn't have any idea of how automation tests work and how helpful this is to an organization?

A pragmatic/diplomatic approach:

Maybe you are more successful by implementing automated tests for the new apps only, for the time being, and leaving the old ones alone. Adding tests to existing apps is hard from a business point of view as it's just burned money (seemingly, I know, but that is what your boss will believe).

I would likely try to sit down with the developers of the new apps and try to steer them into BDD/FDD. This means that they do the bulk of the work for getting good tests. If all goes well, their overhead will not be drastically higher, and they will put you out of your misery of having to test manually. If you do it right, they will want to implement their apps in this way, in the future, because of many benefits, especially psychological ones (e.g., removing the fear factor from refactoring).

After everyone is used to this, you can at some point go back to the old apps and infuse some measure of automated tests there as well.

Answer 6

This is - unfortunately - not rare. (The tl;dr version is at the bottom of the answer)

I usually explain it this way: suppose you have an application that does one thing. You add a second feature. Now you have an application that does two things, but to test it, you have to test it at least four ways:

• Function 1 only
• Function 2 only
• Function 1 then function 2
• Function 2 then function 1

Now you add function 3. You need to repeat your four tests, then add at least these new tests:

• Function 3 only
• Function 3 then 1
• Function 3 then 2
• Function 1 then 3
• Function 1 then 2
• Function 1, 2, 3
• Function 1, 3, 2
• Function 2, 1, 3
• Function 2, 3, 1
• Function 3, 1, 2
• Function 3, 2, 1

And that doesn't include repeating functions (like 3, 3, 1, 2)

What happens is that the amount of time needed to do thorough manual regression increases exponentially with each new addition. The same thing happens with each new configuration flag: one on-off configuration flag requires 2 sets of tests. Two independent on-off flags require 4 sets of tests. Three such flags, 8 sets of tests. And so on.

If it takes an hour to test each feature per configuration flag, your manual test time quickly becomes unworkable.

Now, if it takes say, 2 hours per application of manual regression each release, and releases occur on average once a month per application, for 16 applications that's 32 hours of manual regression per month - almost a quarter of your working time - except that next month, it will take longer because there's more to test.

If you can automate the regression - let's suppose it takes you 40 hours to build a working framework that's reasonably DRY and then you can add new feature regression in 2 hours per feature. Suppose the tests take an hour to run, and maintenance is about an hour per week.

If you run that suite nightly, you're covering 32 hours manual work in an hour. Over a week of regression runs, you've more than recovered the time needed to create the test framework, and you're catching regression bugs within a day of them being introduced - which means there are fewer problems going to customers. That saves money and builds customer goodwill. It also gives you more time to spend on exploratory testing, which means new features will be better tested and more stable.

TL;DR - Once you have automated regression testing in place, you will find regression bugs sooner. The ROI is a mix of fewer bugs reaching the customer, greater customer goodwill towards the company, and better use of your testing time.

Answer 7

There are too many variables in order to answer your question properly. Anyone answering can only give advice and opinions. In order to give a proper answer, we would need to understand a few more facts:

• complexity of product
• software development life cycle
• stability and predictability of product changes

The traditional sense of automation as a tool to avoid lots of manual testing isn't a good selling feature. Manual and exploratory type testing will always find odd bugs that automation won't find. The value from automation finding odd regression in an edge case that popped up once in 6 months is nice, but usually not worth the effort to automate.

That being said, the best way i've seen automation prove useful to managers, executives and development teams is when automation is an internal product and part of the development tools and process in a continuous integration/continuous delivery type environment. Read this testops PDF to get a good picture of how it could work and how to explain to your boss.

• Firstly, Unit and integration tests
• should be automated, maintained and integrated into tools by the development teams.
• Secondly, Development team system tests
  • also integrated into build tools
  • QA/TestOps/Automation teams should be part of the development teams to create frameworks and tools for the development teams to use as a system testing framework to test functionality that can't be covered within unit and integration testing
  • working within the development team and closely tied to product managers, create a test plan to automatically test the main functionalities and key use cases

In the end, you get an artifact that has a proven quality and will not waste a manual testers time by installing, configuring and using a broken product.
Eventually a good goal is to create a rule that development teams own their end to end quality. This will push the maintenance of automated framework items like selenium page objects, api calls, etc earlier in the software development life cycle and encourage confidence in quality company wide.

Answer 8

Test automation saves time and money. Here is why:

• shorter development cycles
  • users can access new version earlier
  • company remains competitive because it can implement required features faster
• regression
  • tests are executed over and over (locally and by the CI). Already existing features are still working while the code base is growing. You can be sure that there are no "side-effects". Of course, it also depends on the test coverage (nevertheless, don't try to achieve 100% code coverage. It does not mean that there are no bugs.).
• bugs are found earlier
  • the earlier you can find a bug the cheaper is the fix. This paper gives a good overview about the software quality. It's older but imho still relevant. Look at the Figure 5-4. Cost Reductions of Detecting Bugs and Fixing Them Faster (Example Only) and you will see how important the test automation is.

Answer 9

You need to step back and consider the bigger picture.

The bigger picture is called DevOps. DevOps is not something one person does while everyone else carries on as before. DevOps is a holistic cultural change, a paradigm shift in the mind-set. So how do you win hearts and minds? You need to sell the benefits to all the stakeholders.

The business need to see the impact on the bottom line. The value of automated testing lies in regression testing. Developers tend to test only the thing they’ve implemented. That’s fine if the thing is a new thing but very often the thing is an old thing and any changes to it can have unforeseen consequences.

Automated regression testing means being sure that what gets released into production is as fit for purpose as what’s already in production. Quote Samsung as an example of what happens when a release goes wrong. Now, offer management a way to reduce testing and their kneejerk reaction is to assume they’ll need fewer people.

Not so. DevOps is Lean for IT. The object is to reduce lead times to free people to pursue added value. Both management and developers need to understand that automated testing does not mean people are out of a job but rather that they’re now free to pursue those blue sky projects they’ve been putting off, those product enhancing features that add value to both the business offering and the developers’ skillset: it’s boring testing v’s exciting development, hidden factory v’s adding value. The biggest obstacle you might face is if that manager you’re trying to sell this to sees manual testing team as the only reason his job exists. Buy that manager a copy of The Phoenix Project. Good luck.

Answer 10

The main problem is the justification of spending time automating test. And it also sounds like your boss is not into the details of the QA process. This is not unusual between technical people and managers. Your boss probably is more interested in the results of your testing, so you should talk to him in a way he understands.

Managers usually have a focus on 3 elements:

1. time: duration of the project
2. budget: the money it costs
3. fte: the people that are working on the project

The automation part could help a lot as it can reduce duration and the manpower needed to perform tests. Testing gives developers a tool to improve the quality. The quality of the product should be interesting stuff for you boss.

Your boss will probably ask how will automation reduce the duration and manpower. Be keen that you do not do additional tasks with automation, because it increase you scope and this is not what the boss wants to hear (as you increase duration/costs). Talk about the testing required by QA to guarantee some basic level of quality, if he does not do these test the project has a risk (angry customers, damages, additional work from fixes).

Regression testing One of the test most suitable for automation is the regression test. They should cover the important features of the product to minimize the risk of delivering a bad product. Those tests are repetitive as they are mostly done after each build and the test performed are mostly stable as they are only tweaked a bit. You can do it with unit test but also with integration/e2e tests in selenium. This way you can easily detect if existing functionality was broken. A boss will probably demand this tests when a customer complains a lot about previously working functionality breaking. If customers complain about this happening but the boss doesn't know, find out and inform him. Automated regression testing will help finding these issues before release. If there is no regression at all, regression test probably cost more then they will save. In this case manual testing is not required to, if they still are required automated them.

Example business case as justification for doing QA and automation in a project Create a business case for you automation idea, this will make clear you are on the right track. Take into account what is essential to test, those would be items with high risks attached to them. Show which test will cover these risks. Now you have list of tests that could be automated. Think of a realistic amount of times the tests will be performed, you can use this to show how much time automation would save your project.

Example reasons: You want to meet a basic level of quality within a project and there are some simple test you have to each day. You could do them manually or automated them. Automating test is only useful if you perform them multiple times, so there is a balance between creating the test and the time saved if you would always do them manually.

For example if you have to do some manual testing for 3 browsers; you have to spend 10 minutes per browser. In case you would automate it you would spend 1 hour automating it with a script, but with every run you would save 28 minutes. So if you have to do these test each day manual testing would cost you 30 minutes, in a week that would be 2,5 hours. When automated you would be busy for 1 hour on the first day, the rest of the week would cost you almost nothing adding up to ~1 hour. This saves you 1,5 hours in one week. If your boss bills 50 dollars per hour on work you would save the project 75 dollars in just one week.

QA: What saves even more money Testing is just a part of QA, mostly performed when the code is already written. So what can improve quality earlier on, before the code was written? A developer would probably say TDD/BDD. Someone from QA would probably say he should do more intakes/reviews on project deliverables. Both are good ideas.

Answer 11

Automate or not my tests ? is a big question. And it become a necessity in a new word that is in continues changements. Before starting with this question, in my opignon, we should start with why you want to automate your tests. We do it because the methodologies of software developement and testing are changed and goes specially to Agile. When you say Agile, you say sprints and cycle of delevery very quick. At this rithme the manual tests can't follow. And at the same occasion we can speak about increasing quality in general by all the types of tests. And automate tests can be by Unit tests at the bigining and then the integration one for scripts and then functional one. The two first one are used to be done by developpers and functional one by manual testers. So here we speaks about coulture and strategy of testing.
Now for manual testers we can give a simple test : "do a same scenarios ten of times per week." It is hard and annoying. and each time you pay for doning these tests specially if you ask to an external company to do that. By automating a scenraio. You can write it one and then you launch it 10 ten per day. So in term of budget you have a the biggest part is the setup of the tests and then the the execution is free. The maintenace is not so expensive if you use best practeces and design patterns and specially factorise codes in order to change one line of code and you correct all the tests. Until now we gain only on frequency. No it is not only that. In fact, the same tests can be executed with many data. so it will be driven by data. The same tests can be executed in many browsers, os ... So the 10 per day can be * (4 browsers) * (3 os) * (3 data for examlpe) , and these cobinations can't be done by human. Hope My answer can help you and sorry for errors.

Answer 12

The concept to explain is

Investment

---

Make a good enough case and the business will ask you for the automation

You use resources in the short term (writing automation instead of manual testing) which takes up more time, so that in the long term you can use that automation to help the business move faster, respond to market quicker, reduce bugs and downtime, etc.

The concept is simple. Whether or not it is actually appropriate for a given situation such as yours will depend. I suggest you ask yourself and the business these additional questions. Consider them the ones your boss would want you to ask. Use your own answers to ask the business these questions and explore the topic further:

• What is the current release frequency ?
• How much do automation engineers cost ?
• Does the company have or plan to use CI/CD ?
• Are automation engineers generally available ?
• How much is the product changing now (churn) ?
• Will the release frequency increase in the future ?
• How much will the product change going forward ?
• How much revenue is the product generating now?
• How long will the product be used for in the future (if known) ?
• How much revenue is the product expected to make in the future ?
• Can we get and afford 2 engineers to avoid an obvious bus problem ?
• Are multiple browsers used? OS ? Versions? Devices? Tablets ? Languages ?

You need to understand the business goals well and then present automation in terms of how it will help those business goals. You should present analysis of why automation is needed in the long term. Make sure you've run the figures by people beforehand. Automation is seductive (because it bring many advantages) but just be aware of creating a need for positions that require paying $100k per person. Don't hide that cost when making your case!

One more point (question):
Does the 9 developer team use and understand the value of automation such as their unit tests well?