Transition from all manual testing to automated testing

Question

Right now my company only does manual testing. We've talked about starting some automated testing for a while but it hasn't taken off at all. It seems like automated testing would be a good way to help our QA staff focus on the challenging problems and leave the routine fixes and tests to the automation system. Does anyone have recommendations on how I could:

1. Encourage the adoption of automated testing (in regards to management mostly)
2. Identify some current tests that would be easily replaced with automated testing
3. Choose a good automated testing software (we apparently have Automation Anywhere but I have no idea if this is a good product to start automated testing with)

Answer 1

I've worked a fair amount with automated and manual testing, and my perspective for your three questions is:

1. How much trouble do you have with regression? This is probably your biggest potential time/cost saver. Anything requiring long, boring regression tests is a good target - just be careful you don't over-sell. There's a large ramp-up cost that ensure that your starting phase will be less effective and more costly than normal operations. Once you've got a basic framework in place and some simple (preferably simple with good ROI - what that is depends on your application) tests running, you've got a guarantee that this feature is working for each release, without needing to do anything else.

2. I partly hit the current tests question above: your best bet for starters is something that's used often, has long or tedious or time-consuming regression testing, may break frequently, or is mission-critical. As a general rule, start with something easy that fits two or more of these criteria: if your application calculates tax, any transaction with any tax scenario is a good starting point. Your basic goal should (in my clearly very modest opinion!) be to hit the 80/20 rule: automate the 20% of your application that's got 80% of your usage. (Logon/logoff and session management are other good, easy targets).

3. Good automation software depends on your application and your needs. The key things you need to consider are whether or not you're dealing with a web application (there are many free or low cost tools available to work with web applications), a Windows application, a mobile app, Linux, Mac, etc. You need something that can look into the software you're testing at least enough to identify the various components you're interacting with and call their public methods. It's a LOT easier to maintain automation that call something like "SubmitButton.ClickButton()" than something that interacts based upon pixel location on the screen.

Good luck, and happy automating.

Answer 2

I recently left Microsoft where we had a lot of automation for a smaller company with just a bit of automation to help begin the process of automating a significant amount of our testing. Fortunately management here had already realized they needed to step up their automation, however even after acknowledging that they needed more automation, actually providing the resources necessary to get that automation up and running has been a bit more difficult.

Because my company had already had some automation successes, and also some automation failures, one of the things that I knew I would need to do is build confidence in our automation both among developers and management as well as other stakeholders. For this reason, I picked a few simple projects with big returns as the first few projects to tackle to show some noticeable increases in productivity and turn around time. The first 2 tools I created actually weren't even necessarily automation. The first was a tool to aid manual testing by inserting data into a log file that would make the data easier to work with when run through our product which parses that log and inserts it into a database. This tool was very simple, but allowed us to do way more in depth testing, increasing our confidence while also speeding up the process. The second was just modifying some of the existing automated tests and adding some tools to use a single mechanism for a) kicking off the automation (mstest) b) logging and reporting results and c) being run build over build. The third one was a tool for automated deployment validation that I outlined in this answer: Deployment Testing. In summary, when trying to convince management (and anyone else) that automation can speed up testing and improve your confidence level, it's nice to have some concrete examples, so a couple of quick tools like this that have already added value can be really useful in convincing management to allow you to do more.

The next pieces I plan on tackling are a UI automation framework (an abstraction on top of the current selenium/webdriver build) and then going a level deeper to test at the HTTP and database layers.

I am not familiar with Automation Anywhere, however I do know that the correct suite of automation tools depends largely on what you are testing. Are you only talking about UI automation, or are you talking about other additional automation? What sort of UI is it - web page, windows app, flash, silverlight, etc? What OS's/Browsers/environments will you need to run your automation in? More often than not, the success of your automation will have more to do with your implementation than it does on the underlying framework you are using. An example of a common mistake is using the existing framework out of the box with no abstraction on top of it, often using tools like recorders to record and playback test cases. While this will get you up and going quickly, it will also make maintenance and stability of your automation difficult if not impossible. Take the time to invest in building out a proper abstraction and it will make developing and maintaining your automated tests much simpler.

Answer 3

First, if you look in the upper right-hand corner of this web page, you will see a white box labelled "search". If you click on that box, type "automation", and hit Return, you will see links to many questions and answers about automation. I believe it will be worth your time to scrutinize what you see there.

This is a big subject, but if it were me, I would first decide whether my primary motivation is to save time or find more bugs.

If your primary motivation is to save time, you should start by automating some common, time-intensive setup tasks, for example, generating test data or automatically installing and configuring your product. These things are a good place to start because they are measurable, which may be important to your management. If you can report quantifiable success with this, it may be easier to justify additional kinds of automation. In my experience, automating setup tasks also requires less programming skill than automating tests.

If you have serious quality problems now, and you suspect that automated tests may help you find more bugs, you should identify some specific areas where automation may help you. There are many strategies for identifying tests to automate, e.g. areas that are particularly buggy, areas that are particularly time-consuming or error-prone to test, or areas with a combinatorial set of possible inputs but which have easily-predicted outputs.

Be forewarned that despite the claims of tool vendors and automated testing evangelists, automated testing is not a magic bullet. Moreover, like all software, automated tests require an up-front investment followed by ongoing maintenance. Approach automation as you would any large project: start small, be honest about your results, and be prepared to switch tactics if you are not achieving your goals.

Answer 4

1. Stress the long term savings of time as new features are added. You could potentially automate your regression and be able to spend time thoroughly testing new features and catching bugs that would potentially cost more money if they were not found. It is rare to see money saved up-front, which usually makes it tough to sell management on automated testing. Expect a challenge. ;)

2. If there is an API-only portion of your software, you can focus on testing that as proof of concept without having to deal with UI-based testing. This can be prototyped rapidly on low-cost tools like Watir (Ruby) and get you started quickly.

3. This is a definite rabbit-hole of opinions on testing tools. Pick the right one for the job. Without knowing more about your product, it'd be hard to give a good opinion. There are vendors that sell extremely costly systems and there are frameworks and APIs that are free. If cost is an issue,and it usually is the way you asked it, let that be your guide.

Answer 5

1. Encourage the adoption of automated testing (in regards to management mostly)

-- // --

1. Identify some current tests that would be easily replaced with automated testing

a) First of all, start from the basic CRUD pattern (http://en.wikipedia.org/wiki/Create,_read,_update_and_delete) operations.

I am confident, that your application has several items that could be CRUD.

For example, a tests to: 1. Create user

1. View user details

2. Update user details

3. And delete the user

And that would be the major first priority functionality.

If you have a very basic CRUD coverage for each entity in your application – than you got a good test coverage.

b) Don’t spend a time to make the automation test the same as the manual one. You will never done this.

c) Find the most boring tests and automate them. Use MSI (http://en.wikipedia.org/wiki/Windows_Installer) command lines to perform silent installation. Create DOS/Powershell/Jscript command lines that help you to deploy the application, copy files and prepare the test environment automatically.

d) Find out where you are wasting a lot of time and try to automate this action or a set of actions

1. Choose a good automated testing software (we apparently have Automation Anywhere but I have no idea if this is a good product to start automated testing with)

Please, keep in mind, that the good automation is the same as a good app development. Please consult with your senior developer about this. It’s easy to control the application if it has 200 lines of source code. It’s hard to control the app if it has 20 000 lines. Your tests could be bigger.

Answer 6

1. Download Hudson or preferably TeamCity Pro for free.
2. Point it to your source code and ask it to run the compile.
3. Write the simplest noddy test you can think of.
4. Ask the CI server to run the test.

The above will take a few hours after work one night, but you can't have a second test without a first.

Even a continuous compile nets your team benefits of better feedback when it doesn't compile (think I forgot to check in a file), so your quids in after step 2.

It's far easier to argue the case to management when you've got an automated test system up and running to show them.

Do or do not, there is no try.

Answer 7

I would like to take a crack at answering the last question regarding the most appropriate tool to use. Inevitably, most companies have already made investments in some form of automation already. These are mostly shell scripts, that run some custom application, and then parse the output. When you choose a tool, the tool will come with its own reporting infrastructure, that often assumes it is being introduced into a vacuum. It is important to choose a tool, that can easily coexist (i.e. report into) with your planning and reporting infrastructure. Automated tests can produce lots of data (not just pass/fail, but also more interesting stuff about the testing conditions), that are useful to store and analyze later. This is, what I would term the next stage of automation maturity. When selling to management, it is important to show the benefits of having long term access to this data.