Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Right now my company only does manual testing. We've talked about starting some automated testing for a while but it hasn't taken off at all. It seems like automated testing would be a good way to help our QA staff focus on the challenging problems and leave the routine fixes and tests to the automation system. Does anyone have recommendations on how I could:

  1. Encourage the adoption of automated testing (in regards to management mostly)
  2. Identify some current tests that would be easily replaced with automated testing
  3. Choose a good automated testing software (we apparently have Automation Anywhere but I have no idea if this is a good product to start automated testing with)
share|improve this question
add comment

7 Answers

up vote 7 down vote accepted

I've worked a fair amount with automated and manual testing, and my perspective for your three questions is:

  1. How much trouble do you have with regression? This is probably your biggest potential time/cost saver. Anything requiring long, boring regression tests is a good target - just be careful you don't over-sell. There's a large ramp-up cost that ensure that your starting phase will be less effective and more costly than normal operations. Once you've got a basic framework in place and some simple (preferably simple with good ROI - what that is depends on your application) tests running, you've got a guarantee that this feature is working for each release, without needing to do anything else.

  2. I partly hit the current tests question above: your best bet for starters is something that's used often, has long or tedious or time-consuming regression testing, may break frequently, or is mission-critical. As a general rule, start with something easy that fits two or more of these criteria: if your application calculates tax, any transaction with any tax scenario is a good starting point. Your basic goal should (in my clearly very modest opinion!) be to hit the 80/20 rule: automate the 20% of your application that's got 80% of your usage. (Logon/logoff and session management are other good, easy targets).

  3. Good automation software depends on your application and your needs. The key things you need to consider are whether or not you're dealing with a web application (there are many free or low cost tools available to work with web applications), a Windows application, a mobile app, Linux, Mac, etc. You need something that can look into the software you're testing at least enough to identify the various components you're interacting with and call their public methods. It's a LOT easier to maintain automation that call something like "SubmitButton.ClickButton()" than something that interacts based upon pixel location on the screen.

Good luck, and happy automating.

share|improve this answer
    
I like the reference to the 80/20 rule. That is where we could save the most time testing simple things in our software with and it is the easiest to start/justify to management. –  Andrew Shipe Aug 26 '11 at 19:56
add comment

I would like to take a crack at answering the last question regarding the most appropriate tool to use. Inevitably, most companies have already made investments in some form of automation already. These are mostly shell scripts, that run some custom application, and then parse the output. When you choose a tool, the tool will come with its own reporting infrastructure, that often assumes it is being introduced into a vacuum. It is important to choose a tool, that can easily coexist (i.e. report into) with your planning and reporting infrastructure. Automated tests can produce lots of data (not just pass/fail, but also more interesting stuff about the testing conditions), that are useful to store and analyze later. This is, what I would term the next stage of automation maturity. When selling to management, it is important to show the benefits of having long term access to this data.

share|improve this answer
add comment

I recently left Microsoft where we had a lot of automation for a smaller company with just a bit of automation to help begin the process of automating a significant amount of our testing. Fortunately management here had already realized they needed to step up their automation, however even after acknowledging that they needed more automation, actually providing the resources necessary to get that automation up and running has been a bit more difficult.

Because my company had already had some automation successes, and also some automation failures, one of the things that I knew I would need to do is build confidence in our automation both among developers and management as well as other stakeholders. For this reason, I picked a few simple projects with big returns as the first few projects to tackle to show some noticeable increases in productivity and turn around time. The first 2 tools I created actually weren't even necessarily automation. The first was a tool to aid manual testing by inserting data into a log file that would make the data easier to work with when run through our product which parses that log and inserts it into a database. This tool was very simple, but allowed us to do way more in depth testing, increasing our confidence while also speeding up the process. The second was just modifying some of the existing automated tests and adding some tools to use a single mechanism for a) kicking off the automation (mstest) b) logging and reporting results and c) being run build over build. The third one was a tool for automated deployment validation that I outlined in this answer: Deployment Testing. In summary, when trying to convince management (and anyone else) that automation can speed up testing and improve your confidence level, it's nice to have some concrete examples, so a couple of quick tools like this that have already added value can be really useful in convincing management to allow you to do more.

The next pieces I plan on tackling are a UI automation framework (an abstraction on top of the current selenium/webdriver build) and then going a level deeper to test at the HTTP and database layers.

I am not familiar with Automation Anywhere, however I do know that the correct suite of automation tools depends largely on what you are testing. Are you only talking about UI automation, or are you talking about other additional automation? What sort of UI is it - web page, windows app, flash, silverlight, etc? What OS's/Browsers/environments will you need to run your automation in? More often than not, the success of your automation will have more to do with your implementation than it does on the underlying framework you are using. An example of a common mistake is using the existing framework out of the box with no abstraction on top of it, often using tools like recorders to record and playback test cases. While this will get you up and going quickly, it will also make maintenance and stability of your automation difficult if not impossible. Take the time to invest in building out a proper abstraction and it will make developing and maintaining your automated tests much simpler.

share|improve this answer
    
I like your examples of simple projects with big returns; this is a wise way to begin. I also agree that that success is more a function of your implementation than having the right framework. –  user246 Sep 3 '11 at 20:17
add comment
  1. Encourage the adoption of automated testing (in regards to management mostly)

-- // --

  1. Identify some current tests that would be easily replaced with automated testing

a) First of all, start from the basic CRUD pattern (http://en.wikipedia.org/wiki/Create,_read,_update_and_delete) operations.

I am confident, that your application has several items that could be CRUD.

For example, a tests to: 1. Create user

  1. View user details

  2. Update user details

  3. And delete the user

And that would be the major first priority functionality.

If you have a very basic CRUD coverage for each entity in your application – than you got a good test coverage.

b) Don’t spend a time to make the automation test the same as the manual one. You will never done this.

c) Find the most boring tests and automate them. Use MSI (http://en.wikipedia.org/wiki/Windows_Installer) command lines to perform silent installation. Create DOS/Powershell/Jscript command lines that help you to deploy the application, copy files and prepare the test environment automatically.

d) Find out where you are wasting a lot of time and try to automate this action or a set of actions

  1. Choose a good automated testing software (we apparently have Automation Anywhere but I have no idea if this is a good product to start automated testing with)

Please, keep in mind, that the good automation is the same as a good app development. Please consult with your senior developer about this. It’s easy to control the application if it has 200 lines of source code. It’s hard to control the app if it has 20 000 lines. Your tests could be bigger.

share|improve this answer
add comment

First, if you look in the upper right-hand corner of this web page, you will see a white box labelled "search". If you click on that box, type "automation", and hit Return, you will see links to many questions and answers about automation. I believe it will be worth your time to scrutinize what you see there.

This is a big subject, but if it were me, I would first decide whether my primary motivation is to save time or find more bugs.

If your primary motivation is to save time, you should start by automating some common, time-intensive setup tasks, for example, generating test data or automatically installing and configuring your product. These things are a good place to start because they are measurable, which may be important to your management. If you can report quantifiable success with this, it may be easier to justify additional kinds of automation. In my experience, automating setup tasks also requires less programming skill than automating tests.

If you have serious quality problems now, and you suspect that automated tests may help you find more bugs, you should identify some specific areas where automation may help you. There are many strategies for identifying tests to automate, e.g. areas that are particularly buggy, areas that are particularly time-consuming or error-prone to test, or areas with a combinatorial set of possible inputs but which have easily-predicted outputs.

Be forewarned that despite the claims of tool vendors and automated testing evangelists, automated testing is not a magic bullet. Moreover, like all software, automated tests require an up-front investment followed by ongoing maintenance. Approach automation as you would any large project: start small, be honest about your results, and be prepared to switch tactics if you are not achieving your goals.

share|improve this answer
    
+1 for the points on saving time. I would like to add that the use cases to be frequently tested in all the upcoming releases are good candidates for automation –  Aruna Aug 24 '11 at 22:46
3  
As an reformed automated testing evangelist, while I am a strong advocate for automated testing, I do NOT advocate doing it like most companies implement software testing: insert quality at the end by utilizing technique X in testing. Automated testing is not just another cop-out for doing GOOD software development. It is a tool, and a VERY good tool, but a tool that needs to be used properly, diligently, and wisely or it will be more of an impediment to development and less of a help. –  TristaanOgre Aug 25 '11 at 12:49
    
I like the point about automated testing not being a magic bullet. That's how it has been sold to some of the more experienced people I'm working with and based on the testing we do now we could spend years setting up automated tests that still wouldn't be as effective. The lure of automated testing to them is a complete replacement of what we do now and it was nice to be reminded of why that's a terrible idea. –  Andrew Shipe Aug 26 '11 at 19:57
add comment
  1. Download Hudson or preferably TeamCity Pro for free.
  2. Point it to your source code and ask it to run the compile.
  3. Write the simplest noddy test you can think of.
  4. Ask the CI server to run the test.

The above will take a few hours after work one night, but you can't have a second test without a first.

Even a continuous compile nets your team benefits of better feedback when it doesn't compile (think I forgot to check in a file), so your quids in after step 2.

It's far easier to argue the case to management when you've got an automated test system up and running to show them.

Do or do not, there is no try.

share|improve this answer
add comment
  1. Stress the long term savings of time as new features are added. You could potentially automate your regression and be able to spend time thoroughly testing new features and catching bugs that would potentially cost more money if they were not found. It is rare to see money saved up-front, which usually makes it tough to sell management on automated testing. Expect a challenge. ;)

  2. If there is an API-only portion of your software, you can focus on testing that as proof of concept without having to deal with UI-based testing. This can be prototyped rapidly on low-cost tools like Watir (Ruby) and get you started quickly.

  3. This is a definite rabbit-hole of opinions on testing tools. Pick the right one for the job. Without knowing more about your product, it'd be hard to give a good opinion. There are vendors that sell extremely costly systems and there are frameworks and APIs that are free. If cost is an issue,and it usually is the way you asked it, let that be your guide.

share|improve this answer
2  
Also, I forgot to mention that, rather than trying to automate everything from the start, start small and automate the simple stuff first. Get that working first and prove the cost benefits in time and effort saved. It may then be easier to improve your suite and/or get additional test help to continue the process at that point. –  Jim Munro Aug 29 '11 at 12:06
    
Big +1 for simple stuff first. If you try to automate the more difficult tests, when they fail and there's something wrong in your test (as opposed to what you're testing) the naysayers will be all up in your face. "A ha! We told you this convoluted testing thing was faulty! Allow us to ignore the 50 bug reports already submitted and focus on this one, complicated thing that failed!" And honestly, I'm not exaggerating... –  corsiKa Aug 30 '11 at 16:24
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.