6

My office doesn't currently have any kind of automated testing or anything that runs on our web site updates prior to release. We really need something. I'm not entirely sure what options are available out there, but we really could use something that allows us to script a series of tests and conditions under which those tests pass or fail. Ideally something open source or more affordable would be fantastic. What options do you use in your office and what works well? If given the opportunity to change how testing is done in your environment, would you change it?

Update

Currently we do everything manually. Of course this has dozens of problems with the biggest being time consuming and inconsistent. We update weekly or bi-weekly and really need to be able to press a few buttons or so to make sure everything's functioning as intended. The only thing I can imagine is something where I can define what the "working" status is on specific screens. Cross-browser would also be ideal, but if we can't do that it's still a step up to have some automated testing.

In regards to what kind of tests I'm not entirely sure how to define that. There are numerous sections of our web application that have pretty specific outcomes/functionality so I'm having trouble being broad with an answer there. I suppose the test would be successful if one of the following were true:

  • A specific dom element existed on the page
  • what's on the page visually matches a screenshot we took previously, thus it's a match

Update

A lot of the answers below don't discuss solutions, but indeed question the need to test in itself. There is a lot of value in these answers as you need to be sure that automated testing is the right solution to whatever the problem is you're trying to solve for. For my needs, I've decided to go with: http://www.telerik.com/automated-testing-tools.aspx

Improve this question
2
  • 3
    Could you perhaps edit this question a bit - as is, you're finishing with two very open-ended questions that, while interesting, aren't really applicable to your main question. I'd also like to see a little more about the kind of tests you want to run before making any suggestions about tools you and options you could use.
    – Kate Paulk
    Commented Nov 16, 2011 at 19:43
  • What sort of testing do you do at the moment - none, some ? How often do you need new releases, is your current app buggy or stable ? why do you think you need automated testing, what problem is this going to solve ?
    – Phil Kirkham
    Commented Nov 16, 2011 at 22:50

7 Answers 7

Reset to default
5

It is a good idea to explore automated testing, but it is difficult to create a system that will allow to you "press a few buttons or so to make sure everything's functioning as intended". Automation is rarely a complete or easy replacement for manual testing. Test automation requires a significant up-front investment and an ongoing maintenance cost. It also requires skills that a manual tester will not necessarily have.

There are two ways to use Selenium: as a capture/replay tool or as a programming API. Capture/replay tools never work very well (watch http://www.youtube.com/watch?feature=player_embedded&v=s_CUPs6xAWw for a good explanation) and I do not recommend using one. Selenium is a decent programming API for writing UI tests, but it requires programming skills.

If you are interested in automated testing, I recommend hiring a programmer for your test team. Instead of launching a grand project to replace all your manual testing with automation, I also recommend starting with a small pilot project, because you will learn things as you go.

At this moment, there are 143 questions in this forum that mention the word "automation". I suggest that you read at least some of them to familiarize yourself with some of the issues.

Improve this answer
5

Currently we do everything manually. Of course this has dozens of problems with the biggest being time consuming and inconsistent

I'm afraid this sentence in 6 months time will readily apply to any automation effort built on a weaker manual process

Currently we automate everything. Of course this has dozens of problems with the biggest being time consuming and inconsistent

Sometimes automation looks to be an answer, but it is almost always a better long term investment to look at your process and ask some questions about why you are in the position you are, when you need to consider automation just to keep up (This is not the goal of automating, and is a large warning sign for your team that automating won't achieve your goal)

I get the impression from what you are saying that you may be locked into a permanent regression test cycle to hit very short release deadlines; my advice would be to look at ways to improve this process and set a sustainable pace, if possible package up slightly larger releases, still get changes to test early, but release to production less regularly. I advise speaking to your developers about what they could be doing up front in terms of checking there changes as they will be the amongst the most qualified in your business to advise what level of automation is possible with your particular product.

I would recommend speaking with your developers and product managers to work the kinks out of the manual process and highlight the risk points at a high level before investigating specific tools, only then will you be able to asses the real cost/benefit of automating.

Improve this answer
5
  • You're right, we are currently updating frequently... we're in the process of padding our times we quote to our customers so we have more time to prevent issues. Our site is over 500k lines of code with 4 developers. There are a lot of moving parts that change just enough for them to need their own testing. I'm trying to make testing less time consuming and more accurate.
    – Ben
    Commented Nov 18, 2011 at 13:38
  • 1
    If you have the option it might be worth trying to get involved in the upfront analysis of the changes coming in, it might let you suggest grouping them together to allow focused testing rather than having to check all areas of the application for little changes each release.
    – AndyC
    Commented Nov 18, 2011 at 16:05
  • Agreed. Rather than jump to the "automate everything!" solution, you should spend some time analyzing your problem. Maybe there are other incremental ways you can improve the development lifecycle.
    – Stephen Gross
    Commented Nov 18, 2011 at 16:53
  • Good suggestions - Our application is large and uses a lot of libraries that are frequently changing. Whenever these libraries change we do need to run through and test the majority of our application. I'm looking at a possible solution now (Telerik) that would allow us to create/group tests which would be fantastic.
    – Ben
    Commented Nov 18, 2011 at 17:21
  • Obligatory quote from Alan: "You should automate 100% of the things that should be automated."
    – corsiKa ♦
    Commented Jul 19, 2013 at 20:03
3

If your biggest problems are time and inconsistencies then I would recommend spending some time to look at your existing processes (read: way of working) and see what can be improved for very little or no cost and yet provide a very visual benefit to your whole office.

Once you have an understanding on how you work you will be in a much better position to understand how tests can be introduced that cover regression of the weekly builds; And only then can you introduce tests that allow you to become more effective.

For now I wouldn't worry too much about test automation - it's only a means to an end - concentrate on what you know.

Improve this answer
2

First thing you need to do is to get a handle on what kind of tests you're running. Making a series of automated tests will do nothing if the automated tests are being written with poor quality.

I have worked with TMap (www.tmap.net/en), which is a sort of testing manifesto written by Sogeti, a large consulting company in The Netherlands. They have a series of white papers and checklists you can peruse, including determining whether you have the correct resources for testing and what kind of testing you should be doing.

Once you get a feeling that your manual testing is on the right track and you want to automate a series of repetitious actions, THEN you can start investigating what kind of automated testing tool would be best for you. You should ask yourself some key questions:

  1. What type of platform am I doing the majority of my testing on?
  2. What platforms are we potentially going to use in the future?
  3. What do I want to accomplish most from my automated tests?

Keep in mind that writing automated testing will be as time-consuming, if not more so, than the manual testing you're doing now. As was mentioned in a different answer on this thread, there is no such thing as a silver bullet for writing test scripts, especially when it comes to automated testing. Something as simple as adjusting the size of your screen can throw your tests into havoc if the automated program is looking for an x/y position on your screen to click.

As far as getting started with automated testing, Selenium is indeed a good freeware tool for testing web based applications. Keep in mind that in the beginning, you'll be doing all your testing in FireFox. If your main browser platform is IE, you will have to do spot checks, because the way some screens draw in FireFox are different than from those in IE (I should know - my last assignment called for a lot of manual testing, because we only supported IE, but the developers kept testing on FireFox or Chrome). If you need to test desktop applications, you can try AutoHotKey, which offers a lot of automated power without having to squint at the screen trying to figure out what the code means. If you have programmers who can help, you might be able to use Cucumber with Ruby - providing that you have people who can quickly debug your Ruby code.

Improve this answer
0

I use WinTask. It's not Open Source, but is affordable.

You might also want to look at Selenium.

Improve this answer
5
  • I downloaded Selenium and can't figure out how to use it. There's no icons or anything in Firefox for it.
    – Ben
    Commented Nov 16, 2011 at 23:29
  • Yup. Without knowing a lot more about what you are trying to do, what you are capable of doing, how much you could spend, etc - it's hard to make a recommendation. You might wish to check out some of these to see if they meet your needs: strazzere.blogspot.com/2010/04/…
    – Joe Strazzere
    Commented Nov 17, 2011 at 0:59
  • 3
    Selenium is not that hard to figure out if you have a dev background ( I have ) If you're struggling with Selenium then it suggests that automation is going to be tough unless you have people who know what they are doing. Beware of snake oil automation tool vendors...
    – Phil Kirkham
    Commented Nov 17, 2011 at 8:21
  • @Webnet Selenium is an API for developing automated tests for browser-delivered applications. Unless you're already a developer with knowledge of how to install and leverage an API, Selenium is going to be very confusing for you.
    – Stephen Gross
    Commented Nov 18, 2011 at 16:51
  • The testers on my team find WinTask far simpler than Selenium. Your mileage may vary...
    – Joe Strazzere
    Commented Nov 18, 2011 at 20:51
0

Presently at my company we are using the open source tool WATIN (cf. http://watin.org/ or WATIR if you prefer Ruby, http://watir.com/) along with Cruise Control .NET (cf. http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET or http://cruisecontrol.sourceforge.net/ if you prefer Java). WATIN allows you to test against the browser of your choice.

Improve this answer
-1

I use IMacros, A simple recording tool. We can use importing from csv. A simplest JavaScript is used in Macros.

Improve this answer
1
  • Hi Nikunj, could you go into more detail on how this could help the original poster solve their problem? As it stands, your answer isn't terribly helpful.
    – Kate Paulk
    Commented Jul 19, 2013 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.