Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions.

learn more… | top users | synonyms (2)

32
votes
11answers
1k views

Should programmers run automated tests written by QA?

Perhaps programmers have their own unit tests, perhaps as the result of doing Test-Driven Development. They run these tests very often; perhaps every few minutes, but at least a few times each day. ...
25
votes
4answers
2k views

Are Selenium functional tests reliable enough to be worthwhile?

I'm finding that regardless of how solid my tests scripts are, I still have tests that fail (falsely), even though they worked just fine before. I'm not talking about if markup changes (although that ...
21
votes
7answers
870 views

What is meant by “Automated tests don't find new bugs”?

As I understand it, this sentence means that automated tests don't find new bugs in existing code (though they might catch a new bug in new code that breaks old code). Is this correct? Or is this ...
17
votes
7answers
1k views

Has automated UI regression testing helped you in finding critical defects in system?

I have heard a lot of "abuse" about automated testing, especially UI bases automated tests. As UI is very fragile and prone to change (especially in agile shops). Personally my automated tests have ...
17
votes
13answers
13k views

Under what circumstances is Sikuli better for test automation than Selenium or Watir?

The Sikuli website says: Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). Under what circumstances is Sikuli a better technology ...
17
votes
14answers
9k views

What programming languages are good for novice testers to learn?

I'm looking for a programming language to recommend to a friend considering teaching herself software testing. I have a friend who is interested in learning software testing, and I believe strongly ...
16
votes
6answers
485 views

How to deal with automation naysayers?

In my team, there are a number of experienced QA people. I work mostly on developing tools to make the QA process easier for these guys, such as developing test automation. The uptake has been slow, ...
16
votes
13answers
10k views

Compare screenshots of rendered web pages

In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a ...
16
votes
7answers
762 views

Are your UI automated tests dependent or ordered?

While writing UI automated tests, I've come across situations where test dependency looked like a must. For example, say I have two tests, where the second is dependent on the success of the first. I ...
15
votes
7answers
2k views

Does automating your manual tests give you good automated tests?

I notice that the tag wiki for the "automated testing" tag contains the following sentence: "Commonly, test automation involves automating a manual process already in place that uses a formalized ...
14
votes
10answers
2k views

Can every test be done by automation?

Can every test be done by automation? Or is there anything that cannot be done by automation?
14
votes
8answers
2k views

How do you calculate your return on investment on automated tests?

I've been looking at calculating the ROI on automated test on a project that I've been working on. However, I'm a little stuck when it comes to determining a good method for this. Simply counting the ...
14
votes
3answers
865 views

What's the recommended practice for separating unit tests from integration tests for automated testing?

On a project I've been working on, we've been steadily increasing the code coverage through tests. Some of these tests were done as integration tests, and some as unit tests. The integration tests can ...
14
votes
5answers
477 views

Are Exhaustive “Brute Force” UI Tests Worth It?

Lately I've been asked to write some automated GUI tests of the following style: Open the application Open ALL dialog boxes In each dialog check for ... Are these types of tests worthwhile? What ...
14
votes
2answers
803 views

Good resources for building a QA team

Our corporation has, up until this point, largely lacked a 'quality mentality' of any kind. From no testers, they went to two testers - but 5 or 6 product teams still rely on a mix of developer and ...
13
votes
11answers
1k views

In house built testing tools

I've been having to deal with a lot of testing tools lately for testing tasks that should/must be automated. These include things like performance/load testing, webservice testing, vulnerability ...
12
votes
7answers
1k views

Is it realistic to try to hire someone who is a developer and a tester?

I've been trying to hire a tester for a few months without success. I really want to find someone who is both a tester and a developer, not because I want them to do two roles, but because the kind of ...
12
votes
5answers
2k views

What is the difference between “automated testing” and “automated regression testing”?

"Automated Testing" as a term seems to have a number of different uses. For some, it means built in unit tests that, when a new build is run, execute to make sure code refactors have not broken ...
12
votes
7answers
2k views

Transition from all manual testing to automated testing

Right now my company only does manual testing. We've talked about starting some automated testing for a while but it hasn't taken off at all. It seems like automated testing would be a good way to ...
12
votes
7answers
556 views

Should QA be able to code tests?

Having a development background, I am convinced that record and replay testing tools have their drawbacks, especially by making tests brittle. See Automated Testing != Record-Playback Tool for a good ...
12
votes
6answers
778 views

In agile (Scrum) practices, where do I start with making Q&A (testing) more efficient if the team is transitioning from waterfall to agile?

My team has recently transitioned from the traditional waterfall method to practicing Scrum. Being the QA lead, really the only tester on a small team, how can I get the process more aligned with ...
11
votes
9answers
9k views

Whether to choose Selenium over Watir or Lightweight Test Automation when testing web applications created using c sharp?

I am currently researching testing frameworks to test c sharp developed web applications. I have been leaning more towards the use of Selenium over Watir or Lightweight Test Automation but hesitant ...
11
votes
6answers
1k views

How do I convince someone who's bought the Record-Playback line that object oriented framework is essential?

Here's the basic setup - over a number of years the testing team at my workplace has developed a sophisticated (probably rather more complex than strictly necessary) object-oriented, data-driven ...
11
votes
4answers
202 views

If I didn't use TDD but want to transition to agile, should I go back and create those unit tests?

In an transitioning agile environment, if there was no Test Driven Development (no unit tests created), should you create unit tests to catch up on technical debt? And to what extent? Should they ...
11
votes
9answers
758 views

Performance testing - a way in?

We have an a browser based application that is built on the Microsoft Stack, specifically .NET 3.5. It also uses Ajax, iframes, jQuery etc. At the moment we are just guessing at the concurrency ...
11
votes
4answers
321 views

Testing interactions with external systems

I'm trying to find a way to test the interactions of our platform with external servers, like Twitter, for example. On our online photo sharing application, we have, for example features which let ...
11
votes
5answers
339 views

How often do you run long automated user acceptance tests?

Let's say you have product or project that is coming near to a next release and that has previous versions released a while ago and that has a big suite of automated user acceptance tests that just ...
11
votes
12answers
3k views

Tools for testing iPhone apps

I was wondering if there any automation tools out there that can help with the process of testing an iPhone application on the device itself and not on a simulator. Ideally, it would have simulating ...
10
votes
9answers
839 views

Can someone be a good automated tester and be terrible programmer?

I'm currently working as an automated tester. Mostly I do GUI testing in SilkTest, which I like and think I'm fairly good at. Recently, I've been assigned a COM-testing project for our application's ...
10
votes
5answers
357 views

What Are Some Examples of Why Explicit Sleep Statements Are Bad

I've read various places that explicit sleep statements are a bad practice in automated tests, even if I've used them occasionally. They can be annoying (creating unnecessary waiting in tests) but I ...
10
votes
9answers
2k views

What is a good website to hack around using Selenium?

I want to learn Selenium, but I don't know what to code. Sure, I can throw Google some queries, I can fill out some forms, press buttons, find divs and links, etc. But I think this stuff is pretty ...
10
votes
8answers
278 views

Testing of automated scripts

How much testing of your automated scripts do people generally do? Is it best to treat the automated test suite as a separate project or part of the Agile development cycle of the product your are ...
10
votes
5answers
222 views

What is best practice for test failures that have a low priority fix?

If I have a test that fails due to a known bug, but that bug is deemed lower priority to fix than other work, what should be done with the test? I can think of 2 options: Leave the test in the test ...
10
votes
3answers
2k views

What are some tools for fuzz testing?

Fuzz Testing as defined by Wikipedia is: a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer ...
10
votes
2answers
619 views

Test Automation for SAP

I'm evaluating a number of test automation framworks to implement on a SAP project. Right now I'm looking at Fitnesse, which we believe we can write some fixtures for, to call SAP Green Hat, which ...
10
votes
3answers
963 views

Is jQuery faster than CSS/Xpath selectors for IE 8.0 in Selenium1?

I am planning to run my existing Selenium 1.x tests in IE 8.0. After checking few blogs and reading this, I tried to run my tests using jQuery locators with the method mentioned here. My tests uses ...
9
votes
13answers
698 views

Should automated tests be efficient?

I've written and edited a bunch of automated GUI tests. In some cases, the tests are inefficient (eg to select an item from a list, instead of a clever search, a dialog is opened and starting fro the ...
9
votes
11answers
3k views

What are good guidelines to determine when to automate a test?

When it comes to test automation, I use a very arbitrary method based on experience that 80% of all tests should be automated. This has worked well in practice, but is based more on "voodo" than ...
9
votes
7answers
2k views

Should I try the evaluation version of HP Quality Center?

I'm trying to determine if I should spend some time working with a trial version of HP Quality Center. For a number of reasons, my time is very valuable to me right now both at work and in my ...
9
votes
10answers
18k views

Best free (or very cheap) automation tool for Windows GUI testing?

I used to use vTask Studio, but it seems they've abandoned it. No updates in almost a year.
9
votes
4answers
544 views

How can I report coverage on an n-dimensional test matrix?

I have a test matrix for my product's installation, which is n-dimensional, where n is about 11 or 12 at this point. For instance, the OS is one dimension, and the DB is another, and the Reporting ...
9
votes
4answers
413 views

Should unit tests be executed as part of a build?

I worked at a company where unit tests ran as part of the build. The build failed if the unit tests failed. What are the benefits and disadvantages of that practice?
9
votes
2answers
675 views

Layered Architecture of Test Automation Software

I was introduced to a 5-layer-model for GUI test automation frameworks, that was considered common knowledge. But I would be interested in a source describing this model. The layers are the following: ...
9
votes
4answers
304 views

Test Automation for different Locale?

I have begun with automation for a product which is in one language and soon to be ported in another. The only good thing is - Test Automation is just begun. So there are two things which worry me - ...
9
votes
3answers
173 views

Testing Unfamiliar Software

One thing I've picked up about testing software is to think about the software from the user perspective, and to sometimes "use" the software the way a "normal" user would. However, what if the ...
9
votes
4answers
992 views

Load testing tools vs. making your own

I've never needed to work with commercial load running tools before, as our test infrastructure is pretty immature here and I've used internal tools in the past. We're working on a .NET technology ...
9
votes
1answer
158 views

How do I tune the probabilities in a Markov chain based load test to produce specific hit rates?

I am writing a load test for a web application. I have a few month's worth of access logs that tell me relative hit rates of various URLs. However, the access logs do not contain enough information ...
9
votes
2answers
506 views

Maintaining automation framework for different browsers like IE and FF

How to maintain automation framework for different browsers like IE and FF? In my current set up, I maintain two different configuration properties files, one with xpath locators that FF supports ...
8
votes
8answers
377 views

Bugs keep reappearing in our GUI software during development. How should this be addressed?

We have a software product has a lot of usability features, and we're constantly changing it. We have issues with things like persisting scroll position which seem to get fixed, but then are broken ...
8
votes
4answers
296 views

Good strategy for automated regression where initial state can't be guaranteed or predicted

The situation I need to start building automated regression for a very large web application with thousands of users. It is impossible to start from a known state, or even to be sure that data ...