Tag Info

Hot answers tagged

25

My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs. But what they do do is two key things: They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs. They give you ...


23

You can try building your Selenium tests using the Page Object pattern, more info on: http://code.google.com/p/selenium/wiki/PageObjects


16

Regression testing is specifically looking for bugs in functionality that was previously working and has "regressed" to a non-working state. I've never heard "retesting" used except in a common-sense way: Retesting is simply testing something again after it has been previously tested, and is a more general term since retesting doesn't say anything about ...


14

Any ideas for reducing test time for GUI tests? One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are. I once joined an organization that ran automated overnight tests ...


14

> The integration tests can sometimes take a long time, > thus discouraging users from running the entire test > suite prior to checking in For the checkin-runs you can mark the long-running tests with their own category and tell the test-runner to exclude those long-runners You may also look at Is there a way to separate long running (e.g. ...


12

The truth is for a lot of people, there isn't a difference. A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly ...


12

The crucial issue IMO is to acknowledge that your pages will need to be adjusted in order for the tests. To pick up on the Selenium issue, instead of using XPath to structure your selectors, add ids to the pages and use those in the tests. I've also found the Page Object pattern to give good results: Within your web app's UI there are areas that your ...


12

You need to understand what are the core issues with these repeated breakages before you decide how to fix the problem. Relying on (more, improved, better, faster, etc) testing may not be the most effective solution here. It might not even help at all. Do you have code merge issues? Are fixes becoming undone because of that? Do you have new bugs ...


11

There are a number of key things that I have learnt are key to have easy to maintain, reliable automated tests. Map your controls in a single place Your controls should be mapped into a single location, so that if a control changes, you only need to change it in a single location. Having it in a single location allows you to make a one line code change if ...


11

Welcome to SQA, Rosa. I am not sure your question makes sense; QA is an abbreviation for "Quality Assurance", and often is used as a synonym for testing. ISTQB is a type of certification for testers. See also Do ISTQB/ISEB Testing Certificates prove someone can test? and http://en.wikipedia.org/wiki/Caps_lock.


10

To supplement glowcoder's answer, mainly with more examples, automated regression tests are a subset of automated testing, but there are other kinds. Some examples of non-regression automated tests are: Any automated test that will be thrown away and not used to test for regressions is not a regression test, including unit tests that are not used later ...


9

I've been in this situation before. We were running real-time tests for similar reasons. I wouldn't optimize prematurely, but if you see a looming problem I also wouldn't wait until it's already an issue. Some things we did: Parallelized testing across many machines. Make it easy to select a subset of your tests, so you can break one test run into ...


9

You're asking a unicorn question. Why? Because you've skipped over the point of all that work. It doesn't matter what proportion of different types of testing you do, it doesn't matter what your organisation is, and it doesn't matter what technologies you use, if you don't know what information your stakeholders want to discover from your testing. That's ...


8

It's not exactly clear what question you are asking, but let me take a stab. I would deal with it by creating a bug report. In it, I would mention what you are seeing in Commit A, and Commit B. I'd mention that the combination of the two pushes performance past the prescribed limits. From a QA point of view, it's not important at all in which Commit the ...


8

Extending and maintaining automated tests costs time, but if your team are slaves to updating tests you are doing it wrong. :) One of the biggest reasons to create automated test coverage is to create a fast feedback cycle loop. Letting the test-team fix the tests afterwards slows down this process and will always lag behind. There is a high risk the tests ...


7

Ideally there should be a regression or "hardening" sprint before the project ends, and in the dream world, a hardening sprint every few cycles. I'd suggest that automated tests of the steel thread of the sprint be developed as the sprint progresses, and any bug fixes get an automated test as well. This way, you build your automated regression as the ...


7

Yes, however this is one of those circumstances when it depends, an the final answer on "how efficient" will be different for everyone. Why it matters If you use test automation for a smoke or BTV test, then you want to be able to execute broad coverage quickly If you need to debug or re-run a test to see if a failure occurrs you don't want to be sitting ...


7

You should not edit the tests to pass. There are still defects in the product, and running the tests and having them fail on those points continues to provide data that the issues are not fixed. Since the development team has accepted the bugs and scheduled (although not solidly) them to be addressed, modifying the tests to no longer report the defects ...


6

I've worked as a solo tester on a small agile team as well, and I've found the two things that help most with regression testing are automation and risk-based test prioritisation. Having an automated test suite with good coverage is definitely very helpful. However, I wouldn't recommend relying entirely on automated test for regression, as there are some ...


6

I repeatedly apply three principles: Hide incidental details. Any detail that isn't directly related to the purpose of your test belongs somewhere other than in your test. Hide it somewhere, such as in a variable or in a method, and give it a good name. Name every important idea. Why does the test code log in in as "F.D.Gumby" rather than as some other ...


6

The previous posters are correct - automation can never find new bugs, that can only be done by sapient testing - however automation will find regression faults, and often not the ones you are looking for. However, sapient analysis of the results of automation tests can find significant issues. I cite an example for my last project where we had close to ...


6

Discussing your automated test needs with your developers up front can often help. A developer should not have tip toe around the application with fear of breaking our tests, but if they can proactively make design choices that make it easier for us to maintain them then that can be a huge time saver.


6

With Kanban you don't really have the problem of the sprint ending before you are happy that a feature is fully tested and working, so there is no need for an equivalent hardening sprint. Instead you would hold the work item in the test state until you have a robust set of automated test that verify the acceptance criteria. Limiting the work in progress ...


6

One possibility here is to build into your testing the ability to flag a failure as a "Known issue" which is then reported with each run of your automation. I've gone into more detail in my answer to the question user246 linked - I'd recommend you check into my comments and the other responses there.


5

First of all keep in mind: there will be always some maintenance. Whatever approach you will use, your tests will change over time. Many people will suggest to skip GUI automation as it is very brittle. It is true to some degree, many times it is just something they have heard from someone. Sometimes people say that because they have tried it once, expected ...


5

You have to understand the following... Automation will never find NEW bugs. It will find bugs if your system is updated. It will find bugs if the system is down for whatever reason and you are sleeping in your bed and you have automation enabled overnight. It will send you an urgent email that you will read in the morning that the site was down for an ...


5

Efficiency is relative to your needs. Rather than ask, "Should automated tests be efficient?", it might be more interesting to ask, "When does automated efficiency matter?" For example, if you expect a developer to run your automated test before checking in a change, it matters a great deal how long it takes that test to run. You could probably poll your ...


5

It is hard to draw reliable conclusions about an organization from two paragraphs in StackExchange, but it sounds to me as if you're going too fast. The fact that you are constantly changing the product suggests it is fairly new. With new products and new companies, when most of your users are early adopters, it may be more important to release quickly ...


5

These definitions will vary from one organization to the next. In the testing jobs I've had, regression testing has meant testing that verifies that features that used to work still work. In that context, a regression is a bug that did not exist in an earlier version of the product. In the testing jobs I've had, regression testing occurs after all new ...


4

Some good answers added already, but I haven't seen this distinction yet, so I'll add it to the pile: You may automate some tests for initial verification of the build against requirements - but another way of describing that is that you may decide to conduct specification workshops, with the goal of distilling Key Examples, that you can then turn into ...



Only top voted, non community-wiki answers of a minimum length are eligible