37
My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs.
But what they do do is two key things:
They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs.
They give you ...
34
You can try building your Selenium tests using the Page Object pattern, more info on:
https://github.com/SeleniumHQ/selenium/wiki/PageObjects
19
> The integration tests can sometimes take a long time,
> thus discouraging users from running the entire test
> suite prior to checking in
For the checkin-runs you can mark the long-running tests with their own category
and tell the test-runner to exclude those long-runners
You may also look at Is there a way to separate long running (e.g. ...
18
The truth is for a lot of people, there isn't a difference.
A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly ...
18
There are a number of key things that I have learnt are key to have easy to maintain, reliable automated tests.
Map your controls in a single place
Your controls should be mapped into a single location, so that if a control changes, you only need to change it in a single location. Having it in a single location allows you to make a one line code change if ...
17
Regression testing is specifically looking for bugs in functionality that was previously working and has "regressed" to a non-working state. I've never heard "retesting" used except in a common-sense way: Retesting is simply testing something again after it has been previously tested, and is a more general term since retesting doesn't say anything about ...
17
I think it is not a good idea because regression testing is carried out once you complete integration and sanity testing followed by system testing.
No, it is not necessarily a bad idea.
You should not consider it as a bad idea solely based on the order in which tests are carried out.
By definition:
Regression testing is a type of software testing which ...
15
Any ideas for reducing test time for GUI tests?
One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are.
I once joined an organization that ran automated overnight tests which ...
15
You need to understand what are the core issues with these repeated breakages before you decide how to fix the problem.
Relying on (more, improved, better, faster, etc) testing may not be the most effective solution here. It might not even help at all.
Do you have code merge issues? Are fixes becoming undone because of that?
Do you have new bugs ...
14
The crucial issue IMO is to acknowledge that your pages will need to be adjusted in order for the tests. To pick up on the Selenium issue, instead of using XPath to structure your selectors, add ids to the pages and use those in the tests.
I've also found the Page Object pattern to give good results:
Within your web app's UI there are areas that your ...
13
To supplement glowcoder's answer, mainly with more examples, automated regression tests are a subset of automated testing, but there are other kinds. Some examples of non-regression automated tests are:
Any automated test that will be thrown away and not used to test for regressions is not a regression test, including unit tests that are not used later for ...
12
Ideally there should be a regression or "hardening" sprint before the project ends, and in the dream world, a hardening sprint every few cycles.
I'd suggest that automated tests of the steel thread of the sprint be developed as the sprint progresses, and any bug fixes get an automated test as well. This way, you build your automated regression as the ...
11
I've been in this situation before. We were running real-time tests for similar reasons. I wouldn't optimize prematurely, but if you see a looming problem I also wouldn't wait until it's already an issue. Some things we did:
Parallelized testing across many machines. Make it easy to select a subset of your tests, so you can break one test run into ...
11
You're asking a unicorn question.
Why?
Because you've skipped over the point of all that work. It doesn't matter what proportion of different types of testing you do, it doesn't matter what your organisation is, and it doesn't matter what technologies you use, if you don't know what information your stakeholders want to discover from your testing.
That's ...
10
The previous posters are correct - automation can never find new bugs, that can only be done by sapient testing - however automation will find regression faults, and often not the ones you are looking for. However, sapient analysis of the results of automation tests can find significant issues.
I cite an example for my last project where we had close to ...
10
You should not edit the tests to pass. There are still defects in the product, and running the tests and having them fail on those points continues to provide data that the issues are not fixed.
Since the development team has accepted the bugs and scheduled (although not solidly) them to be addressed, modifying the tests to no longer report the defects ...
10
One possibility here is to build into your testing the ability to flag a failure as a "Known issue" which is then reported with each run of your automation.
I've gone into more detail in my answer to the question user246 linked - I'd recommend you check into my comments and the other responses there.
10
"Yes"
If you have an automated regression suite, you should definitely re-run it. If your organization is really on the ball, this will happen automatically along with all the unit tests and other tests as part of your continuous integration process.
If you don't have that luxury, then if you don't see a major risk, then testing only around the areas that ...
9
I repeatedly apply three principles:
Hide incidental details. Any detail that isn't directly related to the purpose of your test belongs somewhere other than in your test. Hide it somewhere, such as in a variable or in a method, and give it a good name.
Name every important idea. Why does the test code log in in as "F.D.Gumby" rather than as some other ...
9
Extending and maintaining automated tests costs time, but if your team are slaves to updating tests you are doing it wrong. :)
One of the biggest reasons to create automated test coverage is to create a fast feedback cycle loop.
Letting the test-team fix the tests afterwards slows down this process and will always lag behind.
There is a high risk the tests ...
answered Jul 3 '14 at 12:07
Niels van Reijmersdal
22.5k2 gold badges34 silver badges83 bronze badges
8
I've worked as a solo tester on a small agile team as well, and I've found the two things that help most with regression testing are automation and risk-based test prioritisation.
Having an automated test suite with good coverage is definitely very helpful. However, I wouldn't recommend relying entirely on automated test for regression, as there are some ...
8
With Kanban you don't really have the problem of the sprint ending before you are happy that a feature is fully tested and working, so there is no need for an equivalent hardening sprint. Instead you would hold the work item in the test state until you have a robust set of automated test that verify the acceptance criteria. Limiting the work in progress ...
8
It's not exactly clear what question you are asking, but let me take a stab.
I would deal with it by creating a bug report. In it, I would mention what you are seeing in Commit A, and Commit B. I'd mention that the combination of the two pushes performance past the prescribed limits.
From a QA point of view, it's not important at all in which Commit the "...
8
My Advice:
Do not communicate your disagreement by email
Schedule a time to meet with your manager and have the conversation with them.
Emails do not work well for this sort of communication as it is not clear what the all important 'tone' is and it is also one-way at a time which does not work very well for difficult conversations.
The manager may also ...
7
Discussing your automated test needs with your developers up front can often help.
A developer should not have tip toe around the application with fear of breaking our tests, but if they can proactively make design choices that make it easier for us to maintain them then that can be a huge time saver.
7
First of all, keep in mind: there will be always some maintenance. Whatever approach you will use, your tests will change over time. Many people will suggest to skip GUI automation as it is very brittle. It is true to some degree; many times it is just something they have heard from someone. Sometimes people say that because they have tried it once, expected ...
7
You have to understand the following...
Automation will never find NEW bugs.
It will find bugs if your system is updated. It will find bugs if the system is down for whatever reason and you are sleeping in your bed and you have automation enabled overnight. It will send you an urgent email that you will read in the morning that the site was down for an ...
7
Yes, however this is one of those circumstances when it depends, an the final answer on "how efficient" will be different for everyone.
Why it matters
If you use test automation for a smoke or BTV test, then you want to be able to execute broad coverage quickly
If you need to debug or re-run a test to see if a failure occurrs you don't want to be sitting ...
7
As a general rule, retesting in the staging environment is done more as an integration/sanity test than a full retest because the QA environment will likely have different code (due to other changes that aren't being pushed to the staging environment yet).
The usual considerations are:
The staging environment is kept as close to production as possible, and ...
7
Push to make everything testable in the integration environment. The location you are testing in should mimic the production environment as closely as possible. Most payment providers should also have a testing interface, even credit-card validation has fake numbers to use.
If the devs can confirm they fixed it, use their environment, maybe even pair test ...
answered Dec 16 '15 at 15:33
Niels van Reijmersdal
22.5k2 gold badges34 silver badges83 bronze badges
Only top voted, non community-wiki answers of a minimum length are eligible
