Tag Info

Hot answers tagged

25

My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs. But what they do do is two key things: They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs. They give you ...


16

Regression testing is specifically looking for bugs in functionality that was previously working and has "regressed" to a non-working state. I've never heard "retesting" used except in a common-sense way: Retesting is simply testing something again after it has been previously tested, and is a more general term since retesting doesn't say anything about ...


14

Any ideas for reducing test time for GUI tests? One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are. I once joined an organization that ran automated overnight tests ...


13

> The integration tests can sometimes take a long time, > thus discouraging users from running the entire test > suite prior to checking in For the checkin-runs you can mark the long-running tests with their own category and tell the test-runner to exclude those long-runners You may also look at Is there a way to separate long running (e.g. ...


12

You need to understand what are the core issues with these repeated breakages before you decide how to fix the problem. Relying on (more, improved, better, faster, etc) testing may not be the most effective solution here. It might not even help at all. Do you have code merge issues? Are fixes becoming undone because of that? Do you have new bugs ...


11

The truth is for a lot of people, there isn't a difference. A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly ...


10

Welcome to SQA, Rosa. I am not sure your question makes sense; QA is an abbreviation for "Quality Assurance", and often is used as a synonym for testing. ISTQB is a type of certification for testers. See also Do ISTQB/ISEB Testing Certificates prove someone can test? and http://en.wikipedia.org/wiki/Caps_lock.


9

I've been in this situation before. We were running real-time tests for similar reasons. I wouldn't optimize prematurely, but if you see a looming problem I also wouldn't wait until it's already an issue. Some things we did: Parallelized testing across many machines. Make it easy to select a subset of your tests, so you can break one test run into ...


8

It's not exactly clear what question you are asking, but let me take a stab. I would deal with it by creating a bug report. In it, I would mention what you are seeing in Commit A, and Commit B. I'd mention that the combination of the two pushes performance past the prescribed limits. From a QA point of view, it's not important at all in which Commit the ...


7

Ideally there should be a regression or "hardening" sprint before the project ends, and in the dream world, a hardening sprint every few cycles. I'd suggest that automated tests of the steel thread of the sprint be developed as the sprint progresses, and any bug fixes get an automated test as well. This way, you build your automated regression as the ...


7

Yes, however this is one of those circumstances when it depends, an the final answer on "how efficient" will be different for everyone. Why it matters If you use test automation for a smoke or BTV test, then you want to be able to execute broad coverage quickly If you need to debug or re-run a test to see if a failure occurrs you don't want to be sitting ...


7

You're asking a unicorn question. Why? Because you've skipped over the point of all that work. It doesn't matter what proportion of different types of testing you do, it doesn't matter what your organisation is, and it doesn't matter what technologies you use, if you don't know what information your stakeholders want to discover from your testing. That's ...


7

You should not edit the tests to pass. There are still defects in the product, and running the tests and having them fail on those points continues to provide data that the issues are not fixed. Since the development team has accepted the bugs and scheduled (although not solidly) them to be addressed, modifying the tests to no longer report the defects ...


6

With Kanban you don't really have the problem of the sprint ending before you are happy that a feature is fully tested and working, so there is no need for an equivalent hardening sprint. Instead you would hold the work item in the test state until you have a robust set of automated test that verify the acceptance criteria. Limiting the work in progress ...


6

The previous posters are correct - automation can never find new bugs, that can only be done by sapient testing - however automation will find regression faults, and often not the ones you are looking for. However, sapient analysis of the results of automation tests can find significant issues. I cite an example for my last project where we had close to ...


5

I've worked as a solo tester on a small agile team as well, and I've found the two things that help most with regression testing are automation and risk-based test prioritisation. Having an automated test suite with good coverage is definitely very helpful. However, I wouldn't recommend relying entirely on automated test for regression, as there are some ...


5

You have to understand the following... Automation will never find NEW bugs. It will find bugs if your system is updated. It will find bugs if the system is down for whatever reason and you are sleeping in your bed and you have automation enabled overnight. It will send you an urgent email that you will read in the morning that the site was down for an ...


5

Efficiency is relative to your needs. Rather than ask, "Should automated tests be efficient?", it might be more interesting to ask, "When does automated efficiency matter?" For example, if you expect a developer to run your automated test before checking in a change, it matters a great deal how long it takes that test to run. You could probably poll your ...


5

It is hard to draw reliable conclusions about an organization from two paragraphs in StackExchange, but it sounds to me as if you're going too fast. The fact that you are constantly changing the product suggests it is fairly new. With new products and new companies, when most of your users are early adopters, it may be more important to release quickly ...


5

These definitions will vary from one organization to the next. In the testing jobs I've had, regression testing has meant testing that verifies that features that used to work still work. In that context, a regression is a bug that did not exist in an earlier version of the product. In the testing jobs I've had, regression testing occurs after all new ...


5

One possibility here is to build into your testing the ability to flag a failure as a "Known issue" which is then reported with each run of your automation. I've gone into more detail in my answer to the question user246 linked - I'd recommend you check into my comments and the other responses there.


4

Automated tests should be just efficient enough, and no more. "This creates tests that run much longer than they could, but is this a problem? Most of the time the tests I run are run overnight unattended, so it's not too relevant if they take 4 or 10 hours to run. On the other hand, even small efficiencies could make a difference." Not trying to use ...


4

Scaling is always important to keep in mind. If you doubled the number of tests (which will happen eventually) to so now it goes up to 20 hours, would this be a problem? Maybe. You would start it at 5 when you leave, and get no feedback until after lunch the next day. Another thing I'd consider is how fast could they be? If you were to take them from 4-10 ...


4

I think you should work towards fast feedback. Using parallel test execution your testrun can theoretically take as long as your slowest test. A testrun which completes in minutes instead of hours helps you work in smaller faster cycles, catch errors faster, improve your testsuite faster etc. Developers want to know a change they've made caused a bug in ...


4

I assume that you have a suite of unit tests the developers are running which is essentially your first line regression suite. The purpose of a functional regression test suite is to help ensure that code churn didn't destabilize existing functionality, especially across dependencies. In general, the functional regression test suite should be ran every ...


4

There's a lot of ways you can go here - if you want to stay with manual testing you can look to improve your ability to find and report problems, or you can add load testing or functional regression testing to your skill set. Depending on where you're based, the most effective method for you could vary - you'll want to balance between something you enjoy ...


4

As a general rule, retesting in the staging environment is done more as an integration/sanity test than a full retest because the QA environment will likely have different code (due to other changes that aren't being pushed to the staging environment yet). The usual considerations are: The staging environment is kept as close to production as possible, ...


3

Typically when regression testing is talked about, it's tests that ensure features don't break from one release to another. Automated testing is a big category. It includes, but is not limited to regression testing with it's sub categories like smoke/bvt testing, unit tests, scripted UI automation. There are lots of other kinds of automation. Monkey ...


3

Some good answers added already, but I haven't seen this distinction yet, so I'll add it to the pile: You may automate some tests for initial verification of the build against requirements - but another way of describing that is that you may decide to conduct specification workshops, with the goal of distilling Key Examples, that you can then turn into ...


3

According to my view, "Scrum is the best agile approach. In scrum, each sprint produces an increment, which is a potentially releasable product. Here, each increment must satisfy all acceptance criteria and pass the different categories of tests. Regression tests are a tiresome activity, especially in an agile process, which is characterized by nonstop ...



Only top voted, non community-wiki answers of a minimum length are eligible