Hot answers tagged automated-testing
27
Should QA push for programmers running QA's automated tests?
Yes, but I'd also suggest that if the programmers don't seem to, it's probably more productive to find out why before pushing harder. What's preventing them? Access to proprietary test tools? Difficult to setup easily on dev environment? Results not meaningful or easy to read? Too slow to ...
26
No.
You can't automate everything.
You can't automate people's reactions (emotions) to your software.
You can't automate things you don't think of. (eh? eh?)
You can't automate users' thought patterns going from feature to feature.
You can't run a "fun factor" metric (applies to games more than enterprise software, but still.)
And that's just in 2 ...
22
My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs.
But what they do do is two key things:
They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs.
They give you ...
22
I think it's difficult to make through-the-UI tests reliable. The challenge comes down to the difficulty of reliably controlling and observing the variables that matter to your tests. Whether this is worthwhile depends on your ability to make your test code more resilient, and on the value of being able to run the tests automatically.
Asynchrony. For web ...
17
I think attempting to automate everything without thinking about what you're actually going to test is a horrible (but unfortunately common) practice.
My standard line is that you should automate 100% of the tests that should be automated. Figuring out which tests to automate (or not to automate) is tha hard part. Testers frequently waste time attempting ...
16
To answer your question, yes, if someone has the skill then QA should code tests.
Beyond the simple question is a bigger issue. As a professional test engineer you need to know when automated/coded tests are right to use on a project. Do you need full on automation, a quick script, some home-grown tool or do you need a manual test plan executed by a ...
16
If test automation does the job it is intended to, when a defect is introduced, it should be found and fixed very quickly, hopefully within a single daily cycle.
If this is how the automation suite is working, then automation should become a "barrier" that prevents new bugs from being created in code that is already working and tested.
When you think of it ...
16
It's not actually a true "either this or that" question since you can actually use both together via watir-webdriver.
Firstly:
If we are talking about testing at the browser level, you can pretty much completely ignore the language used to run the server side. There's no C# remaining once the page is rendered to HTML+Javascript and sent over HTTP to the ...
16
A hard coded sleep statement is generally supposed to represent some sort of mocked delay in the application that doesn't exist during testing.
The harmful aspect of this is that a hard coded value can't represent the complexity that actually exists in what its mocking.
Take for example a network delay. Your production system usually takes 4 seconds to go ...
15
Yes. Over the last few years I have built, re-built and evolved testing tools for a number of things:
A Windows autumation library on top of UIAutomation
A full C# based testing stack built on top of Watin and the UIAutomation library
Control generators
Test case management systems
Defect tracking systems
Various intergraion tools to work with TFS, JIRA or ...
14
Any ideas for reducing test time for GUI tests?
One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are.
I once joined an organization that ran automated overnight tests ...
13
I would suggest that you go back and do those unit tests when you are required to refactor that area of legacy code. The approach for do this is described in this question.
When you are working with legacy code with no unit tests, its the same principle and techniques to add them regardless if you are doing TDD or not.
I highly recommend Working ...
13
We run unit tests prior to check-in and rerun them as part of a larger test suite on each daily build (feature branch) and weekly build (main branch; multiple feature branches aggregated).
Some advantages we have found include:
Fewer build breaks, and virtually zero build breaks in main
Additional test coverage for component/integration level testing
...
13
I think you're selling yourself short, here - you've said that in a language and skillset with which you're familiar and current you do well, but you're struggling with a language and skillset you have worked with in the past but where you've become rather rusty.
That's normal. Seriously. Automation and programming are complementary skills but they're also ...
12
The best place to start is over at http://www.thebraidytester.com/ where Michael Hunter describes the stack he built for testing Microsoft Expression in number of articles, papers and blog posts.
I have used this approach as the inspiration for my own watin based stack which is available at http://testingstax.codeplex.com
Essentially the framework breaks ...
11
I definitely feel your pain. As noted in a question I had (I'll link in a minute) I too work for a 10-digit revenue company, and our primary software has 0 automated tests for over a million lines of production code.
It boils down to the same philosophy that a lot of developers have about using external libraries rather than rolling their own: we fear what ...
11
> The integration tests can sometimes take a long time,
> thus discouraging users from running the entire test
> suite prior to checking in
For the checkin-runs you can mark the long-running tests with their own category
and tell the test-runner to exclude those long-runners
You may also look at Is there a way to separate long running (e.g. ...
11
For Test Automation purposes, it's always been my belief that once you know 2 scripting languages, you are in very good shape.
Being comfortable with 2 languages means you know how to automate, and that you know how to learn a new scripting language. As a hiring manager, I will be able to assume that you could easily pick up another new language as needed.
...
10
If they can be run unattended, run them over the weekend every week.
That'll give you enough run-time, and enough break between runs to fix bugs and perhaps improve the automation.
As for analyzing the results, that's a different issue. I have found it useful to group the results into features or categories. That way, I can quickly see from the result set ...
10
I don't like the statement because it assumes that the only thing you can automate are regression tests. If you rewrote the statment as "Regression tests don't find new bugs", then I think it's accurate.
I often write automation (e.g. performance, data driven, or model-based tests) that find new bugs - and I think the automation effort is incomplete without ...
10
Who would be administering QC? If it's going to be you, I'd recommend you give it a miss for now. I don't think it's particularly suited to small teams, or Agile teams: it's aimed at large companies, with waterfall development methodologies, and pretty much the whole design is aimed at that - you might find yourself having to swim against the current a lot ...
9
In my experience ... Execute, YES. Maintain NO.
Context for my comment. I am assuming Programmers in your question = developers of application, not testers with programming skills
I think that there is an inherent motivation that testers, (even dedicated software engineers in test) have that developers don't, and that is that writing, running and ...
9
Solution Overview
To solve this problem, you will want to intelligently select a manageable set of combinations based on a pairwise coverage approach (explained below) or a more thorough variation of combinatorial test design.
Glowcoder and user246 have good points. I particularly like testerab's comment for reasons that will become clear in a minute.
...
9
The easiest way I can think of is to base it on how many hours the qa people are saving by not having to regression test all the scenarios covered by the automated test. If the response is they wouldn't regression test it, the answer to that that they should. The ideal scenario is that the entire application should be tested from one release to the next, ...
9
The truth is for a lot of people, there isn't a difference.
A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly ...
9
I think its hard to determine what is a "Best Practice" in this field since many aspects of software/hardware under test tend to be customized to the environments they are developed for what works in one environment does not work in another. There are generics, such as highly repetitive tasks that have low return or have standard results which make good ...
9
I've been in this situation before. We were running real-time tests for similar reasons. I wouldn't optimize prematurely, but if you see a looming problem I also wouldn't wait until it's already an issue. Some things we did:
Parallelized testing across many machines. Make it easy to select a subset of your tests, so you can break one test run into ...
9
Your automated tests are currently set up to answer the question "What isn't working as expected?"
Now you want them to answer the question "What isn't working that I don't already know about?"
Your solution is to change the automated tests so that they account for the "known" bugs in one of two ways:
You could comment out the tests which find every ...
9
To answer the question "are they worthwhile?", you need to explore:
How expensive (in terms of time and money) are they to create, maintain, and execute?
What value are these tests providing? Are they finding bugs? Providing confidence?
Are there other, less expensive, ways to provide similar value?
I once worked at a software company which built desktop ...
9
You mentioned abstracting your test code from the framework. The page object pattern is one way that this can be done and is fairly common. It is popular with Selenium but can be applied to any UI automation: http://code.google.com/p/selenium/wiki/PageObjects.
You can avoid using xpath that contains the entire path (or even part of the path) to the UI ...
Only top voted, non community-wiki answers of a minimum length are eligible
