Hot answers tagged

40

I think it's difficult to make through-the-UI tests reliable. The challenge comes down to the difficulty of reliably controlling and observing the variables that matter to your tests. Whether this is worthwhile depends on your ability to make your test code more resilient, and on the value of being able to run the tests automatically. Asynchrony. For web ...


36

No. You can't automate everything. You can't automate people's reactions (emotions) to your software. You can't automate things you don't think of. (eh? eh?) You can't automate users' thought patterns going from feature to feature. You can't run a "fun factor" metric (applies to games more than enterprise software, but still.) And that's just in 2 ...


33

Should QA push for programmers running QA's automated tests? Yes, but I'd also suggest that if the programmers don't seem to, it's probably more productive to find out why before pushing harder. What's preventing them? Access to proprietary test tools? Difficult to setup easily on dev environment? Results not meaningful or easy to read? Too slow to run? ...


33

We cannot automate CAPTCHA, as it is not meant to be automatized. How to deal with CAPTCHA? Either ask dev team for a workaround, like configure CAPTCHA in test environment in such a way it will always accept 1 specific value. Ask developer to disable the CAPTCHA module in testing environment. If your are using custom CAPTCHA module, you can ask developer ...


31

My real world experience of running test suites of 1000+ tests every day on large web systems is that your hunch is right and that they don't find that many bugs. But what they do do is two key things: They free up the valuable time that testers would otherwise have to spend regression testing to do exploratory testing which does find bugs. They give you ...


31

In addition to what everyone else has said, it's absolutely realistic. With respect to what the hiring agencies are telling you, here are some reasons you could find difficulty convincing a developer that they want to be part of the test team: if your automation specialists are paid significantly less than your developers, you'll be asking any developer ...


25

You don't, that is the whole idea behind a good CAPTCHA. CAPTCHA stands for: Completely Automated Public Turing test to tell Computers and Humans Apart So, by definition, resolving a CAPTCHA cannot be automated: Otherwise it could not tell computers and humans apart and hence fails being a CAPTCHA. Howto handle a CAPTCHA in a test environment: If ...


23

If test automation does the job it is intended to, when a defect is introduced, it should be found and fixed very quickly, hopefully within a single daily cycle. If this is how the automation suite is working, then automation should become a "barrier" that prevents new bugs from being created in code that is already working and tested. When you think of it ...


22

I think attempting to automate everything without thinking about what you're actually going to test is a horrible (but unfortunately common) practice. My standard line is that you should automate 100% of the tests that should be automated. Figuring out which tests to automate (or not to automate) is tha hard part. Testers frequently waste time attempting ...


22

It's not actually a true "either this or that" question since you can actually use both together via watir-webdriver. Firstly: If we are talking about testing at the browser level, you can pretty much completely ignore the language used to run the server side. There's no C# remaining once the page is rendered to HTML+Javascript and sent over HTTP to the ...


22

A hard coded sleep statement is generally supposed to represent some sort of mocked delay in the application that doesn't exist during testing. The harmful aspect of this is that a hard coded value can't represent the complexity that actually exists in what its mocking. Take for example a network delay. Your production system usually takes 4 seconds to go ...


21

I think you're selling yourself short, here - you've said that in a language and skillset with which you're familiar and current you do well, but you're struggling with a language and skillset you have worked with in the past but where you've become rather rusty. That's normal. Seriously. Automation and programming are complementary skills but they're also ...


19

To answer your question, yes, if someone has the skill then QA should code tests. Beyond the simple question is a bigger issue. As a professional test engineer you need to know when automated/coded tests are right to use on a project. Do you need full on automation, a quick script, some home-grown tool or do you need a manual test plan executed by a ...


18

I would suggest that you go back and do those unit tests when you are required to refactor that area of legacy code. The approach for do this is described in this question. When you are working with legacy code with no unit tests, its the same principle and techniques to add them regardless if you are doing TDD or not. I highly recommend Working ...


18

I work as a Manual tester on a big project. My project follows Agile methodology. I test a Web-based application. I sometimes ask my self , that am i giving any value to the stakeholders? So, the value that you are giving to stakeholders is the reduction in the amount of defects that go to production. The cost of fixing a defect drastically increases over ...


17

> The integration tests can sometimes take a long time, > thus discouraging users from running the entire test > suite prior to checking in For the checkin-runs you can mark the long-running tests with their own category and tell the test-runner to exclude those long-runners You may also look at Is there a way to separate long running (e.g. ...


17

There are plenty of testers who have learned to develop and there are also many testers who began their careers as developers. It is absolutely possible to find these kinds of people. There may be other hindrances however, such as availability in your area who have all of the skill-sets that you require, especially when it comes to specialized skill-sets. ...


16

Yes. Over the last few years I have built, re-built and evolved testing tools for a number of things: A Windows autumation library on top of UIAutomation A full C# based testing stack built on top of Watin and the UIAutomation library Control generators Test case management systems Defect tracking systems Various intergraion tools to work with TFS, JIRA or ...


16

We run unit tests prior to check-in and rerun them as part of a larger test suite on each daily build (feature branch) and weekly build (main branch; multiple feature branches aggregated). Some advantages we have found include: Fewer build breaks, and virtually zero build breaks in main Additional test coverage for component/integration level testing ...


16

For hacking WebDriver without real production purpose, you need to find some task that really motivates you. Don’t forget, WebDriver – is not only about test automation, people use the tool (not so widely) for many different purposes: crawling some data from websites, semi-automating real job-related tasks. For instance, a girlfriend of friend of mine ...


16

Writing unit tests is not difficult - as the saying goes, it is matter of simple programming :-) So if you are competent programmer, and are willing to learn necessary skills and patterns, you can do it as QA engineer. But IMHO (and best practices say that) developers are much better suited to write unit test - because unit tests use internal calls to ...


16

I think the main reason why companies are not doing automated testing is because the "Return on Investment" is hard to prove. Also its hard to prove that you will have less defects in production due to these tests, since they will catch the obvious issues, but not the complex dependency issues that actual users might run into. The automated testing mindset ...


15

In my experience ... Execute, YES. Maintain NO. Context for my comment. I am assuming Programmers in your question = developers of application, not testers with programming skills I think that there is an inherent motivation that testers, (even dedicated software engineers in test) have that developers don't, and that is that writing, running and ...


15

I definitely feel your pain. As noted in a question I had (I'll link in a minute) I too work for a 10-digit revenue company, and our primary software has 0 automated tests for over a million lines of production code. It boils down to the same philosophy that a lot of developers have about using external libraries rather than rolling their own: we fear what ...


15

In my experience, the best developer/testers bring what I call the boredom herusitic to software testing. Inother words, they often automate rote tasks so they can focus on testing and analysis of the software. An example I use often is the "add contact" feature in an instant messaging or email client. To test the feature, I'd need to try long names, ...


15

Any ideas for reducing test time for GUI tests? One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are. I once joined an organization that ran automated overnight tests which ...


14

I don't like the statement because it assumes that the only thing you can automate are regression tests. If you rewrote the statment as "Regression tests don't find new bugs", then I think it's accurate. I often write automation (e.g. performance, data driven, or model-based tests) that find new bugs - and I think the automation effort is incomplete without ...


14

The truth is for a lot of people, there isn't a difference. A regression test is something that is going to check your test results against some requirements. Good examples of this are unit tests and functional tests. They tell you whether or not your application has regressed in its functionality (hence, regression.) Of course that assumes it was properly ...


14

The best place to start is over at http://www.thebraidytester.com/ where Michael Hunter describes the stack he built for testing Microsoft Expression in number of articles, papers and blog posts. I have used this approach as the inspiration for my own watin based stack which is available at http://testingstax.codeplex.com Essentially the framework breaks ...


13

I'd take a look at Google QualityBots. It's generally used for comparing websites on multiple versions of Chrome, but looks very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source. Here is an article about it on Google's testing Blog And ...



Only top voted, non community-wiki answers of a minimum length are eligible