How much testing of your automated scripts do people generally do?
Is it best to treat the automated test suite as a separate project or part of the Agile development cycle of the product your are testing?
Being the sole QA person writing and running the automated tests suite can you really have confidence in the robustness and correctness of the scripts?
Any advice?
I don't think testing test cases themselves, beyond running them and getting a code review, makes sense. However, testing reusable test code makes some sense; the question is whether it makes enough sense to justify the time that would be invested. I think that question needs to be answered component by component.
I don't generally do testing of test code beyond running test cases as I create them to ensure they behave as expected. I've come to realize that the test cases I run are implicitly testing my test harness / fixtures every time those test cases are run. I do execute a test run after making significant changes to test code on a previously tested project, to ensure that I am not getting different test results on the same piece of code. Getting different results would suggest a test code bug, so regression tests for other features end up also being regression tests for my test code. In general, I find tests and test code tend to "test themselves", since most test bugs cause the test checks to fail. The biggest exception is the test harness, which exists before any tests do, and gets some basic tests to ensure that passes and fails are recorded correctly, data is properly loaded from the DB, etc.
I have occasionally found test bugs where the test case wasn't testing what I thought it was. However, these are rarely caused by testing code bugs IME (after the first successful run) and are usually caused by either a bad specification or bad test data (e.g., running the wrong file through the test). These kinds of bugs are difficult to test for; bad specs are best found through intense curiosity, and bad test data is usually only found when a bug doesn't make a test fail but should have, and is detected through some other means (often another test case that implicitly tested that behavior).
IME, all testing needs to get on the backlog in one form or another, but specifics don't matter as much. We currently make test harness changes their own user stories, but fixtures and test cases are tasks on the relevant user story being tested. I've done it the other way, with all testing tracked as its own project, but that tends to hide testing a bit more. I think the main thing is that testing gets written down somewhere where it will be taken seriously and get attention when people are making their scheduling decisions.
I know my tests are good quality when I see my tests finding lots of bugs and very few bugs getting through. I can guess my tests are good quality when I see a strong correlation between how many bugs I find and what the quality of the product seems to be. There have also been occasions where I have "parallel tested" software, myself automating and a black-box tester using exploratory techniques. I've found that we find the same number of bugs, with 80% being the same issues and about 20% each being different, but of similar severity and priority, in the same time frame. This has given me a great deal more confidence in my test automation skills.
Some testers do more than this, however, using techniques like mutation testing. Mutation testing is when a set of artificial (perhaps random) small changes are made to the system under test. The question then becomes, does your test catch those changes? If the production code is well-designed without dead code, you should be picking up those regressions. "Beautiful Testing" has a chapter on mutation testing in part III of the book (chapter 18).
Unfortunately, where I work the answer to your first question is "not enough".
Your second question, I would say that you should treat the automation as part of the Agile development cycle of the application under test. The testing of user stories is automated as they are developed and accessible to you, and each run of the scripts then provides regression of all previous test cases automated (I would recommend scheduling the automation separately so you can sign off on stable development and have that running while working with new functionality).
For your third question, if you're not able to have anyone else review your test scripts, their ability to function and generate correct responses is your degree of confidence. That means you need to have defined and validated baselines that you know are accurate, as well as correct workflows. If your time to update the scripts to account for changes to the application is more than your new development time, that's a reasonable indication that your scripts could be more robust. For instance, if you need say 4 hours to add a new test, regenerate baselines and verify that your new baselines are accurate, but it takes you 6 hours to update the other parts of your script to handle changes to the application flow or structure, you don't have a very robust suite and could probably stand to make your scripts more modular.
Hopefully that gives you somewhere to start looking.
When I have written GUI automation test suites, I have found it best to treat it as a separate project.
At a test logic level, I have not found it a good use of time to write automated tests for the automated tests. However, at a lower level I have found it very useful.
For example, I may write a common component for handling win32 dialogs, which will detect the dialog, read the message, and close the dialog correctly. When I use this in a test script, I will want it to cater for the following cases:
and, most importantly:
So I can create a simple web page that simulates each dialog scenario as well as a simple test, and test that my dialog handling component can handle each case. Then when I make updates to it, I can run the test again and know that I haven't broken anything. It's much faster than running my entire test suite again and I have some reassurance of robust code driving my test suite.
Automated tests, depending upon the extent of complexity that you're doing with them, are almost a development project in themselves. For myself, I have always submitted my automated test code for a code review to a peer to have them check my logic and see if there are any problems. I do also try and make sure that I execute each code path at least once to make sure that everything works cleanly.
Personally, as an automated regression suite, I would prefer automated tests to be treated as their own project. If there's an Agile development going on, there need to be tasks to adjust/add to/create new automated code for the project.
To your first question, it's impossible to automate everything, and there are many schools of thought on how to decide what to automate, but my general rule of thumb is to automate a sampling of the test cases for each feature based on their requirement priority. Some tests are just important than others. "Can I add / edit / and delete this record" is a more important question to answer than "Are the records sorted correctly?".
To your second, I think it depends on the type of automation, and how time consuming it is to add addtional tests to your suite. GUI automation requires functional features and is difficult to include in the agile process I feel. Unit testing and functional testing at lower levels can be included in the development effort and testers can collaborate with developers in the creation of tests.
And to your third, have your automated tests found bugs? Have they prevented some show-stopping issues from reaching the customer? Does your department rely on your tests to check functional areas to save time from performing manual regression?
If you answered yes to any of those, I think you should be proud of the work you've done and take confidence from that. That said, you shouldn't stop trying to improve the work you've done to make it even better. Schedule some time to meet with the developers on your team to explain what your tests cover, and how they work. They might have some great ideas for you.
I would answer them with respect to my job position -
I do make sure that my tests fail when there is a failure, to do so I need to simulate false failures in scripts to check if would really fail.
In my job it is part of agile development life cycle. Many a times I work on automating scripts, while dev is doing development (though I would not have access to testing hooks till UI is available)
That's little tough, I usually ask others (outside my team) to review my code, execute it and give me feedback if some thing could be better. But if none is available to you then review you own code. May be after a while u have written it. You yourself might find better ways. Well I do...
To get your questions in order (since I go through this exercise often)
Enjoy, and have fun!
In my previous project we ran a set of unit tests on the core of the testing scripts every time we submitted something the source control system. The objective was to make sure we didn't break important basic functionality (loggers, parsers etc.) that can affect the whole automation.
