Suite of automated tests - owned by devs or testers?

Question

I'm working on a project that involves 4 developers and three QAers.

I've tried to encourage the other devs to write tests for every new feature they have added, and we have built up a suite of around 50 Selenium UI tests. I don't doubt that this has improved the quality of the product but the developers have become slaves to getting the tests working and I don't think we know enough about Selenium yet to really use it efficiently.

I'm thinking that for the next project I will pass automation of tests and building an automated regression suite over to the QA people. One of them has had good success using the Selenium add-on for Firefox before so the skills and the will do exist. Is this the right decision to make?

Answer 1

Extending and maintaining automated tests costs time, but if your team are slaves to updating tests you are doing it wrong. :)

One of the biggest reasons to create automated test coverage is to create a fast feedback cycle loop. Letting the test-team fix the tests afterwards slows down this process and will always lag behind. There is a high risk the tests will always be an after thought and are untrusted by the developers.

I suggest using a BDD framework. My previous team combined Cucumber and Selenium into a testing framework. When a developers checks in new code the current tests should not break. Also you want new coverage for new features. Schedule the tests in a continuous integration cycle to make sure the tests do not break.

• Testers write test-cases
• Developers automate them

Of-course testers can also automate them and developers can extend tests with more corners cases, but keep in mind that you need good developers to create a maintainable testing framework.

Use best of both worlds and pair program (1 dev + 1 qa) on implementing the more complex test-cases.

Answer 2

Why either/or? Work as a team with testers providing test ideas and expertise that the devs can code up. I would also shy away from an emphasis on GUI automation and get the devs working at a unit/system level unless they are doing this already

Answer 3

Here's how I would approach this kind of situation:

• Unit tests - Evangelize the living daylights out of unit testing. Any kind of business logic automation should be handled by unit tests, as should any other automated tests whose goal is to check the functionality of a single unit of code. This is developer-level code, but tester guidance is essential if you want your unit tests to be useful.
• API/Integration tests - These work more or less the same way as unit tests from the developer perspective, but they test slightly bigger chunks of code. Again, coded by your developers with tester input on what makes a good test.
• "Headless" end-to-end tests - These are lighter-weight than full-GUI tests, but will not find problems with components and validation code. They're also not possible in all circumstances. They're somewhere between API-level and full GUI-level automation.
• End-to-end tests - These are usually your GUI-level automation tests. They can be coded by developers with tester input, or coded by testers with developer input.

As a general rule you want most of your automation to be at the unit test level because it's the fastest, lightest-weight level to work with and it's relatively trivial to set up most build environments to run unit tests on every compile/build. Thousands of unit tests can be run in the time it takes to complete a single GUI-level test.

The next biggest number of tests should be the API/Integration level tests. They're slower than unit tests, but may still be able to run with every build. Depending on how they're designed, they may or may not require the executable to run.

End to end tests in any form should be the smallest grouping for several reasons:

• they're orders of magnitude slower than unit-level and API-level tests.
• they're more likely to report false positives than lower-level tests (as a rule, the more pieces you have to link together in automation, the more likely you are to get a failure that isn't caused by a bug)
• they're more difficult to maintain than unit-level and API level tests (because they're dealing with more complexity).

Your developers and testers are working together on the software - why wouldn't they work together on the automated testing? Who does what is a matter of who is best suited to which task, but the goal is still the same.

Answer 4

It depends on what sort of resources you can afford. Test Automation, done well, is not a part time effort. If you can't afford to have a Toolsmith on your team to work on the Automation Framework you will need help, or expect it to take a long time, or expect it to take effort to keep it going as it gets reprioritized. While most Testers do not spend a lot of time coding and learning to code, many do and they can help provide the assistance, and boost, your team may need. I'm no Developer but I do know HOW to make a Test Framework and get it working, with input from Developers as needed. After all many Developers did not know how to code at some point, they needed to learn how to do it as well. It's also a skill that many Testers need to work on, as the thought process is different but you can usually make a very effective Framework utilizing Coding Skills and Practices with Testing Skills and Practices.

If you want a Framework that does what YOU want, QA needs to own it. QA needs to run it, and QA needs to direct it. You should get buy in from your Developers as well, some Code Review (this is code after all) will be good now and again not just on the Test but on the Framework as well. Make it robust, and most of all make it useful. Other teams should be able to run it, even Developers should have a way to do some localized Smoke Testing in their environments to get their buy-in, and show the usefulness of the Framework for them - get them catching issues early on. The last Framework I used (a BDD Web Driver that used SpecFlow) was able to be used by QA, DEV and the BA's - I just had to show them how to run it. I am not a Developer but I do a lot of automation.

It's good you have the Developers working on Unit Tests, and in the Framework, but remember their goal is to Develop a solution to a problem and your role is to show that their solution solves the problem.

While the browser add-ons are useful, look at something like WebDriver for Selenium and another tool like Cucumber or SpecFlow. Build a framework that will allow you to run a suite of tests, or a select few, so you can cover Regression, Functional and other tests as needed. You also want reports coming from your tests, you should be able to see the historical Pass/Fails for Tests so you know if they are valid, failing only at specific times or show that your pass rate is getting better - management will like that! Any tool takes time to learn, you just need to dedicate that time to your team and give them the resources to improve.

Answer 5

I think it is the right decision to let the QA Engineers develop the test automation. Developers aren't testers and do not have the know-how to create good GUI test cases.

But you can involve the developers in the development of test automation, if needed. They have useful knowledge about programming and can help you to setup the infrastructure of your test automation framework. If you have troubles or do not know how to develop something, ask them.