Test Automation in Agile?

Question

Since Agile is iterative and many test-last tools are record and playback-style, and therefore can't really be applied in an Agile environment, this brings up a few issues.

• the target for test is changing, rapidly - making it very difficult to automate
• by the time the target code has settled, the sprint/iteration is over, and you're on to new tasks (at least in the PM/BA's mind)
• resource management is difficult - 'if project A is 'done' (shipped) why are you now doing automation for it' may be asked.

What strategy can be applied in an agile environment to ease the pain of these moving targets / project-resource misalignment?

Answer 1

An answer - as Phil implied, is that you take every record and playback tool you can find, and burn them in a fiery pit of despair.

I probably got myself a down vote for that, so I'll try to earn it back.

Good Agile teams test constantly - not just at the end. If you include test design as part of feature design (and consider how the feature will be tested as it's designed), you won't have to write GUI automation - you can automate at the controller level, or through another abstraction.

Automating the GUI directly is rarely a good idea - even on "test-last" teams. Instead focus on what "quality" looks like for the feature, and work as a team to implement all of the tasks necessary to get to "done".

Answer 2

It really depends on what you're trying to achieve with your automated tests. The answer should drive your approach. Are you trying to:

• reduce regressions?
• reduce the amount of repetitive manual checking your testers have to do? (This may be the same thing as reducing regressions.)
• please a manager with some magic numbers?

Some ideas:

• Test first. Make your automated tests drive the development instead of the other way around. (Think about writing tests that will tell you when you're done, rather than tests that check what you've done after it's complete.)
• Change your definition of done to include the existence of tests (or some level of test coverage, or whatever metric works for you or your manager).

Generally, I would try and avoid record and playback style tools if possible (I'm assuming you're talking about something like Selenium IDE here.) In my experience they tend to be both slow and fragile. I've never gotten much return on investment from those types of tools.

Answer 3

The Assumption "Agile is iterative and many test-last tools are record and playback-style" is incorrect

• Tools are developed / adopted for managing repetitive tests, reusuable tests
• The feature developed might be in iterative phases, In such cases automation need to be developed for stable feature
• Automation has its own resource allocation, planning and execution. Automation benefits are typically in multiple cycles. Do not mix manual test efforts vs automation resource allocation
• Automation strategy need to be planned based on project resources, timelines. The points you have highlighted are perceptions, They do not represent a maturity of automation area

Answer 4

It sounds rather like you have the typical "agile-but..." implementation going.

There are ways around it that avoid the overhead and issues that go along with GUI-based automation (which can be done without record/playback - the things I'm doing with Microsoft's CodedUI right now were probably never considered when they devised the tool).

A few thoughts here:

• Unit testing - is this happening as part of the development? Good unit testing reduces the amount of automation that needs to happen at higher levels (you don't need to exhaustively test processing logic but can focus on integration issues and missed scenarios).
• Is there an API to work with? If there is, you can be building your automation with placeholders for the new fields until you know definitively what they'll be (if you're really lucky, there's a consistent naming convention that will allow you to get the correct field names most of the time. So far I haven't been lucky, but maybe one day...).
• How well does your existing automation support plugging in new functionality? It doesn't matter whether you're dealing with GUI automation or not: if you've got well-structured automation code, new functionality can be very easy to add.
• Does your GUI automation allow you to add new tests around existing functionality easily? While it's better, as Alan says, to avoid GUI automation if you can, that isn't always possible.
• Can the functional/GUI automation be handled by developers as well? This requires automation-complete being part of the definition of done, but if it can be done helps to prevent future issues.

For your issues with automating last and not having time for it, I'd recommend the term "technical debt". I presume the automation after code-complete is partly due to a changing GUI, and partly because the automation is going into a regression suite. What happens if there's any skimping on that automation is that you'll find customer-reported bugs clustering in the areas of the application that don't have good automation coverage.

My experience with this is probably typical - a large, complex business-to-business application where the pace of development and a chronically short-handed test team left gaping holes in automated regression. Most of the regression had to be GUI-based and large chunks of the application are grandfathered in from before object-oriented coding was a gleam in anyone's eye. They're not unit tested because they're not unit testable. The majority of customer bug reports were in the areas that didn't have automated regression running against them. The rest were edge cases - sometimes so far on the edge it wasn't worth automating them ("if the operating system crashes during this operation the data is corrupted, but only if you have the configuration that only one person in the known universe uses because it breaks every business rule in the system"). The more automation holes we were able to fill in, the fewer simple bugs escaped to customers.

=======

Answer 5

I write automation code with Robot Framework in an agile development environment. So long as you have an idea of the implementation of the code in test, you can write automation scripts before the code is deployed. This should be an exercise in logic and workflow.

Once the code is live you can update your script with specific identifiers for each keyword. This approach pre-supposes that you have a library of existing keywords for testing your application.

Often times we finish automation after a sprint is over - we run all automated scripts nightly so the script still gets run plenty. Furthermore, even without a nightly automation run, your automation scripts will be viable during regression sprints.

Answer 6

In our test team, we are working alongside developers in agile teams and we use (to some degree) a test system like you're referring to. There are a few reasons why this isn't as much of a problem as you'd think.

1) Many GUI test applications are pretty tolerant of changes to the UI and can be modified without re-recording. For example, if the developer goes back and changes a textbox to a dropdown, this isn't more than a 2 minute chance to the test to use the new element. Moving or renaming elements is fairly inconsequential.

2) In most Agile teams (certainly any I've worked with), user stories are geared toward completing user goals. This means that very often a piece of an interface is completed in accomplishing a user story and it doesn't change very much after that. Once the tests are written for it, they don't change much either.

As for the last two bullet points, that's all about culture. Testing needs to be part of the project, not something that people do once the project is done. In our teams, user stories aren't complete until the tests have been written and successfully run - that includes all automation. The way I usually put it is this: If my tests aren't written and haven't all succeeded, how do you know you're done coding?

Answer 7

Test automation in agile can be a nightmare if it's not done with appropriate analysis. There are certain tips that can help a lot in agile testing. Please check the link. http://www.wikihow.com/Make-Test-Automation-Effective-in-an-Agile-Environment