Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

As example - a new project is going to be started. Team is almost ready. There is also requirement regarding software testing - is to automate it.

The question is: 1. How to estimate how is good future application to automate it (as there is no application yet) ? 2. Generally - when exactly to start automated testing - exact software life cycle point ?

share|improve this question
    
This kind of implies that there is a bucket that is "automated testing" that is homogeneous. With different kinds of automated tests (Xunit tests, front end automation, load tests) all start at different points in the life cycle, and the starting point changes drastically by paradigm. –  corsiKa Feb 25 at 17:31
add comment

7 Answers 7

up vote 0 down vote accepted

Well, by start of automation I am presuming you mean actually running the tests. A good time for this is when the environment is stable and the tests have been written. However, your already doing a lot right by planning (or attempting to plan) your automation process with the application.

As for what point in the cycle, it will typically be toward the middle of the testing cycle. But that really depends on the type of development cycle, how closely your automation team works with your development team and how well your test cases and utilities have been developed to work with the system.

My suggestion, instead of trying to quickly launch automation into the first and second sprint/release/whatever, try and make sure that you make quality, long term scripts that are easily adjustable and even easier to update. You will be a hero and it will make your job a lot easier long term. This will be the best scenario possible for your team as it will allow them more time to write new scripts instead of maintaining code.

As for the requirements to automate it, I hate to be the bearer of bad news but it looks like the business is dooming you to failure and likely has unrealistic goals for what automation can do and what it is for. Automation, in my opinion, should be the step where the company says 'everything that worked before this code still works'. I would suggest you talk to the company, if possible, regarding having some manual testing, especially in the early phases. It will drastically improve the quality of the software.

share|improve this answer
add comment

It depends.

If you're automating against an API, you can start as soon as the API calls are stable (the call names and required fields are decided).

If you're required to do GUI automation, there's generally no point starting until the GUI is stable - for agile projects that usually means that GUI automation will run a sprint behind application development, and in waterfall projects GUI automation often starts during the test phase (and may be handled as a separate project).

The automator's role early in the project should be to work with developers to a) help them design good quality unit tests, and b) ensure that as the application is built it's designed with automation and testability in mind (things like making sure there are hooks for GUI automation to access custom components, building in APIs to minimize the need for GUI automation, and so forth).

share|improve this answer
add comment

I have good experience starting automation with unstable or non-existent API, sometimes using mocked API- calls returning pre defined values.

Starting automating tests only when the API is stable means you'll have a big gap with no automatic tests at all, whereas if you start early you'll probably need to do some refactoring but your automation will be in a working state much faster.

This also works well in Agile (or Agile like) projects, or even with longer-cycle projects where you get pre releases or partially working releases. You can test both your code and the project itself.

share|improve this answer
add comment

As you already have a REQUIREMENT (hopefully a document or series of documents), you already have something to test. Does the requirement provide sufficient information to tell when you have in fact developed something that satisfies the requirement?

share|improve this answer
add comment

Start automating at the moment when the requirement becomes the next requirement to implement.

You can start automating before you implement each requirement. And finish the automation as part of the work to implement each requirement.

Before you implement a requirement, the team can create examples of the requirement in action. Start by writing each example on a sticky note or on a whiteboard.

Building these examples together helps the team understand the requirement more clearly, and to know that others on the team also understand the requirement (in a reasonably similar way).

Begin the implementation process by automating the examples as automated tests. You can go a long way here before the implementation exists.

First, recast each example in some executable form (e.g. Cucumber's Given/When/Then form or even a plain old JUnit test method). Take care to retain the meaning of the example (because it forms the basis of the team's understanding of the requirement).

Next, implement the test steps as far as you can. You can at least declare a method to implement each test step in your test.

You can go a very long way even before you implement the requirement. The trick is to pretend (as far as you can) that the requirement were already implemented, and write the test.

Of course, the limit here is that there is no implementation to interact with, so you'll be missing some information. That's okay. Write your test as if you had that information.

In places where you've declared some test helper method but don't yet know the details to implement it, make the method throw some kind of "not implemented yet" exception.

In places where you know you'll need some kind of identifier (e.g. to identify some yet-to-be-implemented GUI element), create a variable or constant and give it an obviously bogus value as a placeholder.

Later, as you implement the requirement, you will fill in these methods and variables with the appropriate information. And then your test is ready to run.

Start automating right away. Almost all of the work of automating the tests can be done before the requirement is implemented.

And the work to automate the tests helps you and the team to understand the requirements.

So start automating at the moment when the requirement becomes the next requirement to implement.

share|improve this answer
add comment

You need some kind of stability otherwise you spend too much time refactoring tests. It depends what the end goal is and how you are contributing to meet that. If you can test an API or commandline before a UI arrives then you may be able to complete a lot of functional testing and have performance tests standing ready.

share|improve this answer
add comment

How to estimate how is good future application to automate it (as there is no application yet)

Its not about estimation but selection of tool.

Typically if team knows what to test, it has some ideas about what should be expected out to automation tool. Personally I would like to list all parameters and criteria which I would like to see in tool and assign weights to it. E.g.

  • How easy to add new case
  • How easy to modify the tool or extend the tool
  • How costly the tool is
  • How much effort will require in maintaining or am I better of developing in house tool.
  • Is team familiar/comfortable with tool, its reporting
  • How good debugging support tool has.

Once you have relevant parameters list is ready, go ahead evaluate all possible tools which you get by doing google, talking with peers, by asking team-members who earlier had worked with some automation tools, if possible get the demos and try to see if it is useful This exercise will help in getting management buy in as well it will help you in identifying what is possible to automation and what you would like to do manual way.

. Generally - when exactly to start automated testing - exact software life cycle point ?

If you are following Agile or continue integration software methodology, better start planning automation when you start planning for testing. Try to implement automation as soon as you have some steady results are available.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.