Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Problem

We and devs are an Agile team and finally managed to work in parallel on development and testing within an iteration. When developers were working on new features, we were automating test scenarios, preparing test data, implementing fixtures and talking to developers to expose necessary interfaces for those fixtures. So, when developers delivered the features implemented, we already had tests automated to run. That was very encouraging.

However, we found some tests failing because of errors in the tests, like:

  • NullPointerExceptions in fixtures
  • Some configuration required by the tested component was not provided by the test
  • Some small differences in naming convention of expected results (devs called them differently)

Since devs delivered the feature on the last day of the iteration, we didn't have much time to fix our tests and stories were carried over for the next iteration. What are the ways to address this problem?

Proposed solutions

  1. Make devs deliver stories for testing quickier, so we have time to verify our tests as well.
  2. Test our tests before getting the feature from devs. Some problems could be, in fact, avoided with unit testing of fixtures, but others would require mocking the system/feature. That's a lot more work from our side.
  3. We focus on automating integration tests. Those are quite high-level tests, but maybe we are still too tightly coupled with interfaces which the devs expose to us and instead should we focus on less-coupled end-to-end tests?
share|improve this question
1  
It sounds like you suffer from the same kinds of pains that developers face during integration testing. –  user246 Jun 3 '13 at 19:19

2 Answers 2

The situation you had given here is really generic and it is in many organizations. For your above problem I can suggest some possible solutions which I had seen.

  • According to your solution 1, you can ask for the developers to do their dev tasks quickly if possible so that maximum testing coverage can be given.

  • In case of new feature which the devs will be adding to your existing system, make sure that after their development you can do : Regression Testing (to check that the new feature will not affect the existing functionality of the system) and also make sure that the new feature is completely tested by your QA team until it is completely stable.

(I know that in Agile, you have much less time for complete testing but that is how the quality can be best achieved without having the stories/tests to be carried over the next iteration)

Rest your approach is perfectly fine of developing the automated tests until the fature is being developed by the devs. That's really awsome and helpful.

The main advantage you may get by having these solutions implemented is completely stable feature can be released and probability of automated test failure will be less.

Second Answer

Let me tell you the workflow for my team in short.


1. Developers (abroad) first develop the feature and then it is ready to test.
2. Then that feature comes to a feature testing team.
3.The feature testing team is completely responsible for manual testing first, and then the development of automated tests.
4. After that the feature testing team will declare that feature as stable.
5. The new feature can then be released and before releasing it may come to Regression Testing team for regression.

In this way, the testing can be done.

Thanks.

share|improve this answer
    
Can you elaborate on your second point? Particularly, what do you mean by "make sure the new feature is completely tested until it is completely stable"? Do you mean I should start automation only when the feature is developed and stable? –  dzieciou Jun 3 '13 at 17:28
    
Yes. This is because once the feature is stable, you have the expected behavior and based on that you can develop the automated tests so that the probability of failure is minimal. –  talktokets Jun 4 '13 at 3:09
    
So what QA should do before devs complete development? My understanding was that QA works in parallel to feature development. Once features gets completed, there's not much time within iteration to automate tests. –  dzieciou Jun 4 '13 at 4:24
    
I think there is some misunderstanding. You are right, but my point of view above is regarding the development of automated tests. When the feature is being developed by devs QA is involved in Manual Testing. While doing manual testing, QA usually opens new issues those are to be handled by devs during development. Is that answered your question? –  talktokets Jun 4 '13 at 4:35
    
This answer my question. Thank you. But this approach means I will have even less time to automate tests, as I will be testing the same feature both manually and automatically. How do you handle that in your team? –  dzieciou Jun 6 '13 at 5:36
  1. You can unit test your own tests as well (we start sounding very meta here - test your tests that test the code) :)

  2. Provide sample input that your tests can use and parse as what should be expected valid/invalid inputs. Confirm the output.

  3. Build tests for the software that expects input. Treat it as test-driven development - the software passes when it passes these regression/acceptance tests that you've constructed.

  4. as devs build or design features, they could give you the component names, or settle on a naming convention that they sign off on. That way they have to stick to it - or they've failed your test for having the wrong component name.

  5. have one iteration with manual tests, and propose automating the previous feature in the next iteration. It means you're one step 'behind' in a way, but it also means you have the option of always having something to test against. Depends what your deadlines are like and how strict your company is with Agile workflow.

The joys of Agile mean you may have trouble fitting this into your environment, as we all do, but it's a case of working with what you have and seeing which best fits your environment / working habits.

share|improve this answer
    
Re 2: You meant "provide to devs"? –  dzieciou Jun 3 '13 at 17:21
    
Re 3: We sometimes do so, but I think it does not resolve the problem alone. Passing or failing test won't drive the development correctly if the fault is in the test, not the system under test. –  dzieciou Jun 3 '13 at 17:22
    
Re 5: I think that would make sense in GUI-based system. We have backend system tests, so automated tests are often easier to perform than manual ones, so in our case that could be a waste of time. But I think the idea here would be to make tests that are more black-box, more decoupled from the code, and so not that much depending on internal details. –  dzieciou Jun 3 '13 at 17:25
    
@dzieciou - for 2 - no, I mean to your tests - to test your tests, as it were. For 3 - yeah, it's tricky, unfortunately, but it works for some. And at least it'll show a fail - you have to assume, at least initially that your test is correct once you've built it, and then you can investigate once a fail occurs. For 5, backend does make it a bit easier, especially with frequent code drops. –  Mark Mayo Jun 3 '13 at 23:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.