How to test your tests without having the system under test?

Question

Problem

We and devs are an Agile team and finally managed to work in parallel on development and testing within an iteration. When developers were working on new features, we were automating test scenarios, preparing test data, implementing fixtures and talking to developers to expose necessary interfaces for those fixtures. So, when developers delivered the features implemented, we already had tests automated to run. That was very encouraging.

However, we found some tests failing because of errors in the tests, like:

• NullPointerExceptions in fixtures
• Some configuration required by the tested component was not provided by the test
• Some small differences in naming convention of expected results (devs called them differently)

Since devs delivered the feature on the last day of the iteration, we didn't have much time to fix our tests and stories were carried over for the next iteration. What are the ways to address this problem?

Proposed solutions

1. Make devs deliver stories for testing quickier, so we have time to verify our tests as well.
2. Test our tests before getting the feature from devs. Some problems could be, in fact, avoided with unit testing of fixtures, but others would require mocking the system/feature. That's a lot more work from our side.
3. We focus on automating integration tests. Those are quite high-level tests, but maybe we are still too tightly coupled with interfaces which the devs expose to us and instead should we focus on less-coupled end-to-end tests?

Answer 1

Although it is an old question, I would like to provide my thoughts on this for future readers.I was in similar situation numerous times, in fact, its very common situation for test engineers working in Agile.

Our journey was somewhat as below:

1). We all as a team realised this issue. Dev and test agreed and we planned to move large no. of tests from low/middle level (integration level) to unit level and aimed to have only 1 E2E UI test per user story.

2).Focused to design more black box automated tests without relying on the internals.This is true for any level of tests.

3).However, We made sure(by code review) all the boundary/edge cases are fully covered in unit tests only.Also in unit tests we found we needed to focus more to test public interfaces, not directly the internals.

4).We(testers) worked more closely with devs and helped them in developing/testing/reviewing unit tests.This period we moved more towards TDD(but not exactly). At this point, We also started thinking actually "Are we developers or testers"?

5).After few iterations, we ended up having the majority of tests as unit tests and fewer integration tests and very few E2E tests.

At this point as a team, we found ourselves in a far better situation and were overall satisfied.(Happy Ending :) )

Above all (I think which helped), we had to blur the boundary between who is "Developer" and who is "tester" in the team.

I believe Project Nirvana comes when quality becomes a mindset/culture in the team not just having an individual as "Tester".

Answer 2

1. You can unit test your own tests as well (we start sounding very meta here - test your tests that test the code) :)

2. Provide sample input that your tests can use and parse as what should be expected valid/invalid inputs. Confirm the output.

3. Build tests for the software that expects input. Treat it as test-driven development - the software passes when it passes these regression/acceptance tests that you've constructed.

4. as devs build or design features, they could give you the component names, or settle on a naming convention that they sign off on. That way they have to stick to it - or they've failed your test for having the wrong component name.

5. have one iteration with manual tests, and propose automating the previous feature in the next iteration. It means you're one step 'behind' in a way, but it also means you have the option of always having something to test against. Depends what your deadlines are like and how strict your company is with Agile workflow.

The joys of Agile mean you may have trouble fitting this into your environment, as we all do, but it's a case of working with what you have and seeing which best fits your environment / working habits.

Answer 3

The situation you had given here is really generic and it is in many organizations. For your above problem I can suggest some possible solutions which I had seen.

• According to your solution 1, you can ask for the developers to do their dev tasks quickly if possible so that maximum testing coverage can be given.

• In case of new feature which the devs will be adding to your existing system, make sure that after their development you can do : Regression Testing (to check that the new feature will not affect the existing functionality of the system) and also make sure that the new feature is completely tested by your QA team until it is completely stable.

(I know that in Agile, you have much less time for complete testing but that is how the quality can be best achieved without having the stories/tests to be carried over the next iteration)

Rest your approach is perfectly fine of developing the automated tests until the fature is being developed by the devs. That's really awsome and helpful.

The main advantage you may get by having these solutions implemented is completely stable feature can be released and probability of automated test failure will be less.

Second Answer

Let me tell you the workflow for my team in short.

1. Developers (abroad) first develop the feature and then it is ready to test.
2. Then that feature comes to a feature testing team.
3.The feature testing team is completely responsible for manual testing first, and then the development of automated tests.
4. After that the feature testing team will declare that feature as stable.
5. The new feature can then be released and before releasing it may come to Regression Testing team for regression.

In this way, the testing can be done.

Thanks.