Where should test fixture code live?

Question

I'm working in a primarily Windows / .NET system. I've written a test framework in C# that includes a test harness that runs fixtures (anything that implements the ITestFixture interface). These fixtures are the interface between my framework and the SUT (System Under Test), and sometimes they have dependencies on SUT methods or objects. Nothing else in my framework depends on the code of the SUT. I am testing multiple projects, which have relatively little to do with each other, and will be testing more and more over time.

Right now, I have a test solution in .NET that has all of my test framework code . . . and all of the fixtures, and all of their dependencies. The result is a giant test framework solution with a ton of dependencies. This doesn't seem like a very good way to do things in the long-term.

Another alternative that we are discussing is having me create a "Common.Test" project that essentially includes the interface for the fixtures, and maybe a few utility functions. Developers would then include a "SUT.Test" project (where SUT is whatever the system under test is called) that includes their fixtures and depends on the Common.Test project. Fixtures would be compiled as part of their solutions, and the test harness would load their fixtures from their .dll files dynamically.

Do either of these options (fixtures in the test solution with tons of dependencies, or fixtures in the SUT code that are dynamically loaded by the harness) sound better than the other to experienced automation testers? Are there other options that I'm missing that should be considered? If you use fixtures, what does your team do?

Answer 1

Personally I use a combination of approaches.

I have all my tests inherit from a base test class, and it has a lot of the common objects and contains any setup and teardown code.

I also have an “observer” class that all the layers of my stack can reference and that contains any utility classes that are necessary for execution. The observer also serves as an in-memory data store for holding test data objects instead of having to pass them up and down between my layers.

This approach allows me to enforce separation of concerns where my tests no nothing of how the actual application is implemented, not the underlying automation engine that actually drives the application.

If you want to have a look it is open sourced at testingstax.codeplex.com

if I had to call external dll's and I want to be loosley coupled I would look to use .net reflection to load them dynamically on the fly. That way I can have my cake and eat it too :-)

Answer 2

In my company the test fixtures reside in a seperate folder called component_test_tools and it is created and maintained by the DEV. Our test case call these fixtures. When the application code/object changes and the DEV forgets to update the respective fixture, all the test cases calling those fixtures fail. The DEV needs to ensure that whenever he changes the code and objects, the fixture is also updated.

These dependencies caused some invalid failures in the test run. The testers depend on DEV to fix the fixtures which was a real issue that I faced. The DEV considered automation fixture bugs to be of low priority compared to application code bugs and it was challenging to get them attend to these issues.