Should I perform tests during fixture?

Question

I'm automating testing for a fairly complex system involving multiple VMs, routers, and switches. Because of this complexity, the setup/teardowns sometimes fail (a node did not boot up correctly, or one of its component is messed up for example), so we tend to add verifications during the setup : is DHCP running ? is OSPF converging ? etc.

It's good because :

• when setup fails, we can just cancel all the tests, saving time.
• when teardown fails, we perform additional cleanup before the next testsuite

But it also has many drawbacks :

• the fixture scripts are more complex
• the fixtures take longer
• the testing and fixture code are not isolated
• it causes code duplication, because some verifications we do in the fixtures are also embedded in some other tests : the basic DHCP verifications we do in our setup for testsuite_ospf are also part of testsuite_dhcp for example.
• it's hard to debug an issue that happened during fixtures, because as already said, they are fairly complex

What is usually considered to be the best practice ?

EDIT

Answers are suggesting that I should mock components that I am not actually testing, but I don't think that this is something realistic in my case. I didn't want to be specific about what I am testing, but maybe I should have.

I am testing an SDN solution made of :

• a policy engine, controlled by a rest API.
• controllers
• virtual and/or physical openflow switches

For testing purpose we also have :

• physical and virtual machines connected to the nodes
• a traffic generator

Let's say I want to test QoS on the virtual switch. My setup consists in :

1. configuring a network underlay
2. configuring the policy engine, the constrollers, and the switches and checking that they see each other correctly (openflow, json-rpc, and tcp sessions are up, ntp is synchronised, etc.)
3. starting VMs, configuring policies to allow them to talk to each other or not, and checking that they are up and able to ping each other.

Then I start testing QoS, which involves all the components of the chain : I configure QoS rules on the policy engine via the the rest API, the controllers pull them and configure the switches via openflow, and then I can do my provisionning or traffic tests.

As you can see the setup includes many verifications, which was the point of my initial question.

Now, I think the answers given (that I should mock some components depending on what I am testing) can't apply here because :

• Although I am doing feature testing, a feature generally involves the whole system, not only a single component.
• Even if I can isolate the components I am testing, it's pretty difficult to mock an openflow controller for example.

Answer 1

To expand on bish's answer a bit:

Your fixtures aren't supposed to be testing end-to-end functionality of all the subsystems - that's something for manual testing and/or automated functional testing and should be done as little as possible (because it's slow and prone to error - you do that as the top-level sanity testing).

Each fixture is - or should be - testing a single subsystem. If, for instance, you're testing the main controller of the system, you don't actually care what the various nodes are doing: you care that the controller responds correctly to various inputs from those nodes.

So you write mock implementations of the node interfaces (or use the ones the unit tests use) that will respond to calls from the controller instead of needing to rely on the real nodes.

Similarly, if you're testing how Node A responds to a controller call, you use a mock of the controller interface to send the call you want to test.

If you keep all your mocks in helper libraries, you're not having to reuse code because you're using the library calls from whichever node's tests you're running.

It's more work up-front to get this in place, but a lot less work long-term because it allows you to simplify your automation fixtures.

UPDATE - Added in response to the edits to the question

I'm leaving what I initially posted because it could be helpful to someone else who isn't in quite such a difficult situation.

In view of the added information, I'd probably look at it this way:

• Libraries - If you need to use it in more than one place, extract it into a library and keep your code as DRY as you can. Using the example you gave, have a separate DHCP library that can be called by either the OSPF or the DHCP tests. That reduces the likelihood of changes to one set of duplicate or near-duplicate code not propagating to a different set.
• Look to the 80/20 rule - I've said this in several places now, but it bears repeating here. With the complexity of your system and the speed and false-positive issues that come with not being in a position to mock and isolate, you want to focus on the 20% or so of the system that gets the most use. I'm going to guess that the actual service and communications is the part that you want to focus on, which leads to the next point.
• Use "canned" configurations - In a similar situation to yours (in my case there was no unit testing and no way to isolate anything), I used pre-built configurations a lot, but ran a several-hour configuration run once a week to rebuild the configuration data set. In essence this means that most of the time, you'll leave your components and VMs running and your test run will start with checking that everything is communicating as it should be, then perform the QOS test.
• Consider dependent tests - This is an ugly situation, and one I've been in before. You're pretty much forced to run end-to-end tests by your circumstances, and those are inherently slower and more error-prone. One way to deal with the speed issue is to build in dependencies and make your validations smarter. It does mean if something wipes the test run early, the later tests don't run, but under most circumstances you can do a lot more testing if you do the setup once and change configuration on the fly as the test runs proceed.
• Do as much mocking as your situation permits - because it's faster and removes a point of failure.

Whatever method you choose, don't be afraid of getting it wrong - no matter how you handle this, you're going to be refactoring and finding better ways to handle the system for as long as you're testing it. It's better to have error-prone automation that you understand (and believe me, I've been there) and can improve than nothing at all.

Answer 2

To isolate your subsystems you should use mocks to give wanted answers like "connection is up / connection is down" with just 'return true / false' and test the behavior. With fixtures you know all states of your systems and are not depenend from the actual connections etc.

As I am writing from my mobile phone I can't give you some external links and I'm sorry for my formatting.

Answer 3

Coming back to share a solution that I find suitable. I wrote a framework based on nose2 (a python testing framework), that allows me to write hierarchical setups and teardown, thanks to such, a nose2 plugin.

start_vms
  test_vms_are_started
  configure interfaces
      test_interfaces
      configure_dhcp
          test_dhcp
          configure_nat
              test_nat
              [...]
                 # then starts the actual test of my feature
                 test_1
                 test_2
                 test_3
              [...]
          deconfigure_nat
      deconfigure_dhcp
   deconfigure_interfaces
stop_vms

This way I achieve :

• setup/teardown and test codes are separated
• the setup/teardown scripts are small independant bites that I can re-use, thus limiting duplication.
• I know exactly where things fail

The main drawback of this is that setups take a lot of time. So for script development, and quick regressions, I'm using canned configurations as suggested by Kate.