I'm automating testing for a fairly complex system involving multiple VMs, routers, and switches. Because of this complexity, the setup/teardowns sometimes fail (a node did not boot up correctly, or one of its component is messed up for example), so we tend to add verifications during the setup : is DHCP running ? is OSPF converging ? etc.

It's good because :

• when setup fails, we can just cancel all the tests, saving time.
• when teardown fails, we perform additional cleanup before the next testsuite

But it also has many drawbacks :

• the fixture scripts are more complex
• the fixtures take longer
• the testing and fixture code are not isolated
• it causes code duplication, because some verifications we do in the fixtures are also embedded in some other tests : the basic DHCP verifications we do in our setup for testsuite_ospf are also part of testsuite_dhcp for example.
• it's hard to debug an issue that happened during fixtures, because as already said, they are fairly complex

What is usually considered to be the best practice ?

I'm automating testing for a fairly complex system involving multiple VMs, routers, and switches. Because of this complexity, the setup/teardowns sometimes fail (a node did not boot up correctly, or one of its component is messed up for example), so we tend to add verifications during the setup : is DHCP running ? is OSPF converging ? etc.

It's good because :

• when setup fails, we can just cancel all the tests, saving time.
• when teardown fails, we perform additional cleanup before the next testsuite

But it also has many drawbacks :

• the fixture scripts are more complex
• the fixtures take longer
• the testing and fixture code are not isolated
• it causes code duplication, because some verifications we do in the fixtures are also embedded in some other tests : the basic DHCP verifications we do in our setup for testsuite_ospf are also part of testsuite_dhcp for example.
• it's hard to debug an issue that happened during fixtures, because as already said, they are fairly complex

What is usually considered to be the best practice ?

EDIT

Answers are suggesting that I should mock components that I am not actually testing, but I don't think that this is something realistic in my case. I didn't want to be specific about what I am testing, but maybe I should have.

I am testing an SDN solution made of :

• a policy engine, controlled by a rest API.
• controllers
• virtual and/or physical openflow switches

For testing purpose we also have :

• physical and virtual machines connected to the nodes
• a traffic generator

Let's say I want to test QoS on the virtual switch. My setup consists in :

1. configuring a network underlay
2. configuring the policy engine, the constrollers, and the switches and checking that they see each other correctly (openflow, json-rpc, and tcp sessions are up, ntp is synchronised, etc.)
3. starting VMs, configuring policies to allow them to talk to each other or not, and checking that they are up and able to ping each other.

Then I start testing QoS, which involves all the components of the chain : I configure QoS rules on the policy engine via the the rest API, the controllers pull them and configure the switches via openflow, and then I can do my provisionning or traffic tests.

As you can see the setup includes many verifications, which was the point of my initial question.

Now, I think the answers given (that I should mock some components depending on what I am testing) can't apply here because :

• Although I am doing feature testing, a feature generally involves the whole system, not only a single component.
• Even if I can isolate the components I am testing, it's pretty difficult to mock an openflow controller for example.