I'm automating testing for a fairly complex system involving multiple VMs, routers, and switches. Because of this complexity, the setup/teardowns sometimes fail (a node did not boot up correctly, or one of its component is messed up for example), so we tend to add verifications during the setup : is DHCP running ? is OSPF converging ? etc.
It's good because :
- when setup fails, we can just cancel all the tests, saving time.
- when teardown fails, we perform additional cleanup before the next testsuite
But it also has many drawbacks :
- the fixture scripts are more complex
- the fixtures take longer
- the testing and fixture code are not isolated
- it causes code duplication, because some verifications we do in the fixtures are also embedded in some other tests : the basic DHCP verifications we do in our setup for
testsuite_ospfare also part of
- it's hard to debug an issue that happened during fixtures, because as already said, they are fairly complex
What is usually considered to be the best practice ?
Answers are suggesting that I should mock components that I am not actually testing, but I don't think that this is something realistic in my case. I didn't want to be specific about what I am testing, but maybe I should have.
I am testing an SDN solution made of :
- a policy engine, controlled by a rest API.
- virtual and/or physical openflow switches
For testing purpose we also have :
- physical and virtual machines connected to the nodes
- a traffic generator
Let's say I want to test QoS on the virtual switch. My setup consists in :
- configuring a network underlay
- configuring the policy engine, the constrollers, and the switches and checking that they see each other correctly (openflow, json-rpc, and tcp sessions are up, ntp is synchronised, etc.)
- starting VMs, configuring policies to allow them to talk to each other or not, and checking that they are up and able to ping each other.
Then I start testing QoS, which involves all the components of the chain : I configure QoS rules on the policy engine via the the rest API, the controllers pull them and configure the switches via openflow, and then I can do my provisionning or traffic tests.
As you can see the setup includes many verifications, which was the point of my initial question.
Now, I think the answers given (that I should mock some components depending on what I am testing) can't apply here because :
- Although I am doing feature testing, a feature generally involves the whole system, not only a single component.
- Even if I can isolate the components I am testing, it's pretty difficult to mock an openflow controller for example.