0

Let's say we have a piece of Software, with a bunch of constants in a configuration file that sets certain arbitrary values, like a timeout of 300 seconds.

Now we have a script for automated checks in form of unit tests of the software, one of them being a check for if the timeout works properly.

Most of the time, I see the timeout value being hardcoded as a magic number into the test script, as the developers try to separate software from tests as much as possible.

But in my opinion, the configuration of the software is not part of the software itself, as the values are arbitrary, and thus not only could be used, but actually should be used by the tests.

The advantage would be, that if the timeout values change, no test would need to be adapted at all.

But all developers are reluctant to do that, any idea why?

0

The developers on my team (and the testers as well) generally do exactly this (pull the value we are checking from the config to ensure that the feature works as designed). We also have a suite of configuration unit tests to verify that the values are set as expected for key values. We've enjoyed the benefits you mention in the majority of our tests, while limiting hard-coded configuration checks to a small set of very fast tests. For system integration / acceptance tests we grab the configuration from the product, to avoid duplicate configuration in our test configs.

The largest benefit is a separation of concerns. A good test should test one thing only. One test verifies that the feature works (the timeout happens when the config says it should); another test verifies that the value is correct. Hard-coding the value is testing two things at once.

While there is a risk of getting the values wrong twice, hard-coding the values in multiple tests doesn't really help with this (the test-writer is likely using copy-paste if they get it wrong twice, and will continue doing so if doing it more than twice) and does increase the maintenance costs of the test suite significantly, which discourages thorough testing.

Some of the configuration tests can look a bit silly; it's almost redundant to just look up a configuration value and check that it equals a hard-coded value. However, they do work and have saved us from bad config changes; and more elegant tests can also verify logic and configuration consistency (e.g., ensuring that no production environment URLs are in any non-prod configurations).

ETA: Testing configuration permutations does not generally make a ton of sense for our particular project. On projects where it did matter, I still took this approach and use the product configurations in the tests (increases code reuse with non-configuration tests). However, I also added asserts to verify the values under test are what the test expected, and those asserts had values or logic about the values hard-coded / stored in test configuration. I really don't see why you would hard-code something you could pull from a config unless the correctness of the value itself was currently under test.

| improve this answer | |
0

When the same value is checked all the time autotest becomes nearly useless (no new knowledge of the product). Unless you check that the default value of timeout is set correctly in config-file. And in this case configuration value could not be used by tests and your devs are right.

But you are checking that the timeout works properly so it's not good for your autotest to check only the value provided by default.

If you have one check of this timeout in your test suite it would be preferable to check a randomly set timeout within a range of valid values. And after the test reset timeout to the system default value (300 seconds) or to the value before test (cause generally autotest should keep software in the same state as it was before test, if they are not integrated in dependent sequence of tests).

If you have two check, another one could be done with invalid value (if the system supposed to handle this case somehow).

There could be even more checks e.g. if error in this configuration feature has high impact on company reputation.

| improve this answer | |
  • Yeah, I didn't mean to just use the value from the config file, but you can use it for creating relative values with offsets, like Timeout + 500ms, -500ms, etc. – tuexss May 12 '16 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.