9

In my limited experience, configuration changes (small 1-3 line changes to a project config file), either completely break the program or do not seem to bring noticeable changes to the user-experience.

  • As a QA should I be worried about my developers telling QA that only cursory smoke testing is required for a small configuration deployment?
  • Are there any implicit, more nasty, and intermittent types of bugs that could be introduced through configuration changes?

I want to be a thorough QA contributor but also allow my team's developers to move quickly through the deployment and CI pipeline.

This is the particular change that brought about my question:

from

<add key="LightningQueueSettings.OldestMessageInProcessedHistory" 
value="10m" />

to

<add key="LightningQueueSettings.MapSize" value="1073741824" />

This occurred in a JasperService.exe.config file. This change allows tons more messages from our database to propogate to this queue (it was being overflowed).

|improve this question
  • Can we please have more specifics? How "small" is it? Why do you think it is "small"? – Yu Zhang Jul 7 '17 at 22:20
  • I think it is 'small' because it doesn't seem like it touches other code. I need to research this JasperService more but since I am new to the team I did not want to stop a deployment because of my ignorance. – gordlonious Jul 8 '17 at 5:04
  • 1
    Since that's a completely different key, it looks like it might hit a slightly different code path for determining which records to process off the queue. If you haven't tested what the MapSize setting does, you will likely want to at least do a little bit of testing to verify that this setting does what everyone thinks it does . . . – ernie Jul 10 '17 at 0:32
7

It all depends on what you're configuring, and if the configuration mechanism itself was tested before.

If it's something small, like a timeout interval, number of retries, etc, then I'd agree it likely doesn't need full testing before changing the config in prod.

On the other hand, if it's bigger/riskier (i.e. changing a DB connection string to a new instance), then exercising it with the new configuration might make sense. Even in this case though, if it was tested before, then a smoke test can be sufficient if you've verified the DB copy is correct (i.e. all the data is there, stored procs, ACLs, etc).

It's all about gauging the risk. My gut on this is as folks move to more cross-functional teams, embedded testers, etc, there should be near consensus on whether something is low risk or high risk and what kind of testing it might need. Sounds like you're still seeing your role as a bit of a gatekeeper, which is fine. Just make sure you raise the flag if you don't agree with the team, if you think there's significant risk.

|improve this answer
2

I think your question:

What should a QA in software be worried about when a small configuration deployment is being pushed without testing?

Already contains the answer:

When a small configuration deployment is being pushed without testing, a QA in software should be worried.

I myself have seen more than 1 instance of a 'small change' having disastrous consequences and completely breaking the entire application. In fact "small change, shouldn't affect anything" are now trigger words for me to test. Of course most of the time you will probably be fine not to test, but that one time it breaks... can be disastrous to your business... as many companies have found and are in the news almost daily with the usual byline 'due to insufficient testing, x, y, z happened'.

Some the areas that can break when releasing software with untested deployment changes:

  • scripts
  • sessions
  • asset display
  • authentication
  • dependencies
  • code compilation
  • asset compilation
  • performance issues
|improve this answer
2

I've seen too many times when a tiny, seemingly insignificant change resulted into a broken application.

No matter the size of a change, you've got a new version of your application - it should be tested as thoroughly as after any other change.

Addressing the comments

Depending on the size and criticality of the application, established continuous build&test practices and available resources, testing each and every release might or might not be practical. But, if it does not slow the development cycle down, aim for not differentiating changes by the potential impact.

|improve this answer
  • 1
    Testing every change thoroughly will bring development to a halt. It makes sense in a critical application, but not in a free service. If anything the problem implies on a problem with the development culture- the change wasn't reviewed, there are no sanity checks etc. – Rsf Jul 8 '17 at 5:15
  • @Rsf not necessarily to a halt, depending on a size of an app, build&test processes, how often the changes happen etc. I just wanted to stress the importance of that "tiny change may have a big impact", nothing else. – alecxe♦ Jul 8 '17 at 5:20
  • The emphasis should be on testing each release. If a release contains 25 changes that doesn't mean test each one, you can run 1 set of user acceptance testing to cover all of them in one go. Having said that each of those 25 changes should be accompanied by unit tests. – Michael Durrant Jul 8 '17 at 11:02
  • 1
    Also most of the testing should be automated to help address the speed and turnaround issues. – Michael Durrant Jul 8 '17 at 11:10
1

OP: you seems to see your role in QA as Quality Assurance. That is wrong. You are in Quality Assistance business.

Your role is to keep communication between all interested parties, so concerns of all parties are heard and risks are managed. It is not up to you to decide if configuration is too trivial to test: there are people with much more relevant information and skills: your developers, your system and DB administrators, etc. Your role is to make sure they all stakeholders agree on the level of risk, and business is accepting their decision, while all steps of the process of developing, testing and deploying the bug fix are followed (or exception is granted for valid business reasons).

Even if all stakeholders agreed that no testing is necessary, quite often as QA you may want to make some testing. More than once in my experience such testing uncovered a real bug. :-) This is more related to having a "feel" for your application, knowing where dark corners are where bugs might be lurking. Or you know that some developers make better own testing, while others rely more on QA to test the heck out of the app.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.