What to do if a QA tester mistakenly deleted a table from the QA Database?

Question

QA tester mistakenly deleted a table from QA testing env database. Should this be a big deal?

Answer 1

If you read enough tech Twitter or other tech blogs, you'll come across the sentiment:

"You're not a true software developer until you delete production!"

Yes, QA testers are developers. And while you didn't take down production, it's close enough!

There is also the old XKCD comic about dropping a database.

This is to say, it happens and it's better to do so in an isolated environment like a QA environment instead of production! Personally, I've done something similar with an rm -rf command of the production website files from a Linux terminal. Scary? Yes! But we were able to recover it pretty quickly.

Keep in mind, that the QA environment is supposed to be used for testing and to find ways the software is likely to fail, but do so in a safe and protected way.

I say protected because this environment isn't supposed to be pristine. It's not production and it's always in flux. You deploy changes to it all the time with untested code/features, with bug fixes, etc. To me, the worst thing you've done is cause some delays and inconvenience to the team. So no, this should not be a big deal. The database should either have a backup which you can restore from, or a way to start fresh. I've been on teams where we had a fresh virtual QA environment for each Jira story/feature. This was nice to have. Even if you're not starting fresh each time, cleaning up the QA environment and starting fresh can be good!

How can you recover from this? Talk to your devs or your DevOps team to explain the situation and they can restore the database or setup a fresh database install!

More importantly, what can you learn from this?

• Was this database drop an accident? Were you not paying attention? Did you not understand the SQL command? You're likely feeling embarrassed or ashamed of doing this. Commit to understanding why this happened, to learn from it so it doesn't happen again in the future.
• Was this on purpose? There are forms of testing where this is a legitimate edge case to test to see how well you can recover from it. This can be a form of chaos testing, performance testing, or security testing (see OWASP top ten). If this is the situation, great, you found a major highly impactful but that needs to be addressed immediately!
• If you've already talked to the team, what did they say? What was the feedback? Were you blamed for this? Did they just shrug and say, "it happens, no big deal?" How fast were you able to recover the database? Can you highlight ways the process can be improved to recover faster or to prevent it from occurring again?

Ultimately, this is a learning lesson for you and the team.

Answer 2

It shouldn't be a big deal.

Consider another event. You deploy a build to your QA test environment and it has a defect which leads to rows being deleted or even a table being dropped. Should this be a big deal?

You should be able to somehow restore data in a QA environment. This could mean having a "golden" dataset that you can restore or having scripts that populate the database or snapshot backups that can be restored from early points in time or ways to drop and fully recreate the necessary data that can then be populated manually. It depends on the complexity of the system you are testing.

Answer 3

If they were not supposed to be able to delete a table, then your QA tester has done their job. That's a pretty major bug and potential security hole; report it, and give your QA tester a gold star.

If they were supposed to be able to delete a table, simply reset your test environment. QA environments are meant to be trashed. QA testers should be able to get a fresh test environment with a push of a button. They should be doing this after every test run to ensure tests don't interfere with each other; each test should be isolated.

Answer 4

It depends on the context. But not in a bad way for your QA. First of all, it could be a good panic test, I would say. :) If this is a table that should not be deleted, how did it happen? In a good db model, tables can't be dropped that easily and you always have way to return/restore/migrate/etc easily again.

Answer 5

I would fire such a QA ;-)

Jokes apart, such things happens.

So, it's a big deal if:

1. you are not prepared for setting up test environment (basically, answer from @ThomasOwens)
2. you have not constrained access to parts of test environment (answer from @spikey_richie)

Sometimes QAs help discover issues not only in the system under test but also in the testing infrastructure. I would thank this QA and sit together to plan how to avoid such issues in the future (see 1 and 2).

Answer 6

It could be considered an extreme negative test. If you don't want to have QA perform these sorts of tests, you should provide different logins with different levels of permissions.

Answer 7

What is the scope of this testing?

If the person doesn't need to be performing destructive tests, then I'd question if they should have access to the database to do this.

But at the same time, test environment setups can be automated; they can be easily backed up. So it may be that if this is a mistake, the tester can revert back to the previous days database, or they can re-build the db from scripts.

Answer 8

Couple of things.
First of all, restore your database from a backup. If you don't backup your test database, simply reset it. If this means that you have other testers who have lost quite a bit of work, well, maybe consider a backup strategy for selected databases. Learn from every mistake.

Next, re-asses your permissions. I firmly believe that QA should absolutely have the ability to do this. However, it should also be restricted. Create groups for each of your major permission groups, Read, Write, Alter, Admin, etc. It's fine for all of your QA's to be in the Read and Write groups. The other groups, only have people in those groups when required. Maybe keep a couple of people in the group all the time that can perform ad-hoc operations quickly without your support or whomever to add the person to the group. Yes, it can be slower like this, but, on the teams where we've done this, we found less 'accidents' and didn't actually slow our velocity so long as we were planning correctly.

So long as the deletion wasn't malicious, it was a learning experience, either from them, or, you as a company.

Answer 9

First of all, Don't panic, it's a QA environment, not the Production environment, The Database can be restored from the backup.

Now first check, if the QA engineer, who has deleted the table has the right to delete it or not, if not, and still, he can delete then it's a P0 bug and it should be addressed immediately.

You should also check the same issue on other QA environments as well.

However, here I would say that a QA engineer has done a great job, QA is all about trying to break the application to make it a more robust and high-quality application. Software qa consulting members are trained to do so.

QA environments are meant to be tested thoroughly with random testing. And if delete rights are given to the QA engineer to test delete scenarios, then there must be some training document having steps to restore the table or any kind of KT recording should be there for the engineer to simply restore the table. It's not a big deal. So depending on both cases, we can conclude, whether is it a bug or not. Further, there must be some log location as well where you can find out, how the table got deleted, and which user deleted it along with other details.

Answer 10

QA users should not get the rights to delete any table. Before testing, Check list should be prepared for the ways of testing in the database. If the table was deleted, it should be recovered using the DB admin.