Deleting created data from Automated Tests?

Question

So what's the best way to go about this? Lets pretend im testing localhost, or a remote site. That either has a mysql or postgreSQL database.

I have a few automated tests that create things (X,Y,Z widgets). However deleting these would require clicking a link based on the xpath where it resides on a list of widgets to delete.

This seems....very brittle.

Is there a better way around this? Our main goal is to strengthen regression testing after pushing a new feature to a site that is sort of in "maintenance/final mode"). So resetting the database seed file is possible (This is mostly rails sites).

I would use database cleaner, but I cannot find a good tutorial on using it with capybara/poltergeist. Plus im not sure it works on remote sites (Since im not spinning up a capybara server, im hitting a remote site even if it's just localhost).

Or are tests that "add" things just not a good idea in general, or I just have to deal with using brittle tests to delete the items?

Answer 1

Do not clean data

Usually it adds more problems than it solves:

• If you clean data in tests, the tests may fail (e.g. process is killed) and thus the DB won't be clean.
• If you clean data with external scripts - you'd have to write and maintain them separately. Even more - you'd have to not forget to run them from your local machine.
• This doesn't solve issues with concurrency. Imagine you, your colleague and CI running the tests at the same time. They will step on each other toes.
• If a bug is found, you want to keep the records to investigate. So if you decide not to clean in case of failure, the environment will be unavailable for testing until the investigation is over.

Isolate your test data

• Create new resources instead of reusing old data.
• Randomize your testing to generate new data that doesn't clash with old records. E.g. when you create a user: username = alphanumeric(1, 30).
• Follow the test pyramid and do not write a lot of System tests. Write more of Unit and Component tests that do not leave traces after themselves.

Q1: Leaving artifacts you add one more cause of failure, which is not relevant to production code
A1: If we concentrate only on bugs in code - the tests may fail both because of bugs in production code and in tests. By randomizing you make the state of the SUT vary which may find the bugs - those that you could've missed accidentally. From this perspective yes - you add more cause of failures. But finding bugs (both for prod and for test code) is beneficial, so we should be good from this perspective.

A case where this can add troubles is performance (see below).

Q2: Leaving artifacts may have performance implication - DB cannot contain infinite amount of data
A2: One of the good parts of not-cleaning the DB is that with time you're going to have big volumes of data and that would also implicitly test the performance. But if no one cares about the performance it may cause the flakiness to the tests (e.g. if that dialog doesn't show up for several secs). What we can do is to periodically clean the env - on one of my projects we did this once in couple of years.

You should also note, that if we do a proper Test Pyramid we don't have a lot of system tests that leave the state - therefore we won't be overwhelmed with the data any soon.

Answer 2

Why not just delete whole database and copy it from clear master copy?

Unless you want to test deleting those artifacts, wiping the whole database and replacing it with a clean copy seems fastest and rock-solid solution.

re @Stanislav Bashkyrtsev answer: but to create randomized test which adapt to existing data is more work (== more possibility to add a bug to your tests, adding false positive or negative), and artifact leftovers after failed test might break tests which would pass on clean data. Leaving artifact you add one more cause of failure, which is not relevant to production code.