3

So what's the best way to go about this? Lets pretend im testing localhost, or a remote site. That either has a mysql or postgreSQL database.

I have a few automated tests that create things (X,Y,Z widgets). However deleting these would require clicking a link based on the xpath where it resides on a list of widgets to delete.

This seems....very brittle.

Is there a better way around this? Our main goal is to strengthen regression testing after pushing a new feature to a site that is sort of in "maintenance/final mode"). So resetting the database seed file is possible (This is mostly rails sites).

I would use database cleaner, but I cannot find a good tutorial on using it with capybara/poltergeist. Plus im not sure it works on remote sites (Since im not spinning up a capybara server, im hitting a remote site even if it's just localhost).

Or are tests that "add" things just not a good idea in general, or I just have to deal with using brittle tests to delete the items?

1

Do not clean data

Usually it adds more problems than it solves:

  • If you clean data in tests, the tests may fail (e.g. process is killed) and thus the DB won't be clean.
  • If you clean data with external scripts - you'd have to write and maintain them separately. Even more - you'd have to not forget to run them from your local machine.
  • This doesn't solve issues with concurrency. Imagine you, your colleague and CI running the tests at the same time. They will step on each other toes.
  • If a bug is found, you want to keep the records to investigate. So if you decide not to clean in case of failure, the environment will be unavailable for testing until the investigation is over.

Isolate your test data

  • Create new resources instead of reusing old data.
  • Randomize your testing to generate new data that doesn't clash with old records. E.g. when you create a user: username = alphanumeric(1, 30).
  • Follow the test pyramid and do not write a lot of System tests. Write more of Unit and Component tests that do not leave traces after themselves.

Q1: Leaving artifacts you add one more cause of failure, which is not relevant to production code
A1: If we concentrate only on bugs in code - the tests may fail both because of bugs in production code and in tests. By randomizing you make the state of the SUT vary which may find the bugs - those that you could've missed accidentally. From this perspective yes - you add more cause of failures. But finding bugs (both for prod and for test code) is beneficial, so we should be good from this perspective.

A case where this can add troubles is performance (see below).

Q2: Leaving artifacts may have performance implication - DB cannot contain infinite amount of data
A2: One of the good parts of not-cleaning the DB is that with time you're going to have big volumes of data and that would also implicitly test the performance. But if no one cares about the performance it may cause the flakiness to the tests (e.g. if that dialog doesn't show up for several secs). What we can do is to periodically clean the env - on one of my projects we did this once in couple of years.

You should also note, that if we do a proper Test Pyramid we don't have a lot of system tests that leave the state - therefore we won't be overwhelmed with the data any soon.

  • I was more thinking about just wiping the DB, but you do bring up a good point. I guess im just not experienced enough with Capybara/Selenium to know "how" to do that yet (Randomizing test data) – Mercfh Jun 1 '16 at 14:03
  • @Stanislav Bashkyrtsev Leaving artifacts you add one more cause of failure, which is not relevant to production code. – Peter M. Jun 1 '16 at 17:08
  • @Peter Masiar, I've added more info to the question. – Stanislav Bashkyrtsev Jun 2 '16 at 5:40
  • @StanislavBashkyrtsev good points, although I don't agree with your perspective (but I think it also depends on the application itself) – FDM Jun 2 '16 at 5:44
  • @FDM, I haven't seen apps where it would be inapplicable. And as for the large and very complex systems - this is must have, the project will suffer without randomization and proper pyramid in the tests. There are times when you can't easily randomize (e.g. generating chemical molecules), but in most cases it can be used everywhere. – Stanislav Bashkyrtsev Jun 2 '16 at 5:53
2

Why not just delete whole database and copy it from clear master copy?

Unless you want to test deleting those artifacts, wiping the whole database and replacing it with a clean copy seems fastest and rock-solid solution.

re @Stanislav Bashkyrtsev answer: but to create randomized test which adapt to existing data is more work (== more possibility to add a bug to your tests, adding false positive or negative), and artifact leftovers after failed test might break tests which would pass on clean data. Leaving artifact you add one more cause of failure, which is not relevant to production code.

  • This is something i've considered, but im trying to keep it as low overheard as possible. But it's def. a possibility. – Mercfh May 26 '16 at 17:34
  • Anonymous downvoter, care to enlighten us of your reasons? – Peter M. May 31 '16 at 14:42
  • Yep. When I read this answer I found the idea as the worst possible one. Since it's so bad and so many people repeat this again and again, I downvoted. Then I realized that my downvote is not fair - it's not an offtopic and it tries to answer the question. But by that time it was late. SE won't allow to change the vote unless you edit your answer. Sorry for that, I feel bad :( – Stanislav Bashkyrtsev Jun 1 '16 at 16:30
  • Seems that you did some research on "not wiping DB". I think that people repeat it because it is common sense (at least IMHO). Do you have some references from experts why it is so bad? None of local experts expressed opinion either way so far. So I am curious what I am missing. – Peter M. Jun 1 '16 at 17:11
  • Um.. I don't know who you call the "experts", but I've been doing randomized testing for several years and I listed the problems in my answer :) As for other resources - a book on Continuous Delivery sheds some light on the test isolation: amazon.com/… I don't remember if it talks specifically about randomization, but this is an important book in the history of software development anyway. – Stanislav Bashkyrtsev Jun 2 '16 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.