Creating a dummy database for automated functionality testing

Question

I have a continuous integration build server (using Team City). When people check code in, I'd like to run a set of tests which run through a set of functional cases such as:

• Using an administrator account, I can create a document
• If I belong to the 'Editor' role, then I can access document x and document y.

and so on.

We have a number of unit tests which cover individual functions, but we really want this quick 'smoke test' to see if anything discrete has slipped through the net.

Should I:

1. Create a 'dummy database' with pre-populated users, permissions, documents, and create the tests to use data from this database?
2. Use some sort of mocking framework for this? What are the advantages / disadvantages here?

Or, is my thinking completely off?

Appreciate your thoughts.

Answer 1

If you are curious about the tradeoffs around mock objects, there are entire websites and books devoted to the subject. I will restrict my answer to the specific problem you described.

I never replace a real system with a mock system unless the real system does not meet my needs; a mock system requires time to write and maintain, and when a test fails, you will have to ask yourself whether the problem is in the product or in your mock system. You have not mentioned anything that would prevent using a dummy database for your smoke test. While you did not specify whose database technology you are using, most database vendors provide an easy and reasonably fast way to restore a database from a backup and to rename a database (e.g. from "DB_backup" to "test_123").

(If you suspect there are reasons why using a dummy database will not meet your needs, please revise your question to include those issues.)

Answer 2

One small thought. You might want to do both.

1. Testing against "production" data helps you find some problems.

2. Testing against a special mock database may help you test the limits.

Example of no2. (Part of it is real world, my experience). Everyone knows that there are a maximum of 24 hours per day. Or? Assume that you want to calculate hours worked. And I work past midnight. It might never have happened (yet) in production data, but one day it will. Or, well, how many hours are there in a day when day-light savings time is changed. There are probably both 23 and 25 hour days in addition to 24. A mock database could contain these "at the limits" data which you might not find in a production data.

I have learned the hard way to always define limits on data, and to prohibit input of data outside the limits. The 24 hour per day is one example of how thinking through the limits carefully showcases the use of defining limits. Well, my two cents.

Answer 3

What you can do is run your tests within a transaction and assert your expected outcomes. When you are finished you can roll back the transaction. This will leave your database in the original state. This is easy to set up using spring with junit. You would have to have your developers implement this test for you. You can also use Selenium 2 or Webdriver to run your application in a web browser if it's a web application.