What's the best approach to asserting dynamically generated values with selenium?

Question

Suppose you have automation tests (my example uses Selenium) to assert complex business logic for a web site. What is the best way to assert that thousands of different test cases are outputting the correct values in the software? I thought about just copying the business logic that the actual software uses for the values we're expecting, but then wouldn't that just be re-inventing the wheel?

Example:

The goal is to assert that shipping is correct for all order scenarios. eCommerce site has 50,000 products (let's suppose). The shipping costs are determined based on the shipping provider and rates for each client, extra handling fees, base shipping rates whether the user is a wholesaler, retailer, might factor in product quantity thresholds depending on types of items ordered, etc..

Should we move this business logic (from our site server code) to our assert business logic in our Selenium code library when the automation test cases run for all products? Or should we hard code the shipping cost for thousands of products given the parameters of those test cases?

Let me rephrase the question...

All of the test cases I've done in the past have been making sure site features don't break when enhancements are made to our software (ie: all pages in a workflow work and don't crash--like placing an order, create a product given a set of values and making sure the product has the same values when editing the product, redirects work properly, etc..). Those test cases are simple to write. However, with what I'm asking in this question, I'd like to assert/validate that shipping costs (essentially output values based on a set of parameters) are correct based on our business logic. What is the best methodology/approach for this validity testing of business logic? Do I simply move the business logic from our server code to our Selenium class library code to make sure shipping is correct for all products? Or should we have a spreadsheet/matrix of what our parameters are and the expected values and hard code in our Selenium test cases? I expect the business logic to get more complex in the future, but we need to make sure the shipping costs are coming out the same for products we don't expect the business logic to change in the future.

Pricing is more complex than shipping (with discounts, etc..) so I'd like to use the same methodology for that as well.

Answer 1

No, you should neither copy nor implement the complex business logic. (See the answers to How to test an Excel macro that selects, groups, and counts data using SQL query? about the downsides of that practice.)

I suspect you do not have the resources to test the behaviour of 50,000 products. Here are some alternatives:

1. Test the business logic against synthetic, canonical products. If you know the business logic -- not the implementation but the expected behaviour -- you could devise a test suite to exercise it. If you run into combinatorial problems because the number of variables is too large, you may want to try an All-Pairs approach. See What criteria do I apply to a highly complex development project to warrant getting "real world" data? for more on that subject.
2. Sample the behaviour for specific products. If you know the business logic but do not want to address the combinatorial problem, you can select a set of products to test that you think are especially important. Your selection criteria might be based on their financial importance, or on their similarity to other products, or some other business-relevant metric. I personally do not like this approach, but I have seen organizations use it.
3. Regression-test the behaviour for specific products. If you do not know the business logic, but you have reasons to believe that the current results are correct for a set of products, you can capture those results and compare them against the results of future releases.

If possible, I recommend performing these tests at the API level rather than the UI level.

Finally, if the business logic is truly complex, you should investigate whether it can be tested at a lower level, i.e. a component at a time. (I will not dwell on whether that would be considered unit testing, or whether such testing is the province of developers or testers. If it makes sense to test at a lower level, your team will need to figure that out for themselves.)

Answer 2

MacGyver,

One of the first places to look is what level of coverage of the business logic exists in unit tests. If the unit tests are covering your boundary logic, fee logic and so forth, then you don't need to do more than run tests and ensure that the end data is correct.

For the case where you're looking at the correct end date, I'd recommend a harness that will process a test step independently, can string them all together, and pull from a data store to generate multiple transactions. Your data could be something like:

"STEPKIND=NEWSALE, ACTION=SELECTNEW" "STEPKIND=PICKITEM, ACTION=(ItemName)"

When all the transactions run, you pull the data from the system database and run a cell-by-cell comparison. Since while you're building your transaction set you're also building your baseline, the only changes that need to happen after this are adding new transactions and their baseline data - I use this method (with a different tool) for a specialized point of sale system and linked e-commerce site where the correct calculation of shipping amounts, taxes, customer discounts and the like is essential. It works. Any time a refactor breaks calculations, my team knows about it.