Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Suppose you have automation tests (my example uses Selenium) to assert complex business logic for a web site. What is the best way to assert that thousands of different test cases are outputting the correct values in the software? I thought about just copying the business logic that the actual software uses for the values we're expecting, but then wouldn't that just be re-inventing the wheel?

Example:

The goal is to assert that shipping is correct for all order scenarios. eCommerce site has 50,000 products (let's suppose). The shipping costs are determined based on the shipping provider and rates for each client, extra handling fees, base shipping rates whether the user is a wholesaler, retailer, might factor in product quantity thresholds depending on types of items ordered, etc..

Should we move this business logic (from our site server code) to our assert business logic in our Selenium code library when the automation test cases run for all products? Or should we hard code the shipping cost for thousands of products given the parameters of those test cases?

Let me rephrase the question...

All of the test cases I've done in the past have been making sure site features don't break when enhancements are made to our software (ie: all pages in a workflow work and don't crash--like placing an order, create a product given a set of values and making sure the product has the same values when editing the product, redirects work properly, etc..). Those test cases are simple to write. However, with what I'm asking in this question, I'd like to assert/validate that shipping costs (essentially output values based on a set of parameters) are correct based on our business logic. What is the best methodology/approach for this validity testing of business logic? Do I simply move the business logic from our server code to our Selenium class library code to make sure shipping is correct for all products? Or should we have a spreadsheet/matrix of what our parameters are and the expected values and hard code in our Selenium test cases? I expect the business logic to get more complex in the future, but we need to make sure the shipping costs are coming out the same for products we don't expect the business logic to change in the future.

Pricing is more complex than shipping (with discounts, etc..) so I'd like to use the same methodology for that as well.

share|improve this question
    
I think I understood all of your question except for the last sentence. Would you mind phrasing the last sentence in a different way? –  user246 Jan 25 '12 at 16:37
    
I made the last paragraph more clear hopefully.. –  MacGyver Jan 25 '12 at 16:49

2 Answers 2

up vote 5 down vote accepted

No, you should neither copy nor implement the complex business logic. (See the answers to How to test an Excel macro that selects, groups, and counts data using SQL query? about the downsides of that practice.)

I suspect you do not have the resources to test the behavior of 50,000 products. Here are some alternatives:

  1. Test the business logic against synthetic, canonical products. If you know the business logic -- not the implementation but the expected behavior -- you could devise a test suite to exercise it. If you run into combinatorial problems because the number of variables is too large, you may want to try an All-Pairs approach. See What criteria do I apply to a highly complex development project to warrant getting "real world" data? for more on that subject.
  2. Sample the behavior for specific products. If you know the business logic but do not want to address the combinatorial problem, you can select a set of products to test that you think are especially important. Your selection criteria might be based on their financial importance, or on their similarity to other products, or some other business-relevant metric. I personally do not like this approach, but I have seen organizations use it.
  3. Regression-test the behavior for specific products. If you do not know the business logic, but you have reasons to believe that the current results are correct for a set of products, you can capture those results and compare them against the results of future releases.

If possible, I recommend performing these tests at the API level rather than the UI level.

Finally, if the business logic is truly complex, you should investigate whether it can be tested at a lower level, i.e. a component at a time. (I will not dwell on whether that would be considered unit testing, or whether such testing is the province of developers or testers. If it makes sense to test at a lower level, your team will need to figure that out for themselves.)

share|improve this answer
    
We know the values in our system are correct, but we're planning on adding new logic to pricing and shipping, so I like #3! –  MacGyver Jan 25 '12 at 17:40

MacGyver,

One of the first places to look is what level of coverage of the business logic exists in unit tests. If the unit tests are covering your boundary logic, fee logic and so forth, then you don't need to do more than run tests and ensure that the end data is correct.

For the case where you're looking at correct end data, I'd recommend a harness that will process a test step independently, can string them all together, and pull from a data store to generate multiple transactions. Your data could be something like:

"STEPKIND=NEWSALE, ACTION=SELECTNEW" "STEPKIND=PICKITEM, ACTION=(ItemName)"

When all the transactions run, you pull the data from the system database, and run a cell-by-cell comparison. Since while you're building your transaction set you're also building your baseline, the only changes that need to happen after this are adding new transactions and their baseline data - I use this method (with a different tool) for a specialized point of sale system and linked ecommerce site where correct calculation of shipping amounts, taxes, customer discounts and the like is essential. It works. Any time a refactor breaks calculations, my team knows about it.

share|improve this answer
    
When you say baselines, are you simply saying take past shipping costs from orders placed in a live database (that we know is correct for the product combinations in each order), sync that data to our QA database as baselines and compare those shipping costs against the shipping costs of what our dev site outputs? And in the future, if we add new logic to shipping, make new baselines in our QA database? –  MacGyver Jan 25 '12 at 17:05
2  
I'm saying a store of known good costs from orders - whether they're live or not doesn't matter, so long as you know they're correct and they sample the application's behavior. If the logic changes but the costs don't you don't need to change anything - your tests will check that the new logic still gets the same result. If the costs change, you update your baselines so your tests stay correct. So if Order 367 should have shipping of $5.78 then that's what your baseline says. –  Kate Paulk Jan 25 '12 at 19:45
    
thanks for your input. I gave you some up-votes. ;-) –  MacGyver Jan 25 '12 at 20:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.