4

I have been interviewing for a "Test Automation Lead" position and in the last two companies I was asked to design a test framework for a given system during the interview. I didn't quite pull it off during the interviews, so here's what I was able to come up with so far. I'd appreciate any feedback or thoughts on the following.

Problem Statement: Consider a credit card billing system that generates bills based on customer credit-card usage data and sends these bills out to the customers via email.

There's an admin website where an admin can (after logging in) choose the billing period (e.g. current month, last month, or any arbitrary from_date and to_date) and then click a button to generate bills. This form submit (button-click) makes a REST call (say an HTTP POST) with "from_date" and "to_date" as URL query parameters to a server. This REST call triggers the bill generation batch process. The batch process queries a MySQL DB and generates the bills.

The bills are formatted as simple HTML tables that are mailed to all the relevant customers in the DB for the given time period. Each bill contains details like total dues and minimum payment amount and the payment due date.

Give an outline of how you would design and structure your test automation framework. What classes and class-hierarchies do you foresee, what helper classes might you use? Your design should ideally be DRY and avoid code duplication. When you start writing performance tests, can your framework reuse your existing functional test code to some extent? How would you add UI (e.g. Selenium-based) tests to your existing functional-verification framework?

Component-level testing

I started talking about how I would test components in isolation first (I guess I like to think bottom-up tackling the easier sub-problems first). I would have a BillingRestHelper class that would abstract the lower-level details of making the HTTP call into a method sort of like "def trigger_billing(from_date, to_date)" which would return the HTTP status (200/400/403 etc). I'd then run data-driven tests for various input/output combinations and assert accordingly for +ve and -ve test cases. At this purely-REST-level I am not concerned with the correctness of the bills or even if they actually generate.

For just the UI-level, I would have some Selenium-based framework that would check element positioning, login/logout, ability to submit the biling form, catch any JS errors etc.

I assumed that the batch process could take hours or more for a real-world situation where a bank might have millions of customers. So to make things simple initially, I would have a well-controlled test environment consisting of credit-card transaction data for just 10 fake/sample customers in my test DB for the batch process. So after I trigger the billing generation, I don't have to sleep for hours (potentially) to wait for the emails to be sent out.

I'd set my test-DB to an initial state from some well-known SQL file that will generate fixed, pre-determined billing data (that I can assert on later). Then I'd start up the billing server against this test-DB before actually triggering any billing.

Once the 10 emails for 10 test-customers are sent to their test-email-accounts, I would programmatically login to email, open the bill HTML, assert on pre-known correct values of total amount due, due date etc. This would test the correctness of the actual bill generation+mailer component.

end-to-end tests

I would then work on end-to-end tests that start from Selenium, all the way to verifying correctness of the generated bills.

nonfunctional tests

Then I would design and run perf-tests and tests against realistic data with lots of customers.

class design?!?

This seemed to be okay but didn't impress the interviewer, who was looking for a high-level class design. I think he wanted to see which class inherits from what, do I use interfaces or abstract classes, or something like that. I'm not from a Java background, and I haven't really built my Ruby+Rspec/Python+PyTest-based test frameworks in a top-down manner like this before. I consider too much OOP and design patterns to be overkill for most test frameworks where a few solid specs (Rspec style) can do the job easily and quickly (but I could be wrong). Comments on this are welcome.

So I muttered something about a SQLHelper class for setup that would load the well-known test data into MySQL. An EmailHelper class for handling all aspects of email. A BillVerifier class for doing the verification of the actual bill data from the email.

This didn't go too well and I want to learn how to solve this kind of problem in an interview situation :)

Help! I've been trying to google this: how to design a test automation framework for a given system, but haven't found much yet. Could someone please point me to any well-known or well-designed test automation suites for any opensource project on Github? Are there any relevant resources elsewhere on the Web, or even better, could someone directly help with the given billing-system problem?

  • Re: well known test automation suites, then the WikiMedia foundation would be worth looking at: mediawiki.org/wiki/Quality_Assurance/Browser_testing - uses Ruby+Rspec so should be familiar. I am not sure what your main problem is though - your question talks about a specific scenario, but it sounds like your issue is - well, I'm not sure. Did the job state it required a Java background, or were you surprised by that in interview? Are you looking for feedback on whether your normal design approach is right? Or? – testerab Feb 23 '15 at 19:14
  • @testerab: The job didn't require a Java background. I was just never asked to "design a test framework for 'System X'" in an interview before. I tried to answer it as best I could, but it's not really something I thought was really feasible in the space of an interview. Wanted to know if someone else here has encountered this sort of question and how they would answer it. Also wanted to know if my approach made sense. – nonbeing Feb 24 '15 at 15:28
2

I am not sure about the billing system problem, however I've found these resources very helpful in understanding and setting up test automation.

http://safsdev.sourceforge.net/FRAMESDataDrivenTestAutomationFrameworks.htm

Also, specifically to Python + Pytest based frameworks that help in abstraction of test cases check out Robot Framework. It has detailed documentation and has a repo in github as well,

http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html

The project has moved to http://robotframework.org/

Hops this helps, good luck!

2

Looking at your problem statement, we've 3 areas to do testing on functionality.

  1. UI - Admin Login
  2. Webservices - REST calls
  3. Batch Processing (DB or Files)

If the interviewer is looking for a framework on this... you've to build a framework on all 3 areas. Only Selenium doesn't work here. UI, you need to use Selenium Webdriver, WebServices you need to use SOUPUI or JDK Objects and Batches (MyBatis and Core Java data structures)

You must choose the right framework which accomplishes all areas without breaking UI to DB.

Cucumber - It's suitable to couple with selenium webdriver, JDK service objects, Batch with mybatis framework.

Class level design must be on these three areas and mostly you'll be using core components of selenium, webservice object operations, mybatis framework usage. This would be different based on your thought process during division. It would be least priority if you've given above answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.