I have been interviewing for a "Test Automation Lead" position and in the last two companies I was asked to design a test framework for a given system during the interview. I didn't quite pull it off during the interviews, so here's what I was able to come up with so far. I'd appreciate any feedback or thoughts on the following.
Problem Statement: Consider a credit card billing system that generates bills based on customer credit-card usage data and sends these bills out to the customers via email.
There's an admin website where an admin can (after logging in) choose the billing period (e.g. current month, last month, or any arbitrary from_date and to_date) and then click a button to generate bills. This form submit (button-click) makes a REST call (say an HTTP POST) with "from_date" and "to_date" as URL query parameters to a server. This REST call triggers the bill generation batch process. The batch process queries a MySQL DB and generates the bills.
The bills are formatted as simple HTML tables that are mailed to all the relevant customers in the DB for the given time period. Each bill contains details like total dues and minimum payment amount and the payment due date.
Give an outline of how you would design and structure your test automation framework. What classes and class-hierarchies do you foresee, what helper classes might you use? Your design should ideally be DRY and avoid code duplication. When you start writing performance tests, can your framework reuse your existing functional test code to some extent? How would you add UI (e.g. Selenium-based) tests to your existing functional-verification framework?
Component-level testing
I started talking about how I would test components in isolation first (I guess I like to think bottom-up tackling the easier sub-problems first). I would have a BillingRestHelper class that would abstract the lower-level details of making the HTTP call into a method sort of like "def trigger_billing(from_date, to_date)" which would return the HTTP status (200/400/403 etc). I'd then run data-driven tests for various input/output combinations and assert accordingly for +ve and -ve test cases. At this purely-REST-level I am not concerned with the correctness of the bills or even if they actually generate.
For just the UI-level, I would have some Selenium-based framework that would check element positioning, login/logout, ability to submit the biling form, catch any JS errors etc.
I assumed that the batch process could take hours or more for a real-world situation where a bank might have millions of customers. So to make things simple initially, I would have a well-controlled test environment consisting of credit-card transaction data for just 10 fake/sample customers in my test DB for the batch process. So after I trigger the billing generation, I don't have to sleep for hours (potentially) to wait for the emails to be sent out.
I'd set my test-DB to an initial state from some well-known SQL file that will generate fixed, pre-determined billing data (that I can assert on later). Then I'd start up the billing server against this test-DB before actually triggering any billing.
Once the 10 emails for 10 test-customers are sent to their test-email-accounts, I would programmatically login to email, open the bill HTML, assert on pre-known correct values of total amount due, due date etc. This would test the correctness of the actual bill generation+mailer component.
end-to-end tests
I would then work on end-to-end tests that start from Selenium, all the way to verifying correctness of the generated bills.
nonfunctional tests
Then I would design and run perf-tests and tests against realistic data with lots of customers.
class design?!?
This seemed to be okay but didn't impress the interviewer, who was looking for a high-level class design. I think he wanted to see which class inherits from what, do I use interfaces or abstract classes, or something like that. I'm not from a Java background, and I haven't really built my Ruby+Rspec/Python+PyTest-based test frameworks in a top-down manner like this before. I consider too much OOP and design patterns to be overkill for most test frameworks where a few solid specs (Rspec style) can do the job easily and quickly (but I could be wrong). Comments on this are welcome.
So I muttered something about a SQLHelper class for setup that would load the well-known test data into MySQL. An EmailHelper class for handling all aspects of email. A BillVerifier class for doing the verification of the actual bill data from the email.
This didn't go too well and I want to learn how to solve this kind of problem in an interview situation :)
Help! I've been trying to google this: how to design a test automation framework for a given system, but haven't found much yet. Could someone please point me to any well-known or well-designed test automation suites for any opensource project on Github? Are there any relevant resources elsewhere on the Web, or even better, could someone directly help with the given billing-system problem?