What to test when "testing" a PDF download?

Question

What should I test if I have created a scenario with Python (bottle + mako) and web front-end to download a PDF file for registered users? I can use selenium to start Firefox, login and trigger the PDF file download, and then (theoretically, I didn't try it) I can use a tool like textract to extract to text from the PDF file and inspect that it has the expected text and not an error message.

However the nature of my use case is that the file won't download at all if something is wrong and I'm rendering HTML onto a PDF using the python tool pdfkit:

import pdfkit
to_pdf_file = '/tmp/output.pdf'
from bottle import static_file
css = '/srv/http/python.dev/python-common/assets/static_files/css/styles.css'
pdfkit.from_string(render_template("oncall/dayview_reload", values), to_pdf_file, css=css)
return static_file('output.pdf', root='/tmp', download='output.pdf')

So it feels rather pointless to test the PDF instead of testing the HTML because in practice I would only test that the third-party library pdfkit is working on, not testing my code.

I feel that I might as well test only the HTML that it renders and prints the correct result, or would it be important to actually test that the PDF is generated and prints the correct data?

Or should I use some other approach to make my test "right" ?

My test code looks like the following right now, it starts selenium and makes the download, but I feel that I might as well test only the HTML instead of the PDF.

class ViewTest(unittest.TestCase):
@classmethod
def setUp(cls):
    cls.driver = webdriver.Firefox()
    cls.driver.implicitly_wait(20)
    cls.driver.maximize_window()
    cls.driver.get("http://python.dev/oncall/daily_reload?oncall_customer_ids=417&oncall_customer_id=417")


def test_1(self):

    username_input = self.driver.find_element_by_name("email")
    username_input.send_keys('[email protected]')
    password_input = self.driver.find_element_by_name("password")
    password_input.send_keys('ex4mple1')
    self.driver.find_element_by_css_selector('.btn.btn-primary.btn-block.btn-lg').click()

    print("done")

Answer 1

As I understand this, you have an application that takes some set of data per user, renders it to PDF, and allows the user to download the generated PDF file. If there are any errors, the PDF is not generated.

You are correct that in testing the PDF, you are essentially testing the third party library the app uses to generate it.

You are also pulling too many things into a single test by doing this:

• the code that generates the data that is sent to the PDF file,
• the code that sends the data to PDF (if the data is manipulated between being retrieved and being sent to the PDF generation module),
• the code that handles the returned PDF or error

I'd handle it this way:

• Login should be part of the setup routine, since you will need to log in for all tests in this scenario.
• One test should check the data prior to generating the PDF file, failing if either the data is not correct or an error is returned.
• If the data is manipulated in any way before being sent to the third-party PDF generator, the outcome of the manipulation should be checked.
• Rather than checking the content of the PDF programmatically, I'd suggest a manual sanity check that the template you're using works with the data you're generating. This is a once-and-done kind of check that shouldn't need to be repeated unless the size and format of the data used changes.
• A check which I would consider optional because you're covering the intermediate steps and can do a single manual check for it is that you receive a PDF file at the end of the process.
• If possible, you should also test with bad data and check the HTML output for the expected error messages.

Your goal when unit testing or UI testing your application is not just to check that it works as expected, but that the application can handle defined error conditions and fails with some grace when faced with undefined error conditions.

Answer 2

We tend to create a test to check the download file is not empty for exports that are the same as the screen.

Some psuedo code:

driver.downloadFile();
file = getFile(pathToFile);
assert(file.size > 0)

If the PDF is the Core of your product I would invest time to verify its contents. Either visual with Sikuli or a PDF parser.

If it is just a handy export for users, but not business critical, I would think a test like this is enough. Does the file generate, check. I would only start adding tests once defects or regressions start flowing in.