Better approach to a complex test and report file downloads using python

Question

I've created a very complex script using Python3 and YAML (I keep all the configurations in YAML) that in effect tests all potential file downloads for my environment. Sort of end to end test. Due to specific patenting behind, I can't share too much of code, but I am having around 37 different property combinations that download a different file, and each file can be downloaded as zip/gzip, or CSV/XML, so technically 37 X 4 is the total number of my files. To make it "simpler", I need to check that file with license key for combination 1, works only for 1, not for 2-37, and with it we are growing to a relatively big number of tests, currently around 380 in the script, 70ish being happy path tests, and the rest negative tests.

This script was so far run manually, and I'm now trying to convert it and automate with Jenkins, where I'd need to print out some nice report.

I have the following script structure in one big .py file (760 lines):

• a few methods used to read configurations from YAML
• a few methods building URLs reusing config from YAML for files that will be downloaded
• a few methods using requests to download files and their MD5s using built URLs

• the method that executes a single test that then compares calculated MD5 from downloaded file and MD5 that is served from the server

  def executeTest(testName,conf, compareFlag="", fromThread=""):
  if compareFlag=="Compare":
      testPrintableName = getTestName(conf, testName)
      urlForFile = testUrlBuilder(conf, testName, "")
      urlForMD5 = testUrlBuilder(conf, testName, "MD5")
      if printHeadersAndPaths:
          print (urlForFile)
      timeDlStart = time.time()
      filePath = getFile(urlForFile, "")
      timePassed = time.time() - timeDlStart
      #print (timePassed)
      if printHeadersAndPaths:
          print (urlForMD5)
      md5Path = getFile(urlForMD5, "")
      #print(filePath)
      #print(md5Path)
      if ((filePath == "Site Maintanance") and (md5Path is not None)):
          print(testPrintableName + " Attempted to get file from site under Maintanance")
      elif ((filePath is not None) and (md5Path is not None)):
          if ((filePath == "Invalid Index Attempt") and (md5Path == "Invalid Index Attempt")):
              print(testPrintableName + " Attempted to get wrong file with license for different Index type")
          elif ((filePath == "Invalid Licence Attempt") and (md5Path == "Invalid Licence Attempt")):
              print(testPrintableName + " Attempted to get wrong file with invalid licence (eg Beta licence to get Alpha file)")
          elif ((filePath == "Invalid LicenSe Attempt") and (md5Path == "Invalid LicenSe Attempt")):
              print(testPrintableName + " Attempted to get wrong file with invalid license (eg Beta license to get Alpha file)")
          elif ((filePath == "Invalid Key Attempt") and (md5Path == "Invalid Key Attempt")):
              print(testPrintableName + " Attempted to get wrong file with license for different File Type")
          elif ((filePath == "Site Maintanance") and (md5Path == "Site Maintanance")):
              print(testPrintableName + " Attempted to get from site under Maintanance")
          elif ((filePath == "Nothing") and (md5Path == "Nothing")):
              print(testPrintableName + " Attempted to get special data - where we return Nothing")
          elif ((filePath == "Invalid License Gama At Alpha") and (md5Path == "Invalid License Attempt At Gama")):
              print(testPrintableName + " Attempted to get Gama Data with wrong license")
          else:
              cal = calculateMD5(filePath)
              ff = readMD5fromFile(md5Path)
              if (cal.strip() != ff.strip()):
                  print ("Test "+ testPrintableName + "MAYBE FAILED - Calculated MD5 " + str(cal) + " does not match the one grabbed from file " + str(ff) + fromThread)
              else:
                  print ("Test " + testPrintableName + " PASSED " + lastModifiedComparedToNow(lastModified, requestTime) + " in " + str(timePassed) + " seconds " + fromThread)
  
      else:
          if (filePath is not None and filePath != "Invalid Index Attempt"):
              print ("Test "+ testPrintableName + "FAILED - " + "File was not downloaded properly " + fromThread)
          if (md5Path is None and md5Path != "Invalid Index Attempt"):
              print ("Test "+ testPrintableName + "FAILED - " + "MD5 file was not downloaded properly " + fromThread)
  else:
      pass
  

• a load of methods that execute set of tests by name grouped in meaningful methods

  def happyPathsPOIMaps(conf):
  executeTest("POIMapCsvCommaZip", conf, "Compare")
  executeTest("POIMapCsvSemicolonZip", conf, "Compare")
  executeTest("POIMapCsvPipeZip", conf, "Compare")
  executeTest("POIMapCsvXMLZip", conf, "Compare")
  
  executeTest("POIMapCsvCommaGz", conf, "Compare")
  executeTest("POIMapCsvSemicolonGz", conf, "Compare")
  executeTest("POIMapCsvPipeGz", conf, "Compare")
  executeTest("POIMapCsvXMLGz", conf, "Compare" )
  

Each test (again, coming from YAML) contains all the parameters used in building URL for certain file type, that is then tested. Tests in YAML look like this:

#format: 'testName,environment,licenseNameToTest,serverUrlExtendionToUse,CompressionFormat,FileFormat,DataFileType,MD5,Date'
POIMapCsvCommaZip: '"Test Get POI Map",testsrv,POIMap, getPOIMap, zip,xml, , '

YAML contains all servers and test combinations, and it's currently at 500 lines.

And now, finally, my questions:

• Is there any good testing framework that I could use with python 3 to rewrite upper "executeTest" method and get decent reporting
• Is there a more pythonic way how to test such described test cases (using assertions?)
• Is there a way to split YAML to multiple files and load them all "as one file" without actually merging them (like sepparate server configs from happy path tests and negative tests, and then just have at the beginning of YAML - load a.yml,b.yml...

I'm open to suggestions, this does the job at the moment, but I have a feeling that it is overly complex and that there must be a way to make it simpler. Maybe separate methods that are tested in a separate file?

Thanks in advance for not closing it as not constructive enough.

Answer 1

Just the first thought without a deep dive into the problem:

• use assert statements instead of if and then print, e.g.:

  actual = calculateMD5(filePath).strip()
  desired = readMD5fromFile(md5Path).strip()
  assert actual == desired
  

• use pytest testing framework as it is the most popular, advanced and feature-rich at the moment, understands the "assert" statements and have a test discovery which may allow you to leave your program structure as is or close to what it is now. And, of course, you'll get the desired reporting
• you can then research pytest.fixtures as a solution for parameterizing your tests and make your YAML files be implemented as fixtures for your tests

In the long run, I would probably still switch to having your tests formalized as unittest.TestCase classes and running them via pytest anyway.

As a side note, consider posting your current code to the Code Review site - first, make sure the code is working, of course. You may get a lot of valuable feedback.

Answer 2

Adding to the answer above, I'd like to add this.

Primarily, you're responsible for telling the unittest framework when to log, via assertion messages and exceptions. This is similar to your myriad of print statements.

Ideally, this would be managed using Python's unittest framework and exceptions. As well as this, in Python it's generally good practice to ask forgiveness rather than permission. That is, instead of checking beforehand, catch the exception raised when the fields are None, and use that to trigger exception messages.

I'd recommend looking into the built-in unittesting framework that Python comes with, as it should cover everything you want it to. This here is a good place to start

YAML is not capable of including other files, so you'll have to do that yourself.

Examples

This is some of my own work, a class which builds various fixed length integers from any given one, as well as both big endian and little endian values

#                       MIT License
# 
# Copyright (c) 1/06/17 Ennis Massey
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
import unittest
import logging
from unittest import TestCase

from midisnake.integers import IntBuilder, LengthException

logger = logging.getLogger(__name__)


class TestIntBuilder(TestCase):
    def test_repr(self):
        self.intb_inst = IntBuilder(bytearray(b'\x01\xA4'))
        memory_address = id(self.intb_inst)
        self.assertEqual(repr(self.intb_inst), "<midisnake.integers.IntBuilder at 0x{mem_addr:x}, raw: 0x1a4, "
                                               "little endian: "
                                               "41985, big endian: 420, byte length: 2, C type: uint16>".format(
            mem_addr=memory_address))
        self.intb_inst = IntBuilder(bytearray(b'\x2A'))
        memory_address = id(self.intb_inst)
        self.assertEqual(repr(self.intb_inst), "<midisnake.integers.IntBuilder at 0x{mem_addr:x}, raw: 0x2a, "
                                               "little endian: "
                                               "42, big endian: 42, byte length: 1, C type: uint8>".format(
            mem_addr=memory_address))

    def test_str(self):
        self.intb_inst = IntBuilder(bytearray(b'\x01\xA4'))
        memory_address = id(self.intb_inst)
        self.assertEqual(str(self.intb_inst), "41985LE : 420BE : 0x1a4B".format(
            mem_addr=memory_address))
        self.intb_inst = IntBuilder(bytearray(b'\x2A'))
        memory_address = id(self.intb_inst)
        self.assertEqual(str(self.intb_inst), "42LE : 42BE : 0x2aB".format(
            mem_addr=memory_address))

    def test_nullbyte(self):
        with self.assertRaises(LengthException):
            self.intb_inst = IntBuilder(bytearray(b''))

    def test_init(self):
        self.intb_inst = IntBuilder(bytearray(b'\x01\xA4'))
        self.assertEqual({
            'big_endian': self.intb_inst.big_endian,
            'byte_length': self.intb_inst.byte_length,
            'c_type': self.intb_inst.c_type,
            'little_endian': self.intb_inst.little_endian,
            'original_data': self.intb_inst.original_data
        }, {
            'big_endian': 420,
            'byte_length': 2,
            'c_type': 'uint16',
            'little_endian': 41985,
            'original_data': bytearray(b'\x01\xA4')
        })
        self.intb_inst = IntBuilder(bytearray(b'\x2A'))
        self.assertEqual({
            'big_endian': self.intb_inst.big_endian,
            'byte_length': self.intb_inst.byte_length,
            'c_type': self.intb_inst.c_type,
            'little_endian': self.intb_inst.little_endian,
            'original_data': self.intb_inst.original_data
        }, {
            'big_endian': 42,
            'byte_length': 1,
            'c_type': 'uint8',
            'little_endian': 42,
            'original_data': bytearray(b'\x2A')
        })
        self.intb_inst = IntBuilder(bytearray(b'\xff\xff\xff\xff\xff\xff\xff\xff'))
        self.assertEqual({
            'big_endian': self.intb_inst.big_endian,
            'byte_length': self.intb_inst.byte_length,
            'c_type': self.intb_inst.c_type,
            'little_endian': self.intb_inst.little_endian,
            'original_data': self.intb_inst.original_data
        }, {
            'big_endian': 18446744073709551615,
            'byte_length': 8,
            'c_type': 'uint64',
            'little_endian': 18446744073709551615,
            'original_data': bytearray(b'\xff\xff\xff\xff\xff\xff\xff\xff')
        })
        self.intb_inst = IntBuilder(bytearray(b'\xff\xff\xff\xff'))
        self.assertEqual({
            'big_endian': self.intb_inst.big_endian,
            'byte_length': self.intb_inst.byte_length,
            'c_type': self.intb_inst.c_type,
            'little_endian': self.intb_inst.little_endian,
            'original_data': self.intb_inst.original_data
        }, {
            'big_endian': 4294967295,
            'byte_length': 4,
            'c_type': 'uint32',
            'little_endian': 4294967295,
            'original_data': bytearray(b'\xff\xff\xff\xff')
        })
        self.intb_inst = IntBuilder(bytearray((999999999999999999).to_bytes(128, 'big')))
        self.assertEqual({
            'big_endian': self.intb_inst.big_endian,
            'byte_length': self.intb_inst.byte_length,
            'c_type': self.intb_inst.c_type,
            'little_endian': self.intb_inst.little_endian,
            'original_data': self.intb_inst.original_data
        }, {
            'big_endian': 999999999999999999,
            'byte_length': 128,
            'c_type': None,
            'little_endian': 179767638237020898356623490710434418536775593352576159933757128944707257350954536914063245326939442175767480235509437650677415131917209873130951247687383631600519862940215216702259024034572879454308448988996237931693908948093169681562364347544258390966030456380791142075624025472352408185492254992022911320064,
            'original_data': bytearray(
                b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\r\xe0\xb6\xb3\xa7c\xff\xff')
        })

This should give you an idea of what the average Python testing file looks like

Hope this helps :)