How to handle unit test / integration test for a Installation / Setup Application that can be used in multiple platform such as Windows, Linux, Unix

Question

I'm handling a code repository that is responsible to deploy / install / upgrade agent that are running on different platform, windows, unix, linux.

The code base already has some unit test to cover the behavior of important component.

I'm now thinking how I could increase the test coverage and confidence of the setup / upgrade by introducing integration testing, and integrate with travis.

What are the available test framework that I could explore on this type of application?

===== Edit #1 =====

The application I'm looking at implementing integration testing as the follow characteristic.

• The code base is written in ruby.
• It is a setup / installer, the main functionality to bring the state of the machine to a desired stated. For example, we would deploy a file, set the file permission or managed the content of the file if the Shasum does not match.
  • Most of the codes are platform dependent, and create other processes to perform the job. For example chmod.
• It calls external remote APIs, to fetch and verify configuration / data.

Is there any best practice, or code base out there that does this kind of integration testing?

Answer 1

Travis docs explain how to setup your job for multiple OS:

language: c

os: # Will run on Linux and Mac
  - linux
  - osx

compiler:
  - gcc
  - clang

addons:
  apt:
    packages:
      - graphviz

before_install: # You can use $TRAVIS_OS_NAME to perform OS specific actions.
  - if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew update          ; fi
  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then apt update; fi

Answer 2

Windows and Linux are really different.

The main question that you did not address was WHY ?

You say "It is a setup / installer, the main functionality to bring the state of the machine to a desired stated" but any such state is really different on windows vs unix.

Why do you need to run this Ruby code on both windows and non-windows machines. That is not clear to me. The description "I'm handling a code repository that is responsible to deploy / install / upgrade agent that are running on different platform, windows, unix, linux" needs more explanation. What specifically is the agent? What services does it perform and how does it do that? Installers on the two operating systems are essentially completely different with different commands and approaches. It is not a 1:1 translation. The approaches and methods themselves can be vastly different.

Any sort of infrastructure creation is going to be vastly different on Windows and Unix and you have to address them separately. It's like comparing two cars - a Ford and a BMW. They both are cars, have an engine, transport people, etc. However 'one part' does not just fit both cars. There countless reasons behind this, not least capitalism. A comparison that shows how this does works in other systems can be seen in plumbing where standards mean that different manufacturers can supply pipes because they will all be standard fittings, e.g. 1/2", 1", etc.

Conclusion: No single tool will address this other than in a large "if windows, else linux" way. So just have two programs and maintain them separately. The common thread of 'install software' is not enough to bind them perhaps.

Answer 3

The integration testing, as well as any testing, as a general theory, is done against some requirements. In your case you need to have some (good) architectural / design requirements, and based on these requirements you can design the integration tests strategy.

Those requirements have to present at a minimum:

• the list of objects / modules / units / ...;
• the clear interfaces between said units;
• the allowed ranges for the data;
• etc.

Once you have the requirements, you can take into consideration how to build the test cases and which tools are more suitable for the task.

Why are requirements a pre-requisite? Simply because, without them, you cannot know / prove that a(ny) test cases succeeded or failed.