Designing an open source test framework without compromising security

Question

Context

The robot framework website mentions robotframework-autoitlibrary in its external library listing and it points to http://code.google.com/p/robotframework-autoitlibrary/.And I found a git repo https://github.com/nokia/robotframework-autoitlibrary and when we do pip install this is the library that's getting installed.

Problem:

So in our day to day work life, we search for solutions to make things work. We might reuse npm modules, pip modules, etc. which are developed and published by individuals.

In the above context, the documentation points to a different module than what I have installed which I accidentally noticed when things were not working.

Question

I asked the robot framework slack-community and most saying it's safe to use, and many are using it ( like I did ).

But how will one know that an open-source package or module is safe for our test framework? How to decide what to use on the basics of fact than just majority opinion.

We as test engineers try different packages to make things work. We go for tutorials to see how to read CSV, how to connect to the database, etc but don't have much knowledge about what happens inside the 3rd party modules that we use in the framework

Answer 1

I'm afraid the general answer would be "you have to rely on the community" since the code is visible to huge amount of people, a lot of people review it and anyone is free to publish their concerns about the safety of any particular piece of code. This gives quite a strong belief of there is no backdoors added intentionally.

This of course does not protect from any potential involuntary impact. I would say that the best way to protect from such impact is isolate your environment. So that you limit the resources which are available from machine that executes your test code, add proxies, firewalls, etc.

Answer 2

Adding to @Alexey R. answer.

You should definitely rely on the community for evaluating and finding security and other issues. The same applies not only for test frameworks but also for tools and packages used in your product in production.

But that doesn't make you immediately resilient to problems, scroll through NPM's advisories and you'll see critical problems that were detected while in production.

You need to develop and use a layer of practices, processes and tools above the community to make yourself safer.

Here's some common best practices-

• Have a dedicated owner of the tool. Someone that will be responsible for keeping track on alerts and updates for tool.

• Don't use the latest and greatest version, choose a version that is proven and tested for a while.

• Don't auto-update tools and packages in your tests

• Use local repositories like jfrog artifactory and block access to the external repository. This will allow you to enforce which version is used and protect against changes made to older versions.

• Use common sense. Always download from official sites, check who the developer is, verify the downloaded file checksum etc.

Answer 3

Developers have the same issue when using open-sources libraries in the application you are building. Most developers also do not have the knowledge of most security risks this brings into their application.

This is a serious issue, this article gives some real world examples where security was breached in open-source libraries:

How Hackers Infiltrate Open Source Projects

"It's not only heard of, it's happening all the time around us. We know of such actions from history and there's no reason to believe that it's not still going on," says Erez Yalon, head of security research for Checkmarx.

_{Source: How Hackers Infiltrate Open Source Projects}

As a tester I think you should help the development team and yourself by assessing the risks and take appropriate mitigations measures.

For example: If you have an application which processes sensitive data, consider segregating the test-network from the production-network. This way the tools are sandboxed within its own environment. Worst case now is that the test-tooling could be an attack vector into the test-environment, here hackers could steal the application, but not any sensitive data.

It boils down to securing your development/test/production environment, some things that come to mind:

• Monitor (or by default block) in/out going traffic. If tools have backdoors, or if they are leaking information, you should be able to prevent this.
• Test on temporary machines, so that attackers only have a very short time window todo harm

Use commons sense and estimate the risks based on your context. Building healthcare application needs different security awareness compared to building a game like https://slither.io :)