My team uses a security testing tool called Burp Suite. We currently try to manually run it once a release. One of my tasks is to determine if/how we can make that into more of an automated process. It does have an API, as do other pen testing tools but there are some challenges that I forsee that I don't have good answers for and I wanted to see if anyone else has been successful with security testing automation and how they approached it.
Let me start by providing a simplified view of our manual process:
- Define the attack surface. Find all of the urls we want to attack (use the tool's crawler, and then add any additional orphaned urls).
- Modify the settings to filter out some of the common security vulnerabilities that we know are not issues with our product to reduce the noise.
- Run the tool and get back a ton of warnings/errors
- Go through the warnings/errors as time allows to determine if there are actual security bugs or not.
The main problems I see with security testing automation are:
- Each set of penetration tests has multiple actual tests (thousands of them) that it performs and are not logged as individual tests, so I get multiple pass/fail/warning for each "test". It is hard to push these results into our existing reporting infrastructure.
- The results from one set of tests can produce multiple informational messages/warnings/errors that are weighted by severity as well as likelihood of being an actual error. Going through those results is a time consuming manual process and since the results are "squishy" it is difficult to translate to a pass/fail.
- Once I investigate a failure and determine it is not an actual bug, I can filter out similar failures for future runs, but I may end up having a different but similar issue in a later build that actually is a security bug that because of my filtering I no longer get an error.
Has anyone successfully automated some or all of this process? I believe that there are some pieces we could automate or maybe use our tool more intelligently, but at the end of the day I don't see a way around a pretty intensive manual process of sifting through the results. I would love to be proven wrong if anyone has had success in automating this process, or reducing the amount of manual labor involved.