Is there a successful approach to automating security testing?

Question

My team uses a security testing tool called Burp Suite. We currently try to manually run it once a release. One of my tasks is to determine if/how we can make that into more of an automated process. It does have an API, as do other pen testing tools but there are some challenges that I forsee that I don't have good answers for and I wanted to see if anyone else has been successful with security testing automation and how they approached it.

Let me start by providing a simplified view of our manual process:

1. Define the attack surface. Find all of the urls we want to attack (use the tool's crawler, and then add any additional orphaned urls).
2. Modify the settings to filter out some of the common security vulnerabilities that we know are not issues with our product to reduce the noise.
3. Run the tool and get back a ton of warnings/errors
4. Go through the warnings/errors as time allows to determine if there are actual security bugs or not.

The main problems I see with security testing automation are:

1. Each set of penetration tests has multiple actual tests (thousands of them) that it performs and are not logged as individual tests, so I get multiple pass/fail/warning for each "test". It is hard to push these results into our existing reporting infrastructure.
2. The results from one set of tests can produce multiple informational messages/warnings/errors that are weighted by severity as well as likelihood of being an actual error. Going through those results is a time consuming manual process and since the results are "squishy" it is difficult to translate to a pass/fail.
3. Once I investigate a failure and determine it is not an actual bug, I can filter out similar failures for future runs, but I may end up having a different but similar issue in a later build that actually is a security bug that because of my filtering I no longer get an error.

Has anyone successfully automated some or all of this process? I believe that there are some pieces we could automate or maybe use our tool more intelligently, but at the end of the day I don't see a way around a pretty intensive manual process of sifting through the results. I would love to be proven wrong if anyone has had success in automating this process, or reducing the amount of manual labor involved.

Answer 1

Even though we've never tried doing so, I can almost certainly answer your question:

No

Because finding a security flaw is not a procedure - it requires a skill-set. Not everyone can do it - and bot would be the last thing I would expect me to report a real security issue. They're sometimes good to find basic flaws like SQL Injections etc. but integrating them with the infrastructure doesn't look feasible to me, because there's no ROI in doing so.

To support what I said, I can give you an instance of few years back when my programmer found a big security flaw by changing name of cookie to gain access to unauthorized account. This is something you can't expect to be done by bot. This requires intelligence.

Answer 2

So is your code lacking in security testability? I believe Burpsuite should allow creation & running of automated scripts - is it possible to get automated security testing hooks into your code?

If Burpsuite can't do it, there are tools out there that will, although I don't know how they compare to Burpsuite.

In the short term, you are manually checking the logs - you could easily write a script to parse out the actual threats from just the noise. These could be output into an csv / xls file for easy filtering & interpretation.

Answer 3

Is it possible, in a word, yes. However, there will be some investment required.

Personally, I use the tool Acunetix. It's rather customizable and comprehensive. Yes, it has caught security flaws like changing a cookie. Yes, it can be run by command line or a remote request if set to listen. And, yes, it's expensive. I would happily run it on our build servers, except for licensing. We typically run scans on the same application in multiple environments (ie: Dev, PreProd, sometimes Test) and with a few dozen applications that need to be scanned, sometimes running at build time won't work for our release schedules.

A tool like this is great if you want to check against typical threats. Typically, myself, I like to do a quick scan for the OWASP Top 10 in Dev and then a full comprehensive scan (sometimes in the millions of tests) in a PreProd environment. If it finds anything, it gives you exactly what it used and how to reproduce it in a Fiddler like interface. After it's done, it gives you the options to create quick reports for management to look at, or hundreds page reports with all details for the testers and dev's to pour through with steps to correct and idea's for manual testing based off of it.

In the end, it depends on what you want from it. If you want something akin to typical test automation where it will catch regressioned bugs, it's great. If you've done tests on your site before and now want to look for new things, Abhimanyu is absolutely correct, an automated tool can and should be replaced by a dedicated tester specializing in security (and, IMHO, should be done in addition to the tool).