Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

My team uses a security testing tool called Burp Suite. We currently try to manually run it once a release. One of my tasks is to determine if/how we can make that into more of an automated process. It does have an API, as do other pen testing tools but there are some challenges that I forsee that I don't have good answers for and I wanted to see if anyone else has been successful with security testing automation and how they approached it.

Let me start by providing a simplified view of our manual process:

  1. Define the attack surface. Find all of the urls we want to attack (use the tool's crawler, and then add any additional orphaned urls).
  2. Modify the settings to filter out some of the common security vulnerabilities that we know are not issues with our product to reduce the noise.
  3. Run the tool and get back a ton of warnings/errors
  4. Go through the warnings/errors as time allows to determine if there are actual security bugs or not.

The main problems I see with security testing automation are:

  1. Each set of penetration tests has multiple actual tests (thousands of them) that it performs and are not logged as individual tests, so I get multiple pass/fail/warning for each "test". It is hard to push these results into our existing reporting infrastructure.
  2. The results from one set of tests can produce multiple informational messages/warnings/errors that are weighted by severity as well as likelihood of being an actual error. Going through those results is a time consuming manual process and since the results are "squishy" it is difficult to translate to a pass/fail.
  3. Once I investigate a failure and determine it is not an actual bug, I can filter out similar failures for future runs, but I may end up having a different but similar issue in a later build that actually is a security bug that because of my filtering I no longer get an error.

Has anyone successfully automated some or all of this process? I believe that there are some pieces we could automate or maybe use our tool more intelligently, but at the end of the day I don't see a way around a pretty intensive manual process of sifting through the results. I would love to be proven wrong if anyone has had success in automating this process, or reducing the amount of manual labor involved.

share|improve this question
1  
Sam, what's the main goal you're looking to achieve with automating these tests? Is it mainly to save time that you need to spend on other testing tasks, or are you hoping that automation will allow a wider group of people to use the test results? –  testerab Sep 18 '11 at 18:33
    
Mainly to save time and potentially increase the frequency at which we can run the tests and process the results. The utopia would be to have 100% automated runs build over build to find specific new security issues with each build. I'm not sure that opening up the results to a wider audience has much benefit since any "real" security issues found will eventually become bugs that everyone will have access to. –  Sam Woods Sep 22 '11 at 17:33
add comment

3 Answers

up vote 5 down vote accepted

Even though we've never tried doing so, I can almost certainly answer your question:

No

Because finding a security flaw is not a procedure - it requires a skill-set. Not everyone can do it - and bot would be the last thing I would expect me to report a real security issue. They're sometimes good to find basic flaws like SQL Injections etc. but integrating them with the infrastructure doesn't look feasible to me, because there's no ROI in doing so.

To support what I said, I can give you an instance of few years back when my programmer found a big security flaw by changing name of cookie to gain access to unauthorized account. This is something you can't expect to be done by bot. This requires intelligence.

share|improve this answer
    
Could the tool be used to run regression tests (checks) to ensure there's no deprecation in the quality of the security already in place? I agree that exploratory security testing is a cognitive skill. –  DuncN Oct 21 '11 at 20:18
1  
No, I think - because depreciation of quality can be because of not one or two reasons but infinite reasons. I'd like to add though that trivial things like checking for SQL injections etc. can be automated but again the ROI of such an automation has not appealed to me yet because of past experiences. –  Abhimanyu Oct 23 '11 at 15:19
add comment

Is it possible, in a word, yes. However, there will be some investment required.

Personally, I use the tool Acunetix. It's rather customizable and comprehensive. Yes, it has caught security flaws like changing a cookie. Yes, it can be run by command line or a remote request if set to listen. And, yes, it's expensive. I would happily run it on our build servers, except for licensing. We typically run scans on the same application in multiple environments (ie: Dev, PreProd, sometimes Test) and with a few dozen applications that need to be scanned, sometimes running at build time won't work for our release schedules.

A tool like this is great if you want to check against typical threats. Typically, myself, I like to do a quick scan for the OWASP Top 10 in Dev and then a full comprehensive scan (sometimes in the millions of tests) in a PreProd environment. If it finds anything, it gives you exactly what it used and how to reproduce it in a Fiddler like interface. After it's done, it gives you the options to create quick reports for management to look at, or hundreds page reports with all details for the testers and dev's to pour through with steps to correct and idea's for manual testing based off of it.

In the end, it depends on what you want from it. If you want something akin to typical test automation where it will catch regressioned bugs, it's great. If you've done tests on your site before and now want to look for new things, Abhimanyu is absolutely correct, an automated tool can and should be replaced by a dedicated tester specializing in security (and, IMHO, should be done in addition to the tool).

share|improve this answer
add comment

So is your code lacking in security testability? I believe Burpsuite should allow creation & running of automated scripts - is it possible to get automated security testing hooks into your code?

If Burpsuite can't do it, there are tools out there that will, although I don't know how they compare to Burpsuite.

In the short term, you are manually checking the logs - you could easily write a script to parse out the actual threats from just the noise. These could be output into an csv / xls file for easy filtering & interpretation.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.