As is defined in its web page (https://www.owasp.org/index.php/Benchmark) the OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services.
To understand a bit more about the OWASP Benchmark and how Kiuwan performs with it, read our previous post The OWASP Benchmark & Kiuwan. Feel free to comment on it and the little twist we’ve given to the Benchmark as described in the post.
In this post we describe the required steps to run the Benchmark against Kiuwan Code Security in order for you to test it yourself and see how Kiuwan rates against the benchmark.
The Benchmark evaluates 4 indicators:
Tool correctly identifies a real vulnerability (True Positive – TP)
Tool fails to identify a real vulnerability (False Negative – FN)
Tool correctly ignores a false alarm (True Negative – TN)
Tool fails to ignore a false alarm (False Positive – FP)
The test suite can be downloaded from its github page (https://github.com/OWASP/Benchmark/releases/tag/1.2beta). It is composed of 2740 individual tests covering the following vulnerabilities:
|cmdi||OS Command Injection||78|
|crypto||Risky Cryptographic Algorithm||327|
|hash||Reversible One-Way Hash||328|
|weekrand||Use of Insufficiently Random Values||330|
|trustbound||Trust Boundary Violation||501|
Requisites and first steps to run the test suite.
To run the suite you need the following:
- Oracle JDK 8
- Apache Maven 3.1.0
- A Kiuwan account (https://www.kiuwan.com/) and Kiuwan Local Analyzer (https://www.kiuwan.com/docs/display/K5/Kiuwan+Local+Analyzer)
- Internet connection
- Download OWASP-Benchmark from https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.0. This is a fork of the official version (https://github.com/OWASP/Benchmark/releases/tag/1.2beta) where we have added the parsers for kiuwan reports.
Build the sample report.
- Unzip Benchmark-master.zip in a local folder.
- Ensure that JAVA and MAVEN environment are set and in your PATH
- Builds the suite executing the command:
> mvn clean package
- Run bat (assuming that you are running the suite on Windows. Unix scripts are also provided)
This last command generates the HTML reports of the benchmark in the ‘scorecard’ folder. You can find a report for each vulnerability type and a report for each analysis engine evaluated.
The summary report (OWASP_Benchmark_Home.html) looks like this:
Configure your Kiuwan model.
Kiuwan Code Security comes with more than 170 java security checks (called rules in kiuwan terminology), but only a few of them are really necessary to run the suite. You can create a model (see how at https://www.kiuwan.com/docs/display/K5/Models+Manager+User+Guide) copying the default CQM model or create a new one from scratch.
The rules that apply to the OWASP Benchmark suite are:
|Kiuwan Rule||CWE code|
|Avoid non-neutralized user-controlled input composed in a pathname to a resource.||22 73|
|Avoid non-neutralized user-controlled input composed in a command.||78|
|Improper neutralization of input during web content generation (Cross-site Scripting, XSS).||79|
|Avoid SQL code formed with non neutralized user input.||89|
|Avoid non-neutralized user-controlled input in LDAP search filters.||90|
|Weak cryptography, insufficient key length||326|
|The program uses a weak encryption algorithm that cannot guarantee the confidentiality of sensitive data.||327|
|Weak cryptographic hashes cannot guarantee data integrity.||328|
|Standard pseudo-random number generators cannot withstand cryptographic attacks.||330|
|Trust boundary violation.||501|
|Generate server-side cookies with adequate security properties.||614|
|Avoid XPath expressions formed with non neutralized user input.||643|
Two of these rules need a specific configuration. Below you can see the required adjustments:
How the OWASP benchmark score works.
The suite parses the reports exported from different tools to generate the final scores. To achieve this result our fork of original source code includes the parser ‘KiuwanCSVReader.java’.
For the vulnerability type ‘crypto’, which validates the CWE-327 code, Kiuwan has two rules:
CWE-326: Weak cryptography, insufficient key length
CWE-327: The program uses a weak encryption algorithm that cannot guarantee the confidentiality of sensitive data.
The parser ‘KiuwanCSVReader.java’ maps these two codes in the CWE-327, to accommodate the expected results:
Scan the tests suite against Kiuwan.
To scan the suite it takes 3 simple steps:
- create a Kiuwan Model configuring the rules as indicated above. Remember ‘to publish’ the model to be able to use it later.
- create a new application and set your new model (for this tutorial we are using a model called ‘owasp-benchmark’).
4. A few minutes later you can see the results. Login in Kiuwan and navigate to ‘Vulnerabilities’ page:
Obtaining the score card.
Two more steps and you will get the OWASP benchmark score cards.
First of all, go back to Kiuwan and export the results to a csv file (filename is similar to 201_2017-04-22 20-49-52.0_DefectsTable_rfc_4180.csv).
Copy this file in Benchmark-1.2beta\results folder.
Execute the createScorecards.bat script again. You will get a scorecard folder updated with the new reports.