As is defined in its web page (https://www.owasp.org/index.php/Benchmark) the OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services.
To understand a bit more about the OWASP Benchmark and how Kiuwan performs with it, read our previous post: The OWASP Benchmark & Kiuwan. Feel free to comment on it and the little twist we’ve given to the Benchmark as described in the post.
In this post we describe the required steps to run the Benchmark against Kiuwan Code Security in order for you to test it yourself and see how Kiuwan rates against the benchmark.
The Benchmark evaluates 4 indicators:
- Tool correctly identifies a real vulnerability (True Positive – TP)
- Tool fails to identify a real vulnerability (False Negative – FN)
- Tool correctly ignores a false alarm (True Negative – TN)
- Tool fails to ignore a false alarm (False Positive – FP)
The test suite can be downloaded from Github (https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.1). It is composed of 2740 individual tests (source codes in the Java language) covering the following vulnerabilities:
|cmdi||OS Command Injection||78|
|crypto||Risky Cryptographic Algorithm||327|
|hash||Reversible One-Way Hash||328|
|weekrand||Use of Insufficiently Random Values||330|
|trustbound||Trust Boundary Violation||501|
Requisites and first steps to run the test suite
To run the suite you need the following:
- Oracle JDK 8
- Apache Maven 3.1.0 or newer
- Kiuwan account (https://www.kiuwan.com/) and Kiuwan Local Analyzer (https://www.kiuwan.com/docs/display/K5/Kiuwan+Local+Analyzer)
- Internet connection
- Download OWASP-Benchmark from https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.1 This is a fork of the official version (https://github.com/OWASP/Benchmark/releases/tag/1.2beta) where we have added the parsers for Kiuwan reports.
Build the sample report.
- Unzip Benchmark-master.zip in a local folder.
- Ensure that Java and Maven directories are added to your Windows PATH environment variable. This can be checked by executing the commands:
> java -version
> mvn -version
- Build the suite executing the command, from within the unzipped folder:
> mvn clean package
- Create the benchmark Scorecards for the first time: execute from within the unzipped folder:
This last command generates the HTML reports of the benchmark in the ‘scorecard’ folder. You can find a report for each vulnerability type and a report for each analysis engine evaluated.
The main graph on the summary report (OWASP_Benchmark_Home.html) looks like this:
Configure your Kiuwan model.
Kiuwan Code Security comes with more than 175 security checks for Java source code (called rules in Kiuwan terminology), but only a few of them are really necessary to run the suite. You can create a model (see how at https://www.kiuwan.com/docs/display/K5/Models+Manager+User+Guide) copying the default CQM model or create a new one from scratch.
The rules that apply to the OWASP Benchmark suite and need to be ‘Active’ in your Kiuwan model are:
|Kiuwan Rule Name||CWE code|
|Avoid non-neutralized user-controlled input composed in a pathname to a resource||22 73|
|Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’)||78|
|Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’)||79|
|Use bind (or named) parameters in HQL and native SQL queries||89|
|Improper Neutralization of Special Elements used in an SQL Command in iBatis (‘SQL Injection’)||89|
|Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’)||89|
|Content Provider URI Injection||89|
|Avoid non-neutralized user-controlled input in LDAP search filters||90|
|Weak cryptography, insufficient key length||326|
|A hardcoded salt can compromise system security||326|
|Weak symmetric encryption algorithm.||327|
|Weak cryptographic hash.||328|
|Standard pseudo-random number generators cannot withstand cryptographic attacks||330|
|Trust boundary violation||501|
|Generate server-side cookies with adequate security properties||614|
|Improper Neutralization of Data within XPath Expressions (‘XPath Injection’)||643|
Do not forget to PUBLISH your new Kiuwan model.
Two of these rules need a specific configuration. Below you can see the required adjustments:
How the OWASP benchmark score works.
The suite parses the static analysis reports exported from different tools to generate the final scores. To achieve this result our fork of original source code includes the parser ‘KiuwanCSVReader.java’.
For the vulnerability type ‘crypto’, which validates the CWE-327 code, Kiuwan has two rules:
CWE-326: Weak cryptography, insufficient key length
CWE-327: Weak symmetric encryption algorithm.
The parser ‘KiuwanCSVReader.java’ maps these two codes in the CWE-327, to accommodate the expected results:
Scan the OWASP Benchmark suite against Kiuwan.
To scan the suite it takes 4 simple steps:
- Create a Kiuwan Model configuring the rules as indicated above. Remember to PUBLISH the model to be able to use it later.
- Create a new application using your new model (for this tutorial we are using a model called ‘owasp-benchmark’).
- Run the analysis (local or in the cloud) using the “src” part of the unzipped folder of the Benchmark-1.2betaU1.1.zip file as source:
As analysis options, only the Java files need to be analysed. Also, the “Architecture analysis” part is not necessary:
- A few minutes later you can see the results. Login to Kiuwan and navigate to ‘Vulnerabilities’ page:
Obtaining the Kiuwan score card.
Three more steps and you will get the OWASP Benchmark score cards including Kiuwan.
- On Kiuwan’s website: Code Security –> Vulnerabilities, there is a grey option box to the left of VULNERABILITIES. This gives a drop-down list, the last option “CSV” generates a CSV file to the default download location of your browser. The filename is similar to “67110_2017-09-20 17-10-16.0_VulnerabilitiesTable_rfc_4180.csv”
- Copy this file in …\Benchmark-1.2betaU1.1\results folder.
- Execute the createScorecards.bat script again, from within the Benchmark-1.2betaU1.1 folder. You will get a scorecard folder updated with the new reports. The comparison file OWASP_Benchmark_Home.html now has this main graph:
The main statistics are:
In the file with a neme similar to Benchmark_v1.2beta_Scorecard_for_Kiuwan_v2017-09-20_17-10-16.0.html there are the details for Kiuwan: