OWASP Benchmark DIY

As is defined in its web page (https://www.owasp.org/index.php/Benchmark) the OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services.

To understand a bit more about the OWASP Benchmark and how Kiuwan performs with it, read our previous post The OWASP Benchmark & Kiuwan. Feel free to comment on it and the little twist we’ve given to the Benchmark as described in the post.

In this post we describe the required steps to run the Benchmark against Kiuwan Code Security in order for you to test it yourself and see how Kiuwan rates against the benchmark.

The Benchmark evaluates 4 indicators:

Tool correctly identifies a real vulnerability (True Positive – TP)

Tool fails to identify a real vulnerability (False Negative – FN)

Tool correctly ignores a false alarm (True Negative – TN)

Tool fails to ignore a false alarm (False Positive – FP)

The test suite can be downloaded from its github page (https://github.com/OWASP/Benchmark/releases/tag/1.2beta). It is composed of 2740 individual tests covering the following vulnerabilities:

key description CWE code
pathtraver Path Traversal 22
cmdi OS Command Injection 78
xss Cross-site Scripting 79
sqli SQL Injection 89
ldapi LDAP Injection 90
crypto Risky Cryptographic Algorithm 327
hash Reversible One-Way Hash 328
weekrand Use of Insufficiently Random Values 330
trustbound Trust Boundary Violation 501
securecookie Sensitive Cookie 614
xpathi XPath Injection 643

Requisites and first steps to run the test suite.

To run the suite you need the following:

  1. Oracle JDK 8
  2. Apache Maven 3.1.0
  3. A Kiuwan account (https://www.kiuwan.com/) and Kiuwan Local Analyzer (https://www.kiuwan.com/docs/display/K5/Kiuwan+Local+Analyzer)
  4. Internet connection
  5. Download OWASP-Benchmark from https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.0.  This is a fork of the official version (https://github.com/OWASP/Benchmark/releases/tag/1.2beta) where we have added the parsers for kiuwan reports.

Build the sample report.

  1. Unzip Benchmark-master.zip in a local folder.
  2. Ensure that JAVA and MAVEN environment are set and in your PATH
  3. Builds the suite executing the command:

> mvn clean package


  1. Run bat (assuming that you are running the suite on Windows. Unix scripts are also provided)


This last command generates the HTML reports of the benchmark in the ‘scorecard’ folder. You can find a report for each vulnerability type and a report for each analysis engine evaluated.

The summary report (OWASP_Benchmark_Home.html)  looks like this:


Configure your Kiuwan model.

Kiuwan Code Security comes with more than 170 java security checks (called rules in kiuwan terminology), but only a few of them are really necessary to run the suite. You can create a model (see how at https://www.kiuwan.com/docs/display/K5/Models+Manager+User+Guide) copying the default CQM model or create a new one from scratch.

The rules that apply to the OWASP Benchmark suite are:

Kiuwan Rule CWE code
Avoid non-neutralized user-controlled input composed in a pathname to a resource. 22 73
Avoid non-neutralized user-controlled input composed in a command. 78
Improper neutralization of input during web content generation (Cross-site Scripting, XSS). 79
Avoid SQL code formed with non neutralized user input. 89
Avoid non-neutralized user-controlled input in LDAP search filters. 90
Weak cryptography, insufficient key length 326
The program uses a weak encryption algorithm that cannot guarantee the confidentiality of sensitive data. 327
Weak cryptographic hashes cannot guarantee data integrity. 328
Standard pseudo-random number generators cannot withstand cryptographic attacks. 330
Trust boundary violation. 501
Generate server-side cookies with adequate security properties. 614
Avoid XPath expressions formed with non neutralized user input. 643

Rule configuration

Two of these rules need a specific configuration. Below you can see the required adjustments:

Rule: Standard pseudo-random number generators cannot withstand cryptographic attacks.4344


Kiuwan default configuration fires a vulnerability for this rule only for methods whose name match the configured regular expression. This is useful in real code to avoid false positives in non sensitive methods.

On the other hand, to run these tests suite, we need to configure the value to ‘.*’ so all methods in the suite are checked.

Rule: Weak cryptography, insufficient key length.333


Default configuration fires a vulnerability for RSA key size under 2048, so we need configure to 1024 to run the tests suite.

How the OWASP benchmark score works.

The suite parses the reports exported from different tools to generate the final scores. To achieve this result our fork of original source code includes the parser ‘KiuwanCSVReader.java’.

For the vulnerability type ‘crypto’, which validates the CWE-327 code, Kiuwan has two rules:

CWE-326: Weak cryptography, insufficient key length

CWE-327: The program uses a weak encryption algorithm that cannot guarantee the confidentiality of sensitive data.

The parser ‘KiuwanCSVReader.java’ maps these two codes in the CWE-327, to accommodate the expected results:


Scan the tests suite against Kiuwan.

To scan the suite it takes 3 simple steps:

  1. create a Kiuwan Model configuring the rules as indicated above. Remember ‘to publish’ the model to be able to use it later.
  2. create a new application and set your new model (for this tutorial we are using a model called ‘owasp-benchmark’).

f3. run the analysis (local or in the cloud) using the Benchmark-1.2beta.zip file as source (only ‘java’ scan is needed to this suite).



4. A few minutes later you can see the results. Login in Kiuwan and navigate to ‘Vulnerabilities’ page:


Obtaining the score card.

Two more steps and you will get the OWASP benchmark score cards.

First of all, go back to Kiuwan and export the results to a csv file (filename is similar to 201_2017-04-22 20-49-52.0_DefectsTable_rfc_4180.csv).


Copy this file in Benchmark-1.2beta\results folder.

Execute the createScorecards.bat script again. You will get a scorecard folder updated with the new reports.