OWASP Benchmark DIY

As is defined in its web page (https://www.owasp.org/index.php/Benchmark) the OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services.

To understand a bit more about the OWASP Benchmark and how Kiuwan performs with it, read our previous post: The OWASP Benchmark & Kiuwan. Feel free to comment on it and the little twist we’ve given to the Benchmark as described in the post.

In this post we describe the required steps to run the Benchmark against Kiuwan Code Security in order for you to test it yourself and see how Kiuwan rates against the benchmark.

The Benchmark evaluates 4 indicators:

  • Tool correctly identifies a real vulnerability (True Positive – TP)
  • Tool fails to identify a real vulnerability (False Negative – FN)
  • Tool correctly ignores a false alarm (True Negative – TN)
  • Tool fails to ignore a false alarm (False Positive – FP)

The test suite can be downloaded from Github (https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.1). It is composed of 2740 individual tests (source codes in the Java language) covering the following vulnerabilities:


Key Description CWE code
pathtraver Path Traversal 22
cmdi OS Command Injection 78
xss Cross-site Scripting 79
sqli SQL Injection 89
ldapi LDAP Injection 90
crypto Risky Cryptographic Algorithm 327
hash Reversible One-Way Hash 328
weekrand Use of Insufficiently Random Values 330
trustbound Trust Boundary Violation 501
securecookie Sensitive Cookie 614
xpathi XPath Injection 643

Requisites and first steps to run the test suite

To run the suite you need the following:

  1. Oracle JDK 8
  2. Apache Maven 3.1.0 or newer
  3. Kiuwan account (https://www.kiuwan.com/) and Kiuwan Local Analyzer (https://www.kiuwan.com/docs/display/K5/Kiuwan+Local+Analyzer)
  4. Internet connection
  5. Download OWASP-Benchmark from https://github.com/mcprol/Benchmark/releases/tag/v1.2betaU1.1 This is a fork of the official version (https://github.com/OWASP/Benchmark/releases/tag/1.2beta) where we have added the parsers for Kiuwan reports.

Build the sample report.

  1. Unzip Benchmark-master.zip in a local folder.
  2. Ensure that Java and Maven directories are added to your Windows PATH environment variable. This can be checked by executing the commands:
    > java -version
    > mvn -version
  3. Build the suite executing the command, from within the unzipped folder:
    > mvn clean package
  4. Create the benchmark Scorecards for the first time: execute  from within the unzipped folder:
    > createScorecards.bat

This last command generates the HTML reports of the benchmark in the ‘scorecard’ folder. You can find a report for each vulnerability type and a report for each analysis engine evaluated.

The main graph on the summary report (OWASP_Benchmark_Home.html) looks like this:BM_home_orig

Configure your Kiuwan model.

Kiuwan Code Security comes with more than 175 security checks for Java source code (called rules in Kiuwan terminology), but only a few of them are really necessary to run the suite. You can create a model (see how at https://www.kiuwan.com/docs/display/K5/Models+Manager+User+Guide) copying the default CQM model or create a new one from scratch.

The rules that apply to the OWASP Benchmark suite and need to be ‘Active’ in your Kiuwan model are:

Kiuwan Rule Name CWE code
Avoid non-neutralized user-controlled input composed in a pathname to a resource 22 73
Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’) 78
Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’) 79
Use bind (or named) parameters in HQL and native SQL queries 89
Improper Neutralization of Special Elements used in an SQL Command in iBatis (‘SQL Injection’) 89
Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’) 89
Content Provider URI Injection 89
Avoid non-neutralized user-controlled input in LDAP search filters 90
Weak cryptography, insufficient key length 326
A hardcoded salt can compromise system security 326
Weak symmetric encryption algorithm. 327
Weak cryptographic hash. 328
Standard pseudo-random number generators cannot withstand cryptographic attacks 330
Trust boundary violation 501
Generate server-side cookies with adequate security properties 614
Improper Neutralization of Data within XPath Expressions (‘XPath Injection’) 643


Do not forget to PUBLISH your new Kiuwan model.

Rule configuration

Two of these rules need a specific configuration. Below you can see the required adjustments:

Rule: Standard pseudo-random number generators cannot withstand cryptographic attacks4344Kiuwan’s default configuration fires a vulnerability for this rule only for methods whose name match the configured regular expression. This is useful in real code to avoid false positives in non-sensitive methods.On the other hand, to run these tests suite, we need to configure the value to ‘.*’ (dot star) so all methods in the suite are checked.
Rule: Weak cryptography, insufficient key length333Default configuration fires a vulnerability for RSA key size under 2048, so we need configure to 1024 to run the tests suite.Value       RSA/2048 –> RSA/1024 (the rest of Value stays unchanged).

How the OWASP benchmark score works.

The suite parses the static analysis reports exported from different tools to generate the final scores. To achieve this result our fork of original source code includes the parser ‘KiuwanCSVReader.java’.

For the vulnerability type ‘crypto’, which validates the CWE-327 code, Kiuwan has two rules:

CWE-326: Weak cryptography, insufficient key length

CWE-327: Weak symmetric encryption algorithm.

The parser ‘KiuwanCSVReader.java’ maps these two codes in the CWE-327, to accommodate the expected results:


Scan the OWASP Benchmark suite against Kiuwan.

To scan the suite it takes 4 simple steps:

  1. Create a Kiuwan Model configuring the rules as indicated above. Remember to PUBLISH the model to be able to use it later.
  2. Create a new application using your new model (for this tutorial we are using a model called ‘owasp-benchmark’).
  3. fRun the analysis (local or in the cloud) using the “src” part of the unzipped folder of the Benchmark-1.2betaU1.1.zip file as source:
    kla1As analysis options, only the Java files need to be analysed. Also, the “Architecture analysis” part is not necessary:
  4. A few minutes later you can see the results. Login to Kiuwan and navigate to ‘Vulnerabilities’ page:vuln1


Obtaining the Kiuwan score card.

Three more steps and you will get the OWASP Benchmark score cards including Kiuwan.

  1. On Kiuwan’s website: Code Security –> Vulnerabilities, there is a grey option box to the left of VULNERABILITIES. This gives a drop-down list, the last option “CSV” generates a CSV file to the default download location of your browser. The filename is similar to “67110_2017-09-20 17-10-16.0_VulnerabilitiesTable_rfc_4180.csv”
  2. Copy this file in …\Benchmark-1.2betaU1.1\results folder.
  3. Execute the createScorecards.bat script again, from within the Benchmark-1.2betaU1.1 folder. You will get a scorecard folder updated with the new reports. The comparison file OWASP_Benchmark_Home.html now has this main graph:
    The main statistics are:
    In the file with a neme similar to Benchmark_v1.2beta_Scorecard_for_Kiuwan_v2017-09-20_17-10-16.0.html there are the details for Kiuwan: