Security and Quality Indicators in Kiuwan: a practical guide

Kiuwan gathers evidence from application’s source code using in-house develop Kiuwan static analyzers.
Based on the found evidence from static analysis, Kiuwan calculates and presents relevant Software Analytics Indicators that provide accurate information on:

• Software characteristics (ISO-25000 based), such as: Security, Efficiency, Maintainability, Reliability and Portability.
• Global Indicator, a general index that is calculated as a weighted average of the above software characteristics
• Risk associated with the structural security and quality of the software
• Effort to target, or the amount of manpower needed to achieve the defined security and quality targets
• Security Rating, a 5-star grade that indicates how secure your applications are

Kiuwan provides these indicators to help different stakeholders in the SDLC to make informed decisions and manage all kinds of aspects of the development process, with the common goal of continuously improve software and processes.
So, let’s go through them in some detail.
I will follow a bottom-up approach so I would suggest you to keep the reading in order because every concept is based on the previous ones.

Software characteristics

Kiuwan uses CQM as the methodology (ISO-25000 based) for evaluating the software characteristics of a software product.
In order to correlate with software characteristics, CQM provides the next indicators based on evidence found using static code analysis:

• Security. The capability of the software product to protect information and data so that unauthorized people or systems cannot read or modify them and authorized people or systems are not denied access to them.
• Reliability. The capability of the software product to maintain a specified level of estability.
• Efficiency. The capability of the software product to provide appropriate performance relative to available resource
• Maintainability. The capability of the software product to be modified. Modifications may include corrections, improvements or adaptability of the software to changes in environment and in requirements and functional specifications.
• Portability. The capability of the software product to be transferred from one environment to another.

Kiuwan’s CQM indicators are normalized. What does this mean?

Indicators’ Normalization

Normalization allows the comparison of different applications between them, based on the characteristics, regardless of the technologies used to implement them.
This means that you can inspect and compare, for example:

• If the software is more maintainable than it is efficient or reliable,
• Different version of the same application over time (because the meaning of the indicator does not change),
• Compare two different applications, regardless of the technology
  This normalization allows Kiuwan to represent three different regions you can use to assess your software, make decisions and take appropriate actions:
• 0-30 region. The characteristic pointed to by the indicator is in the RED zone. Improvements are needed.
• 30-70 region. Represented in YELLOW, it means that you have to keep your mind on this indicator. Your next moves will depend on your requirements.
• 70-100 region. The GREEN zone. This is the zone where all indicators must be. No critical defects found.

Global Indicator (GI)

Now that you understand the specific software characteristic indicators, you probably miss a general indicator. Here comes the Global Indicator.
Kiuwan Global Indicator (GI) is calculated as a weighted average of the above software characteristics, providing an accurate global metric for the whole application.
GI is calculated through a complex algorithm that takes into account the severity of the defects, the weight of the category, the analyzed code volume and the criticality of the different programming languages.

Global Indicator Customization

Kiuwan allows you to “customize” this algorithm, by modifying its level of demand, the weights of the category, the weight of the programming languages, the level of importance for every defect priority, the repair difficulty, the priority of the rules, etc…

• Importance of the characteristics
  o You can decide that, for certain applications, maintainability is more important than efficiency but less than security
• Importance of the programming languages
  o Depending on your application portfolio, you might decide that in some cases Java defects will be more important than JavaScript ones, but the opposite in other cases
• Importance (priority) of every defect
  o From Very High to Very Low, you can decide how important a defect is and consequently how much it will influence each indicator value
• Repair difficulty
  o You can fine tune remediation efforts for every single defect or group of defects
  Please, note that this customization does not necessarily apply to all the analyses. You can create different customizations for different applications and purposes. Please have a look at https://www.kiuwan.com/docs/display/K5/Models+Manager+User+Guide for further info.

Too many configuration options? Do not get overwhelmed! Kiuwan provides an off-the-shelf model called CQM, with a default calibration of all the above parameters so you can use it immediately. It is a great starting point to start measuring your applications. You can later customize your own model depending on your needs.

QA: Why does the Global Indicator show a lower value if I have reduced the number of defects?

As you probably have already guessed, most of the time, that reduction is because the new analysis introduced higher priority defects than the previous one. Those new defects do have a higher associated penalty and, consequently, the Global Indicator decreases.

Effort to Target (E2T)

CQM provides some default “targets” for the different software characteristics. Targets are considered as desirable values for indicators and are defined at application level.
Comparing the actual indicator value to the target, you will see at a glance how far you are from the desired values. Obviously, you can fine tune these default values (set to 90 by default).
Every defect found by Kiuwan has an associated priority and an effort to programmatically remediate it. Effort is measured in man/ hours.
Taking into account found defects, their priority, remediation effort and targets, Kiuwan calculates the total effort needed to repair the defects and reach the target for every characteristic. The result of this calculation is presented as the Effort to Target indicator (E2T). You think of the E2T as the technical debt you have accumulated in your application due to code problems affecting the different characteristics.

QA: Does Effort to Target indicator mean the effort to correct all the defects?

No. A very common confusion on Effort to Target is to understand it as the effort to remediate ALL the found defects. This is not so, the indicator represents the effort needed to reach the defined targets. If you want to see the total effort, you have it available in the metrics screen for every application.

Risk Index (RI)

Kiuwan’s Risk Index (RI) is a summary indicator that concentrates all the above indicators to provide a measure of the implicit risk associated to the security and quality aspects of your application, according to your defined requirements (targets)..
Kiuwan Risk Index provides a measure on how far you are to get an acceptable security and quality level. Remember that the exact meaning of “acceptable” is defined by you (the “target”).

Risk Index calculation concentrates all the evidence found in the source code of your application, combining Global Indicator (GI), Effort to Target (E2T) and Application Size.
Take into account that every indicator is represented by a different mathematical distribution (Gaussian, Lineal, and Step, respectively), so its interpretation should be made carefully because of the interaction among them.

How should Risk Index be interpreted?

Although the Risk Index scale is 0-100, any value greater than 0 should be observed. Nevertheless, what is most interesting is to monitor its evolution over time, as it indicates the evolution of your application exposure.
Pay attention to risk index evolution over time.
For example, if you have a low Global Indicator, but the Effort needed to get better is low, you are not assuming a high risk, because you are going to repair your problems easily and, consequently, the Risk Index will not be too high. But, if the effort needed to get better is very high the Risk Index will be high too.

QA: If I spend 20% of the total Effort to remediate defects, should I expect a proportional reduction (20%) in Risk Index?

As said above, it will depend on the application size and how far you are from the target.
If your application is small and you have a high effort to target, that probably means that it’s not worth it to spend that effort to correct it and it’s better to build it from scratch. Consequently, you will need a high volume of effort to decrease the risk index.
But if the application size is high, any reduction in risk will be more sensitive to remediation effort, and you will see how the risk index is reduced as you remediate problems.
Questions on this matter can be many. My intention in this post is to provide a quick understanding on the rationale being applied to the calculation of Risk Index.
I can see you are thinking… Oops, this is not easy. Keep on reading…

QA: How can I know what actions to do once I get the results of Kiuwan indicators?

To facilitate this decision making process, Kiuwan provides a “what-if simulator”, that will predict how your indicators will change depending on the remediation effort you can spend or the values you want to achieve for the different characteristics indicators in your next development iteration. The simulation will generate as well a comprehensive action plan with the defects you have to work on to get the predicted results. More information about Action Plans and Simulation (what-if) at https://www.kiuwan.com/docs/display/K5/Action+Plans and https://www.kiuwan.com/docs/display/K5/New+automatic+action+plan.+What+if+analysis

Security Rating

Kiuwan’s Security Rating has been recently introduced.
The reasoning behind it is to surface the application security details by providing a 5-star grade that indicates how secure your application is in terms of the likelihood and impact of found security vulnerabilities.

Security vulnerabilities found by Kiuwan are grouped in a quadrant according to two major axes:

• Impact of the vulnerability, according to the severity of the associated security risk
• Likelihood of the event that could cause the associated security breach

These two axis produce 4 quadrants. Kiuwan summarizes found vulnerabilities for each quadrant.

Kiuwan’s Security Rating is quite strict. It will assign additional stars as related quadrants become “empty” of vulnerabilities.

• Security Rating starts from 1-star (i.e. 0 is not considered) when there is at least 1 Very-High vulnerability (high impact, high likelihood)
• 2-star rating when there is at least 1 High vulnerability (high impact, low likelihood) and none of higher priority
• 3-star and next will follow this schema.

Applications with 5 stars are considered to be secure, whereas those with 1 star are considered to be very insecure.
Please visit https://www.kiuwan.com/docs/display/K5/Security+Summary for more information.
At this time, those of you that are reading this post might be thinking..

QA: How does Security Rating relate to above mentioned Security Indicator?

Although both measure security, each indicator measures a different aspect of security.
While Security Rating provides a rating based on detection of security “holes” (or breaches), Security Indicator provides a measure of “level of protection”. Let’s see the difference with some examples.
Let’s imagine we have 3 buildings with the following indicators:

1. A single family house (sec rating:1, sec indicator: 20)

2. A block of apartments (sec rating:1, sec indicator: 50)
3. A military building (sec rating:1, sec indicator: 70)

Each of them has one high-priority vulnerability (e.g. an open window). Therefore, all of them will have a 1-star Security Rating (they are completely vulnerable).

But then we find that the Security Indicators are different for each of them. How then can these results be understood?

Although all of them are highly vulnerable because of the high priority vulnerability, the different Security Indicators show the different levels of protection mechanisms each of them have.
Building #1 presents a low level of overall protection, #2 a medium level and #3 a high level, based on the quantity and quality of protection mechanisms they have.

The higher the security indicator, the more the protection mechanisms are present and validated by Kiuwan. The lower the security indicator, the more security checks have failed during the analysis.

In the above example, the military building has a high security rating (most of the security checks are ok), but there’s one big vulnerability that drops down the entire Security Rating. Once that vulnerability is resolved, most probably its security indicator and security rating will improve and be much higher.

In the opposite example, the single family house might solve the same vulnerability, but there are so many security breaches present that the rating and indicators will also be more difficult to improve.

Moreover, size matters! The security indicator is also considering the density of security mechanisms and breaches according to the size of the building. It’s not the same to have one window open in a family house with 10 windows, than in a block of apartments with 100 windows. This explains why, even with the same vulnerability, Security Rating is low in both cases bit Security Indicator is higher in the block of apartments.

Thank you for reading and stay tuned for more.