The main purpose of an Audit is to evaluate if the results of an application delivery analysis satisfy a pre-defined set of conditions (checkpoints). Based on the results of that evaluation, the Audit will pass or fail.
In Kiuwan, you can pre-define as many Audits as you want as a set of checkpoints that are evaluated when the Audit is applied. These pre-defined Audits will be available in your account to be assigned to applications. The specific Audit assigned to an application is automatically applied to the results of any delivery analysis of that application.
For example, we have an application with a baseline analysis describing the actual state of the application (current defects and indicator level). And we want to define a corporate policy stating that any delivery (total or partial) of that application must not contain any new defect. In such a case, that delivery should not be accepted.
In this case, we can define an Audit to check that any delivery does not contain any new defect. In case of a new defect, the Audit will FAIL, otherwise it will be OK.
This case is exactly what Kiuwan's Default Audit does, and the delivery will be marked as OK or FAIL depending on analysis results.
Similarly, we might define any other policies. Some examples might be:
All these Audits (and any other you might consider) can be defined in Kiuwan (without any programming) and will be applied automatically to every delivery analysis.
Kiuwan not only marks a delivery as OK or FAIL, it will also specify:
For every delivery analysis, Kiuwan provide a full Audit Report with all this useful information.
Kiuwan Audits are based on Checkpoints.
A checkpoint is a specific (atomic) condition to be met by the analysis. An Audit may contain as many checkpoints as validations you want to check.
Every checkpoint has two possible results:
Besides, depending on its level of compliance, a checkpoint can be classified as:
An Audit will FAIL if any of its mandatory checkpoints fails.
Please see Checkpoint Management for details on how to create and manage them.
Kiuwan provides a library of checkpoint types you can use to define your specific checkpoints when creating an Audit.
Currently, all the checkpoint types are based on the number of defects found in the analyzed delivery.
When defining your checkpoints, you will be able to define thresholds for:
Besides, Kiuwan not only allows you to define the number of defects, it also allows you to define the nature or type of those defects.
When selecting the nature or types of the defects considered in a checkpoint, you can specify the following criteria:
Before explaining the logic applied during Audit evaluation, we need to define a couple of concepts and parameters you can control in the definition of Kiuwan Checkpoints and Audits.
Every checkpoint has an associated Weight that represents the relative weight of the checkpoint in the Audit. The weights you specify translate (automatically) into a percentage contribution to the overall Audit.
For example, if your Audit has 2 checkpoints of equal importance, you should set this value to 1 for both, translating into a 50% contribution for each checkpoint. Now, if you consider that one is 2 times more important than the other, you should set them as 2 and 1 respectively, translating into a 66% and a 33% contribution.
In a Kiuwan Audit you can specify an Approval Threshold. This threshold will represent the minimum percentage of checkpoints contribution to consider the Audit as OK. Independently if they are mandatory or not, only the contribution percentage of each checkpoint is taken into account to evaluate this threshold.
Learn how the Audit evaluation logic works in the next section.
The logic behind of audit evaluation is based on two-steps
Let's see this logic applied to some examples.
Audit Approval Threshold = 75%
Audit will FAIL. Mandatory checkpoint has failed, therefore Audit result is FAIL.
Audit Approval Threshold = 75%
Audit will FAIL. Although mandatory checkpoint is OK, the sum of successful checkpoints (70%) is lower than Audit Approval Threshold (75%).
Audit Approval Threshold = 75%
Audit will be OK. Mandatory checkpoint is OK and the sum of successful checkpoints (80%) is higher than Audit Approval Threshold (75%).
To access Audit Management module, select "Audits Management" option from the configuration drop down menu.
Only users with "Manage audits" privilege will be allowed to access Audit module.
You will go directly to the audit summary page for the default selected Audit.
Kiuwan comes with an off-the-shelf Default audit. This audit cannot be modified by end users but can be used in any application. In fact this is the Audit assigned to any new application in Kiuwan by default.
Default audit comes under Shared Audits section in left panel. Any user-defined audit will be under My audits.
Clicking on any audit name will allow you to view/manage it.
To create a new Audit, click on New link at the end of the Audits list in the left side panel.
You have provide a Name and an optional Description.
Approval Threshold represents the minimum percentage of checkpoints contribution to consider the Audit as OK. After audit execution, this value is used to evaluate if the audit passes or not. Please see Audit evaluation logic above to know how this value is used in audit evaluation.
Click Create Audit to save the new audit and have it available under My audits
Every Audit needs to have at least 1 Checkpoint. Therefore, once you create an audit, the next natural step is to create checkpoints.
For any selected Audit, the Checkpoints tab will show all the defined checkpoints in a table.
To facilitate working with checkpoints when you have many, you can filter them by Name, Type or Mandatory status; in the filter panel above the table.
Clicking on the checkpoint Name you can modify the checkpoint details and definition.
In this page, you can also directly modify the weights of the checkpoints. By introducing integers in the Weight text box, Kiuwan will automatically calculate the contribution percentage of every checkpoint in the audit. This way, you can easily fine tune checkpoint contributions without editing every individual checkpoint.
Similarly, you can also make each checkpoint Mandatory or Optional with the Mandatory checkbox directly in the checkpoint list.
The Add Report Section button, allows you to create sections to group checkpoints. When exporting the audit results to a PDF report, these sections are used to group audit results as well. You can define the order of sections and checkpoints using the arrows in the first column of the checkpoints table.TO move checkpoints across sections just use the arrows in the checkpoint until you place it in the section you want. The order defined here is used just to display results, it doesn't affect the Audit evaluation logic explained above.
To create a new checkpoint, click the Add checkpoint button.
To create a checkpoint you should provide Name and Description.
Weight represents the relative weight of this checkpoint in the Audit.
Every checkpoint has an associated Weight that represents the relative weight of the checkpoint in the Audit. The weights you specify (integer values) will translate (automatically) into a percentage contribution to the overall Audit. Please see Audit evaluation logic to fully understand how this value is used in audit result calculation.
Mandatory checkbox indicates if the checkpoint is Mandatory (checked) or Optional (unchecked).
Maximum threshold indicates the maximun number defects that are allowed. When audit is executed, if the number of defects is higher than this value, the checkpoint will FAIL.
Type combo allows to select between the available checkpoint types. Please see Checkpoint Types for an explanation.
Available checkpoint types:
Sets a maximum number of defects allowed in the application for the specific defect types selected
Sets a maximum number of new defects allowed in the application for the specific defect types selected
Sets a maximum number of defects allowed in the application for the defined languages, categories and priorities
Sets a maximum number of new defects allowed in the application for the defined languages, categories and priorities
Checks if the Global indicator improves the baseline Global indicator
Checks if the Global indicator is above the defined threshold
Checks if the percentage of Duplicated code is above a defined threshold
Checks if the percentage of Very High defects is above a defined threshold
Depending on the selected checkpoint type, you wiil be able to specify that only defects of selected languages, characteristics and priorities will be taken into account when evaluating the checkpoint.
In this example, the checkpoint evaluation will only compare with the threshold those defects of Very High priority belonging to Maintainability or Security in Java or Cobol source files.
In case you prefer that only defects of certain types be considered, click on Search defect types and select the desired rules by searching the Kiuwan rules repository.
After selecting the rules, the checkpoint will check only defects of those types.
Once you have created the audit and its checkpoints, to associate that audit to an application you should open the Applications management module, select the Application and open the dropdown menu.
Selecting Audit option will open a form where you can select available audits.
Selecting an audit will associate that audit to the application, i.e. every delivery analysis on that application will run the audit.
Do not forget to Publish the audit to make it available for use.
By default, Kiuwan default audit is the "default" audit assigned to a new application. You can change the audit assigned to an application at any time as seen above.
But if you want to set an specific audit, a shared one or one of your custom audits, as the default audit for new applications you can do it as follows.
Go to Audits Management and select the audit you want to be "default" audit in the audits selector.
Once selected, open the "sandwich" menu and click on "Set as default audit" option,
After set, the selected audit will be default audit assigned to new applications.
The Audit selector at the left section of Audit Management indicates with an asterisk the default audit.
Every time a delivery analysis es executed, the audit associated with that application will be evaluated, associating a value of OK or FAIL to that delivery analysis.
You can access the audit results in several ways:
Deliveries module displays a full listing of deliveries.
The Status column shows the results of the audit as well as the overall compliance of the analysis with the defined audit (Score column).
Clicking on the "hand" icon under the Status column will open the Audit Results page.
Below picture shows an example for an audit that resulted OK.
Information is displayed about the Overall status, the reasons (Why?), the overall compliance (Audit score), information on Checkpoint results, the Effort needed to pass the audit (if it failed) and specific information about the results of every checkpoint.
A full listing of audit's checkpoints is displayed, along with information on each of them.
For those checkpoints failed, clicking of the arrow will list those defects that caused the checkpoint failure.
You can export Audit results in pdf format clicking on PDF link.
If you are using Kiuwan Local Analyer in GUI mode, after analyzing the delivery the View Results button will take you directly to the Audit Results page.
When using in CLI mode, stdout will display the URL of the Audit Results Page
Also, you can check programmatically the audit status (OK or FAIL) by inspecting the return code of KLA script (agent.cmd or agent.sh).
Please visit Kiuwan Local Analyzer Return Codes for detailed info on return codes.
If you are using Kiuwan Plugin for Jenkins, you can set the build status depending on audit result, for example marking the build UNSTABLE in case the audit fails.
Please, visit Jenkins plugin - DeliveryMode for details on how to configure Kiuwan plugin for Jenkins to set build status depending on audit result.