Deep View Validator Overview

This package contains the stand-alone Deep View Validator that validates object detection, semantic segmentation, or pose (vision) models. This package is capable of validating Deep View RT, Keras, TFlite, and ONNX models using either Darknet or TFRecord datasets. 

Prerequisites

• Deep View Validator is tested using Python 3.10. To install Python, please check here. Support for Python 3.8.0 and higher is coming soon!
• To install Deep View Validator and the model dependencies, please visit Deep View Validator Installation Guide
• To see the dataset formats that is currently supported in Deep View Validator, please visit Deep View Validator Supported Dataset Formats.

Running Deep View Validator

When using Deep View Validator it is needed to take into consideration the model path. For that, validator accepts the path to the model as the first unnamed positional parameter.

Detection Models

The following command provides a template to run a detection model and to evaluate its performance.

deepview-validator <path to the model> --dataset=<path to the dataset>

Note: Substitute the values in <> for the path to the model and the path to the dataset.

modelrunner -c 1 -H <modelrunner target>

deepview-validator 
    <path to the model> 
    --dataset=<path to the dataset> 
    --target=<EVK IP>:<modelrunner target>
    --application=modelclient

Segmentation Models

Note: Segmentation validation is only supported by Darknet datasets using JSON annotation files to contain the segments.

The following command provides a template to run segmentation models.

deepview-validator 
    <path to the model> 
    --dataset <path to the dataset>
    --validate=segmentation

However, Deep View RT segmentation models can also be run using ModelClient library. The first step is to enable modelrunner in a target device such as an EVK.

To enable modelrunner, use the following command in the target device.

modelrunner -c 1 -H <modelrunner target> 

• To run a ModelPack Deep View RT model, use the following command.

deepview-validator  
    <path to the model>
    --dataset=<path to the dataset>
    --validate=segmentation 
    --target=<EVK IP>:<modelrunner target>

• To run a DeepLab Deep View RT model, use the following command.

deepview-validator 
    <path to the model>
    --dataset=<path to the dataset>
    --validate=segmentation 
    --model_segmentation_type=deeplab 
    --target=<EVK IP>:<modelrunner target>

Pose Models

deepview-validator 
    <path to the model> 
    --dataset=<path to the dataset>
    --validate=pose 
    --norm=unsigned

{
 "image": <path to the image>,
 "dimension": [width, height],
 "segment": [
 [
  {
   "polygon": [[x, y], [x,y], ...], [...], # polygon with x,y coordinates.
   "class": int
  },
  { ... }, ...
 ],
 [
  {
   "polygon": [[x, y], [x,y], ...], [...], # polygon with x,y coordinates.
   "class": int
  },
  { ... }, ...
 ]
]}

{
 "image": <path to the image>,
 "dimension": [width, height],
 "boxes": [[xc, yc, width, height, label], [...], ...], # Yolo box format.
 "angles": [[roll, pitch, yaw], [...], ...] # Euler angles by default, but Quaternion is also accepted.
}

Visit this Deep View Validator Validation Output to see examples and explanations of the validation outputs generated by the sample commands shown above. 

To see the plots and the images with visualizations, the --visualize parameter accepts a path to save these results in disk which will create and save into a directory named with the year and time of test. Alternatively, the plots and the images can be viewed using Tensorboard, the --tensorboard parameter accepts a path to save the tfevents file which contains the validation data that can be viewed in Tensorboard. 

To view the results in tensorboard, run the following command.  

tensorboard --logdir=<path to the tensorboard logs>

Appendix

Command Line Parameters

The following parameters are required parameters when running validation.

• model: This is the positional argument that points to the path of model to validate.
• -d, --dataset: This parameter accepts a path to the ground truth dataset to validate the model.

The following parameters specifies the dataset format provided.

• --labels_file: This is an optional parameter that specifies the path to a labels.txt file which validator will use to convert the model indices into string labels.
• --annotation_format: This parameter specifies the format of the annotations in the dataset. Options are ‘yolo’, ‘coco’, ‘pascalvoc’. Default annotation format is in ‘yolo’.
• --absolute_annotations: If this parameter is included in the command line, it means that the user has specified the annotations to be not normalized to the image dimensions. Normalized coordinates are values between 0 and 1.
• --include_background: This is for segmentation purposes which specifies to include background in the evaluations. For example, background to background pixels will be regarded as true predictions.
• --gt_label_offset: This is used for mapping ground truth integer labels to strings if unique string labels are provided via labels.txt or in the YAML file. This is to offset the integer label to the specific string label.
• --local_reader: This specifies to use local validator methods for reading datasets. Otherwise, by default, use deepview-datasets dependency.
• --letterbox: This specifies to perform letterbox transformations on the images prior to model inference.

The following command line parameters controls the model behavior that influences the prediction outputs.

• -e, --engine: This is the type of processor that will be used to run the model. Options are ‘cpu’, ‘gpu’, ‘npu’. Default engine used is ‘npu’. However, certain engines will be automatically specified depending on the availability of the processor or on the model such as "hailort" for Hailo models, "deepviewrt" for ModelClient, or "cuda" if available.
• --detection_threshold: This is the detection score threshold that will be used for NMS. Default detection score threshold is 0.50.
• --detection_score: This parameter is essentially the same as --detection_threshold. However, this is provided such that user can also use this parameter to set the detection score threshold. If both parameters are provided, a ValueError will be raised. Otherwise, one will overwrite the other.
• --validation_threshold: This is the validation score threshold that will filter the predictions to only validate the predictions with scores greater than or equal than the set threshold. Default validation score threshold is 0.00. No secondhand filtering is done by default, as this is already handled by the NMS.
• --validation_score: This parameter is essentially the same as --validation_threshold. However, this is provided such that the user can also use this parameter to set the validation score threshold. If both parameters are provided, a ValueError will be raised. Otherwise, one will overwrite the other.
• --detection_iou: This is the detection IoU threshold used by NMS to fine tune the bounding boxes to be included. The higher the IoU threshold might suggest less predictions because NMS will only consider bounding box predictions with IoU greater than or equal to the threshold. Default detection IoU threshold is 0.50.
• --validation_iou: This is the validation IoU threshold to consider true positives. Validator will match predictions to ground truth using the highest IoU. If the matches have the same labels and the IoU is greater than or equal to the set IoU threshold, then these matches are true positives. However, if the matches have the same labels, but the IoU is less than the set IoU threshold, then the matches are considered as localization false positives. Default validation IoU threshold is 0.50.
• -l, --label_offset: On some occasions, the model indices will not match to the ground truth indices. This parameter gives the user opportunities to offset the prediction indices so that they are much more in line with the ground truth indices. Default label offset parameter is 0.

Mismatches can occur because of the orientation of the labels… For example, If the ground truth indices reflects [person, car] and the prediction indices reflects [background, person, car]. Then the model may predict an index of 2 for car, but the ground truth may store an index of 1 for the same object. In this case, provide a label offset of -1 to match ground truth and predictions.

• -b, --box_format: This specifies the box format to output the prediction bounding boxes. By default, ‘xyxy’ is set. Other options include ‘xywh’ and ‘yxyx’.
• -n, --norm: This is the normalization method to apply for each image being passed to the model. Options include ‘raw’, ‘unsigned’, ‘signed’, ‘imagenet’, ‘whitening’. However, as a default ‘raw’ is assigned. Note: ‘imagenet’ and ‘whitening’ only applies to Deep View RT models.
• -m, --max_detection: This specifies the number of maximum detections to output for each image. By default, Keras models are set to 100 and Deep View RT models are set to 25 which is a VAAL default.
• -w, --warmup: This specifies the number of warmup iterations to exercise the model. Performing warmup iterations ensure model prediction and timing consistencies. By default, warmup is set to 0.
• -s, --nms_type: This specifies the non max suppression to be applied to the model predictions. Options include ‘standard’, ‘fast’, ‘matrix’, 'tensorflow'. For Deep View RT, ‘fast’ is by default. This does not apply to other models. Certain models only have single options such as Keras can only use "tensorflow" NMS. 
• --decoder: This parameter is primarily used for validating Deep View RT detection models using the ModelClient library. If the provided model does not have embedded decoder, then decoder needs to be applied externally within validator to allow model inference. Having this parameter in the command line suggests that the model does not have embedded decoder and must be applied externally.
• --model_box_type: This parameter is primarily used for Deep View RT detection models using the ModelClient library. This specifies the type of the Deep View RT model being validated. Options include ‘modelpack’ or ‘yolo’. By default, it is ‘modelpack’.
• --model_segmentation_type: This parameter is used for validating Deep View RT segmentation models. Currently validator supports Deep View RT segmentation models that are either ‘modelpack’ or ‘deeplab’. Validator needs specification for the type of segmentation model that is being validated. By default, it is ‘modelpack’.
• --offline_annotation_format: When the model provided does not include an extension, validator will assume that the path to the model points to a directory containing text files with prediction annotations. The user needs to specify the format of these annotations. By default, it is set to ‘yolo’. Other options include ‘coco’ and ‘pascalvoc’.
• --class_filter: Filter the model detection classes to allow only the classes present in the ground truth. However, a requirement is that the model classes should be more than the ground truth classes and the matching classes between the dataset and the model should be atleast in the same order. A common use case for this would be to validate multi-class models with a single class dataset.  Ex. A coco model on a people detection dataset.

The following command line parameters are extra options that validator requires for more specification.

• -V, --version: This outputs the version of validator.
• --exclude_plots: This option will exclude the plots from being shown when the visualize or the tensorboard parameter is specified. Furthermore, it will also exclude the data for the plots from the return summary of the evaluation.
• --exclude_symbols: Logging messages on the terminal will exclude symbols for warning, errors, and information.
• --clamp_box: This option will specify to clamp the bounding boxes from both detection and ground truth with height or width less than this value which will modify those smaller bounding boxes with height and width as this value in pixels.
• --ignore_box: This option will ignore any bounding boxes from the detection and ground truth with height or width less than this value in pixels.
• --metric: This option will specify which metric to consider when performing detection to ground truth matches. By default validator will use `iou` to match the detection to ground truth based on highest calculated IoU. Another option is `centerpoint` which will measure the distances between centres of bounding boxes and the smallest distances will get matched.
• --leniency_factor: Specify the criteria to consider center distances. This is the number of times the smallest bounding box diagonal (center to corner) should fit in the box-box center distance. This is necessary to avoid matches of bounding boxes that are visually far apart.
• --disable_auto_offset: Currently Deep View Validator performs auto offset of the model label indices based on its output shape to try to match to the ground truth indices. This parameter disables the auto offset functionality.
• --validate: Validator provides the option to validate detection, segmentation, or pose models. However, validator cannot tell if the provided model file is either a detection, segmentation, or pose model. The user must specify the type of model. By default, it is set to ‘detection’.
• --validate_3d: This specifies to perform 3D detection validation. A 3D dataset needs to be provided for this to work properly.
• --yolov5: This performs YoloV5 matching of detections to ground truths.
• --application: This option will specify which library to run Deep View RT models. Options are only either ‘vaal’ or ‘modelclient’. If ‘modelclient’ is specified, the user must specify the target to the modelrunner that was initialized in a target device such as an EVK. By default, application is set to ‘vaal’.
• --target: When using the ModelClient library, the user needs to specify the modelrunner target that is running in a target device such as an EVK. This will allow the model to use the endpoint for inference.
• --show_missing_annotations: For Darknet datasets, some images may not have a text file pair. Validator will still allow the model to run against those images, but the predictions will be classified as localization false positives. Regardless, this option will notify the user the names of the images that have no annotations.
• --display: This specifies how many images with visualizations (bounding boxes or masks) to display on to tensorboard or to save in the local machine. By default, it is set to -1 suggesting all the images.
• --visualize: This specifies the path to save the images with visualizations locally in disk. If this parameter is not included in the command line, no images will be saved.
• --tensorboard: This specifies the path to save the tfevents file that stores information to be displayed onto tensorboard. If this parameter is not included in the command line, no tfevents files will be saved in disk.

To display the images using tensorboard run the following command.

tensorboard --logdir=<path to the tfevents file(s)>

• --json_out: This specifies the path to save the JSON file containing the validation summary. If the path provided does not exist, it will create the path. It is possible to provide the name of the JSON file in the path, for example, “/home/root/results/myresults.json”. If only the directory is provided, it will name the file as “results.json” under the specified directory.

To command line options can be viewed using this command.

deepview-validator --help

Conclusion

This article has shown how to setup and use Deep View Validator to validate Deep View RT, Keras, TFLite, and ONNX models across different tasks such as object detection, semantic segmentation, and pose.

The purpose of this tool is to evaluate the performance of the model showing the final metrics for precision, recall, and accuracy, and as well as show the class metrics that can be observed with the generated plots for precision vs. recall, confusion matrix, and the class metric histogram. 

Furthermore, the tool can also evaluate the inference speed of the model and additional timing information concerning the application such as the time it takes to preprocess the image either through resizing or letterbox transformations or the time it takes to post process the model outputs into proper bounding boxes, scores, and labels either through NMS operations, index to string label mapping, or the proper reshaping of the outputs.

Additional Articles

The article Deep View Validator Detection Classification shows how the model detections are classified as true positives, false positives, and false negatives.

The article Deep View Validator Matching and Classification Rules describes the established rules for matching detections to ground truth in Deep View-Validator. 

The article Deep View Validator Object Detection Metrics shows how the metrics, and the timings are calculated in Deep View-Validator. 

The article Deep View Validator Semantic Segmentation Metrics shows how the metrics are calculated for segmentation tasks. 

The article Deep View Validator Validation Output describes the validation outputs.