Introduction

This package contains the stand-alone deepview-validator that validates detection, semantic segmentation, or pose models. This package is capable of validating quantized DeepViewRT and float Keras models using either Darknet or TFRecord datasets. 

Step 1: Install Deepview-Validator

It is recommended to set up a python virtual environment before moving forward with the next steps to contain the downloaded libraries within the project.

Upgrade pip to ensure the latest python packages are installed.

pip install --upgrade pip

The validator package can be installed using:

pip install deepview-validator

Step 2: Configure Model Dependencies

Certain models will require certain dependencies to run.

• DeepViewRT models have an extension .rtm which require the VAAL library, but the library installation requires specific steps depending on the machine's processor that is used to run validator.   
  • DeepViewRT models that are being validated in the PC (x86_64 processors) will require Debian installations.
  • DeepViewRT models that are being validated in the EVK (ARMV8 processor) will require VisionPack.  
  • DeepViewRT models can also be run using the ModelClient library which is part of the DeepViewRT library also require the Debian package installations and the library can be installed using a terminal with the command:

sudo apt install deepview-rt-python

from deepview.rt.modelclient import ModelClient

• Keras models have an extension .h5 which require the Tensorflow library and can only be run in machines with an x86_64 processors. The library can be pip installed using the terminal with the command:

pip install tensorflow~=2.8.0

• TensorRT engines have an extension .trt which require TensorRT and PyCuda dependencies.
• TFLite models have an extension .tflite which require the tflite_runtime library and can be installed using a linux terminal with the command:

pip install tflite-runtime~=2.11.0

• If the model has no extension,  deepview-validator will assume it is a directory of text files that contains the model prediction annotations in the format described below:

  cls score xc yc width height

  This is the same as the YOLO format annotations except score immediately follows the class index.

Step 3: Configure Dataset Formats

Currently validation in the EVK (ARMV8 processor) only supports Darknet datasets because TFRecord datasets consume too much memory and poses TensorFlow issues that are still under investigation. Typical machines with x86_64 processors can support both Darknet and TFRecord datasets. 

Dataset annotations for detection can be ‘yolo’ (default), ‘coco’, or ‘pascalvoc’ format.

Deepview-validator accepts a path to a dataset that points to a YAML file or a directory that contains the validation dataset.

• For TFRecord datasets, the directory should contain *.tfrecord files or the YAML file should be structured in the following way:

classes:
 - class_1
 - class_2
 - class_3 
validation:
 path: "path to the tfrecord files"

• For Darknet datasets, the directory should contain images (jpg, png, jpeg, JPG, PNG, JPEG) and annotations (txt or JSON). The following directory structure is also accepted.

Directory Path
     |---------images
                  |---validate
                         |------- image.jpg ....
     |---------labels
                  |---validate
                         |------- image.txt ....
     |---------labels.txt (optional)

The YAML file should be structured in the following way:

classes:
 - class_1
 - class_2
 - class_3 
type: darknet
validation:
 annotations: "path to the annotation text files"
 images: "path to the image files"

Step 4: Run Validation

This section shows how to validate vision models with a specified task: detection, segmentation, or pose. 

Detection Models

The following command provides a template to run a detection model and validate its performance:

deepview-validator "path to the model" --dataset="path to the dataset"

Note: Substitute the values in quotation marks for the path to the model and the path to the dataset.

modelrunner -c 1 -H "modelrunner target"

deepview-validator 
    "path to the model" 
    --dataset="path to the dataset" 
    --target="EVK IP":"modelrunner target" 
    --application=modelclient

Segmentation Models

Note: Segmentation validation is only supported by Darknet datasets using JSON annotation files to contain the segments.

DeepViewRT segmentation models require the ModelClient library. The first step is to enable modelrunner in a target device such as an EVK.

To enable modelrunner, run the following command in the target device:

modelrunner -c 1 -H "modelrunner target" 

• To run a ModelPack DeepViewRT (rtm) model, run the following command:

deepview-validator  
    "path to the model" 
    --dataset="path to the dataset" 
    --validate=segmentation 
    --target="EVK IP":"modelrunner target"

• To run a DeepLab DeepViewRT (rtm) model, run the following command:

deepview-validator 
    "path to the model" 
    --dataset="path to the dataset" 
    --validate=segmentation 
    --model_segmentation_type=deeplab 
    --target="EVK IP":"modelrunner target"

• Keras segmentation models can be run using the following command:

deepview-validator
    "path to the model" 
    --dataset="path to the dataset" 
    --validate=segmentation

Note: Keras segmentation models don't need a modelrunner target running in the background. 

Pose Models

deepview-validator 
    "path to the model" 
    --dataset="path to the dataset" 
    --validate=pose 
    --norm=unsigned

Visit this DeepView-Validator Validation Output to see examples and explanations of the validation outputs generated by the sample commands shown above. 

To see the plots and the images with visualizations, the --visualize parameter accepts a path to save these results in disk which will create and save into a directory named with the year and time of test. Alternatively, the plots and the images can be viewed using Tensorboard, --tensorboard parameter accepts a path to save the tfevents file which contains the validation data that can be viewed in Tensorboard. 

To view the results in tensorboard, run the following command.  

tensorboard --logdir="path to the tensorboard logs"

Appendix

Command Line Parameters

The following parameters are required parameters when running validation.

model: This is the positional argument that points to the path of model to validate.

-d, --dataset: This parameter accepts a path to the ground truth dataset to validate the model.

The following parameters specifies the dataset format provided

--labels_file: This is an optional parameter that specifies the path to a labels.txt file which validator will use to convert the model indices into string labels.

--annotation_format: This parameter specifies the format of the annotations in the dataset. Options are ‘yolo’, ‘coco’, ‘pascalvoc’. Default annotation format is in ‘yolo’.

--absolute_annotations: If this parameter is included in the command line, it means that the user has specified the annotations to be not normalized to the image dimensions. Normalized coordinates are values between 0 and 1.

The following command line parameters controls the model behavior that influences the prediction outputs.

-e, --engine: This is the type of engine that will be used to run the model. Options are ‘cpu’, ‘gpu’, ‘npu’. Default engine used is ‘npu’.

--detection_threshold: This is the detection score threshold that will be used for NMS. It is suggested to specify this parameter low so that validator sees a variety of prediction scores that can be used to cleanly plot the precision vs recall curve from score thresholds 0 to 1. Default detection score threshold is 0.01.

--validation_threshold: This is the validation score threshold that will filter the predictions to only validate the predictions with score greater than or equal than the set threshold. Default validation score threshold is 0.25.

--detection_iou: This is the detection IoU threshold used by NMS to fine tune the bounding boxes to be included. The higher the IoU threshold might suggest less predictions because NMS will only consider bounding box predictions with IoU greater than or equal to the threshold. Default detection IoU threshold is 0.45.

--validation_iou: This is the validation IoU threshold to consider true positives. Validator will match predictions to ground truth using the highest IoU. If the matches have the same labels and the IoU is greater than or equal to the set IoU threshold, then these matches are true positives. However, if the matches have the same labels, but the IoU is less than the set IoU threshold, then the matches are considered as localization false positives. Any mismatching labels are considered as classification false positives. Default validation IoU threshold is 0.50.

-l, --label_offset: On some occasions, the model indices will not match to the ground truth indices. This parameter gives the user opportunities to offset the prediction indices so that they are much more in line with the ground truth indices. Default label offset parameter is 0.

Mismatches can occur because of the orientation of the labels… For example, If the ground truth indices reflects [person, car] and the prediction indices reflects [background, person, car]. Then the model may predict an index of 2 for car, but the ground truth may store an index of 1 for the same object. In this case, provide a label offset of -1 to match ground truth and predictions.

-b, --box_format: This specifies the box format to output the prediction bounding boxes. By default, ‘xyxy’ is set. Other options include ‘xywh’ and ‘yxyx’.

-n, --norm: This is the normalization method to apply for each image being passed to the model. Options include ‘raw’, ‘unsigned’, ‘signed’, ‘imagenet’, ‘whitening’. However, as a default ‘raw’ is assigned.

Note: ‘imagenet’ and ‘whitening’ only applies for DeepViewRT models.

-m, --max_detection: This specifies the number of maximum predictions to output for each image. By default, Keras models are set to 100 and DeepViewRT models are set to 25 which is a VAAL default.

-w, --warmup: This specifies the number of warmup iterations to exercise the model. Performing warmup iterations ensure model prediction and timing consistencies. By default, warmup is set to 0.

-s, --nms_type: This specifies the non max suppression to be applied to the model predictions. Options include ‘standard’, ‘fast’, ‘matrix’. For DeepViewRT, ‘fast’ is by default. This does not apply to other models.

--decoder: This parameter is primarily used for validating DeepViewRT detection models using the modelclient library. If the provided model does not have embedded decoder, then decoder needs to be applied externally within validator to allow model inference. Having this parameter in the command line suggests that the model does not have embedded decoder and must be applied externally.

--model_box_type: This parameter is primarily used for DeepViewRT detection models using the modelclient library. This specifies the type of the DeepViewRT model being validated. Options include ‘modelpack’ or ‘yolo’. By default, it is ‘modelpack’.

--model_segmentation_type: This parameter is used for validating DeepViewRT segmentation models. Currently validator supports DeepViewRT segmentation models that are either ‘modelpack’ or ‘deeplab’. Validator needs specification for the type of segmentation model that is being validated. By default, it is ‘modelpack’.

--offline_annotation_format: When the model provided does not include an extension, validator will assume that the path to the model points to a directory containing text files with prediction annotations. The user needs to specify the format of these annotations. By default, it is set to ‘yolo’. Other options include ‘coco’ and ‘pascalvoc’.

The following command line parameters are extra options that validator requires for more specification.

--exclude_plots: This option will exclude the plots from being shown when the visualize or the tensorboard parameter is specified. Furthermore, it will also exclude the data for the plots from the return summary of the evaluation.

--clamp_box: This option will specify to clamp the bounding boxes from both detection and ground truth with height or width less than this value provided to this value which will modify those smaller bounding boxes with height and width as this value in pixels.

–validate: Validator provides the option to validate detection models or segmentation models. However, validator cannot tell if the provided model file is either a detection model or a segmentation model. The user must specify the type of model. By default it is set as ‘detection’.

–application: This option will specify which library to run DeepViewRT models. Options are only either ‘vaal’ or ‘modelclient’. If ‘modelclient’ is specified, the user must specify the target to the modelrunner that was initialized in a target device such as an EVK. By default, application is set to ‘vaal’.

–target: When using modelclient library, the user needs to specify the modelrunner target that is running in a target device such as an EVK. This will allow the model to use the endpoint for inference.

–show_missing_annotations: For Darknet datasets, some images may not have a text file pair. Validator will still allow the model to run against those images but the predictions will be classified as localization false positives. Regardless, this option will notify the user the names of the images that have no annotations.

–display: This specifies how many images with visualizations (bounding boxes or masks) to display on to tensorboard or to save in the local machine. By default it is set to -1 suggesting all the images.

–visualize: This specifies the path to save the images with visualizations locally in disk. If this parameter is not included in the command line, no images will be saved.

–tensorboard: This specifies the path to save the tfevents file that stores information to be displayed for tensorboard. If this parameter is not included in the command line, no tfevents files will be saved in disk.

To display the images using tensorboard run the following command,

tensorboard --logdir='path to the tfevents file(s)'

–json_out: This specifies the path to save the JSON file containing the validation summary. If the path provided does not exist, it will create the path. It is possible to provide the name of the JSON file in the path, for example, “/home/root/results/myresults.json”. If only the directory is provided, it will name the file as “results.json” under the specified directory.

To see more available command line options to specify model or dataset parameters, run the command:

deepview-validator --help

Conclusion

This article has shown how to setup and use Deepview-Validator to validate DeepViewRT and Keras models across different tasks such as detection, semantic segmentation, and pose.

The purpose of this tool is to evaluate the performance of the model showing the final metrics for precision, recall, and accuracy, and as well as show the class metrics that can be observed with the generated plots for precision vs. recall, confusion matrix, and the class metric histogram. 

Furthermore, the tool can also evaluate the speed of the model generating timing information for the time it takes to load an image, to generating and post process predictions.

To learn more about the validation outputs, visit this Deepview-Validator Validation Output.

To learn more about how model predictions are classified as true positives, false positives, and false negatives, visit this Deepview-Validator Model Prediction Classifications.