Introduction

This article will describe the rules governing the matching algorithm in Deep View Validator which is responsible for matching the ground truths to the detections based on best overlap (IoU) and allows classification of detections into true positives, false positives, and false negatives.

Prerequisites

• Familiarity of the definitions of true positives, false positives, and false negatives in Deep View Validator Detection Classifications.

Rules

Rule 1: Requirements of True Positives

Recap: A detection can be a true positive if it meets the requirements described below.

• The detection label matches the ground truth label.
• The detection and ground truth IoU >= validation IoU threshold.
• The detection confidence score >= detection score threshold (NMS) or validation score threshold (optional).

Note: validation score threshold is optional (default = 0.0) which the user can set to further filter detections based on score after NMS. Any detections with scores below the given score thresholds will be thrown out. 

Rule 2: Requirements of False Positives

Recap: False positives are categorized into localization and classification.

Classification false positives meet the requirements described below.

• The detection label does not match the ground truth label.
• The detection and ground truth IoU >= validation IoU threshold.
• The detection confidence score >= detection score threshold (NMS) or validation score threshold (optional).

Localization false positives meet the requirements described below.

• The detection and ground truth IoU < validation IoU threshold.
• The detection does not overlap any ground truth (IoU = 0).

Rule 3: Requirements of False Negatives

• Any ground truth where the detection IoU < validation IoU threshold.
• Any ground truth that does not overlap a detection.

Rule 4: Unique Matches

There exists only one detection (true positive or classification false positive) that is matched per ground truth. 

Rule 4.1: True Positive Count

The number of true positives should not exceed the number of ground truths nor the number of predictions.

1. Each ground truth may have at most, one true positive.
2. Each prediction may have at most, one true positive.

Rule 4.2: Classification False Positive Count

The number of classification false positives should not exceed the number of ground truths nor the number of predictions.

1. If a ground truth does not have a true positive, it may have at most, one classification false positive.
2. If a prediction does not have a true positive, it may have at most, one classification false positive.

Rule 4.3: Localization False Positives Count

For a dataset without any ground truth to detection overlaps, the number of localization false positives should be the total number of predictions.

Rule 4.4: False Negatives

For cases with no model detections, the number of false negatives is equal to the number of ground truth annotations.

Rule 5: Prioritization Cases

If a detection meets the requirements of a true positive, it will be matched to the ground truth. Otherwise, if a detection meets the requirements of a classification false positive, it will be matched to the ground truth. Otherwise, it is a localization false positive. Consider the following cases.

Rule 5.1: Matching Labels Are Prioritized Over IoU

Considering two detections and one ground truth, the detection with a lower IoU, but with the same label as the ground truth should be matched over a detection with a higher IoU, but with a mismatching label as long as both detections have IoU >= validation IoU threshold. 

The following images provides visualization to this rule. The validation IoU is set to 0.10 and the detection score is set to by default at 0.50 in these cases. 

Playing Cards v7; 000000000127.png

Playing Cards; 000020.png

Otherwise, if these detections, are lower than the validation IoU threshold, they will be regarded as localization false positives and one false negative from the ground truth as shown below.

000000000127.png; Validation IoU threshold = 0.90

000000000127.png; Validation IoU threshold = 0.90

Rule 5.2: Prioritization of Highest IoU Matches

For cases where multiple detections have labels that matches the ground truth, the highest IoU is matched.

The following image shows visualization of this rule where the validation IoU threshold is set to 0.10 and the detection score is set to by default at 0.50. 

Playing Cards V7; 000000000145.png

These detections have the same label as the ground truth, but the IoU varies, each detection overlaps the ground truth. The greatest overlap is taken as the match. Since this match meets the criteria of a true positive, it is classified as such. 

Rule 5.3: Matching Labels, Highest IoU Matches

For cases that concerns both scenarios as the rules described above. A match is based on a matching label and the highest IoU with the ground truth if multiple detections are present with varied labels and varied IoUs.

The following image shows a visualization of this rule following the same validation IoU threshold set to 0.10 and the detection score threshold set to 0.50.

Playing Cards v7; 000000000134.png

In this case, matching labels are prioritized and the highest IoU is taken as the match. 

Rule 5.4: Mismatching Labels, Highest IoU Matches

For cases having detections with matching labels as the ground truth, but do not meet the validation IoU requirements, the detections with mismatching labels are matched if it meets the validation IoU requirements.  

The following image with visualization shows a validation IoU threshold set to 0.50.

Playing Cards v7: 000000000134.png

Despite a detection with a matching label of "ace" as the ground truth being present, it is still regarded as a localization false positive since it does not meet the IoU requirements.  The detection with the highest IoU is matched instead which is with the label "king".

Otherwise, if all detections do not meet the validation IoU requirements, they will be treated as localization false positives and one false negative for the ground truth as shown in the image results below. 

Playing Cards v7: 000000000134.png

Rule 5.5: Matches Meets Validation IoU Requirements

Any detections that do not overlap any ground truth or any detections with IoU < validation IoU threshold will be regarded as localization false positives. 

Playing Cards v7; 000000000148.png

Conclusion

This article has shown the rules established in Deep View Validator that governs the behaviour of the matching and classification algorithms to process the raw detections into their classifications of true positives, false positives, and false negatives. Rules 1-3 shows the requirements of these classifications. Rule 4 describes the limits to the count of these classifications. Rule 5 shows the prioritization of matches to ensure classifications requirements are met by prioritizing finding of true positives first, then classification false positives, and then localization false positives.