Introduction

This article will describe how the segmentation metrics for Deep View Validator is calculated.

Definitions

The equations for precision, recall, and accuracy are still applied for segmentation. 

• Precision = True Predictions / Total Predictions
• Recall = True Predictions / Total Ground Truths
• Accuracy = True Predictions / Total Ground Truths U Total Predictions

Deep View Validator will compute the average metrics for precision, recall, and accuracy. This is calculated as the sum of precision, recall, and accuracy per class divided by the number of classes.

Deep View Validator will also compute the overall metrics for precision, recall, and accuracy. This is calculated based on the prediction mask as a whole.

The next section will describe these computations in more detail. 

Sample

For simplicity, consider the following 5x2 image masks for the ground truths and the predictions. A and B are distinct classes, and BG represents that "Background" class that has the option to be include or excluded from the computations. 

Ground Truths

Predictions

To calculate the average metrics, the metrics will be calculated per class: A, B, BG (Optional).

Class A

The following table shows the classifications for class A. T is denoted as a true prediction, and F is denoted as a false prediction, and NULL means the classes do not involve A. 

Based on the metric definitions above, these are the metrics for class A.

• Precision = 2/3
• Recall = 2/2
• Accuracy = 2/3

Note: For the metrics above, there are 2 true predictions for A, 3 total predictions for A, and 2 total ground truths for A. 

Class B

The following table shows the classifications for class B.

These are the metrics for class B.

• Precision = 2/2
• Recall = 2/5
• Accuracy = 2/5

Note: There are 2 true predictions for B, 2 total predictions for B, and 5 total ground truths for B.

Class BG

Optionally, the background class BG is evaluated. The following table shows the classifications for BG.

These are the metrics for class BG.

• Precision = 3/5
• Recall = 3/3
• Accuracy = 3/5

Note: There are 3 true predictions for BG, 5 total predictions for BG, and 3 total ground truths for BG.

Average Metrics

Based on the metrics per class shown above, the average metrics is calculated based on the following formula.

Following the equations shown above, the average metrics for this sample is shown below.

Without Background

• Average Precision = [(2/3) + (2/2)]/2 = 5/6 ~= 0.83
• Average Recall = [(2/2) + (2/5)]/2 = 7/10 = 0.70
• Average Accuracy = [(2/3) + (2/5)]/2 = 8/15 ~= 0.53

With Background

• Average Precision = [(2/3) + (2/2) + (3/5)]/3 = 34/45 ~= 0.76
• Average Recall = [(2/2) + (2/5) + (3/3)]/3 = 4/5 = 0.80
• Average Accuracy = [(2/3) + (2/5) + (3/5)]/3 = 5/9 ~=0.56

Overall Metrics

The overall metrics is calculated based on the multiclass mask. Optionally, background is included.

The following tables shows the classifications of the whole prediction mask either with or without background.

Without Background

Note: NULL are spots with Background to Background which we do not include as true predictions.

The following shows the overall metrics based on the metric definitions in the first section.

• Overall Precision = 4/5 = 0.80
• Overall Recall = 4/7 ~= 0.57
• Overall Accuracy = 4/7 ~= 0.57

Note: There are 4 true predictions, 5 total predictions, and 7 total ground truths.

With Background

Note: Here background to background predictions would now be considered as true predictions. Also, the number of ground truths would equal the number of predictions.

The following shows the overall metrics based on the metric definitions in the first section.

• Overall Precision = 7/10 = 0.70
• Overall Recall = 7/10 = 0.70
• Overall Accuracy = 7/10 = 0.70

Note: There are 7 true predictions, 10 ground truths, and 10 predictions. Note the number of Union pixels does not necessarily translate to the number of ground truths. An example would be a ground truth being a background class, but the prediction is an object class, in such case, the union pixels would be greater than the ground truth pixels. 

Conclusion

This article has shown how segmentation metrics are calculated in Deep View Validator.