Introduction

From ModelPack Training Guide, we have seen that trainer is capable of generating TensorBoard logs with the aim of showing the training progress. We provide a docker container that encapsulates a TensorBoard entry point deployable at port 6006. In this article we intend to explain how to visualize the training progress with a docker container.

Calling the Logging System

ModelPack logging system is a docker container shared under

deepview/tensorboard:latest

This container is already configured to deploy a TensorBoard in a specific location (/logs). 

docker run -it 
(1)  -p 9999:6006 
(2)  -v /data/outputs/tensorboard/<time-stamp>:/logs 
(3)   deepview/tensorboard:latest

1. Links internal docker port 6006 into 9999 port in the host-PC. Feel free to change the host-PC port, but not the internal one. That one is mandatory
2. Mount the host-pc /data/outputs/tensorboard/<time-stamp> path into /logs. Tensorboard is expecting a logging folder into the preset location /logs. To understand a bit more about the logs folder read ModelPack Training Guide.

3. Container's name, to view available tags refer to the DeepView TensorBoard Page.

NOTE:  if you set logs to the base TensorBoard directory ( /data/outputs/tensorboard/ in 2. ), all runs will be available for analysis.

Inspecting Results

To inspect the results, it is needed to open the browser with the following address: http://ip-host-pc:9999  The browser will open TensorBoard user interface.

In the top menu we can see the following different sections: 

Scalars

Within the scalars ModelPack trainer reports the values for each loss at each step.

• classification loss: Controls the power of correct classification for the model
• localization loss: Tells us how close are our predicted boxes from the ground truth boxes.
• objectness loss: This concept is associated to Yolo model. It computes the confidence of the model for all the predictions. It is like a correctness function that helps to reduce false positives during training. If the model detects higher probability object, but the confidence is too low, then the object won't be as good candidate for a final prediction.
• total loss: Adds all the separated loss and is the one the optimizer uses to train the model. The idea is the model reduces as much as possible the three losses, finding the best trade-off for all of them during the training session.

Images

In the images section we draw all the images from the validation set with overlay bounding boxes (GT and DT).

Text

The text section is used to store all the validation statistics. We start evaluating the weights before training the model and display them into step 0. After completing each training epoch, the model is evaluated again on validation samples and the results are reported in the next step in the following way.

Conclusions

In this article we have presented how to visualize the progress of any training session. Our ModelPack Logging System is capable of visualizing images with overlay boxes as well as capable of reporting an in-depth metrics report.