ModelPack Validation

ModelPack Validation

1. Once the model has completed the training session, we are ready to start validation. ModelPack validation is straightforward when using eIQ Portal. We just need to click the VALIDATE button and go to the validation window.

2. Inside the Validation windows, we are able to select quantization or float validation, and also the validation target. For this exercise we'll select the local target which comes with an internal validator that uses computer CPU by default:

   • Float Model Validation (Do not apply any change on Input/Output data types)
   • Per-Channel Quantization (uint8 inputs and int8 outputs). In this case we need to activate Post Training Quantization and change the input/outputs datatype . Regarding the number of samples, there is an important note to add. To select more samples does not lead us to a better accuracy, actually, to select a lower number of samples is recommended for some datasets. In this case, we recommend to select 5 (see image below to corroborate optimal configuration). 
   • Per-Tensor Quantization (uint8 inputs and int8 outputs). Configuration from per-channel should be the same for per-tensor, Only activate the Per Tensor action button.

3. Expected results. It is expected a tiny drop when evaluating per-tensor quantized model. In this case, we have a drop of less than 1%,  which is a very good tradeoff between accuracy and speed. In some units, per-tensor quantized models tend to be faster than per-channel.  For Maivin and EVK running on an i.MX8MP Plus, it is recommended to use per-tensor instead per-channel if the loss in accuracy is small.