Creating a usable model

How to train and upload model using eIQ?

Step 1: Open eIQ portal and create a new project

Step 2: Using the “Remote Devices” button on the top right add your Vision kit IPADDR and any name

Step 3: To create a dataset start taking images by selecting a remote device from the dropdown menu and then tapping “capture” button  

Step 4: Once you have enough images for the dataset its time to add labels

Step 5: It is smarter to add bounding boxes and adding labels as that image will work for both types

Step 6: Using “Dataset Test Holdout” on top right corner select the % of test images from the dataset

Step 7: Now select “Classification” or “Detection” Model

Step 8: Once model type is selected set meta data to get the best results

Step 9: Train said model and wait till you have a good enough model

Step 10: Now to validate on the vision kit select board ipaddr from “validation target” and hit validate

Step 11: Once validation is complete navigate to http://” ipaddr of board” to test the model in real time

Step 12: To save model hit “Deploy” and “Export” the model

Model Supported:

There are multiple models that the user can use but the most recommended are mentioned below.

1) Mobilenet_ssd_v2 Lite (Quant is recommended)

2) Mobilenet_ssd_v3 Small/Large (Small is recommended). This one in the zoo quantizes poorly, so need to use trimmed float version. (Readme might need updates for where to trim)

Playing Card’s Model:

Introduction:

A total of 2016 pictures were taken to train the models out of which ~2000 was to train the model and rest were to test the said models. To get a more robust model images were taken with different surrounding such for e.g., different background, multiple objects in the frame and different lighting condition. The images were very diverse in terms of how many cards were in one image and had multiple orientations of the cards.

The augmentation tool was used to increase the number of images used for training to generate a higher quality trained model for a relatively small number of training images.   The augmentation tool may be used to change brightness, contract, saturation, orientation etc.

For this Cards performance detection model the following settings were used:

Classification Model:

A classification model is typically used for imaging processing a known region of interest in a captured image.   Classification models have lower computational requirements compared to detection models, so they will have faster inference times.   If the region of interest is fixed for a given use case, then a classification model would typically be a good fit. For example, if the detection of a vehicle in a specific parking spot could use a classification model, while the detection of multiple vehicles in random locations would require a detection model.   

For the included playing cards example, thirteen different classes were defined for Ace – King. When you present a card, it can classify it based on the images which were used to train the model.  This model does not require long training times (roughly 1 hour) and works well with small dataset. Furthermore, in the eIQ portal the classification model can be divided in 3 subcategories:

All the models created below are done on the MCU (Micro Controller Units).

Performance – This model focusses on a speed and gets the lowest possible run times.

Our performance model was trained to 41 epochs with a learning rate of 0.00001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 128,128. The result we got had an accuracy of ~92%. This model when combined with the MCU model is best suited for hardware where memory is an issue.

Balanced – This model focuses on finding the balance between the performance and accuracy model.

Our balance model was trained to 43 epochs with a learning rate of 0.00001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 128,128. The result we got had an accuracy of ~95%.

Accuracy – This model focuses on getting the most accurate results possible; however, the trade-off is to have longer runtime.

Our accuracy model was trained to 41 epochs with a learning rate of 0.00001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 128,128. The result we got had an accuracy of ~96%.

Conclusion:

From the above images we can see that the runtime drastically increases when going from the performance to accuracy model however the accuracy of that model is much better that another model. So, when deciding which model to go with it boils down to what is the application of the model, if the model is required to process data quicker it might be best to go with performance if the problem requires higher accuracy the accuracy makes more sense. Finally, if not sure there is always that balance option to get the best middle ground.

Detection:

A model that learns to report what card can be found in the image and where it is found. The detection model can report more than one card in the image and report the label and confidence it has in its results. To get a highly efficient model high training time is required along with a large data set. 

The model is trained with 13 different classes and can detect any card out of the 13 classes. The training time for this model was extremely high as it took nearly 24 hours to get a useable model. Furthermore, in the eIQ portal the detection model can be divided in 3 subcategories:

All the models created below are done on the MCU (Micro Controller Units).

Performance – This model focusses on a speed and gets the lowest possible run times.

Our performance model was trained to 112 epochs with a learning rate of 0.0001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 320,320. The result we got had an accuracy of ~87%. This model when combined with the MCU model is best suited for hardware where memory is an issue. Training this model took about 12 hours.

Balanced – This model focuses on finding the balance between the performance and accuracy model.

Our performance model was trained to 207 epochs with a learning rate of 0.0001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 320,320. The result we got had an accuracy of ~90%. This model when combined with the MCU model is best suited for hardware where memory is an issue. The model had 37000 steps and took about 17 hours.

Accuracy – This model focuses on getting the most accurate results possible; however, the trade-off is to have longer runtime.

Our performance model was trained to 311 epochs with a learning rate of 0.0001 and batch size 10. Furthermore, the optimizer used was “Adam” and input size was 320,320. The result we got had an accuracy of ~96%. This model when combined with the MCU model is best suited for hardware where memory is an issue. The model had around 61000 steps and took about 24 hours.

Conclusion:

From the above images we can see that the training time change when going from the performance to accuracy model however the accuracy of that model is much better that another model. So, when deciding which model to go with it boils down to what is the application of the model, if the model is required to process data quicker it might be best to go with performance if the problem requires higher accuracy the accuracy makes more sense. Finally, if not sure there is always that balance option to get the best middle ground.