Introduction
This article describes setting up the Maivin Dataset Publisher to prepare for model training an MCAP file which was captured with the Maivin Dataset Recorder. The Maivin Dataset Publisher can publish directly into Deep View Enterprise, export a collection of JPEG images with EXIF annotations, radar cube data, and pcd data into a directory, or export into rerun. The JPEG exporter can also generate appropriate annotations and meta-data in the Darknet YOLO format to use for training a YOLOv7 model.
This article is part of a series of articles documenting the end-to-end workflow for dataset capture, annotation, model training, and deployment back to the Maivin hardware. Refer to Maivin Dataset Overview for more details.
Requirements
Ubuntu 20.04 LTS or 22.04 LTS
Application
maivin-publisher-1.1.0-linux-amd64
Overview
The MCAP file recorded using the Maivin Dataset Recorder is optimized for real-time capture and playback of multiple data streams each running at various frequencies. A dataset to train a model needs to convert this data into a consistent collection of samples and annotations. Not all captured sensors are also relevant for model training, for example it is unlikely you'll need GPS locations at training time. The Maivin Dataset Publisher handles filtering for the relevant data streams (camera, radar, model annotations) and exporting at a consistent frequency.
Exporting the dataset to a consistent frequency involves selecting the reference sensor, for example the camera, and a target frequency such as one frame per second. The publisher will then sample the camera stream from the MCAP file at the desired frequency while also buffering the streams of secondary sensors which were selected for export. On each of the reference sensor frames at the target frequency, the most recent (previous) sample of the secondary sensors will also be captured and all will be saved as a single sample.
Maivin Publisher Settings
The Maivin Publisher has settings available for all sub-commands. All options are optionals.
maivin-publisher -l <LABELS> -t <THRESHOLD> --disable-topics <DISABLE_TOPICS> --time-offset <TIME_OFFSET> --relative-time-offset <> -f <FRAME_RATE> -r <REQ_IMAGE_TYPE> -p <PREFERRED_TOPIC> --skip-messages <SKIP_MESSAGES>
-l, --labels <LABELS>: A plain text file which contains the labels. The format should be label names separated by white space.
-t, --threshold <THRESHOLD>: Specify threshold for the radar point to be inside the box [default: 0.0].
--disable-topics <DISABLE_TOPICS>: Specify topics in MCAPs that should not be used. To specify a topic for all MCAPs simply write the complete topic, in order to disable a topic for a specific MCAP write the position in the list of MCAPs followed by ":" then the complete topic, e.g., "1:/radar/targets". To specify multiple topics enclose them in quotes and separate with white space.
--time-offset <TIME_OFFSET>: Used for manual synchronization of MCAP files The absolute time offset (in nanoseconds) for the (optional) second MCAP file. The second MCAP file is the file which does not contain the main topic. This offset should include the difference created from epochs
--relative-time-offset <RELATIVE_TIME_OFFSET>: Used for relative synchronization of MCAP files. The relative time offset (in nanoseconds) for the (optional) second MCAP file. The second MCAP file is the file which does not contain the main topic. This offset should not include the difference created from epochs or accounted for by the localtime topic
-f <FRAME_RATE>: The frame rate to choose images at. The default is the frame rate of the camera/video in the given MCAP file. A frame rate of 0 will have no effect [default: 0]
-r, --req-image-type <REQ_IMAGE_TYPE>: Specify the image type to use, either Zed image with TYPE replaced with ZED, or Compressed video with TYPE replaced with VIDEO [default: NONE] [possible values: NONE, ZED, VIDEO, JPEG]
-p, --preferred-topic <PREFERRED_TOPIC>: Specify the topic to use as the main topic.
--skip-messages <SKIP_MESSAGES>: Skips a certain number of messages specified in an MCAP. When specifying the number of messages to skip for multiple MCAPs enclose the numbers in quotes separated by white space; each number will correspond to an MCAP at that index.
Deep View Enterprise
Before publishing to Deep View Enterprise please ensure you've configured a project for Bridge In, refer to Publish Data to DVE - Bridge In for details.
To upload to DVE you will need to run the following command
maivin-publisher <MAIVIN PUBLISHER OPTIONS> dve --url <URL to DVE samples upload> --jwt <TOKEN> <Paths to the .mcap file>
--jwt <TOKEN>: Is REQUIRED and can be created through DVE by generating a API Token for a given dataset.
--url <URL>: Is optional since by default it sends data to DVE's sample/upload endpoint. [default: https://dveml.com/samples/upload]
Export to Darknet
In order to export a MCAP file you will need to run the following command
maivin-publisher <MAIVIN PUBLISHER OPTIONS> dir --disable <DISABLE> --out <PATH to where to store the darknet files> <Path to the .mcap file>
--frame-rate <RATE> is optional and is a desired number of frames you would like in a second. If the inputted frame rate is higher than the rate of the camera all frames are kept, same if the rate is 0. [default: 0]
--disable <DISABLE> disables saving particular types of output such as annotations (DARKNET), point cloud data (PCD), or radar cube(RC). When disabling multiple types of output enclose them in a string seperated by whitespace, e.g., "DARKNET PCD RC".
Once the program has exited without error you can find all images and optional annotations in the out-dir specified.
Export to Rerun
This flag of the publisher allows you to export to rerun, export to rerun using an address, or create a RRD file.
In order to export a MCAP file to Rerun
./maivin-publisher -m RERUN <Path to the .mcap file>
A rerun window should appear.
In order to export a MCAP file to Rerun through an address
./maivin-publisher -m RERUN --addr <address to computer> --port <Optional port number> <Path to the .mcap file>
--addr <ADDRESS> IP Address to the computer running rerun you wish to export to.
--port <PORT NUMBER> Optional and defaults to 9876.
A rerun window should appear.
In order to export a MCAP file to a RRD file
./maivin-publisher -m RERUN --out-file <Name of output file> <Path to the .mcap file>
Once the program has exited without error you can find the RRD file in the current directory with the name specified above.
Next Steps
With the dataset exported to either YOLO or Deep View Enterprise, the following articles explain how to train ModelPack or YOLOv7 with these datasets.
Once you've trained a model using one of the above workflows you can deploy to the Maivin.
- Deploy ModelPack to Maivin
- Deploy YOLO ONNX to Maivin (coming soon)
A deployed model's results can be captured with the Maivin Dataset Recorder into the annotation stream. This annotation stream can be published to datasets and audited as part of the model and dataset lifecycle.
Comments
0 comments
Please sign in to leave a comment.