Deep Learning with End to End Gstreamer Pipelines¶
Both Running Simple demos and Running Advance demos uses GStreamer plugins for pre processing, composition and display. Post processing and DL inference are done between appsink and appsrc application boundaries. This makes the data flow sub-optimal because of unnecessary data format conversions to work with open source components.
This is solved by providing DL-inferer plugin which calls one of the supported DL runtime and a post-process plugin which works natively on NV12 format, avoiding unnecessary color formats conversions.
Below are some examples which demonstrates an end-to-end pipeline. These pipelines can be copied as-is and launched on the target
Single Input Single Inference Pipeline¶
Below pipeline reads a video file, decode, perform DL inference, draw boxes on detected objects and display
gst-launch-1.0 \
filesrc location=/opt/edge_ai_apps/data/videos/video_0000_h264.mp4 ! \
qtdemux ! h264parse ! v4l2h264dec capture-io-mode=dmabuf-import ! tiovxmemalloc pool-size=8 ! \
video/x-raw, format=NV12 ! \
\
tiovxmultiscaler name=scale \
\
scale. ! queue ! video/x-raw, width=320, height=320 ! \
tiovxdlpreproc channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
queue ! post.tensor \
\
scale. ! queue ! post.sink \
\
tidlpostproc name=post model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
tiovxmosaic sink_0::startx="<320>" sink_0::starty="<150>" ! \
video/x-raw, width=1920, height=1080 ! queue ! \
tiperfoverlay title="Object Detection" ! kmssink sync=false driver-name=tidss
Fig. 26 Optimized data-flow for object detection with single video input, inference and display¶
Single Input Multi Inference Pipeline¶
Below pipeline reads a video file, decode, perform 4 x DL inference, draw boxes on detected objects, composite and display
gst-launch-1.0 \
filesrc location=/opt/edge_ai_apps/data/videos/video_0000_h264.mp4 ! \
qtdemux ! h264parse ! v4l2h264dec capture-io-mode=dmabuf-import ! tiovxmemalloc pool-size=8 ! \
video/x-raw, format=NV12 ! \
tee name=split \
\
split. ! queue ! tiovxmultiscaler name=scale1 \
split. ! queue ! tiovxmultiscaler name=scale2 \
\
scale1. ! queue ! video/x-raw, width=320, height=320 ! \
tiovxdlpreproc target=0 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
queue ! post1.tensor \
\
scale1. ! queue ! video/x-raw, width=640, height=360 ! post1.sink \
\
scale1. ! queue ! video/x-raw, width=454, height=256 ! tiovxdlcolorconvert ! video/x-raw, format=RGB ! \
videobox qos=True left=115 right=115 top=16 bottom=16 ! \
tiovxdlpreproc out-pool-size=4 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TVM-CL-3090-mobileNetV2-tv ! \
queue ! post2.tensor \
\
scale1. ! queue ! video/x-raw, width=640, height=360 ! post2.sink \
\
scale2. ! queue ! video/x-raw, width=512, height=512 ! \
tiovxdlpreproc target=1 out-pool-size=4 data-type=3 ! \
tidlinferer model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 ! \
queue ! post3.tensor \
\
scale2. ! queue ! video/x-raw, width=640, height=360 ! post3.sink \
\
scale2. ! queue ! video/x-raw, width=320, height=320 ! \
tiovxdlpreproc target=1 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
queue ! post4.tensor \
\
scale2. ! queue ! video/x-raw, width=640, height=360 ! post4.sink \
\
tidlpostproc name=post1 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! mosaic. \
tidlpostproc name=post2 model=/opt/model_zoo/TVM-CL-3090-mobileNetV2-tv ! mosaic. \
tidlpostproc name=post3 model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 ! mosaic. \
tidlpostproc name=post4 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! mosaic. \
\
tiovxmosaic name=mosaic \
sink_0::startx="<320>" sink_0::starty="<180>" \
sink_1::startx="<960>" sink_1::starty="<180>" \
sink_2::startx="<320>" sink_2::starty="<560>" \
sink_3::startx="<960>" sink_3::starty="<560>" ! \
video/x-raw, width=1920, height=1080 ! queue ! \
tiperfoverlay title="Single Input Multi Inference" ! kmssink sync=false driver-name=tidss
Multi Input Multi Inference Pipeline¶
Below pipeline reads a video file and capture from USB Camera, perform 2 x DL inference on each stream, draw boxes over detected objects, composite and display
gst-launch-1.0 \
filesrc location=/opt/edge_ai_apps/data/videos/video_0000_h264.mp4 ! \
qtdemux ! h264parse ! v4l2h264dec capture-io-mode=dmabuf-import ! tiovxmemalloc pool-size=8 ! \
video/x-raw, format=NV12 ! \
tee name=input1 \
\
v4l2src device=/dev/video2 ! image/jpeg, width=1280, height=720 ! \
jpegdec ! tiovxdlcolorconvert ! video/x-raw, format=NV12 ! \
tee name=input2 \
\
input1. ! queue ! tiovxmultiscaler name=scale1 \
input2. ! queue ! tiovxmultiscaler name=scale2 \
\
scale1. ! queue ! video/x-raw, width=320, height=320 ! \
tiovxdlpreproc target=0 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
queue ! post1.tensor \
\
scale1. ! queue ! video/x-raw, width=640, height=360 ! post1.sink \
\
scale1. ! queue ! video/x-raw, width=454, height=256 ! tiovxdlcolorconvert ! video/x-raw, format=RGB ! \
videobox qos=True left=115 right=115 top=16 bottom=16 ! \
tiovxdlpreproc out-pool-size=4 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TVM-CL-3090-mobileNetV2-tv ! \
queue ! post2.tensor \
\
scale1. ! queue ! video/x-raw, width=640, height=360 ! post2.sink \
\
scale2. ! queue ! video/x-raw, width=512, height=512 ! \
tiovxdlpreproc target=1 out-pool-size=4 data-type=3 ! \
tidlinferer model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 ! \
queue ! post3.tensor \
\
scale2. ! queue ! video/x-raw, width=640, height=360 ! post3.sink \
\
scale2. ! queue ! video/x-raw, width=320, height=320 ! \
tiovxdlpreproc target=1 channel-order=1 data-type=3 ! \
tidlinferer model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! \
queue ! post4.tensor \
\
scale2. ! queue ! video/x-raw, width=640, height=360 ! post4.sink \
\
tidlpostproc name=post1 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! mosaic. \
tidlpostproc name=post2 model=/opt/model_zoo/TVM-CL-3090-mobileNetV2-tv ! mosaic. \
tidlpostproc name=post3 model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 ! mosaic. \
tidlpostproc name=post4 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! mosaic. \
\
tiovxmosaic name=mosaic \
sink_0::startx="<320>" sink_0::starty="<180>" \
sink_1::startx="<960>" sink_1::starty="<180>" \
sink_2::startx="<320>" sink_2::starty="<560>" \
sink_3::startx="<960>" sink_3::starty="<560>" ! \
video/x-raw, width=1920, height=1080 ! queue ! \
tiperfoverlay title="Single Input Multi Inference" ! kmssink sync=false driver-name=tidss