How to implement Object Detection in Video with Gstreamer in Python using Tensorflow?

     In this tutorial we are going to implement Object Detection plugin for Gstreamer using pre-trained models from Tensorflow Models Zoo and inject it into Video Streaming Pipeline.

     In previous posts we’ve already learnt How to create simple Gstreamer Plugin in Python and implemented simple Blur Filter as Gstreamer Plugin using OpenCV . Now let’s make a step forward.


How it works?

Setup Environment

  1. Download video (ex: video.mp4) (Note: check additional section to use get stream from raw Youtube link)
  2. Check simple Gstreamer pipeline works with downloaded video well

Note: video/x-raw,format=RGB (capsfilter plugin). Converts output format from previous plugin to input format of next plugin.

  1. Download and Extract ANY model from Tensorflow Models Zoo  (example: ssdlite_mobilenet_v2_coco).
  2. Get code

     3. Setup Environment using commands


Note: Install gst-python . Highly recommended to use guide from “How to install Gstreamer from sources” (Gst-Python Section). And before going to next part check that test case is working and no error being printed


     1. Edit config (format explained)

     2. Launch (with print to console)

Note: <GST_DEBUG=python:4> enables printing logs to console when launching pipeline (gstreamer debugging tools).

     In Terminal you should see something like this:

How to implement Object Detection in Video with Gstreamer in Python using Tensorflow?

Note: App should print something like this (list of dicts with object’s class_name, confidence, bounding_box):


     3. Launch (with object’s bounding boxes over video) 

How to implement Object Detection in Video with Gstreamer in Python using Tensorflow

Note: Look at gst_detection_overlay to check how drawing functions implemented


     The steps to create such a plugin a heavily built on top of this tutorial .

     Let’s dive.

  1. Define buffer format for SRC/SINK of plugin. Since Tensorflow Object Detection models works mostly with RGB define next templates (pay attention to video/x-raw,format={RGB}):

     2. Define properties for simple arguments passing to gstreamer plugin.

     First, you have noticed that plugin has “config” property. Config’s file format is described here (YAML). In general the main purpose of this config is to pass parameters to our Tensorflow Model (ex.: confidence threshold, model’s weights path, device to place graph, labels, etc.)

Note: Labels is also just a file with next format (YAML): <class_id: class_name>. For example:

     Also config file for Gstreamer Plugin is the common practice to pass more than 3 parameters, as it much easier to reuse and extend in future. For example, look at DeepStream SDK plugins.

     To allow Gstreamer Plugin get/set properties define next:

Note: Property “model” is used to pass existing model to plugin, without creating it. It could be useful if you want just to share single model without creating multiple instances.

     3. Implement chainfunc() with Tensorflow model Inference. So the main steps are: Convert Gst.Buffer to np,ndarray, perform inference of image, write objects to Gst.Buffer as metadata (recap: How to add metadata to gstreamer buffer). Check the following code:

     We won’t deep dive much into implementation on how to load Tensorflow model. Just have a look at code here. (If there any questions on those lines of code, leave a comment)


# Gstreamer + Youtube

     To run Gstreamer pipeline from Youtube link (without downloading the video file) we are going to use youtube-dl library. First install dependencies:\

     And let’s edit command from previous examples link described in here (gstreamer commands cheatsheet)

How to implement Object Detection in Video with Gstreamer in Python using Tensorflow?NoteNext command prints link (https) to Youtube Video with specified format (mp4)

And souphttpsrc is a plugin that allows to read buffers from http stream specified by url

     Hope everything works as expected 😉 In case of troubles with running code leave comments or open an issue on Github.

Add a Comment

Your email address will not be published. Required fields are marked *