How to use Gstreamer AppSink in Python

5 min. read |

The most convenient way to integrate video streaming pipeline into your application is through Gstreamer’s appsink plugin. In the following post we’ll explore Python GObject API on how to receive video frames from gstreamer pipeline in Python.

Requirements

Note: gstreamer-python hides a lot of code under-the-hood so developer can reuse it in own apps and focus more on video analytics. For example, recall from previous post How to launch Gstreamer pipeline in Python. There we have to handle explicitly GObject.MainLoop and Gst.parse_launch. With gstreamer-python we simply use GstContext and GstPipeline instead.

Code

Learn how to?

Why use AppSink instead of writing Plugin to access buffers from pipeline?

Recall from “How to write Gstreamer plugin with Python“. There we implemented Blur Filter as a Gstreamer Plugin. In such a case we can access gstreamer buffer inside plugin’s chain() or transform() method. So, let’s have a look on the following table and check when AppSink is suitable:

ApproachProsConsConclusion
AppSink1. Simple to implement;

2. Ability to batch processing
1. Buffer duplication– use for prototypes (pros 1)
– gives more flexibility, less performance (cons 1)
Filter Plugin1. Accessing/editing buffer in-place
1. Require prior gstreamer knowledge;

2. Ability to batch processing only with proper Mixer plugin

3. Gstreamer Python Bindings have some limitations (ex.: working with metadata)
– use for well-defined pipelines (cons 1)
– gives less flexibility (cons 3), but more performance (pros 1)

Note:

  • OpenCV uses approach with appsink/appsrc to pop/push frame buffer from/into gstreamer pipeline
  • Most video-analytics frameworks uses plugins to integrate deep learning models into gstreamer pipeline

Guide

Define Gstreamer Pipeline

For example, we are going to take simple pipeline:

  • videotestsrc generates buffers with various video formats and patterns.
  • capsfilter is used to specify desired video format.
  • appsink with <emit-signals> property enabled. To receive buffers from pipeline in our application.

Note: For any gstreamer pipeline, replace other sink element with appsink instead to be able to receive buffers in your application.

Write Python Script

Import gstreamer into python script

First, initialize GstContext (aka GObject.MainLoop routine)

Then, create GstPipeline (aka Gst.parse_launch)

Subscribe to <new-sample> event, that raises by appsink element.

Implement callback function to handle <new-sample> event. Callback’s arguments are appsink element itself and user data. Callback returns Gst.FlowReturn.

Appsink lets user to receive gstreamer sample (Gst.Sample, wrapper on Gst.Buffer with additional info). For this purpose just emit <pull-sample> with appsink.

Now, let extract Gst.Buffer from Gst.Sample

Also, from gstreamer sample it is possible to extract additional information about video frame, for example, resolution.

Note: import utils from gstreamer-python package to receive number of channels by specific format

Knowing the video frame resolution we can easily to convert Gst.Buffer to numpy array, so we can feed array further to video analytics pipeline.

Note: utils.get_np_dtype implemented in gstreamer-python to get proper np.dtype for any video format.

Place <while loop> to wait until pipeline processing end.

Run example

First, clone and install gst-python-tutorials project on your PC

Now, everything is ready to run test command

You should receive similar output

You can easily play with -p and specify any pipeline you want (resolution, format, plugins, …). Checkout awesome gstreamer pipelines.

Hope everything works as expected 🙂 . In case of any bugs, errors, suggestions – leave comments.

Conclusion

Gstreamer Appsink is simple way to receive video frames from pipelines of any complexity and forward them to further video processing pipeline in Python.

2 Comments

Add a Comment

Your email address will not be published. Required fields are marked *