How to write Gstreamer plugin with Python?

10 min. read |

Next guide shows steps to write Gstreamer Plugin in Python for any Computer Vision, Image Processing task and use it in standard Gstreamer pipeline from command line. Additionally this post explains how to get/set own properties from implemented Gstreamer plugin.

Requirements

Code

Learn how to?

  • create basic gstreamer plugins
  • set/get user defined properties for plugin

Use gstreamer plugins

Preface

Inspired by guide “How to write Gstreamer elements in Python”, where the author shows how to write Audio Source and Filter Elements, I wanted to create simple example on how to write Gstreamer plugins for Computer Vision/Image Processing purpose. Despite mentioned disadvantages of Python implementation of Gstreamer elements it is still faster than any other Python approach in image processing (like OpenCV), and more flexible as you can easily reuse this element in other Gstreamer pipelines without changing any code.

Guide

Extend GstBase.BaseTransform

GstBase.BaseTransform is base class for filter element that does data processing. The main parts or filter element are:

Description

Plugin’s description is stored in gstmetadata field as tuple with the following fields:

Note: detailed explanation of each field

This description is displayed when user calls gst-inspect-1.0. For example:

Pads

General scheme of any filter element looks like the following:

gstreamer element src sink
gstreamer element src sink

Sink and Src are implementations of Gst.Pad. In Pads and Capabilities there is well defined meaning and functions of pads.

To initialize specific pad, – define Gst.PadTemplate first. Gst.PadTemplate describes pad’s name, direction (sink, src), presense (always, sometimes, request), caps. Have a look at the following code to define sink’s Gst.PadTemplate that accepts buffers in Red Green Blue colorspaces format and it’s variations.

The, let do the same for src pad

In order to make pad templates visible for plugin, – just define gsttemplates field

Otherwise you’ll get next CRITICAL error:

If everything OK, you should be able to get next information after running gst-inspect-1.0:

Transform

As buffer flows from sink to src we need to override do_transform_ip (for buffer processing in-place) or do_transform (for out-buffer processing and in-buffer remains unchanged) in order to do custom buffer processing and push it to src’s pad.

In our case the main purpose of plugin is to blur image. So we need to convert Gst.Buffer to numpy array, make it writable (recap: How to make Gst.Buffer writable) and apply Gaussian Blur (using OpenCV) to it. Look how this could be implemented within do_transform_ip:

Properties

Most of gstreamer plugins have properties that are being initialized by user when specifying commands in terminal. As every Gst.Element is derived from GObject we can easily use it’s __gproperties__ metadata to define own properties. It’s well described in official documentation.

So to create own property, – fill the __gproperties__ dictionary (property-name: tuple) with your own properties and it’s description. Common template is the following:

Then properties for gaussian_blur plugin (sigmaX, sigmaY, kernel) could be defined by the following lines of code:

To use own properties just override do_get_property method and include implementation for custom properties handle.

And override do_set_property as well:

With gst-inspect-1.0 defined properties above looks like the following:

Registration

In order to use gstreamer plugin from command line, just put next two lines in the end of the file:

For example, for gaussian_blur plugin previous code is going to be similar to the following:

Run examples

Get repository

First clone the repository and setup environment:

Before running examples, export the following GST_PLUGIN_PATH, so gstreamer can locate newly created plugins:

Note: Explanation for GST_PLUGIN_PATH export

  • $GST_PLUGIN_PATH: previous path with plugin’s libs
  • $PWD/venv/lib/gstreamer-1.0/: path were gst-python was installed (–prefix value) and contains libgstpython*.so
  • $PWD/gst/: path with python/ directory which contains plugins implementation (*.py)

Launch simple command, to check how gstplugin_py works.

gstreamer plugin in python template

Notice that the properties “int-prop“, “float-prop“, “bool-prop“, “str-prop“, “pyobject-prop” successfully set to values specified by command line.

Next, let check gaussian_blur plugin:

Note: To see the difference I used basic video mixing pipeline to put two streams (with/without blur) in one window (Look at gstreamer cheat sheet, to learn more about classic pipelines)

Or let us do the same with a video:

gstreamer plugin in python template

The other option could be to do the same but directly from youtube video using “How to watch Youtube videos with Gstreamer”.

Troubleshooting

1. Remove gstreamer’s cache to reload plugins and display error messages when there is a problem.

2. Check GST_PLUGIN_PATH export as described above if you are receiving:

Conclusion

With gstreamer python bindings you can easily implement plugins for image processing (gaussian_blur). We learned basic steps to implement filter plugin from GstBase.BaseTransform:

  • description
  • pads
  • transform
  • properties
  • registration

Hope everything works as expected 😉 In case of troubles with running code leave comments or open an issue in Github.

15 Comments

Add a Comment

Your email address will not be published. Required fields are marked *