How to draw kitten on video with Gstreamer faster?

     Gstreamer with Python is highly flexible for data visualization. Any video streaming pipeline could be extended with custom Python plugins that allows you to draw any information (bounding boxes, confidences, class names, etc.). Next information is simple comparison of two most popular approaches to draw on gstreamer buffer: PyCairo, OpenCV.

    Requirements

  • Install OpenCV

Otherwise, look at this simple guide to install OpenCV from sources.

WarningIf install from pip you can face next Exception:

How it works?

     Let implement plugin to visualize “walking kitten” on video. (See attachments to get files)

how to draw kitten on video with gstreamer python

how to draw kitten on video with gstreamer python

     Firstly, try to launch code from repository:

     You should get something like this:

how to draw kitten on video with gstreamer python

     Hope everything works as expected. Now let go through implementation. (You can easily extend next approach to solve your tasks)

Explanation

     All code for this project is based on post “How to implement Blur Filter using Gstreamer in Python with OpenCV“. You’ll get a lot of explanations for those parts of code that won’t be covered here.

#1. Implement plugin templates for GstOverlayOpenCv, GstOverlayCairo

     Let make plugin that draw on incoming buffer instead of drawing on output buffer to avoid unnecessary buffer creation. To make this just override next function in GstBase.BaseTransform:

#2. Implement drawing with Cairo

     Have a look at PyCairo documentation to get an intuition how to use it. To implement common drawing tasks with Cairo use next guide.

Put previous code to do_transform_ip() in GstOverlayCairo

Note: To learn how to make Gst.Buffer writable read this.

Note: To draw wih cairo we use cairo.FORMAT_RGB24.  So update Src/Sink caps should have RGBx format.

#3. Implement drawing with OpenCV

     Next function draws transparent image over some other image. Approach was taken from here.


     Put draw_image() to do_transform_ip() in GstOverlayOpenCv

#4. Test with “walking kitten” images and video

     The main goal is to draw “walking kitten” over video and measure its performance to check what plugin GstOverlayOpenCv or GstOverlayCairo works faster. For this purpose have a look at fpsdisplaysink plugin that allows to draw Average FPS on video.

     Gstreamer pipeline looks like this:

     Now launch with OpenCV:

     Now launch with Cairo:

Note: To load images from file with Cairo use next call:

#5. Check performance

     Analyzing results you can see that video with GstOverlayCairo works faster (~103 fps) in comparison with GstOverlayOpenCv (~67 FPS). It is ~35% speed up for your applications. Look at this video (recorded from my PC to compare results)

Note: one of the reason is that in GstOverlayCairo no Gst.Buffer -> np.ndarray conversion.

How to draw image on gstreamer buffer

Note: CPU specs: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz 4/8

Conslusion

     When visualization is performance consuming part – consider Cairo instead OpenCV.

     Hope everything worked well. But if you have any troubles launching this code leave a question below 🙂

2 Comments

Add a Comment

Your email address will not be published. Required fields are marked *