How to draw kitten on video with Gstreamer faster?

     Gstreamer with Python is highly flexible for data visualization. Any video streaming pipeline could be extended with custom Python plugins that allows you to draw any information (bounding boxes, confidences, class names, etc.). Next information is simple comparison of two most popular approaches to draw on gstreamer buffer: PyCairo, OpenCV.


  • Install OpenCV
pip3 install opencv-python

Otherwise, look at this simple guide to install OpenCV from sources.

git clone

cd pycairo

python build
python install

WarningIf install from pip you can face next Exception:

Error: Surface.create_for_data: Not Implemented yet

How it works?

     Let implement plugin to visualize “walking kitten” on video. (See attachments to get files)

how to draw kitten on video with gstreamer python

how to draw kitten on video with gstreamer python

     Firstly, try to launch code from repository:

git clone
cd gst-overlay

# run with opencv
python3 -f ~/attachments/car.mpg  --images ~/attachments/cat --opencv --fps

# run with cairo
python3 -f ~/attachments/car.mpg  --images ~/attachments/cat --cairo --fps

     You should get something like this:

how to draw kitten on video with gstreamer python

     Hope everything works as expected. Now let go through implementation. (You can easily extend next approach to solve your tasks)


     All code for this project is based on post “How to implement Blur Filter using Gstreamer in Python with OpenCV“. You’ll get a lot of explanations for those parts of code that won’t be covered here.

#1. Implement plugin templates for GstOverlayOpenCv, GstOverlayCairo

     Let make plugin that draw on incoming buffer instead of drawing on output buffer to avoid unnecessary buffer creation. To make this just override next function in GstBase.BaseTransform:

def do_transform_ip(self, inbuffer):

#2. Implement drawing with Cairo

     Have a look at PyCairo documentation to get an intuition how to use it. To implement common drawing tasks with Cairo use next guide.

# Initialize stride value in respect to buffer format and width to accelerate image processing 
stride = cairo.ImageSurface.format_stride_for_width(cairo.FORMAT_RGB24, width)

# Create image surface from image buffer (buffer should be mapped with READ | WRITE flags)
surface = cairo.ImageSurface.create_for_data(buffer, cairo.FORMAT_RGB24,
                                             width, height, stride)

# Create cairo context to manipulate with image data
# Documentation:
context = cairo.Context(surface)

# Calculate bottom-right point to place image
overlay = self.overlay()
x = width - overlay.get_width()
y = height - overlay.get_height()

# Documentation:
# Question: How do I paint on surface?
context.set_source_surface(self.overlay(), x, y)

Put previous code to do_transform_ip() in GstOverlayCairo

Note: To learn how to make Gst.Buffer writable read this.

Note: To draw wih cairo we use cairo.FORMAT_RGB24.  So update Src/Sink caps should have RGBx format.


#3. Implement drawing with OpenCV

     Next function draws transparent image over some other image. Approach was taken from here.

def draw_image(source, image, x, y):

    h, w = image.shape[:2]
    max_x, max_y = x + w, y + h
    alpha = image[:, :, 3] / 255.0
    for c in range(0, 3):
        color = image[:, :, c] * (alpha)
        beta = source[y:max_y, x:max_x, c] * (1.0 - alpha)
        source[y:max_y, x:max_x, c] = color + beta
    return source

     Put draw_image() to do_transform_ip() in GstOverlayOpenCv

#4. Test with “walking kitten” images and video

     The main goal is to draw “walking kitten” over video and measure its performance to check what plugin GstOverlayOpenCv or GstOverlayCairo works faster. For this purpose have a look at fpsdisplaysink plugin that allows to draw Average FPS on video.

     Gstreamer pipeline looks like this:

filesrc location=~/attachments/car.mpg ! decodebin ! videoconvert ! gstoverlayopencv name=overlay ! videoconvert ! fpsdisplaysink video-sink=gtksink sync=false

     Now launch with OpenCV:

python3 -f ~/attachments/car.mpg  --images ~/attachments/cat --opencv --fps

     Now launch with Cairo:

python3 -f ~/attachments/car.mpg  --images ~/attachments/cat --cairo --fps

Note: To load images from file with Cairo use next call:

def load_image_cairo(filename):
    assert os.path.isfile(filename), "Invalid filename: {}".format(filename)
    return cairo.ImageSurface.create_from_png(filename)

#5. Check performance

     Analyzing results you can see that video with GstOverlayCairo works faster (~103 fps) in comparison with GstOverlayOpenCv (~67 FPS). It is ~35% speed up for your applications. Look at this video (recorded from my PC to compare results)

Note: one of the reason is that in GstOverlayCairo no Gst.Buffer -> np.ndarray conversion.

How to draw image on gstreamer buffer

Note: CPU specs: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz 4/8


     When visualization is performance consuming part – consider Cairo instead OpenCV.

     Hope everything worked well. But if you have any troubles launching this code leave a question below 🙂


Add a Comment

Your email address will not be published. Required fields are marked *