How to add metadata to Gstreamer buffer in Python?
Gstreamer is flexible plugin based data streaming framework. One of the main advantange of plugins is that developer can pass not only data buffer, but custom metadata which can describe buffer and store any information about buffer. Next information is a simple guide with code templates on how to pass any data from one plugin to another.
Here are some possible use of metadata for Computer Vision applications:
- objects bounding boxes (list of [x, y, width, height])
- objects classes (“person”, “car”, “cat”, “dog”, … )
- objects confidences ([0, 1.0])
- keypoints
- etc.
Note: Metadata do not relies on buffer encoding, format.
To get more information about role of Metadata in Gstreamer read official docs
Problem
It could be difficult (in some cases even impossible) to read/write metadata from gstreamer plugin implemented in Python. (Remember that Gst.Buffer isn’t writable? See “How to make Gst.Buffer writable”)
There are some discussions from forums with problem description and with no solution:
Solution
|
1 2 3 4 5 |
// git clone https://github.com/jackersson/gst-python-hacks.git cd gst-python-hacks cd gst-metadata ./build |
Open how-to-write-metadata-gst-buffer.ipynb in notebook. Let’s go through each step and see how it works:
#1. Create new buffer
|
1 2 |
// buffer = Gst.Buffer.new_wrapped(b"lifestyletransfer") |
#2. Write custom meta with text field
|
1 2 |
// write_meta(buffer, description="Turotial 'How to write metadata in Python Gstreamer plugin'") |
#3. Check that meta exists
|
1 2 3 4 5 |
// buffer_info_meta = get_meta(buffer) print(buffer_info_meta.description.decode("utf-8")) -> Turotial 'How to write metadata in Python Gstreamer plugin' |
#4. Remove meta
|
1 2 3 4 5 |
// is_ok = remove_meta(buffer) print(is_ok) -> True |
Explanation
There are specific requirements from official documentation for handling buffer’s metadata. All those requirements could be combined in C-code template (common for all custom metadata you might want to use).
Have a look at simple example on how to use C-template to create your own Metadata.
#1. Define Metadata Structure
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
// // Structure that holds "description" field to // store custom text data struct GstBufferInfo { gchar* description; }; // Custom structure to be passed between Gstreamer plugins struct GstBufferInfoMeta { // Required as it is base structure for metadata // https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstMeta.html GstMeta meta; // Custom fields GstBufferInfo info; }; |
#2. Register Metadata type.
Gstreamer requires to register Metadata type so it could be used to work with it in Glib ecosystem. Next function registers GstBufferInfoMeta once and returns GType value if success.
Note: GType is a numerical value which represents the unique identifier of a registered type.
|
1 2 3 4 5 6 7 8 9 10 11 |
// GType gst_buffer_info_meta_api_get_type(void) { static const gchar *tags[] = {NULL}; static volatile GType type; if (g_once_init_enter (&type)) { GType _type = gst_meta_api_type_register("GstBufferInfoMetaAPI", tags); g_once_init_leave(&type, _type); } return type; } |
#3. Impelement Metadata init function.
In this function all metadata’s fields (if required) should be set to default values.
|
1 2 3 4 5 6 7 |
// static gboolean gst_buffer_info_meta_init(GstMeta *meta, gpointer params, GstBuffer *buffer) { GstBufferInfoMeta *gst_buffer_info_meta = (GstBufferInfoMeta*)meta; gst_buffer_info_meta->info.description = ""; return TRUE; } |
#4. Implement meta transform.
This is required by Gstreamer to be able to copy medatada from one buffer to another. This is useful in case when some plugin forward through pipeline not the same buffer, but the new one.
|
1 2 3 4 5 6 7 8 9 |
// static gboolean gst_buffer_info_meta_transform(GstBuffer *transbuf, GstMeta *meta, GstBuffer *buffer, GQuark type, gpointer data) { GstBufferInfoMeta *gst_buffer_info_meta = (GstBufferInfoMeta *)meta; gst_buffer_add_buffer_info_meta(transbuf, &(gst_buffer_info_meta->info)); return TRUE; } |
#5. Implement GstMetaInfo
GstMetaInfo provides specific information about Metadata structure. In this we combine all previous steps get_g_type-, transform-, init- functions
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
// const GstMetaInfo *gst_buffer_info_meta_get_info(void) { static const GstMetaInfo *gst_buffer_info_meta_info = NULL; if (g_once_init_enter (&gst_buffer_info_meta_info)) { // Explanation of fields // https://gstreamer.freedesktop.org/documentation/design/meta.html#gstmeta1 const GstMetaInfo *meta = gst_meta_register (GST_BUFFER_INFO_META_API_TYPE, /* api type */ "GstBufferInfoMeta", /* implementation type */ sizeof(GstBufferInfoMeta), /* size of the structure */ gst_buffer_info_meta_init, (GstMetaFreeFunction) NULL, gst_buffer_info_meta_transform); g_once_init_leave (&gst_buffer_info_meta_info, meta); } return gst_buffer_info_meta_info; } |
Now we can use GstBufferInfoMeta as Metadata object. To to read from GstBuffer use following command:
|
1 2 |
// GstBufferInfoMeta* meta = (GstBufferInfoMeta*)gst_buffer_get_meta((buffer), GST_BUFFER_INFO_META_API_TYPE); |
To write GstBufferInfoMeta to GstBuffer use next:
|
1 2 3 4 5 6 7 8 9 |
// gst_buffer_info_meta = (GstBufferInfoMeta *) gst_buffer_add_meta (buffer, GST_BUFFER_INFO_META_INFO, NULL); // copy fields to buffer's meta if (buffer_info->description!=NULL) { gst_buffer_info_meta->info.description = malloc(strlen(buffer_info->description) + 1); strcpy(gst_buffer_info_meta->info.description, buffer_info->description); } |
Additionally, you can remove GstBufferInfoMeta from Buffer
|
1 2 3 4 5 |
// GstBufferInfoMeta* meta = (GstBufferInfoMeta*)gst_buffer_get_meta((buffer), GST_BUFFER_INFO_META_API_TYPE); // https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstBuffer.html#gst-buffer-remove-meta gboolean is_ok = gst_buffer_remove_meta(buffer, &meta->meta); |
So, for now C-template is implemented in next files:
|
1 2 3 4 |
// gst_buffer_info_meta.c gst_buffer_info_meta.h |
To use C-template from Python let compile it into shared *.so library (look at CMakeLists.txt):
|
1 2 3 |
// ./build.sh |
Library file libgst_buffer_info_meta.so is now located in build/ folder.
Now we can call C-functions from shared library using ctypes and Python. Let use next step-by-step guide
#1. Load shared lib
To load libgst_buffer_info_meta.so library use:
|
1 2 3 |
# clib = CDLL("build/libgst_buffer_info_meta.so") |
#2. Define Python version GstBufferInfo Structure (with ctypes) that corresponds to C-version to GstBufferInfoMeta
|
1 2 3 4 5 6 |
# class GstBufferInfo(Structure): _fields_ = [("description", c_char_p)] GstBufferInfoPtr = POINTER(GstBufferInfo) |
#3. Map c-arguments to ctypes-arguments in lib’s functions calls
|
1 2 3 4 5 6 7 8 9 10 |
# clib.gst_buffer_add_buffer_info_meta.argtypes = [c_void_p, GstBufferInfoPtr] clib.gst_buffer_add_buffer_info_meta.restype = c_void_p clib.gst_buffer_get_buffer_info_meta.argtypes = [c_void_p] clib.gst_buffer_get_buffer_info_meta.restype = GstBufferInfoPtr clib.gst_buffer_remove_buffer_info_meta.argtypes = [c_void_p] clib.gst_buffer_remove_buffer_info_meta.restype = c_bool |
#4. Wrap C-calls with Python functions that could be used in your code.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# def write_meta(buffer, description): meta = GstBufferInfo() meta.description = description.encode("utf-8") clib.gst_buffer_add_buffer_info_meta(hash(buffer), meta) def get_meta(buffer): res = clib.gst_buffer_get_buffer_info_meta(hash(buffer)) return res.contents def remove_meta(buffer): return clib.gst_buffer_remove_buffer_info_meta(hash(buffer)) |
Previous functions are used in how-to-write-metadata-gst-buffer.ipynb. Now you can extend Metadata structure and use in your applications. Hope everything works as expected, if not – leave a comment 😉
Additional readings:
Top 5 Coolest projects with Gstreamer
How to use Gstreamer AppSink in Python
clib = CDLL(“/home/pi/installs/gst-python-hacks/gst-metadata/build/libgst_buffer_info_meta.so”)
gives me this error:
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python3.5/ctypes/__init__.py”, line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /home/pi/installs/gst-python-hacks/gst-metadata/build/libgst_buffer_info_meta.so: undefined symbol: _gst_buffer_type
Hi,
Just tried same steps in tutorial but with Python 3.6. And everything worked.
Check same on your PC in own virtual environment 😉
If this won’t work, I’ll try to check it in Docker.
False alarm got it to work, I had been trying to compile the head of gstreamer trunk with good, bad, ugly, omx, etc. with hardware acceleration for my raspberry pi and had some linking issues (putting it nicely) that once cleared up fixed my issue. Thank you for speedy response.
would it be possible to extract metadata from a video stream via rtsp ? Or I would have to write it from a config file and label the metadata for each frame extracted from the video stream?
Hi, Toro
Thanks for question. Proposed method (for writing/reading metadata) in post works only inside single pipeline to pass data from one plugin to another. It won’t work for passing metadata from one pipeline to another, because streaming protocols doesn’t support some custom data transmission (except some of them I suppose).
Using files for such a purpose would be a good choice.
Best regards,
Taras Lishchenko
Hey taras, can I ask a question. Is it possible to obtain the timestamp of when the camera starts recording using gstreamer? or other offsets? I want to obtain the actual time at which each frame was captured.
Hi, Michael
Thanks for your question. There are two options:
1 – Real-World time with timezone using python’s datetime. Due to this example, you can just create simple python plugin, like this and using datetime record timestamp for each incoming buffer.
2 – Real-World time using gstreamer’s running-time. Fixate pipeline’s start time (with datetime lib) as ACTUAL_TIME=START_TIME and each frame increase this time gst_buffer.duration: ACTUAL_TIME += gst_buffer.duration. Or ACTUAL_TIME = START_TIME + gst_buffer.pts (look at for reference)
Additional:
– Getting buffers offset for live source: gst_buffer.pts / gst_buffer.duration
Best regards,
Taras Lishchenko
hey taras, i want to ask if it was possible to obtain the timestamp of when the camera starts recording in gstreamer? how may i go about and find this?
Big help specially in the python implementation of gstreamer.
Thanks 😉