Merge branch 'develop'

.github/workflows/ci.yml

@@ -2,9 +2,9 @@ name: Gstreamer DRP-AI Make CI
 
 on:
   push:
-    branches: [ "master" ]
+    branches: [ "master", "develop" ]
   pull_request:
-    branches: [ "master" ]
+    branches: [ "master", "develop" ]
 
 jobs:
   build:

README.md

@@ -1,77 +1,109 @@
-# GStreamer template repository
+# GStreamer plugin for DRPAI
 
-This git module contains template code for possible GStreamer projects.
+# Video Inference with DRPAI and GStreamer
 
-* gst-app :
-  basic meson-based layout for writing a GStreamer-based application.
+MistySOM RZV2L contains the DRPAI hardware module which is able to run artificial Neural Networks 
+with the focus on low power consumption. To check if this hardware module is present on your device,
+you can look for both `/dev/drpai0` and `/dev/udmabuf0` devices on your linux shell. 
+The Userland Direct Memory Access (UDMA) kernel module is required to provide the trained AI model and 
+the input image to the DRPAI hardware. After activating the hardware, it will use the trained model to
+generate the output which can be read by the UDMA module. While DRPAI is running, the running thread will
+go to sleep. Of course the sleep time varies based on the size of the AI model.
 
-* gst-plugin :
-  basic meson-based layout and basic filter code for writing a GStreamer plug-in.
+MistyWest team has prepared this plugin which can receive any kind of video input,
+such as a file (filesrc), a network stream (udpsrc), or a camera device (v4l2src) and outputs a video 
+with bounding boxes on inferred objects using the DRPAI. Later, this video can be linked to any kind of 
+output, such as the display (autovideosink), a network stream (udpsink), or a file (filesink).
 
-## License
+![GStreamer DRPAI Plugin Chart](img/gst-drpai-chart.png)
 
-This code is provided under a MIT license [MIT], which basically means "do
-with it as you wish, but don't blame us if it doesn't work". You can use
-this code for any project as you wish, under any license as you wish. We
-recommend the use of the LGPL [LGPL] license for applications and plugins,
-given the minefield of patents the multimedia is nowadays. See our website
-for details [Licensing].
+**Note:** At this moment, the plugin is hardcoded to YOLOV2l model. Therefore, you need to have a copy
+of the trained model ([link](models/yolov2))
+with the directory name of `yolov2` inside your working directory for the plugin to work.
 
-## Usage
+The plugin uses the following pad template capabilities for both **src** and **sink** which requires you
+to prepare before your DRPAI element (for example, using a `videoconvert` element):
 
-Configure and build all examples (application and plugins) as such:
+```
+video/x-raw
+    width: 640
+    height: 480
+    format: BGR
+```
+
+The plugin also provides you with the following parameters:
+
+| Name                  | Type                | Default | Description                                                          |
+|-----------------------|---------------------|--------:|----------------------------------------------------------------------|
+| **multithread**       | Boolean             |    true | Use a separate thread for object detection.                          |
+| **log-detects**       | Boolean             |   false | Print detected objects in standard output.                           |
+| **show-fps**          | Boolean             |   false | Render frame rates of video and DRPAI at the corner of the video.    |
+| **stop-error**        | Boolean             |    true | Stop the gstreamer if kernel modules fail to open.                   |
+| **max-video-rate**    | Float [0.001 - 120] |     120 | Force maximum video frame rate using thread sleeps.                  |
+| **max-drpai-rate**    | Float [0 - 120]     |     120 | Force maximum DRPAI frame rate using thread sleeps.                  |
+| **smooth-video-rate** | Float [1 - 1000]    |       1 | Number of last video frame rates to average for a more smooth value. |
+| **smooth-drpai-rate** | Float [1 - 1000]    |       1 | Number of last DRPAI frame rates to average for a more smooth value. |
+
+## How to Build
+
+Configure and build the repository (the sample application and DRPAI plugin) as such:
 
-    meson builddir
+    meson setup builddir
     ninja -C builddir
 
 See <https://mesonbuild.com/Quick-guide.html> on how to install the Meson
 build system and ninja.
 
-Modify `gst-plugin/meson.build` to add or remove source files to build or
-add additional dependencies or compiler flags or change the name of the
-plugin file to be installed.
-
-Modify `meson.build` to check for additional library dependencies
-or other features needed by your plugin.
-
 Once the plugin is built you can either install system-wide it with `sudo ninja
--C builddir install` (however, this will by default go into the `/usr/local`
-prefix where it won't be picked up by a `GStreamer` installed from packages, so
+-C builddir install` (if it wouldn't be picked up by GStreamer, 
 you would need to set the `GST_PLUGIN_PATH` environment variable to include or
-point to `/usr/local/lib/gstreamer-1.0/` for your plugin to be found by a
+point to `/usr/lib64/gstreamer-1.0/` for your plugin to be found by a
 from-package `GStreamer`).
 
 Alternatively, you will find your plugin binary in `builddir/gst-plugins/src/`
-as `libgstplugin.so` or similar (the extension may vary), so you can also set
+as `libgstdrpai.so` or similar (the extension may vary), so you can also set
 the `GST_PLUGIN_PATH` environment variable to the `builddir/gst-plugins/src/`
 directory (best to specify an absolute path though).
 
 You can also check if it has been built correctly with:
 
-    gst-inspect-1.0 builddir/gst-plugins/src/libgstplugin.so
-
-## Auto-generating your own plugin
+    gst-inspect-1.0 builddir/gst-plugins/src/libgstdrpai.so
 
-You will find a helper script in `gst-plugins/tools/make_element` to generate
-the source/header files for a new plugin.
+## Some examples of running the plugin
 
-To create sources for `myfilter` based on the `gsttransform` template run:
+### Read Camera and Show on Screen
 
-``` shell
-cd src;
-../tools/make_element myfilter gsttransform
 ```
+gst-launch-1.0 v4l2src device=/dev/video0 \
+    ! videoconvert \
+    ! drpai show-fps=true log-detects=true smooth-video-rate=30 \
+    ! videoconvert \
+    ! autovideosink
+```
+If your camera supports the BGR format (such as the coral camera), you can modify the camera size in 
+`~/v4l2init.sh` and skip the first `videoconvert` element like this:
+```
+gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, format=BGR \
+    ! drpai show-fps=true log-detects=true smooth-video-rate=30 \
+    ! videoconvert \
+    ! autovideosink
+```
+### Read Camera and Stream on Network
 
-This will create `gstmyfilter.c` and `gstmyfilter.h`. Open them in an editor and
-start editing. There are several occurances of the string `template`, update
-those with real values. The plugin will be called `myfilter` and it will have
-one element called `myfilter` too. Also look for `FIXME:` markers that point you
-to places where you need to edit the code.
+In case you already have the streaming working based on [here](StreamingVideo.md), you can 
+add the drpai element to the `stream.sh` file like this:
 
-You can then add your sources files to `gst-plugins/meson.build` and re-run
-ninja to have your plugin built.
+````
+#!/bin/bash
+[ $1 ] || { echo  "Please specify the destination IP address: ./stream.sh ip" >&2; exit 1; }
 
+./v4l2-init.sh
+echo "Streaming to ${1} with DRPAI..."
 
-[MIT]: http://www.opensource.org/licenses/mit-license.php or COPYING.MIT
-[LGPL]: http://www.opensource.org/licenses/lgpl-license.php or COPYING.LIB
-[Licensing]: https://gstreamer.freedesktop.org/documentation/application-development/appendix/licensing.html
+gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, format=BGR \
+    ! drpai show-fps=true log-detects=true smooth-video-rate=30 \
+    ! vspmfilter dmabuf-use=true ! video/x-raw, format=NV12 \
+    ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 \
+    ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 \
+    ! rtph264pay ! udpsink host=$1 port=51372
+````

bitbake-recipe.bb

@@ -0,0 +1,28 @@
+SUMMARY = "GStreamer DRP-AI plugin"
+SECTION = "multimedia"
+LICENSE = "MIT"
+SRC_URI = "git://github.com/MistySOM/gstreamer1.0-drpai.git;branch=master"
+SRCREV = "${AUTOREV}"
+LIC_FILES_CHKSUM = "file://COPYING.MIT;md5=bba6cdb9c2b03c849ed4975ed9ed90dc"
+MESON_BUILDTYPE = "release"
+
+inherit meson
+
+DEPENDS = "gstreamer1.0 gstreamer1.0-plugins-base drpai"
+
+S = "${WORKDIR}/git"
+PV = "1.0"
+
+do_install_append() {
+    install -d ${D}${ROOT_HOME}/yolov2
+    install -m 0755 ${S}/models/yolov2/* ${D}${ROOT_HOME}/yolov2
+}
+
+FILES_${PN} = "${libdir}/gstreamer-1.0/libgstdrpai.so ${ROOT_HOME}/yolov2"
+FILES_${PN}-dev = "${libdir}/gstreamer-1.0/libgstdrpai.la"
+FILES_${PN}-staticdev = "${libdir}/gstreamer-1.0/libgstdrpai.a"
+FILES_${PN}-dbg = " \
+    ${libdir}/gstreamer-1.0/.debug \
+    ${prefix}/src"
+
+RDEPENDS_${PN} = "gstreamer1.0 gstreamer1.0-plugins-base kernel-module-udmabuf"

gst-app/meson.build

@@ -1,6 +1,5 @@
 app_sources = [
-  'src/main.c',
-  'src/play.c'
+  'src/main.c'
   ]
 
 executable('gst-app', app_sources, dependencies : [gst_dep])