Full screen OpenCV / GtK application in C++ running on Raspberry PI

Introduction

C++, OpenCV and Gtk are a nice triplet to build applications that run on a Raspberry PI, taking images from the camera, process them, display them and have an unlimited user interface. In this article, I will show you a naive path to display camera captures to a full-screen window. In a second article, I will amend some of the shortcomings of this too-simple method. I hope these two articles can help you through the boring problems you need to solve before reaching the place where you can have fun.

I assume that you know about coding but you are not familiar with C++, Gtk or OpenCV. It was my case when I started this little project, and I spent a huge amount of time discovering the specifics of this rich language and its dependencies. As this is not a C++ tutorial, I will only name the concepts and provide links to the explanations.

Here you will find the project sources: https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv. The master branch shows only the very first step of this tutorial. Further steps are in branches Step1, Step2, etc. To have the full version, you can check out the last step, or the last release.

An important element is to be able to build and test your project in your preferred desktop computer or notebook. Raspberry Pi is meant to be an embedded system platform. It is an amazing one but has not the right keyboard, mouse, monitor or amount of memory required to be a comfortable as a development tool.

I’m using the following technical stack:

• Raspberry Pi – The ultimate goal is to launch the application on it.
• CMake – This is the needed ingredient to make your code buildable on most of the platforms. It also exports your project to Xcode, VisualStudio, Eclipse and a long list of others, so you will be able to choose your preferred IDE.
• Mac OS X, Ubuntu, Windows – Where you can write and test your application.
• C++ – Python is fashionable, young, efficient and well supported by Raspberry folks. But I happen to like C++ more. If Python is your thing, then stop reading.
• OpenCV – One very widely used open source (hence the Open) computer vision (hence the CV) library.
• Gtkmm, which is the C++ oriented port of Gtk – Although OpenCV lets you display images on screen, it is somewhat limited when interacting with the user. Being easily compatible, Gtk / Gtkmm are a great complement to OpenCV for building real user interfaces around computer vision applications.
  • From your Mac, open App Store.
  • Browse or search Xcode.
  • Get it or update it.
  • At first launch, it will propose to install more things. Accept them.
  • Go visit https://brew.sh/
  • Follow instructions
  • Sublime Text
  • Real hardcore geeks can configure Vim to be an IDE.
  • C Programming in Linux Tutorial #056 – GDB debugger (1/2)
  • Quick Intro to gdb
  • CppCon 2015: Greg Law ” Give me 15 minutes & I’ll change your view of GDB”
  • Hitchhikers guide to the gdb
  • https://arne-mertz.de/2018/06/cmake-project-structure/: I like this one because it discusses how to integrate those header-only libraries, and uses Catch as an example.
  • https://rix0r.nl/blog/2015/08/13/cmake-guide/: I like this one because it shows in detail how to configure CMake and why, and also acknowledges the difference between library (which is easily unit tested) and application (which is the user interface, and not very easy to unit test).
  • cmake_minimum_required requires a minimum version of CMake. This allows you do confidently use the features existing in the specified version, or show an explicit error message if installed version is too old.
  • project declares the name of the global project, rascam, specifies the current version and list the languages that we are using, namely C++. Historically, the extension for C++ was *.c++, but in some context there are problems with «+», so they switched to cpp or cxx. In CMake, CXX means C++.
  • set(CMAKE_CXX…) sets a few flags that are appropriate for modern C++.
  • find_package(PkgConfig) links to the pkg-config command, so we can use it later on.
  • pkg_check_modules(GTKMM gtkmm-3.0) does the same as executing pkg-config gtkmm-3.0 in the terminal (try it, if you wish), and then copies each section of the answer into variables prefixed with the specified GTKMM. Therefore, after this command we have three variables named GTKMM_LIBRARY_DIRS, GTKMM_INCLUDE_DIRS and GTKMM_LIBRARIES.
  • Then we declare a target. One project can have multiple targets. In general, targets comprise executables or libraries which are defined by calling add_executable or add_library and which can have many properties set (see distinction between a “target” and a “command”). As this is a single target project, it is perfectly convenient to define the target directly in the main CMakeLists.txt file. We list the source files to compile to create the target.
  • target_include_directories sets the list of folders where to look for the files referred in sources as include “…”.
  • target_link_libraries sets the list of libraries the linker (see about what do linkers do) has to link to complete the build.
  • How C++ Works: Understanding Compilation
  • Stack Overflow – How does the compilation/linking process work?
  • virtual means that, in case of inheritance, the method called will be that of the dynamic type. Read more about what is dynamic type of object, virtual functions, and why most of the time it is a good idea to have virtual destructors.
  • default means that we define this destructor to have a default implementation. The default implementation of a destructor is able to deallocate all defined members of the class, which are m_box, m_button, m_label1 and m_label2 (code>buttonClick is a method therefore it does not need allocation or deallocation). By specifying the default implementation, we spare ourselves the need to define the destructor.
  • The function has to be declared as virtual in one of the ancestors of our class.
  • We need to declare it again, in our class, with the exact same signature and accessibility.
  • We may add the override keyword on the function declaration, to acknowledge the fact that we’re overriding an existing method from a parent class. The compiler shows an error when we use override on a method that doesn’t exist or it is not virtual.
  • Some indications about version number and author.
  • A key NSCameraUsageDescription with the description on why your application needs the camera. The message you write here is displayed to the user, who has to accept it. If he doesn’t accept, the system terminates your application.
  • An official documentation, quite old, about Bundles-And-Frameworks
  • The original Info.plist template.
  • void on_size_allocate(Gtk::Allocation& allocation) – This method is called every time the size of the widget changes. This happens the very first time it is displayed, and every time some action of the user or the system makes the size change.
  • bool on_draw(const Cairo::RefPtr& cr) – This method is called every time the area, or part of the area, contained in the widget has to be redrawn. It receives a reference to a Cairo context. The method can use it to render any drawing or graphic. In our case we’re going to use it to copy the image captured from the camera.
  • First we check that the current width and height are not zero. This may happen during startup phase. Calling resize with a 0 dimension would terminate instantly the application so we better avoid it.
  • videoCapture.read(webcam) copies the captured image into the provided cv::Mat, in this case webcam. If the provided material is not initialised, or it is not of the appropriate size, then the method will perform necessary tasks, and reserve needed memory.
  • resize copies the image from source (webcam) to destination (output), changing its size. If the destination is not initialised or not of the right size it performs necessary tasks and memory reservation. To keep the variable in scope, we also declared it as a class property: first call will initialise the variable, next calls can just reuse it.
  • Both webcam and output are class properties so they keep in scope as long as the class instance does. They will need to be initialised only during the first call to on_draw but will keep their values across the subsequent calls. This helps noticeably the performance.
  • create_from_data creates a new Gtk Pixbuf object, which contains an image in a format compatible with Cairo. The long list of parameters are to be taken as a recipe to convert OpenCV‘s Material to Gtk‘s Pixbuf.
  • set_source_pixbuf sets the bitmap source to use for next call to paint
  • Finally, paint does the painting.
  • Raspberry –> Preferences
  • Open Interfaces tab.
  • Enable the Camera
    • See more about syntax here: 
    • See original explanation here: 
  • Original article: Force Raspberry Pi output to composite video instead of HDMI
  • Official doc: Configure composite video
  • More official: Configuring the Raspberry Pi
  • If you haven’t done this yet, then remove the build directory of OpenCV.
  • Start again the building procedure as described above BUT…
  • Before launching make, check the cmake log, and verify the Gtk version is linked with. Look for something like this:
  • The more visible one is that the image is deformed to match the size of the window. This can be easily solved using a more sophisticated algorithm to calculate a size that would fit in the window but keep the original aspect ratio.
  • A second one, very annoying if you plan a First Person View application is the perceptible lag between real life and the images stream. Lag seems variable depending on the camera and the light conditions. The reason is cameras having an image buffer; as we are retrieving one image every now and then, we’re always consuming the oldest image in the buffer. To solve this we should let the capture process drive the window refresh, instead of using a timer.
  • Code has a general lack of decoupling that makes everything very dependent on everything else. This defect isn’t very present in such a simple application, but it will show as soon as we try to solve the lag problem. Also, part of the code should be a pure OpenCV process that you could copy/paste from some other blog, without needing to adapt it to the CameraDrawingArea that is dependent on both OpenCV and Gtk libraries.
  • Finally, you should be able to unit test the OpenCV processing to increase the type-compile-debug cycle.