Full screen OpenCV / GtK application in C++ running on Raspberry PI

Introduction

C++, OpenCV and Gtk are a nice triplet to build applications that run on a Raspberry PI, taking images from the camera, process them, display them and have an unlimited user interface. In this article I will show you a naive path to display camera captures to a full screen window. In a second article I will amend some of the shortcomings of this too-simple method. I hope these two articles can help you through the boring problems you need to solve before reaching the place where you can have fun.

I assume that you know about coding but you are not familiar with C++, Gtk or OpenCV. It was my case when I started this little project, and I spent a huge amount of time discovering the specifics of this rich language and its dependencies. As this is not a C++ tutorial, I will only name the concepts and provide links to the explanations.

Here you will find the project sources: https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv. The master branch shows only the very first step of this tutorial. Further steps are in branches Step1, Step2, etc. To have the full version, you can check out the last step, or the last release.

An important element is to be able to build and test your project in your preferred desktop computer or notebook. Raspberry Pi is meant to be an embedded system platform. It is an amazing one, but has not the right keyboard, mouse, monitor or amount of memory required to be a comfortable as a development tool.

I’m using the following technical stack:



  • Raspberry Pi – The ultimate goal is to launch the application on it.
  • CMake – This is the needed ingredient to make your code buildable on most of the platforms. It also exports your project to Xcode, VisualStudio, Eclipse and a long list of others, so you will be able to choose your preferred IDE.
  • Mac OS X, Ubuntu, Windows – Where you can write and test your application.
  • C++Python is fashionable, young, efficient and well supported by Raspberry folks. But I happen to like C++ more. If Python is your thing, then stop reading.
  • OpenCV – One very widely used open source (hence the Open) computer vision (hence the CV) library.
  • Gtkmm, which is the C++ oriented port of Gtk – Although OpenCV lets you display images on screen, it is somewhat limited when interacting with the user. Being easily compatible, Gtk / Gtkmm are a great complement to OpenCV for building real user interfaces around computer vision applications.
    • Installing your Development environment

      The first thing you need is a working development environment. I’ve tried the three majors, Mac OS X, Windows and Linux. Linux instructions also apply to Raspberry.

      Development environment for Mac OS

      In Mac OS world, the default IDE is Xcode. The other packages are installed via Homebrew.

      C++ – Xcode

      You have a Mac, so you’re entitled to use Xcode for free. I know that plenty of people who hate it, and they have their reasons. If you know better, then use the IDE you prefer. If not, then use Xcode:

      • From your Mac, open App Store.
      • Browse or search Xcode.
      • Get it or update it.
      • At first launch, it will propose to install more things. Accept them.

      Homebrew

      It is a package manager for Mac OS X, and the easiest way to install the rest of packages you need.

      If you have already an installed version of Homebrew, now it is a good moment to verify, update and fix all that is needed. Following command will give you plenty of suggestions:

      $ brew update
      $ brew doctor
      

      Follow instructions until you have all of them fixed (well, don’t get too paranoid in any case; fix all that you can).

      pkg-config

      This package helps to link your project to the OpenCV and GtK libraries (see Wikipedia on pkg-config).

      To install it:

      $ brew install pkg-config
      

      When installation is finished, you can check if all is right by typing:



      $ pkg-config --version
      > 0.29.2
      $ pkg-config --list-all
      > zlib                                zlib - zlib compression library
      > ...                                 etc...
      > ...                                 etc...
      > ...                                 etc...
      > harfbuzz-icu                        HarfBuzz text shaping library ICU integration
      

      OpenCV and GtK

      We install OpenCV with third parties contributions. You won’t need them at the beginning, but they won’t bother you either. And we use Gtkmm3, which is the GtK for C++:

      brew install opencv3 --with-contrib
      brew install gtkmm3
      

      To verify that libraries are now in place:

      $ pkg-config --list-all | grep opencv
      > opencv                              OpenCV - Open Source Computer Vision Library
      $ pkg-config --list-all | grep gtkmm
      > gtkmm-3.0                           gtkmm - C++ binding for the GTK+ toolkit
      

      Git

      You probably had it installed when you ran brew doctor, and it required to execute the following:

      $ xcode-select --install
      

      To verify that you indeed have it:

      $ git --version
      git version 2.17.2 (Apple Git-113)
      

      Development environment for Windows

      TODO

      Development environment for Linux (Debian)

      To write this part, my initial intention after installing a fresh Ubuntu distribution was to set up any well known IDE and click around until I could compile, debug and run the project. Boy, was I not naive! Several days later, I have had discarded Clion because it is a paying application, I had unsuccessfully tried Code::Blocks, Eclipse, and lost my patience before trying Qt Builder — yet, I have not been able to find an acceptable IDE to just do the basic stuff with reasonable simplicity. And then I found this Stack Oveflow question «C++ IDE for Linux?» where accepted and most voted answer states that «UNIX is an IDE. All of it».

      C++ – Linux is an IDE

      Let’s accept that in Linux we are not going to get a fancy IDE and content ourselves with a text editor. Two popular choices are:

      The tool to debug code is gdb. It seems bleak, but you can actually do quite a job with it. Have a look on those videos:

      In the first step, a little bit below, I will give short specific instructions for compiling with debug symbols, and debug the programs with gdb.

      apt-get

      This is the official package manager of most Linux distribution. Most popular applications are distributed through it and it is most usually pre-installed.



      pkg-config

      In a linux distribution, there is the highest chance that pkg-config is already installed. Anyway, you can give it a try; should it be already installed it will just tell you:

      sudo apt-get update
      sudo apt-get install pkg-config
      

      CMake

      You will need it to compile OpenCV, so you better install it now:

      $ sudo apt-get install cmake
      

      OpenCV

      As a prerequisite, you need to have the following libraries from apt-get:

      # required:
      sudo apt-get install \
         libgtk-3-dev \
         pkg-config \
         libavcodec-dev \
         libavformat-dev \
         libswscale-dev 
      # Optional:
      sudo apt-get install \
         python-dev \
         python-numpy \
         libjpeg-dev \
         libpng-dev \
         libtiff-dev \
         libjasper-dev \
         libdc1394-22-dev
      

      Once you’ve installed all prerequisites, fetch the latest version of OpenCV sources, and unzip it:

      $ wget https://github.com/opencv/opencv/archive/3.4.1.zip
      $ unzip 3.4.1.zip
      

      Prepare and compile:

      $ cd opencv-3.4.1
      $ mkdir build
      $ cd build
      $ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local ..
      $ make -j4  # Number of processors. Don't use more that you computer has.
      

      Go for a walk; this takes ages. If process breaks, you can launch again by just retyping:

      $ make -j4
      

      When compilation is done, complete the installation:

      $ sudo make install
      $ sudo ldconfig
      

      To check if the library is available as a dependency:

      $ pkg-config --list-all | grep opencv
      > opencv                         OpenCV - Open Source Computer Vision Library
      

      You can now delete the sources folder; you don’t need it any more:

      $ cd ..
      $ cd ..
      $ rm -rf opencv-3.4.1
      $ rm 3.4.1.zip
      

      Gtkmm

      Gtkmm is the Gtk for C++:



      $ sudo apt-get install libgtkmm-3.0-dev
      

      To check if the library is available as a dependency:

      $ pkg-config --list-all | grep gtkmm
      gtkmm-3.0                      gtkmm - C++ binding for the GTK+ toolkit
      

      Git

      To install Git:

      $ sudo apt-get install git
      

      And to verify that Git is present:

      $ git --version
      > git version 2.7.4
      

      The project’s folder structure

      There are several great articles that discuss about the best folder structure for a project built with CMake:

      In this first stage (there is more on the second part of the article), I’m proposing a simplified folder structure. Projects having multiple executables and libraries need one folder per module, and the structure is much more complex. In this case, as there will be one single executable, I’m just having one root folder, called src and, in it, one folder for each type of file – sources, headers and resources.

      .gitignore  <-- Ignore xcode, codeb, build folders.
      /src  <----- All sources are here
         CMakeList.txt  <-- The CMake configuration file.
            /cpp  <----- Contains the C++ source files.
            /hpp  <----- Contains the C++ header files.
            /res  <----- Contains resource files.
      /build  <----- Contains the temporary build files. 
                       Not under version control
      /xcode  <----- Contains the XCode project.
                       Not under version control.
      /codeb  <----- Contains the Code::Blocks project. 
                       Not under version control. 
      

      Step 1 – A single-window application

      In this first step we build a very simple single-window application using CMake to configure the project and Gtk to display windows. Source code is available as master branch at: https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv. To retrieve it:

      Configure the project with CMake

      This is the CMakeLists.txt that goes on the src folder:

      # src/CMakeLists.txt
      cmake_minimum_required(VERSION 3.3 FATAL_ERROR)
      
      # Project name and current version
      project(rascam VERSION 0.1 LANGUAGES CXX)
      
      # Enable general warnings
      # See http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
      
      # Use 2014 C++ standard.
      set(CMAKE_CXX_STANDARD 14)
      set(CMAKE_CXX_STANDARD_REQUIRED ON)
      set(CMAKE_CXX_EXTENSIONS OFF)
      
      # Must use GNUInstallDirs to install libraries into correct locations on all platforms:
      include(GNUInstallDirs)
      
      # Pkg_config is used to locate headers and files for dependency libraries:
      find_package(PkgConfig)
      
      # Defines variables GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS and GTKMM_LIBRARIES.
      pkg_check_modules(GTKMM gtkmm-3.0)
      
      # Adds GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS to the list of folders the
      # compiler and linker will use to look for dependencies:
      link_directories( ${GTKMM_LIBRARY_DIRS} )
      include_directories( ${GTKMM_INCLUDE_DIRS} )
      
      # Declares a new executable target called rascapp
      # and lists the files to compile for it:
      add_executable(rascapp
          cpp/main.cpp    
          cpp/main-window.cpp
      )
      
      # Adds the folder with all headers:
      target_include_directories(rascapp PRIVATE hpp)
      
      # Link files:
      target_link_libraries(rascapp
         ${GTKMM_LIBRARIES}  
      )
      

      Let’s review the commands one by one:

      • cmake_minimum_required requires a minimum version of CMake. This allows you do confidently use the features existing in the specified version, or show an explicit error message if installed version is too old.
      • project declares the name of the global project, rascam, specifies the current version and list the languages that we are using, namely C++. Historically, the extension for C++ was *.c++, but in some context there are problems with «+», so they switched to cpp or cxx. In CMake, CXX means C++.
      • set(CMAKE_CXX...) sets a few flags that are appropriate for modern C++.
      • find_package(PkgConfig) links to the pkg-config command, so we can use it later on.
      • pkg_check_modules(GTKMM gtkmm-3.0) does the same as executing pkg-config gtkmm-3.0 in the terminal (try it, if you wish), and then copies each section of the answer into variables prefixed with the specified GTKMM. Therefore, after this command we have three variables named GTKMM_LIBRARY_DIRS, GTKMM_INCLUDE_DIRS and GTKMM_LIBRARIES.
      • Then we declare a target. One project can have multiple targets. In general, targets comprise executables or libraries which are defined by calling add_executable or add_library and which can have many properties set (see distinction between a “target” and a “command”). As this is a single target project, it is perfectly convenient to define the target directly in the main CMakeLists.txt file. We list the source files to compile to create the target.
      • target_include_directories sets the list of folders where to look for the files referred in sources as include "...".
      • target_link_libraries sets the list of libraries the linker (see about what do linkers do) has to link to complete the build.

      In case you want to read more about compilation steps:

      The Gtk startup

      All C++ executables need a main method as entry point. To launch a GtK application, you need to specify a main window, which will act as a base for all other user interaction, including to open more windows.



      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #include "main-window.hpp"
        
      int main(int argc, char* argv[])
      {
          auto app
              = Gtk::Application::create(argc,
                                         argv,
                                         "raspberry-cpp-gtk-opencv");
        
          MainWindow mainWindow(300, 300);
          return app->run(mainWindow);
      }

      chevron_right

      
      

      Declaring a variable as auto specifies that the type of the variable that is being declared will be automatically deduced from its initialiser. This is called type inference.

      The call to Gtk::Application::create initialises a new GtK execution context, passes the Program Arguments, and sets the application id (that can be anything, but convention says that it should contain your domain name in reverse, followed by its name, like in ch.agl-developpement.cpp-tutorial.raspberry-cpp-gtk-opencv)

      Once you have a context ready you can use it to open a window. The window that can be an instance of any class that extends Gtk::Window. In this case our main window is called quite explicitly mainWindow and it is an instance of class MainWindow, that we will define later.

      The way we define (see the difference between define and declare) the mainWindow variable makes it an automatic variable (not same as auto type inference modifier seen before). Automatic variables are initialised and allocated in memory at the point of definition, and uninitialised and deallocated automatically when their scope is ended (read more about scope). On a local variable, this happens at the closing ‘}’ of their code block. We don’t need to care about deleting automatic variables (see, in contrast, Dynamic memory allocation with new and delete, or also dynamic memory allocation here at geeks for geeks).

      Whenever the type of variable is a class (as opposed to a basic type like int), the class constructor is called with the construction arguments and returns the initialised instance. In this case, construction arguments are the initial size of the window.

      The call to run will return when the user closes the window. This gives to Gtk execution context the opportunity to control the exit status.

      The basic GtK Window

      In Gtk, all buttons, labels, checkboxes and elements that displays or interacts with the user are called Widgets. Here is a gallery of Widgets.

      Then, gtkmm adds its own sauce:

      gtkmm arranges widgets hierarchically, using containers. A Container widget contains other widgets. Most gtkmm widgets are containers. Windows, Notebook tabs, and Buttons are all container widgets. There are two flavours of containers: single-child containers, which are all descendants of Gtk::Bin, and multiple-child containers, which are descendants of Gtk::Container. Most widgets in gtkmm are descendants of Gtk::Bin, including Gtk::Window

      See Gtkmm official documentation.



      MainWindow class declaration – the header

      class MainWindow : public Gtk::Window declares MainWindow as a class, and as descendant of, or inheriting from, Gtk::Window. The public modifier makes all public members of the ancestor to be also public members of the descendant. Public inheritance is the most usual way of extending a class (in contrast to private inheritance). We are providing MainWindow to the Gtk execution context, which is expecting all functionalities of a Gtk::Window to be available, so we avoid restricting their access.

      As a descendant of Gtk::Window, MainWindow is a single-child container that can only contain one Gtk::Widget. I choose it to be a Gtk::Box, which is a Gtk::Container so, in turn, it is able to contain several widgets. In this occasion I want them to be one Gtk::Button and two Gtk::Label, to illustrate the Packing mechanism that GtK uses to stack together a bunch of controls. For MainWindow to respond to clicks on the button, we declare a buttonClick method (could be any other name, but keep them meaningful) — I’m explaining later how to connect the ‘click’ signal to it.

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #ifndef MAIN_WINDOW_H
      #define MAIN_WINDOW_H
        
      #include <gtkmm.h>
        
      class MainWindow : public Gtk::Window {
      public:
          MainWindow(int width, int height);
          virtual ~MainWindow() = default;
        
      private:
          void buttonClick();
          Gtk::Button m_button;
          Gtk::Box m_box;
          Gtk::Label m_label1, m_label2;
      };
        
      #endif

      chevron_right

      
      

      The two first methods are the constructor and the destructor. As said before, constructor is called every time we declare a new instance of this class. This constructor accepts initial width and height of the window as parameters.

      Whenever an instance is destroyed, the class destructor is called to perform all needed clean up tasks. This destructor has two important modifiers:

      • virtual means that, in case of inheritance, the method called will be that of the dynamic type. Read more about what is dynamic type of object, virtual functions, and why most of the time it is a good idea to have virtual destructors.
      • default means that we define this destructor to have a default implementation. The default implementation of a destructor is able to deallocate all defined members of the class, which are m_box, m_button, m_label1 and m_label2 (code>buttonClick is a method therefore it does not need allocation or deallocation). By specifying the default implementation, we spare ourselves the need to define the destructor.

      The memory model of C++ has no garbage collector. Reserving and freeing memory is relies on the presence of class constructors and destructors and on a strategy called Resource Acquisition Is Initialisation, or RAII.

      MainWindow class definition – the source

      Declaration of MainWindow requires two further definitions – the class constructor and the buttonClick method. Here is the code:

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #include "main-window.hpp"
      #include <iostream>
        
      MainWindow::MainWindow(int witdh, int height)
          : m_button("Hello World"), m_box(Gtk::ORIENTATION_VERTICAL), m_label1("First Label"), m_label2("Second Label")
      {
          // Configure this window:
          this->set_default_size(witdh, height);
        
          // Connect the 'click' signal and make the button visible:
          m_button.signal_clicked().connect(
              sigc::mem_fun(*this, &MainWindow::buttonClick));
          m_button.show();
        
          // Make the first label visible:
          m_label1.show();
        
          // Make the second label visible:
          m_label2.show();
        
          // Pack all elements in the box:
          m_box.pack_start(m_button, Gtk::PACK_SHRINK);
          m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
          m_box.pack_start(m_label2, Gtk::PACK_EXPAND_WIDGET);
        
          // Add the box in this window:
          add(m_box);
        
          // Make the box visible:
          m_box.show();
      }
        
      void MainWindow::buttonClick()
      {
          std::cout << "Hello World" << std::endl;
      }

      chevron_right

      
      

      The full name of the constructor is MainWindow::MainWindow, meaning it is the function called MainWindow (the name of the constructor is the same as the class) that belongs to the class MainWindow. It takes width and height as parameters, and starts by calling the constructors of all declared members (read more about constructor member initialiser lists).

      Then we set the window’s default size by calling this->set_default_size. The this is a pointer to the current class instance. As it is a pointer, we use -> to access its members. As class MainWindow extends Gtk::Window, this instance contain all public or protected members of this class and those of the the parent class. This is called inheritance. The use of this pointer is actually optional, and the following snippet would also be valid:

      .

      // Configure this window (now, without the 'this'):
      set_default_size(witdh, height);
      

      Next we want to react to the m_button being clicked. One way to achieve this is to connect a function to the click signal of the m_button instance (to see all available signals of a widget, you can check the official documentation, for example, the Gtk::Button has only the clicked signal). Connecting functions to widget’s signals allows to process the event from anywhere in the code, as long as you have access to the widget and you can provide the address of the function.



      m_button.signal_clicked().connect([address of the method to call]);
      

      The address of the function could be as simple as placing the function’s name (see about the address of a function). Our case is more complex because the function we want to specify is a member of a class. Member functions need to be provided with the this pointer, so it can access other members of the same instance. To build the pointer to a method of a specific instance, we use sigc::mem_fun:

      sigc::mem_fun(                    // Convert member function to function pointer.
         *this,                         // The address of the instance.
         &MainWindow::buttonClick   // The address of the method.
      );
      

      By default, widgets are not visible. We need to set their visibility explicitly by calling show(). We do this for all widgets, including the m_box.

      Next, we pack all elements in the box. During initialisation we already set the box to Gtk::ORIENTATION_VERTICAL, which means that widgets will be positioned one below the other, using all available horizontal space. pack_start adds a new widget to the box, Gtk::PACK_SHRINK specifies that the widget’s vertical size should be as small as possible (Gtk::ORIENTATION_VERTICAL already specifies that all widgets in the box should have a width as big as possible, so the result will be widgets very wide and very flat.)

      In contrast, Gtk::PACK_EXPAND_WIDGET specifies that the widget should take all available space. This makes the m_label2 to expand when the user expands the window (you can test this shortly).

      The last step is to make the m_box itself visible.

      As for the buttonClick method, we just display a message to the standard output.

      Launching application in XCode

      Start by checking out the source code in some folder where you want place the project. Enter the checked out folder and create a xcode folder that will contain all working data for Xcode. Enter the working folder, create the Xcode project:

      $ cd go-to-your-working-folder
      $ git clone https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv.git
      $ cd raspberry-cpp-gtk-opencv
      $ mkdir xcode
      $ cd xcode
      $ cmake -G Xcode ../src
      $ make
      

      Now open Xcode, from the menu, do a File and Open and navigate for the xcode folder, and open one file called rascam.xcodeproj (notice that rascam is the name of the project specified in Cmake. The navigator shall open, showing one folder per target. Some of the targets, like ALL_BUILD and ZERO_CHECK are Xcode internals. There should be one target called rascapp, which is the name of the executable target in Cmake. You will find there your sources and headers. Open any one that you fancy, and place a break point. Then select the rascapp target in the non-descript drop list in the tool bar and, finally, play. If all is right, the application should execute. It is possible that the windows is not on top, so you don’t see it directly.

      Launching application in VisualStudio

      Launching application in Linux

      Clone the example project, configure it with debug symbols, and build it:



      $ cd go-to-your-working-folder
      $ git clone https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv.git
      $ cd raspberry-cpp-gtk-opencv
      $ mkdir build
      $ cd build
      $ cmake -DCMAKE_BUILD_TYPE=Debug ../src
      $ make
      

      To debug with gdb, assuming that you’re still in build folder:

      $ gdb ./rascapp
      [Now you're in gdb]
      b main-window.cpp:10  # Place a break-point on line 10 of this file.
      run                   # Run the program. It will stop at the break point.
      where                 # It will show the stack trace
      list                  # It will show some context.
      print width           # Displays the value of this variable
      n                     # Step over
      s                     # Step into
      c                     # To continue the program
      q                     # Quit gdb 
      

      Step 2 – A bit of refinement

      This second step adds a couple of small improvements to the application. First one is to react to key events. As the application will be on the Raspberry Pi, and not always easily accessible via a mouse, it may be handy to have some keyboard shortcuts, in particular to toggle full size window or close application. The second improvement is to log events in the system log. Applications that run from command line can cout messages that are directly visible. However, if you launch your window application from a user desktop, you will not see the console, nor the result of cout.

      Logging to syslog

      As window application are usually launched from the desktop, the content produced via cout is not easily visible. Instead, you can submit syslog messages (see also a syslog example). It is quite easy to do, and works across all platforms.

      In Linux (Ubuntu), you can see the system logs via the System Log application. In Mac OS X it is the Console application. In Windows you can use the Event viewer.

      Reacting to keyboard events

      Another way to react to events is to override a specific function in the Widget, that is called whenever that event happens. For example, to react to keyboard events in the MainWindow we have to override the on_key_press_event method. To override a function in C++:

      • The function has to be declared as virtual in one of the ancestors of our class.
      • We need to declare it again, in our class, with the exact same signature and accessibility.
      • We may add the override keyword on the function declaration, to acknowledge the fact that we’re overriding an existing method from a parent class. The compiler shows an error when we use override on a method that doesn’t exist or it is not virtual.

      on_key_press_event is declared in class Gtk::Widget as follows:

      /// This is a default handler for the signal signal_key_press_event().
      virtual bool on_key_press_event(GdkEventKey* key_event);
      

      In consequence, we have to re-declare it our class.

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #ifndef MAIN_WINDOW_H
      #define MAIN_WINDOW_H
        
      #include <gtkmm.h>
        
      #include "camera-drawing-area.hpp"
        
      class MainWindow : public Gtk::Window {
      public:
          MainWindow(int width, int height);
          virtual ~MainWindow() = default;
        
      protected:
          bool on_key_press_event(GdkEventKey* event) override;
        
      private:
          void buttonClick();
          bool probablyInFullScreen;
          Gtk::Button m_button;
          Gtk::Box m_box;
          Gtk::Label m_label1;
          CameraDrawingArea cameraDrawingArea;
      };
        
      #endif

      chevron_right

      
      

      The Widget where the event is originated is the first to receive it on its on_xx_xxx_event() event handler. If it returns true, the event is considered handled and nothing more is done about it. On the contrary, if the handler cannot do anything about the event, it returns false making the event to be offered to the same handler of its parent widget, and parent’s parent, and so on (see about event propagation in the official documentation). The default implementation of handlers like on_key_press_event is to do nothing but returning false, so events are propagated unless otherwise specified.

      [Ctrl] + [C] exits the application, [F] or [f] toggle the full screen mode, and [esc] turns the full mode off:

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #include "main-window.hpp"
      #include <syslog.h>
        
      MainWindow::MainWindow(int witdh, int height)
          : probablyInFullScreen(false), m_button("Hello World"), m_box(Gtk::ORIENTATION_VERTICAL), m_label1("First Label")
      {
          // Configure this window:
          this->set_default_size(witdh, height);
        
          // Connect the 'click' signal and make the button visible:
          m_button.signal_clicked().connect(
              sigc::mem_fun(*this, &MainWindow::buttonClick));
          m_button.show();
        
          // Make the first label visible:
          m_label1.show();
        
          // Make the second label visible:
          cameraDrawingArea.show();
        
          // Pack all elements in the box:
          m_box.pack_start(m_button, Gtk::PACK_SHRINK);
          m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
          m_box.pack_start(cameraDrawingArea, Gtk::PACK_EXPAND_WIDGET);
        
          // Add the box in this window:
          add(m_box);
        
          // Make the box visible:
          m_box.show();
        
          // Activate Key-Press events
          add_events(Gdk::KEY_PRESS_MASK);
      }
        
      void MainWindow::buttonClick()
      {
          syslog(LOG_NOTICE, "Hello world!");
      }
        
      bool MainWindow::on_key_press_event(GdkEventKey* event)
      {
          switch (event->keyval) {
          // Ctrl + C: Ends the app:
          case GDK_KEY_C:
          case GDK_KEY_c:
              if ((event->state & GDK_CONTROL_MASK) == GDK_CONTROL_MASK) {
                  get_application()->quit();
              }
              return true;
        
          // [F] toggles fullscreen mode:
          case GDK_KEY_F:
          case GDK_KEY_f:
              if (probablyInFullScreen) {
                  unfullscreen();
                  probablyInFullScreen = false;
              }
              else {
                  fullscreen();
                  probablyInFullScreen = true;
              }
              return true;
        
          // [esc] exits fullscreen mode:
          case GDK_KEY_Escape:
              unfullscreen();
              probablyInFullScreen = false;
              return true;
          }
        
          return false;
      }

      chevron_right

      
      

      By default, the events are not captured. add_events(Gdk::KEY_PRESS_MASK) activates the capture of keyboard events.



      The functions to toggle full screen are fullscreen() and unfullscreen(), both belong to the Gtk::Window class (here we’re omitting the this pointer). The documentation warns that you shouldn’t assume the window is definitely full screen afterward, because other entities (e.g. the user or window manager) could toggle it on or off again, and not all window managers honor those requests. That’s why we have added a probablyInFullScreen status, that conveys the possibility that window is not in the expected state.

      To exit the application we are not directly calling exit(), instead we use get_application()->quit() to tell the Gtk execution context to call exit. It will, eventually, but first it is going to close all windows and open devices for us.

      Step 3 – Display camera capture – Rough way

      To capture images we’re using OpenCV. If you just want to capture an image from the camera, you could use other libraries, but the final objective is to apply computer vision algorithms to the captured image. Hence, OpenCV is the best alternative.

      Bundling Mac OS X application

      For security reasons, Mac OS X forces applications to request user’s permission before using the camera. The procedure for this is to include a Info.plist file with the necessary content:

      • Some indications about version number and author.
      • A key NSCameraUsageDescription with the description on why your application needs the camera. The message you write here is displayed to the user, who has to accept it. If he doesn’t accept, the system terminates your application.

      The procedure is to include a MacOSXBundleInfo.plist.in file, which is a template; some of the entries can be governed by properties in the CMakeLists.txt file, and others you can put yourself. The file is placed in src/res folder and has the following content:

      <?xml version="1.0" encoding="UTF-8"?>
      <!DOCTYPE plist PUBLIC 
          "-//Apple Computer//DTD PLIST 1.0//EN" 
          "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
      <plist version="1.0">
      <dict>
          <key>CFBundleDevelopmentRegion</key>
          <string>English</string>
          <key>CFBundleExecutable</key>
          <string>${MACOSX_BUNDLE_EXECUTABLE_NAME}</string>
          <key>CFBundleGetInfoString</key>
          <string>${MACOSX_BUNDLE_INFO_STRING}</string>
          <key>CFBundleIconFile</key>
          <string>${MACOSX_BUNDLE_ICON_FILE}</string>
          <key>CFBundleIdentifier</key>
          <string>${MACOSX_BUNDLE_GUI_IDENTIFIER}</string>
          <key>CFBundleInfoDictionaryVersion</key>
          <string>6.0</string>
          <key>CFBundleLongVersionString</key>
          <string>${MACOSX_BUNDLE_LONG_VERSION_STRING}</string>
          <key>CFBundleName</key>
          <string>${MACOSX_BUNDLE_BUNDLE_NAME}</string>
          <key>CFBundlePackageType</key>
          <string>APPL</string>
          <key>CFBundleShortVersionString</key>
          <string>${MACOSX_BUNDLE_SHORT_VERSION_STRING}</string>
          <key>CFBundleSignature</key>
          <string>????</string>
          <key>CFBundleVersion</key>
          <string>${MACOSX_BUNDLE_BUNDLE_VERSION}</string>
          <key>CSResourcesFileMapped</key>
          <true/>
          <key>NSHumanReadableCopyright</key>
          <string>${MACOSX_BUNDLE_COPYRIGHT}</string>
          <key>NSCameraUsageDescription</key>
          <string>This app requires to access your camera to retrieve images and perform the demo</string>
      </dict>
      </plist>
      

      These are some of the resources I used to found out how to do bundles in Mac OS X:

      Linking to OpenCV

      After installing OpenCV following the previous instructions, you can link your application to it in CMake.

      # src/CMakeLists.txt
      cmake_minimum_required(VERSION 3.3 FATAL_ERROR)
      
      # Project name and current version
      project(rascam VERSION 0.1 LANGUAGES CXX)
      
      # Enable general warnings
      # See http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
      
      # Use 2014 C++ standard.
      set(CMAKE_CXX_STANDARD 14)
      set(CMAKE_CXX_STANDARD_REQUIRED ON)
      set(CMAKE_CXX_EXTENSIONS OFF)
      
      # Must use GNUInstallDirs to install libraries into correct locations on all platforms:
      include(GNUInstallDirs)
      
      # Pkg_config is used to locate header and files for dependency libraries:
      find_package(PkgConfig)
      
      # Defines variables GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS and GTKMM_LIBRARIES.
      pkg_check_modules(GTKMM gtkmm-3.0) 
      link_directories( ${GTKMM_LIBRARY_DIRS} )
      include_directories( ${GTKMM_INCLUDE_DIRS} )
      
      # OpenCV can be linked in a more standard manner:
      find_package( OpenCV REQUIRED )
      
      # Compile files:
      add_executable(rascapp
          cpp/main.cpp
          cpp/main-window.cpp
          cpp/camera-drawing-area.cpp
      )
      
      # Add folder with all headers:
      target_include_directories(rascapp PRIVATE hpp)
      
      # Link files:
      target_link_libraries(rascapp
          ${GTKMM_LIBRARIES}
          ${OpenCV_LIBS}
      )
      
      # Apple requires a bundle to add a Info.plist file that contains the required
      # permissions to access some restricted resources like the camera:
      if (APPLE)
          set_target_properties(rascapp PROPERTIES
              MACOSX_BUNDLE TRUE
              MACOSX_FRAMEWORK_IDENTIFIER org.cmake.ExecutableTarget
              MACOSX_BUNDLE_INFO_PLIST ${CMAKE_CURRENT_SOURCE_DIR}/res/MacOSXBundleInfo.plist.in
          
              # This property is required:
              MACOSX_BUNDLE_GUI_IDENTIFIER "rascapp-${PROJECT_VERSION}"
          
              # Those properties are not required:
              MACOSX_BUNDLE_INFO_STRING "rascapp ${PROJECT_VERSION}, by jmgonet@agl-developpement.ch"
              MACOSX_BUNDLE_LONG_VERSION_STRING ${PROJECT_VERSION}
              MACOSX_BUNDLE_BUNDLE_NAME "rascapp"
              MACOSX_BUNDLE_SHORT_VERSION_STRING ${PROJECT_VERSION}
              MACOSX_BUNDLE_BUNDLE_VERSION ${PROJECT_VERSION}
          )
      endif()
      

      A graphic widget

      Gtk::DrawingArea is a widget that holds a graphic area to display custom drawings or bitmaps. We define a CameraDrawingArea that extends this widget, and copies the image captured from the camera into the graphic area:

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #ifndef CAMERA_DRAWING_AREA_H
      #define CAMERA_DRAWING_AREA_H
        
      #include <gtkmm.h>
      #include <opencv2/highgui.hpp>
        
      class CameraDrawingArea : public Gtk::DrawingArea {
      public:
          CameraDrawingArea();
          virtual ~CameraDrawingArea();
        
      protected:
          bool on_draw(const Cairo::RefPtr<Cairo::Context>& cr) override;
          void on_size_allocate(Gtk::Allocation& allocation) override;
        
          bool everyNowAndThen();
        
      private:
          sigc::connection everyNowAndThenConnection;
          cv::VideoCapture videoCapture;
          cv::Mat webcam;
          cv::Mat output;
          int width, height;
      };
      #endif

      chevron_right

      
      

      We’re going to extend this class and override two of its methods:

      • void on_size_allocate(Gtk::Allocation& allocation) – This method is called every time the size of the widget changes. This happens the very first time it is displayed, and every time some action of the user or the system makes the size change.
      • bool on_draw(const Cairo::RefPtr& cr) – This method is called every time the area, or part of the area, contained in the widget has to be redrawn. It receives a reference to a Cairo context. The method can use it to render any drawing or graphic. In our case we’re going to use it to copy the image captured from the camera.

      The cv::Mat is a class that contains a OpenCV image. We’re using two of those: one contains the image captured from the camera, and the other contains the image resized to the Widget current size.



      The width and height contain the current Widget size.

      VideoCapture is a OpenCV access to the video camera.

      everyNowAndThenConnection is a way to call everyNowAndThen() method at regular time intervals, similarly as we set the response to the button click, in previous steps.

      This is the

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #include "opencv2/core.hpp"
      #include "opencv2/highgui.hpp"
      #include "opencv2/imgproc.hpp"
        
      #include "camera-drawing-area.hpp"
        
      CameraDrawingArea::CameraDrawingArea()
          : videoCapture(0)
      {
          // Lets refresh drawing area very now and then.
          everyNowAndThenConnection = Glib::signal_timeout().connect(sigc::mem_fun(*this, &CameraDrawingArea::everyNowAndThen), 100);
      }
        
      CameraDrawingArea::~CameraDrawingArea()
      {
          everyNowAndThenConnection.disconnect();
      }
        
      /**
       * Every now and then, we invalidate the whole Widget rectangle,
       * forcing a complete refresh.
       */
      bool CameraDrawingArea::everyNowAndThen()
      {
          auto win = get_window();
          if (win) {
              Gdk::Rectangle r(0, 0, width, height);
              win->invalidate_rect(r, false);
          }
        
          // Don't stop calling me:
          return true;
      }
        
      /**
       * Called every time the widget has its allocation changed.
       */
      void CameraDrawingArea::on_size_allocate(Gtk::Allocation& allocation)
      {
          // Call the parent to do whatever needs to be done:
          DrawingArea::on_size_allocate(allocation);
        
          // Remember the new allocated size for resizing operation:
          width = allocation.get_width();
          height = allocation.get_height();
      }
        
      /**
       * Called every time the widget needs to be redrawn.
       * This happens when the Widget got resized, or obscured by
       * another object, or every now and then.
       */
      bool CameraDrawingArea::on_draw(const Cairo::RefPtr<Cairo::Context>& cr)
      {
        
          // Prevent the drawing if size is 0:
          if (width == 0 || height == 0) {
              return true;
          }
        
          // Capture one image from camera:
          videoCapture.read(webcam);
        
          // Resize it to the allocated size of the Widget.
          resize(webcam, output, cv::Size(width, height), 0, 0, cv::INTER_LINEAR);
        
          // Initializes a pixbuf sharing the same data as the mat:
          Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(
              (guint8*)output.data,
              Gdk::COLORSPACE_RGB,
              false,
              8,
              output.cols,
              output.rows,
              (int)output.step);
        
          // Display
          Gdk::Cairo::set_source_pixbuf(cr, pixbuf);
          cr->paint();
        
          // Don't stop calling me.
          return true;
      }

      chevron_right

      
      

      In the constructor we start by initialising videoConstructor to the default device. Then we set up a connection to call the everyNowAndThen method every 100ms (10 x per second). Later we call the connection to disconnect it during the destruction process.

      The everyNowAndThen() method invalidates the whole area of the widget. This is an indirect way to provoke a call to on_draw.

      on_size_allocate keeps track of the current size of the Widget. Just in case the parent class did implement something important in its own implementation, we call the parent method.

      Everything we put in place before is so on_draw is called regularly, and we can use the reference to Cairo::Context to paste the image captured from the camera:

      • First we check that the current width and height are not zero. This may happen during startup phase. Calling resize with a 0 dimension would terminate instantly the application so we better avoid it.
      • videoCapture.read(webcam) copies the captured image into the provided cv::Mat, in this case webcam. If the provided material is not initialised, or it is not of the appropriate size, then the method will perform necessary tasks, and reserve needed memory.
      • resize copies the image from source (webcam) to destination (output), changing its size. If the destination is not initialised or not of the right size it performs necessary tasks and memory reservation. To keep the variable in scope, we also declared it as a class property: first call will initialise the variable, next calls can just reuse it.
      • Both webcam and output are class properties so they keep in scope as long as the class instance does. They will need to be initialised only during the first call to on_draw but will keep their values across the subsequent calls. This helps noticeably the performance.
      • create_from_data creates a new Gdk::Pixbuf object, which contains an image in a format compatible with Cairo. The long list of parameters are to be taken as a recipe to convert OpenCV‘s Material to Gtk‘s Pixbuf.
      • set_source_pixbuf sets the bitmap source to use for next call to paint
      • Finally, paint does the painting.

      Placing the drawing area

      The last step is to place the widget we just created in the main window. We need to declare it in the MainWindow header:

      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #ifndef MAIN_WINDOW_H
      #define MAIN_WINDOW_H
        
      #include <gtkmm.h>
        
      #include "camera-drawing-area.hpp"
        
      class MainWindow : public Gtk::Window {
      public:
          MainWindow(int width, int height);
          virtual ~MainWindow() = default;
        
      protected:
          bool on_key_press_event(GdkEventKey* event) override;
        
      private:
          void buttonClick();
          bool probablyInFullScreen;
          Gtk::Button m_button;
          Gtk::Box m_box;
          Gtk::Label m_label1;
          CameraDrawingArea cameraDrawingArea;
      };
        
      #endif

      chevron_right

      
      

      And place it in the box, just the same as a label or a button:



      filter_none

      edit
      close

      play_arrow

      link
      brightness_4
      code

      #include <syslog.h>
      #include <unistd.h>
        
      #include "main-window.hpp"
        
      MainWindow::MainWindow(int width, int height)
          : probablyInFullScreen(false), m_button("Hello World"), m_box(Gtk::ORIENTATION_VERTICAL), m_label1("First Label")
      {
          // Configure this window:
          this->set_default_size(width, height);
        
          // Connect the 'click' signal and make the button visible:
          m_button.signal_clicked().connect(
              sigc::mem_fun(*this, &MainWindow::buttonClick));
          m_button.show();
        
          // Make the first label visible:
          m_label1.show();
        
          // Make the second label visible:
          cameraDrawingArea.show();
        
          // Pack all elements in the box:
          m_box.pack_start(m_button, Gtk::PACK_SHRINK);
          m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
          m_box.pack_start(cameraDrawingArea, Gtk::PACK_EXPAND_WIDGET);
        
          // Add the box in this window:
          add(m_box);
        
          // Make the box visible:
          m_box.show();
        
          // Activate Key-Press events
          add_events(Gdk::KEY_PRESS_MASK);
      }
        
      void MainWindow::buttonClick()
      {
          syslog(LOG_NOTICE, "User %d says 'Hello World'", getuid());
      }
        
      bool MainWindow::on_key_press_event(GdkEventKey* event)
      {
          switch (event->keyval) {
          // Ctrl + C: Ends the app:
          case GDK_KEY_C:
          case GDK_KEY_c:
              if ((event->state & GDK_CONTROL_MASK) == GDK_CONTROL_MASK) {
                  get_application()->quit();
              }
              return true;
        
          // [F] toggles fullscreen mode:
          case GDK_KEY_F:
          case GDK_KEY_f:
              if (probablyInFullScreen) {
                  unfullscreen();
                  probablyInFullScreen = false;
              }
              else {
                  fullscreen();
                  probablyInFullScreen = true;
              }
              return true;
        
          // [esc] exits fullscreen mode:
          case GDK_KEY_Escape:
              unfullscreen();
              probablyInFullScreen = false;
              return true;
          }
        
          return false;
      }

      chevron_right

      
      

      Installing on Raspberry Pi

      Last but not least, you need to install the project on a Raspberry Pi.

      Activate camera

      The easiest way is to do it from the desktop:

      1. Raspberry –> Preferences
      2. Open Interfaces tab.
      3. Enable the Camera
      4. To check that the camera works, type:

        $ raspistill -f -t 0
        

        You should be able to see images from camera. Exit with Ctrl+C.

        Driver v4L2

        In Raspberry, the v4L2 driver offers the standard interface to the camera that OpenCV needs to capture images. The driver is installed by default, but not active. To activate it:

        $ sudo modprobe bcm2835-v4l2
        $ ls /dev/video0
        

        If the device /dev/vide0 is present, it means that the driver is active. To activate it by default when the Raspberry starts, edit file /etc/modules and add the following snippet at the bottom:

        bcm2835-v4l2
        

        Install tooling

        To be able to download and build the project, you need the same tooling as in your development platform: Git, Cmake, pkg-config and the libraries Gtk and OpenCV. To install them, follow the procedure for the Linux environment, in this same article.

        Autorun

        You may want Raspberry Pi to launch your application at startup. The easiest way I found to auto-run a GUI application in Raspberry is to create a desktop file in the autostart directory. In a brand new installation, this directory doesn’t exist and you need to create it:

        $ mkdir ~/.config/autostart
        $ vim ~/.config/autostart/rascapp.desktop
        

        The content of “rascapp.desktop“ should be similar to:

        [Desktop Entry]
        Name=raspberry-pi-camera-display
        Exec=/home/pi/raspberry-cpp-gtk-opencv/build/rascapp
        Path=/home/pi
        Type=application
        

        The Path key is the root folder for the application, when accessing files by a relative path.



        • See more about syntax here:
        • See original explanation here:

        Avoid console blanking

        Console blanking affects you if you’re using ssh to execute commands. If, for some time, you don’t type anything
        in the console, it will close.

        To avoid it, edit file /boot/cmdline.txt and append the paramter consoleblank=0

        See original explanation here: https://www.raspberrypi.org/documentation/configuration/screensaver.md

        Avoid idle screen

        Display-only applications, will typically have no user interaction, so screen may go idle leaving you in the dark. To prevent this, the simplest approach is to install a screen saver and then configure it to NOT run:

        $ sudo apt-get install xscreensaver
        

        After this, screensaver application is in Preferences, in desktop menu. Use the appropriate options to prevent screen saver.

        Enabling composite video out

        If you’re placing the Raspberry on board of some mobile device with a FPV transmitter, you probably want to use the Analog Video Out. Edit the /boot/config.txt file and modify the entries as following:

        ...
        sdtv_mode=2
        ...
        hdmi_force_hotplug=0
        ...
        

        When hdmi_force_hotplug is set to 1, the system assumes that there is a HDMI device present, so
        it never activates video composite output.

        When hdmi_force_hotplug is set to 0, the system will use composite video unless it detects HDMI device. So,
        Raspberry will still use the HDMI monitor if it is connected during boot sequence.

        See more about this:

        Troubleshooting

        I hope you don’t need this section.

        Gtk-ERROR **: GTK+ 2.x symbols detected.

        If you get this error message:

        Gtk-ERROR **: GTK+ 2.x symbols detected. 
        Using GTK+ 2.x and GTK+ 3 in the same process is not 
        supported
        

        You may accidentally linked OpenCV with Gtk2. As this project uses Gtk3, there is a conflict. You need to be sure that OpenCV is linked with Gtk3.

        Start by uninstalling Gtk2 and ensuring Gtk3 is present:

        $ sudo apt-get remove libgtk2.0-dev
        sudo apt-get install libgtk-3-dev
        sudo apt-get auto-remove
        sudo apt-get install libgtkmm-3.0-dev
        

        Build again OpenCV:

        • If you haven’t done this yet, then remove the build directory of OpenCV.
        • Start again the building procedure as described above BUT…
        • Before launching make, check the cmake log, and verify the Gtk version is linked with. Look for something like this:
        ...
        --   GUI: 
        --     GTK+:                        YES (ver 3.22.11)
        --       GThread :                  YES (ver 2.50.3)
        --       GtkGlExt:                  NO
        --     VTK support:                 NO
        ...
        

        Conclusion

        This example is a working proof of concept on how write a cross-platform application based on two very popular libraries, OpenCV for processing computer vision and Gtk for user interface. However, there are a list of shortcomings that need to be addressed before you can use it as a base for the more complex application that you may have in mind:

        • The more visible one is that the image is deformed to match the size of the window. This can be easily solved using a more sophisticated algorithm to calculate a size that would fit in the window but keep the original aspect ratio.
        • A second one, very annoying if you plan a First Person View application is the perceptible lag between real life and the images stream. Lag seems variable depending on the camera and the light conditions. The reason is cameras having an image buffer; as we are retrieving one image every now and then, we’re always consuming the oldest image in the buffer. To solve this we should let the capture process drive the window refresh, instead of using a timer.
        • Code has a general lack of decoupling that makes everything very dependent on everything else. This defect isn’t very present in such a simple application, but it will show as soon as we try to solve the lag problem. Also, part of the code should be a pure OpenCV process that you could copy/paste from some other blog, without needing to adapt it to the CameraDrawingArea that is dependent on both OpenCV and Gtk libraries.
        • Finally, you should be able to unit test the OpenCV processing to increase the type-compile-debug cycle.

        My next article covers those topics.



        My Personal Notes arrow_drop_up

        Electronics and Computer Science engineer in finance by day and embedded systems by night I like to explore around embedded systems onboard of model cars and planes Im using Microchip PIC micro controllers, Raspberry PI, GtK, OpenCV and C/C++

        If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

        Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.




        Article Tags :
        Practice Tags :


        4


        Please write to us at contribute@geeksforgeeks.org to report any issue with the above content.