EN 
DE Possibilities and limitations
Authors: Prof. Dr. Marianne von Schwerin, Daniel Klitzke, Ulm University of Applied Sciences
Contribution – Embedded Software Engineering Congress 2016
Image processing techniques are used in many industrial sectors. Often, this involves the use of hardware specifically tailored to the respective requirements and custom-developed software. However, with the increasing performance of smartphones, these devices can also be used for image processing tasks. The example of traffic sign recognition in an autonomous model car demonstrates that image processing can be implemented very effectively with a standard smartphone. However, the smartphone-based system is not real-time capable, and the hardware is not designed for continuous operation, so compromises have to be made.
Introduction
Modern smartphones are equipped with powerful multi-core processors and several gigabytes of RAM. They can be adapted for image processing tasks with manageable effort, thus offering versatile applications in industrial settings. Furthermore, they offer numerous communication interfaces and can therefore be used in distributed systems. Battery operation makes them particularly attractive for mobile applications, and the integrated, increasingly high-quality cameras also make smartphones interesting for image processing tasks.
Application example
At Ulm University of Applied Sciences, an RC car was developed as a remote-controlled, electrically powered vehicle (see Figure 1, PDF), which can perform autonomous driving maneuvers. It can follow a track marked on the ground, and it also reacts appropriately to traffic signs and traffic lights by evaluating camera images.
The main control unit is an Arm Cortex-M3 microcontroller, which receives control commands via Wi-Fi and controls the electric drive motors accordingly. A Google Nexus 5 smartphone is mounted at the front of the vehicle. This smartphone can follow a lane by analyzing images from its internal camera and also uses images from a forward-facing camera to recognize traffic signs and traffic lights. The software on the image-processing smartphone combines the results of these analyses and sends corresponding control commands to the microcontroller.
Implementation of traffic sign recognition
The smartphone used in this example for image processing is a device running Android operating system version 4.4.2. It is equipped with four cores, each clocked at 2.3 GHz, and 2 GB of RAM. The image processing application is implemented as an Android app in Java.
Since a system for object recognition, such as the one implemented here, uses a variety of common image processing algorithms, a powerful software library is employed. OpenCV is a software library for machine vision and machine learning [1] that contains a large number of such algorithms. OpenCV is open source and is licensed under the BSD license [1]. In this project, OpenCV version 3.1 was used. The library is integrated via a Java wrapper provided by OpenCV. This allows OpenCV to be used very easily from within the Java code of the application. Calling a Java function of the wrapper results in a call to a native method implemented in C/C++, where the actual computation is performed, which offers a significant speed advantage compared to a Java implementation.
The following section examines the recognition of a stop sign as an example image processing task for an autonomous vehicle. Traffic sign recognition is divided into a detection step and a classification step. During detection, potential sign candidates are filtered from the camera image based on color. During classification, the identified sign candidates are then validated based on their shape and, if necessary, added to a list of recognized traffic signs (see Figure 2)., PDF).
The individual steps of detection and classification are now implemented using OpenCV. For color-based detection, the image (see Figure 3, PDFThe data is first converted to the HSV color space, which allows for more intuitive color filtering because the hue information for saturation and brightness is stored separately in the Hue channel. This color space conversion is performed by calling the OpenCV function. Imgproc.cvtColor.
Subsequently, all pixels lying within relevant color ranges are filtered out using simple thresholding. OpenCV also provides a filter function for this purpose. Imgproc.inRange for use. A binary image is obtained in which all relevant pixels have the value 255 (white) (see Figure 4)., PDF).
Based on the binary image, individual objects are now segmented from the image. For this purpose, the entire image is scanned, and each rectangular region is checked for the number of its relevant pixels. If this number exceeds a certain value, the rectangle is marked as belonging to an object. Overlapping rectangles are grouped together into larger regions (see Figure 5)., PDFTo avoid having to add up the color values of all pixels in a region each time, the operation is performed on a previously defined value using the OpenCV function. Imgproc.integral Calculated integral image executed.
The previously identified regions, where traffic signs may be located based on their color coding, will now be classified. This means checking whether a traffic sign is actually present in a detected area, and if so, which one. The classification will be performed using a Support Vector Machine (SVM) based on so-called Histogram of Oriented Gradients (HOG) features.
A Histogram of Oriented Gradients is a feature descriptor frequently used in machine vision for shape-based object recognition. It consists of histograms representing the distribution of gradients in a specific image region (see Figure 6)., PDF). In this case, the gradient is a two-dimensional vector that points in the direction of the steepest change at each pixel position (see Figure 7, PDFThe descriptor can be easily calculated using OpenCV; only an object of type HogDescriptor is instantiated, and the feature vector is calculated by calling the method. HogDescriptor.compute calculated.
For the calculated shape features of each sign candidate, it must now be checked which sign class they can be assigned to. A Support Vector Machine (SVM) is a supervised learning algorithm; that is, the algorithm learns from a set of example data a function that maps a specific output to a given input [2]. This means that the SVM was trained on feature vectors for a large number of known signs and negative examples to produce the appropriate output. Here, the libSVM library was used instead of the OpenCV SVM implementation, because the OpenCV Java wrapper had a bug at the time of implementation that prevented saving and loading a trained SVM model.
Result
The developed system exhibits good functional properties but also has limitations. A positive aspect is the large number of algorithms available in OpenCV and their high degree of optimization, which ensures good performance. The integration via a Java wrapper in the presented project also proved largely trouble-free. Unfortunately, this Java wrapper is poorly documented; that is, the official documentation contains only a few basic examples of its use.
The more extensive and algorithmically complex the image processing, the greater the impact on the overall system performance. Using native C or C++ code exclusively in the image processing portion of the application, instead of Java, can significantly improve performance, as demonstrated by tests in the presented system. This is easily achievable by including OpenCV as a native library rather than via the Java wrapper (see [3]). If performance-critical functionalities are still partially implemented in Java, it is advisable to consult the performance tips in the Android Developers documentation [4] or the Best Practices for Performance [5].
A general problem is the use of Android smartphones in real-time systems. Such a solution is unsuitable for hard real-time requirements, as determining a maximum runtime for certain tasks is usually impossible. An example of this is the non-deterministic garbage collector. A real-time capable Android version, such as RTAndroid developed at RWTH Aachen University [6], could remedy this. However, even with this, other real-time problems remain in the overall system, such as the unpredictable transmission time of data over Wi-Fi.
When using a smartphone for image processing, it's crucial to monitor RAM usage, as exceeding the maximum heap size of the runtime environment will cause the app to crash. This maximum heap size varies depending on the smartphone and is determined by the manufacturer. By default, the maximum heap size is defined by the so-called heapgrowthlimit value. This limit can be adjusted by setting the flag. android:largeHeap The heap size can be increased in the Android app's manifest. In this case, the `heapsize` value defines the limit for the heap size. These values are defined by the developer in the `build.prop` file and can be changed if you have root access.
The system's performance is also subject to factors beyond the programmer's control. Tests showed that the smartphone used significantly reduces the maximum clock speed of its processor cores when a critical temperature is exceeded, causing the processing rate to drop from 10.5 to 5.8 frames per second. In the case of the Google Nexus 5, the CPU frequency is throttled from 2300 MHz to 1570 MHz at a temperature of 42 degrees Celsius and to 1190 MHz at 44 degrees Celsius. These values are defined in the thermal-engine.conf file. Other factors, such as the activation of power-saving mode when the battery is low or the activity of other apps running in the background, also affect the smartphone's performance (see Figure 8)., PDF).
conclusion
Modern smartphones are an interesting platform for implementing an image processing system, as Android versions offer modern libraries like OpenCV. However, the implementation of systems that must meet strict real-time requirements is limited because the runtime environment is not real-time capable. Furthermore, it has been shown that smartphone hardware cannot withstand sustained high processor loads due to temperature issues, and the processor clock speed is significantly throttled after only a short time.
Nevertheless, smartphones represent an interesting, powerful and inexpensive platform for the implementation of image processing systems.
literature
| [1] | Itseez, „About | OpenCV„, 2016. [Accessed on 11 3 2016]. |
| [2] | SJ Russell and P Norvig, Artificial intelligence. A modern approach. 3rd ed., Upper Saddle River, NJ: Prentice-Hall, 2010. |
| [3] | opencv dev team, „Introduction into Android Development – OpenCV 2.4.13.1 documentation„, opencv dev team. [Accessed on 29 9 2016]. |
| [4] | Google Inc., „Performance Tips | Android Developers„, Google Inc. [accessed on 29 9 2016]. |
| [5] | Google Inc., „Best Practices for Performance | Android Developers„, Google Inc. [accessed on 29 9 2016]. |
| [6] | Real-Time Android Project,"„RTAndroid | Real-Time Android Project„, 2015. [Accessed on 29 9 2016]. |
Implementation – our training & coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of implementation/embedded and real-time software development.
Training & coaching on the other topics in our portfolio can be found here. here.
Implementation – Expertise
Valuable expertise in the field of implementation/embedded and real-time software development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
