EN 
DE Performance analysis of the real-time capabilities of Linux-based container solutions on ARM
Author: Michael Schnelle, Mixed Mode
Contribution – Embedded Software Engineering Congress 2018
Real-time operating systems
A real-time system is generally defined as a system that must react to an event within a finite and predictable timeframe. Such systems therefore place not only logical but also temporal demands on the outcome. Real-time does not necessarily mean rapid action, but rather adherence to the set time constraints and deterministic behavior. Real-time systems are thus divided into two categories: soft and hard real-time systems.
See Fig. 1 (PDF).
In a soft real-time system, the set time constraints may be violated. Nevertheless, the delivered result is still relevant to the system. The extent to which the benefit of the result is diminished by the delay depends on the use case. In hard real-time, on the other hand, the set time constraints must be adhered to at all times. If this rule is violated, the result has no value for the system.
Linux as a real-time operating system
Linux was not originally designed for use as a real-time or embedded operating system, but its broad hardware support quickly made it attractive for both areas. Early approaches to making Linux real-time capable involved a dual-kernel architecture. These solutions run an additional, smaller real-time kernel on the system. This kernel executes Linux as a low-priority process. Well-known examples of such approaches are RTAI and Xenomai.
Furthermore, Linux, even in its standard version, is now suitable for soft real-time systems. This is due to the development of the Preempt-RT patch, which makes the Linux kernel itself hard real-time capable. Currently, the developers are working on fully integrating the patch into the mainline kernel. The patch focuses primarily on making the Linux kernel fully interruptible, as this is a crucial concept for real-time operating systems.
The standard kernel currently supports three interruption models:
No Forced Preemption (Server), where a task can only be interrupted upon returning from a system call or upon an interrupt. Voluntary Kernel Preemption (Desktop), which introduces additional interruption points. Or Preemptible kernel (Low-Latency Desktop), where the kernel can be interrupted at any time, unless it is in a critical area.
The Preempt-RT patch introduces two additional interrupt points: Preemptible Kernel (Basic-RT) and Fully Preemptible (RT). The latter makes the Linux kernel completely interruptible, except for a few critical sections. Mechanisms such as sleeping spinlocks and rt_mutexes are used for this purpose. Furthermore, interrupt routines are executed as threaded interrupts, allowing a higher-priority userspace process to interrupt an ISR.
Virtualization
Container virtualization is often mistakenly referred to as having lightweight virtual machines, although the underlying techniques are completely different:
Full virtualization uses a software component to form the logical layer between the guest system and the hardware. This software component is called a hypervisor or virtual machine monitor. In practice, a distinction is made between Type 1 and Type 2 hypervisors. (Mandl, 2014)
A Type 1 hypervisor runs directly on the system's hardware and acts as a minimal operating system responsible for creating and managing individual virtual machines. This type of hypervisor therefore requires drivers for hardware access. A Type 2 hypervisor, on the other hand, runs as an application program within an operating system. Hardware access is then handled by the host system's drivers.
Container virtualization (operating system virtualization) isolates processes solely at the operating system level. The host system's kernel creates virtual process spaces in which the processes can run. The isolated processes appear to have access to a complete computing environment, although they are actually isolated user-space instances of an underlying operating system. The advantage is that sharing a kernel prevents significant performance degradation. However, this also means that all isolated instances depend on the host system's kernel, and therefore different operating systems cannot be run on them. (C. Arnold, 2012)
The Linux kernel itself provides several mechanisms for operating system virtualization. Process trees allow processes to be isolated from each other. Container virtualization utilizes this principle to create individual containers.
Container solutions use this mechanism, for example, for the following system resources:
- Mount namespace: The namespace can be assigned its own root directory or private mounts.
- PID Namespace: When a container is created, the container engine creates a new PID namespace and moves the processes to be isolated into this namespace. The first process running in the container receives PID 1. Every subsequent process in the container is then subordinate to this first process. The container itself is restricted to its PID namespace, while the host system can still see all processes.
Control groups (cgroups) are a mechanism in the Linux kernel that allows processes or containers to be allocated only a fraction of certain resources. Processes are assigned to different control groups, whose access to system resources can be limited.
Container solutions
The concept of isolated process instances under Linux has existed since 2001. It only became practical with the introduction of the kernel features cgroups and namespaces. The first implementation to use these mechanisms was LXC (Linux Containers). Docker represents another implementation.
Docker uses a server-client architecture, consisting of the Docker Client and the Docker Daemon. When creating containers, the user interacts with the Docker Client, which then forwards commands to the Docker Daemon. The Daemon subsequently handles all further tasks, such as managing system images (Docker Images) and creating Docker containers. The client and daemon do not necessarily have to run on the same system. Initially, Docker relied on external container solutions like libvirt or LXC to interact with the Linux kernel. Starting with version 0.9, this functionality was replaced by Docker's own implementation. The virtualized operating system is encapsulated in an image, which can be stored and managed in either a private or public registry. A running image is called a container.
Test concept
The hardware used was a BeagleBone Black with a Texas Instruments Sitara AM335x Cortex-A processor. This setup was used to test the real-time capabilities of the container solutions Docker and LXC. Several test scenarios were selected for this purpose:
Cyclictest is a program frequently mentioned in connection with the Preempt-RT patch. It was written as part of the patch's development and can be used to assess a system's real-time capability. To do this, the program creates a thread that measures the current system time and then goes into sleep mode for a defined time interval. Once the timer expires and the thread is selected again by the scheduler, the thread takes the system time again and can calculate a scheduling latency value from the measured times.
See Fig. 2 (PDF).
The program is also used by OSADL to test the real-time capability of the Preempt-RT patch. The result is output as a histogram. Additional information includes the minimum, average, and maximum latency. Below are example histograms for two BeagleBone Blacks from the OSADL test farm. The one on the left uses the Preempt-RT patch, and the one on the right uses a standard kernel.
See Fig. 3 (PDF).
In a second real-time application, the interrupt stopwatch, the measurement for real-time capability is not performed directly on the test system as with cyclictest, but rather via an external measuring device. The basic idea is to simulate a frequently encountered situation in which a real-time system must react to an external signal within a defined timeframe. The processing and reaction to the external signal are to take place in a user-space application, which is run once natively and once within the container solution.
More specifically, an external trigger signal will be generated using a microcontroller. This signal will trigger an interrupt service routine in the Linux kernel, which will set a GPIO pin high and then wake up the real-time application in user space. This application will then set the corresponding GPIO pin back to low. The planned signal flow from trigger to response is outlined below.
See Fig. 4 (PDF).
The time interval I represents the interrupt latency, i.e., the time between the trigger signal being sent and the start of the corresponding ISR on the test system. The time interval S, in turn, represents the time that elapses between the ISR and the response from the user-space application.
At the start of the measurement, the microcontroller should start a counter and detect both the rising and falling edges of the response signal. The current counter value should be stored each time. This allows the interrupt latency and the interrupt-to-process latency to be determined simultaneously. The measured values should then be transmitted via the serial interface to a computer for later processing, and a new measurement should be started.
The principle of this measurement is a common test scenario in the evaluation of real-time systems and is implemented, for example, by OSADL in the form of the Latency Box.
For the evaluation and comparison of the real-time capability of container solutions, the interrupt-to-process time will be of particular interest here as well, since delays could occur due to the operation of the application within a container.
It should be noted that this test procedure may seem redundant at first glance, since the cyclictest yields a similar value. However, external interrupts and timer interrupts are handled differently in the kernel. This means that different parts of the kernel code are involved, and therefore the timing behavior can differ.
implementation
Two different load scenarios were considered during testing. In the first case, the tests were performed on the BeagleBone Black without any additional load. Subsequently, the system was subjected to additional load during the tests. Both programs were run under different load scenarios with 10 million measurements.
See Figures 5 and 6 (PDF)
Conclusion
„"Hypervisor-based VM virtualization and container-based application virtualization are conceptually as different as night and day," [quote] according to the general consensus. This is entirely true. A container is not simply a lightweight virtual machine. Rather, it encapsulates system processes whose visibility is limited by the operating system.
Furthermore, it can be stated that the basic operation of a real-time application in a Linux container is possible. However, the container must be granted certain permissions, which are particularly concerning with regard to ensuring secure operation. Although the limited scope of the measurements prevented the recording of completely irrefutable results, it can be assumed that similar results will be obtained in future tests.
Bibliography
C. Arnold, MJ (2012). KVM Best Practices. dpoint.
Mandl, P. (2014). Basic course in operating systems. Wiesbaden: Springer Verlag.
author
Speaker Michael Schnelle holds a Master's degree in Software Engineering. He has several years of development experience, specializing in application development and security in various aspects. His work has included conducting risk analyses, developing solutions to defend against malicious requests on a social media platform, creating a highly secure signaling system for emergency call units, and generating a secure Linux/Debian distribution. In his current project, he is involved in the development of passenger information systems for public transport companies, focusing on distributed systems, microservices, and test automation.
Contact: michael.schnelle@mixed-mode.de
Download the article as a PDF file
Real-time – MicroConsult Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of embedded and real-time software development.
Training & coaching on the other topics in our portfolio can be found here. here.
Real-time expertise
Valuable expertise in the field of embedded and real-time software development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
