Authors: Jan Altenberg, Heinz Egger, Linutronix GmbH<\/p>\n
Due to its large number of supported CPU architectures, virtually endless driver options, and, not least, its excellent portability and scalability, Linux is one of the most powerful embedded operating systems of our time. Even systems with stringent real-time requirements can be easily implemented with Linux. Various approaches and methods exist for achieving this. But which approach is the right one? And what latency levels can be achieved? This article presents different technologies that enable hard real-time capability under Linux. Furthermore, it demonstrates the jitter and latency levels achievable with these technologies.<\/strong><\/p>\n Basically, there are two approaches to making Linux capable of real-time operation:<\/p>\n In the microkernel approach, all real-time tasks are handled in a separate RTOS, with Linux being scheduled as a low-priority task within this RTOS. Strictly speaking, therefore, one should not speak of real-time with Linux, but rather of real-time alongside Linux.<\/p>\n The so-called in-kernel approach aims to make Linux itself capable of real-time operation (without an underlying microkernel). Several examples of this approach will be presented below.<\/p>\n The Realtime Application Interface (RTAI) is a development of the Technical University of Milan and was created under the auspices of Professor Paolo Mantegazza. RTAI is a classic example of the microkernel approach. The primary design goal of RTAI is, and always has been, to achieve the lowest possible latency on a given hardware platform. This design goal imposes several limitations on RTAI applications. Furthermore, only a relatively small number of target platforms are supported (currently x86, x86_64, and various Arm platforms). In practice, there are hardly any new projects that build upon RTAI. Figure 1 (see\u00a0PDF<\/a>) shows the basic structure of RTAI.<\/p>\n The Xenomai project was founded in 2001. Unlike RTAI, Xenomai allows for relatively easy use of real-time computing in the user space (RTAI only allows this to a very limited extent). A key feature of Xenomai is its skins, which simplify the porting of applications from other real-time systems (e.g., VxWORKS) to Xenomai. Xenomai skins replicate the APIs of these systems. Xenomai currently supports the following architectures: PowerPC32, PowerPC64, x86, x86_64, Blackfin, Arm, and ia64. The central concepts in the Xenomai design are the Xenomai Nucleus, the Interrupt Pipeline (IPIPE), the Hardware Abstraction Layer (HAL), and the System Abstraction Layer (SAL).<\/p>\n IPIPE can be visualized as a virtual interrupt controller. It organizes the system into different domains. Interrupts are received by IPIPE and distributed to the individual domains. Nucleus contains the Xenomai Core functionality. This is responsible for providing all the necessary resources that skins need to emulate the functionality of RTOSs. The Hardware Abstraction Layer contains the platform- and CPU-dependent code. All layers above it (including Nucleus) build upon this. Both the kernel and the application must use custom libraries and a custom API. Therefore, standard Linux libraries cannot be used! (This also applies to RTAI). Figure 2 (see\u00a0PDF<\/a>) shows the concept of Xenomai.<\/p>\n The Realtime Preemption Patch (PREMPT_RT) originated from the work of Ingo Molnar and Thomas Gleixner. Thomas Gleixner, in particular, is the driving force behind the development of PREEMPT_RT today. Unlike RTAI and Xenomai, PREEMPT_RT makes the Linux kernel itself real-time capable. This is achieved primarily through the following mechanisms:<\/p>\n Many mechanisms originally developed in PREEMPT_RT have long since found their way into the mainline Linux branch: High Resolution Timers (high-resolution timers independent of the scheduler tick), priority inheritance, generic interrupt handling for all architectures, and, as early as version 2.6.30, threaded interrupt handlers. Furthermore, the Linux developer community agreed as early as 2006 that Preempt-RT should be integrated into the Linux kernel.<\/p>\n With the work of OSADL and a new working group of the Linux Foundation (RTL, founded in October 2015; founding members include Google, OSADL, Intel, Arm, Texas Instruments, Altera, National Instruments, and others), another major step in this direction has recently been taken. Due to its many advantages and widespread acceptance within the Linux community, PREEMPT_RT has established itself as the de facto standard for real-time Linux in recent years. In addition to its numerous technological advantages, it is worth noting that OSADL is extensively involved in quality assurance for PREEMPT_RT-based systems. For this purpose, countless benchmarks and latency measurements are performed on a wide variety of hardware in a test farm (https:\/\/www.osadl.org\/QA-Farm-Realtime.qa-farm-about.0.html).<\/p>\n Furthermore, the Realtime Preemption Patch offers the significant advantage that real-time applications can be written as POSIX real-time applications. No special API is used. PREEMPT_RT supports a wide variety of architectures (PowerPC, x86, x86_64, MIPS, Arm, etc.).<\/p>\n As shown in Figure 3 (see\u00a0PDF<\/a>As demonstrated, PREEMPT_RT integrates real-time functionality seamlessly into the Linux kernel. Developers of other projects have also recognized the advantages of PREEMPT_RT. Xenomai 3 offers support for PREEMPT_RT, enabling the use of Xenomai skins on PREEMPT_RT kernels.<\/p>\n To compare microkernel and in-kernel approaches, comparative measurements were performed on an Arm Cortex A9 CPU. Xenomai was chosen to represent microkernel technology, and PREEMPT_RT to represent in-kernel technology. The response time to an external signal generated at a frequency of 10 kHz was measured. The OSADL Latency Box was used to perform the measurements.https:\/\/www.osadl.org\/uploads\/media\/OSADL-Latency-Box.pdf<\/a>The response time for both the kernel and the application was measured. All measurements were performed under identical conditions and a CPU load of 100% (generated using the program "hackbench").<\/p>\n Figure 4 (see\u00a0PDF<\/a>) shows the response time to the event in the kernel.<\/p>\n Figure 5 (see\u00a0PDF<\/a>This represents the latency for an application waiting for the event. The worst-case scenario is just under 100 microseconds.<\/p>\n Figures 6 and 7 (see\u00a0PDF<\/a>The response times in the kernel under PREEMPT_RT were shown. Isolating a core can improve the worst-case scenario somewhat. In the measurements for the application, PREEMPT_RT performs slightly better than Xenomai. The worst-case scenario is just over 90 microseconds (see Figure 8).,\u00a0PDF<\/a>). By isolating a core, the result can be improved to 80 microseconds (see Figure 9,\u00a0PDF<\/a>).<\/p>\n Comparing response times within the kernel isn't entirely fair, because with Xenomai we're talking about a microkernel, not the Linux kernel. Strictly speaking, we're not executing the code in the Linux context (as is the case with Xenomai). Therefore, another measurement was performed for PREEMPT_RT: processing the critical code path in the FIQ context (which many Arm-based SoCs offer). The FIQ operates in its own "world," but it's possible to register an FIQ handler from within Linux.<\/p>\n Figure 10 (see\u00a0PDF<\/a>) shows the results of an FIQ-based solution (based on a PREEMPT_RT kernel).<\/p>\n The worst-case scenario was improved to 30 microseconds. This single outlier is most likely due to a hardware problem. On other platforms, response times of less than 10 \u00b5s were achieved using this approach!<\/p>\n With the right extension, Linux possesses excellent real-time capabilities. Due to its widespread acceptance within the developer community and its ease of use, the so-called PREEMPT_RT approach has become the standard. The latency of this in-kernel approach is comparable at the application level to that of microkernels (such as Xenomai). Microkernels can only achieve better latencies within the kernel itself, but it's important to remember that this isn't within the Linux context, but rather within the microkernel (and therefore subject to its limitations and API). Those willing to accept limitations for better latency can also use an FIQ solution with PREEMPT_RT on Arm-based systems. With this approach, latency times of less than 10 microseconds are sometimes achievable.<\/p>\n Overall, PREEMPT_RT offers the best trade-off between low latency and ease of use. Furthermore, it's worth noting that organizations like OSADL continuously monitor the quality of the PREEMPT_RT approach. Another positive development is the Linux Foundation's RTL project, which will fund and drive its integration into the Linux mainline kernel over the next few years. These factors provide users with greater planning certainty for future projects and the maintenance of existing products.<\/p>\n Download the article as a PDF<\/strong><\/a><\/p>\n Do you want to bring yourself up to date with the latest technology?<\/strong><\/p>\n Then find out more\u00a0here<\/strong>\u00a0<\/a>MircoConsult offers training courses\/seminars\/workshops and individual coaching on the topic of Open Source \/ Embedded Software Engineering.<\/p>\n Training & coaching on the other topics in our portfolio can be found here.\u00a0here<\/a>.<\/strong><\/p>\n Valuable expertise in the field of Open Source \/ Embedded Software Engineering is available.\u00a0here\u00a0<\/strong><\/a>Available for you to download free of charge.<\/p>\nAvailable technologies<\/h2>\n
\n
RTAI<\/h2>\n
Xenomai<\/h2>\n
The Realtime Preemption Patch (PREEMPT_RT)<\/h2>\n
\n
Evaluation of the different approaches<\/h2>\n
Xenomai Results<\/h2>\n
Results PREEMPT_RT<\/h2>\n
Conclusion<\/h2>\n
\nOpen Source \u2013 our training & coaching<\/h2>\n
\nOpen Source \u2013 Expertise<\/h2>\n