Authors: Frank Erdrich and Jan von Wiarda, emtrion<\/p>\n
Virtualization offers a solution for running multiple software instances, such as operating systems, on a single system. The individual systems run independently of each other, unaware that other instances are running on the same physical system. This separation provides a degree of security, as a compromised instance, for example, cannot access the resources, such as main memory, of other instances.<\/strong><\/p>\n In industrial environments with embedded systems, such separation was typically achieved using separate computing systems with dedicated communication channels. For example, two processors are mounted on a control unit, communicating with each other via a defined protocol such as Ethernet or SPI. This allows one system to handle communication with the outside world, perhaps in the form of a human-machine interface (HMI), while the second system runs safety-critical software that must adhere to real-time requirements.<\/p>\n With the development of increasingly powerful multi-core processors, whose low power consumption and thus heat generation make them suitable for embedded systems, virtualization has become a viable option for industrial devices. This allows for the replacement of expensive multi-processor solutions with more affordable single-processor systems that offer multiple cores. However, certain prerequisites must be met for a processor to be used in virtualization projects. The following sections will describe the requirements for system virtualization, the different types of virtualization available, and how a solution for an embedded system might look. The focus will be specifically on the ARM-based System-on-Chip (SoC) processors widely used in embedded systems.<\/p>\n Virtualization can be divided into various types, each with different approaches and varying processor support requirements. This section focuses on those types of virtualization that are practical to run on an embedded system. This perspective may change with the advent of more powerful processors.<\/p>\n The important virtualization methods here are partitioning and virtualization using a Type 1 or Type 2 hypervisor. Virtualization through emulation (QEMU) and application virtualization (Java VM) are not considered further here.<\/p>\n Partitioning assigns the resources available in the system to individual instances. An instance can only use the resources it has been allocated. Resource allocation is handled by an administrative instance and can be created and modified by the user, for example, via a descriptive configuration file.<\/p>\n A Type 2 hypervisor is a fully virtualized system. This involves creating a complete virtual system, which, for example, also includes the BIOS on x86-based platforms. Unlike QEMU, where execution within a virtual machine is emulated, the system runs directly on the hardware. The key feature of a Type 2 hypervisor is that it provides the guest operating system with virtual resources such as virtual CPUs, main memory, hard drives, network cards, etc. This distinguishes it from Type 1 hypervisors. A hypervisor is also known as a virtual machine monitor.<\/p>\n Furthermore, a Type 2 hypervisor runs as an application within the host operating system. It distributes the computer's physical hardware resources to the guest systems. The hypervisor may act as an emulator, particularly when certain hardware components, such as a physical network card, cannot be accessed simultaneously. While emulation avoids driver issues, it does result in a certain performance loss.<\/p>\n The principle of full virtualization is based on different privilege levels of a processor and is architecture-dependent. These levels differ in the extent of access rights to CPU registers, main memory areas, and hardware. The hypervisor resides at the level with the highest privileges. Guest operating systems run at lower privilege levels. If a guest attempts to perform higher-privileged operating system operations, the hypervisor intercepts these as exceptions, evaluates them, and handles them accordingly. A guest can only access the hardware and shared system resources through the hypervisor. Well-known examples of software solutions in the field of full virtualization include VMware Workstation, KVM, and VirtualBox.<\/p>\n The advantage of full virtualization is that no changes to the host and guest operating systems are required. Access to system resources is intercepted and managed by the hypervisor and then passed on to the host system. This makes guest execution considerably more performant compared to native execution. Switching privilege levels does result in some performance loss, as each change of level, similar to task switching in a real-time operating system, triggers a context switch. A context switch necessitates saving and restoring various CPU registers. When a privileged instruction is requested by the guest, the hypervisor must provide a wrapper function that forwards the actual execution to the host's kernel API.<\/p>\n Unlike full virtualization with Type 2 hypervisors, paravirtualization with Type 1 hypervisors does not virtualize or emulate any hardware. Instead, the Type 1 hypervisor runs directly on the hardware at the highest privilege level. For this reason, this type of hypervisor is also called a bare-metal hypervisor. Guest operating systems do not access emulated hardware, but rather an API provided by the hypervisor. Physical resources are accessed solely through this API.<\/p>\n The hypervisor acts as a kind of operating system, distributing system resources among the guest operating systems. This allows different operating systems to run separately on the same hardware. In the case of the Xen hypervisor, a host operating system also runs at the second-highest privilege level, preventing its kernel from performing operations that require the highest privilege level. To achieve this, the host kernel communicates with the hypervisor via hypercalls. This is similar to the communication between kernel and user space in operating systems using system calls.<\/p>\n A Type 1 hypervisor requires modifications to the host kernel to support the aforementioned hypercalls. The hypervisor intercepts all system calls from guests and forwards them to the host kernel. The host kernel must then convert each system call into a hypercall and forward it back to the hypervisor, which then executes the actual operation on the hardware. Well-known Type 1 hypervisor systems include VMware ESXi and Xen.<\/p>\n The 32-bit ARMv7 architecture is currently the most widely used architecture in the field of embedded Linux. Alongside this, the 64-bit ARMv8 architecture exists, and its adoption is increasing due to current SoCs from various manufacturers. However, not every SoC with an ARM core is suitable for virtualization. To support virtualization, the ARM Virtualization Extensions and the ARM Security Extensions must be available. These are found, for example, in cores of the Cortex-A7, Cortex-A15, or Cortex-A53 type. The following section attempts to provide a basic overview of virtualization on ARM cores and the associated subsystems of the core. Due to the scope and complexity of the topic, the information is presented only in a rudimentary and sometimes incomplete manner and should preferably be consulted in conjunction with the ARM documentation.ARM Architecture Reference Manual)<\/em>.<\/p>\n The Security Extensions introduce a new CPU mode, the Monitor Mode. Additionally, two security states, the Non-Secure State and the Secure State, are added to the architecture. These two states are called Normal World and Secure World. The CPU is always in one of these two states. Switching to Secure Monitor Mode is done via the new SMC instruction or a hardware exception (IRQ, FIQ, etc.). This mode is later used by the hypervisor to release the CPUs again via PSCI after it has been shut down. Coexistence between the hypervisor and the Secure World is possible without any issues. Among other things, the Security Extensions offer enhanced hardware security features that enable the implementation of DRM, secure payment, and other software solutions. A well-known solution for TrustZone is, for example, OP-TEE.https:\/\/wiki.linaro.org\/WorkingGroups\/Security\/OP-TEE<\/a>).<\/p>\n The Virtualization Extensions also introduce a new CPU mode, the so-called Hyp Mode. The hypervisor runs in this mode. Furthermore, numerous additional registers and hardware functions are introduced, such as the extension of the MMU, the GIC interrupt controller, and other components to include specific virtualization functions. In particular, the virtualization of the non-secure state of the ARM VMSAv7 implementation is supported. Additionally, the new HVC instruction is introduced, which allows virtual machines to communicate with the hypervisor. If an architecture is to support the Virtualization Extensions, the Security Extensions, the Large Physical Address Extension (LPAE), and the Multiprocessing Extensions must be implemented. According to the ARM definition, a virtualized system essentially comprises:<\/p>\n A key advantage of Virtualization Extensions is the introduction of hardware mechanisms that prevent the hypervisor from having to intervene too frequently via software during the execution of guest operating systems. This would massively and negatively impact performance. Thanks to these hardware mechanisms, the hypervisor doesn't have to handle the following tasks, but can delegate them to the guest operating systems:<\/p>\n Unlike the classic translation of virtual memory addresses to physical memory addresses using page tables, virtualization environments involve not just one, but several translation phases. Virtualization environments (VEs) offer so-called translation regimes for memory accesses from the non-secure PL0 and PL1 modes to the non-secure PL2 mode. In the non-secure PL0 and PL1 translation regime, address translation occurs in two phases, called stages. Stage 1 maps the virtual address (VA) to an intermediate physical address (IPA). Typically, the guest VM configures and controls this stage and believes the IPA to be the physical address (PA). Stage 2 then maps the IPA to a PA. This stage is controlled by the hypervisor with the support of the MMU (Module Management Unit), transparently to the guest VM.<\/p>\n Interrupts, however, are not initially assigned to a specific recipient. Such a recipient could be the currently running VM, a VM that is not currently running, or even the hypervisor itself. The Virtual Interrupt (VE) approach stipulates that physical interrupts are first sent to the hypervisor. If the interrupt belongs to a VM, the hypervisor maps a virtual interrupt to that VM. To support this functionality, the interrupt controller needs to be extended. More precisely, VE provides special registers to manage virtual interrupts. There are physical and virtual registers. The hypervisor accesses the physical registers, while the VMs access the virtual ones. The virtual registers are remapped by the hypervisor so that the VM thinks it is accessing the physical registers. The interrupt mask bits I, A, and F in the CPSR belong exclusively to the currently running VM and therefore no longer need to be trapped. Virtual interrupts are forwarded directly to the respective CPU mode (FIQ, IRQ, Abort). The VM manipulates the virtualized interrupt controller. A virtual GIC interface was implemented for this purpose. The VM's Interrupt Service Routine (ISR) communicates with the virtual GIC. Lists containing pending and active interrupts exist for each VM. The virtual GIC communicates with the physical GIC. The functionality of the VM remains unchanged by the virtual GIC. The hypervisor can set IRQs in the HCR register as virtual. Interrupts are configured to trigger a hypervisor trap. The hypervisor can forward an interrupt to the CPU currently assigned to the VM via register lists.<\/p>\n ARMv7 uses memory-mapped I\/O for communication with peripheral devices, meaning the CPU communicates with the devices via device registers mapped to main memory. The MMU (Memory Management Unit) sits between the CPU and the I\/O devices, managing access. The device register addresses are physical memory addresses, meaning the MMU doesn't need to perform any translation but passes the addresses directly. Only access control is important, as this is where the hypervisor comes in to separate the guest systems.<\/p>\n The Jailhouse software project is led by Siemens AG and is implemented as an open-source project under the GPLv2 license. The source code is available on GitHub. There is a master branch and a next branch. In addition to Siemens, ARM and Huawei are also involved in the development. Jailhouse was launched in 2013 and supports ARM as well as x86 and x86-64 architectures.<\/p>\n Jailhouse is a partitioning hypervisor used to implement Linux-based asymmetric multiprocessing (AMP) systems. This means that, in addition to a standard Linux kernel, various other operating systems or bare-metal applications can run on the same hardware, the same SoC, while being kept as strictly separated as possible. The Jailhouse hypervisor ensures isolation between the so-called cells, utilizing the hardware support of modern CPU cores. Each cell runs a guest system with full control over specific system resources, such as CPUs, main memory, and I\/O devices. The hypervisor's sole function is to maintain the strict separation of the cells. The host system, based on a standard Linux kernel, is called the root cell. All other running systems are called non-root cells. The root cell is responsible for starting and controlling the hypervisor and other cells. However, it does not have complete control over all system resources; resources such as CPUs or memory are taken from it when additional cells are created. Jailhouse does not emulate resources that are not present; it simply distributes the available hardware among the individual cells. Therefore, no scheduling takes place between the individual guests. The software running within a non-root cell, be it a Linux kernel, FreeRTOS, or a bare-metal application, is referred to by the developers as an inmate.<\/p>\n A key motivation for developing a partitioning hypervisor is the need for virtualization solutions that enable real-time capabilities. Real-time capability, along with safety and security, are intended to be the defining characteristics of Jailhouse. This includes the goal of maintaining the smallest possible codebase, both to minimize complexity and facilitate understanding of the overall system, and to maintain a high level of code quality. The actual hypervisor comprises only about 12,600 lines of code (LOC), whereas Xen, with nearly 200,000 LOC, is considerably more complex. Of course, Jailhouse cannot be equated with Xen's feature set, but that is not the intention behind the project. A focus is placed on code quality. Since the beginning of development, the codebase has been regularly checked for errors using the static code analysis tool Coverity Scan. In the latest version of the source code, Coverity determined an error density of 0.00. This does not, of course, mean that the software is error-free, but it does indicate very good code quality. For comparison, Xen was found to have an error density of 0.56. A key feature of the hypervisor is that, unlike Xen, it doesn't need to be loaded as a BLOB by the bootloader, nor does it use the Linux kernel itself as a hypervisor, as with KVM. Instead, it can be loaded and configured within an unmodified Linux system. Jailhouse's focus is not on replacing KVM or Xen in the desktop environment, but rather on implementing systems that include critical applications in the area of functional safety. For example, a system could be implemented where a GUI runs under Qt in one cell, while on the same SoC, in another cell, a time- and safety-critical machine control system is executed. Naturally, these processes must not interfere with each other under any circumstances. The Jailhouse hypervisor is designed to ensure this.<\/p>\n A prerequisite for using Jailhouse on ARM-based systems is the presence of the ARMv7 Virtualization Extensions. Furthermore, the SoC must have at least two CPU cores, as partitioning takes place, requiring at least one CPU core to be allocated per cell. The bootloader, in our case U-Boot, must also support use in virtualization environments, meaning it must boot Linux in HYP mode. Additionally, PSCI support is required, enabling offline switching of CPUs.<\/p>\n The hypervisor itself is a BLOB of approximately 30 kilobytes. Its small size reflects the aforementioned intention of maintaining a lean codebase. This BLOB can be copied as firmware into a dedicated RAM area, which must be reserved during the Linux system boot process, using the jailhouse.ko driver module via the kernel API. The kernel parameters mem=958M and vmalloc=512M inform the kernel that the full 1 GiB address space should not be used, but rather that space in the upper range should be reserved for the hypervisor. The driver module also creates a device node, \/dev\/jailhouse, which serves as the interface for the userspace program \/usr\/local\/sbin\/jailhouse. The driver itself contains no hypervisor logic; it merely loads the hypervisor. Various actions, such as starting and stopping a non-root cell, can be performed via the userspace program.<\/p>\n Once the driver is loaded via `modprobe jailhouse` and the hypervisor is activated via the userspace application, it positions itself between the actual hardware and the root cell. The binary file `bananapi.cell` is a configuration file written in C, consisting of several nested structs that describe the respective hardware. It specifies the physical address at which the hypervisor runs, the size of its memory area, and the number of CPUs the hardware comprises. Additionally, the memory-mapped I\/O address of a UART debug console, through which the hypervisor logs output, is entered. Furthermore, all memory-mapped I\/O regions of peripheral devices, such as SPI, MMC, USB, SATA, etc., are entered. The registers of the respective controllers are mapped to these areas, and the hypervisor needs this information to monitor access to the memory areas. The hypervisor always runs in non-secure PL2 hyper mode.<\/p>\n In this CPU mode, the hypervisor has access to important registers used for management. Even after the hypervisor is activated, the root cell initially retains all CPUs for its own use until other cells are started. The hypervisor already monitors every memory access with hardware support and receives all hardware interrupt requests to forward them to the root cell. For interrupt handling, the hypervisor also utilizes the hardware support of the ARM GICv2 interrupt controller. If an exception occurs during the execution of a root or non-root cell, the hypervisor calls a specific exception handler, depending on the current CPU mode, to handle the exception. From this point on, the hypervisor takes over CPU control. The exception handlers are accessed via fixed vector addresses listed in the `hyp_vectors` table.<\/p>\n Depending on the type of exception, the hypervisor takes appropriate action. For example, if a cell accesses a memory address for which it does not have permission, the `arch_dump_exit()` function is called, which ultimately parks the CPU of the affected cell. This means the cell is no longer operational. The other cells are not affected and continue to execute. An unauthorized memory access is referred to as an abort, which is generated by the MMU, i.e., again with hardware support.<\/p>\n Furthermore, Jailhouse distinguishes between exceptions and traps. Exceptions are exceptions handled by interrupts or undefined instructions. Strictly speaking, in the Jailhouse context, there are only two true exception handlers: one for physical interrupts and one for software-generated interrupts. All other exceptions are handled by trap handlers, which are called by other exception handlers. Traps can occur due to accesses to coprocessor registers. Hypervisor Calls (HVCs) can be triggered by cells to communicate with the hypervisor or to request specific functionalities from it. When HVC instructions occur in cells, they are handled by the `arch_handle_hvc()` function. Secure Monitor Calls (SMCs) are also trapped by the hypervisor. These are instructions that can occur in CPUs with security extensions. As described above, these ARM CPUs can be in two different security states: the secure state and the non-secure state. The security state can be changed using the SMC instruction. The hypervisor catches this case. Another trap is triggered, as described above, in the event of unauthorized memory access (data abort).<\/p>\n The first inmate used was a FreeRTOS application, after a modification of a semi-finished solution 2 on GitHub had been carried out in collaboration with the developer. Building on this solution, a bare-metal application was then run as an inmate, which periodically switched the LED on and off.<\/p>\n The second inmate was intended to be a Linux kernel, which proved challenging. First, it was necessary to determine where and how the kernel zImage should be loaded into memory and how it could then be executed without a bootloader. Studying U-Boot code and the ARM boot protocol in the official kernel documentation proved helpful in this regard.<\/p>\n A legitimate question regarding real-time capability concerns the shared system resources of a SoC and how the execution of cell A affects another cell B. When considering CPUs, this primarily concerns the external interfaces to buses and caches. Specifically, the Bus Interface Unit (BIU), the Level 2 cache system including the Snoop Control Unit (SCU) and Generic Interrupt Controller (GIC), as well as the connection to the AMBA 4 bus via the AXI Coherency Extensions (ACE), are shared among the CPU cores. This demonstrates that, on current hardware, interference between different cells cannot be completely avoided, even with the partitioning approach of Jailhouse.<\/p>\n However, partitioning prevents CPU scheduling and the associated context switching. Furthermore, benchmark tests by the Jailhouse Project have shown that the bottlenecks, which are unavoidable anyway, do not have a significant impact on real-time performance. In any case, there is no more efficient approach with the available hardware.<\/p>\n To compare different systems with regard to hypervisor overhead, a benchmark based on the cpuburn application was created. Xen and Jailhouse were examined. This demonstrated that Jailhouse's overhead is minimal and that it utilizes the hardware optimally. Compared to a Linux kernel without a hypervisor, the overhead for Jailhouse amounted to 0.04 %, while for Xen it was 5.71 %. This is likely primarily due to Xen's CPU scheduling.<\/p>\n With Jailhouse, Jan Kiszka from Siemens provides a stable and fast, real-time capable way to partition a multi-core SoC in such a way that multiple guest systems can run in parallel without interfering with each other. This offers a way to run or virtualize security applications alongside insecure software components on a single SoC, thereby reducing hardware requirements and associated costs. Understandably, it is essential to verify for each project whether the performance losses resulting from virtualization, however small, affect real-time capability.<\/p>\n Download the article as a PDF<\/strong><\/a><\/p>\n Do you want to bring yourself up to date with the latest technology?<\/strong><\/p>\n Then find out more\u00a0here<\/strong>\u00a0<\/a>MircoConsult offers training courses\/seminars\/workshops and individual coaching on the topic of implementation\/embedded and real-time software development.<\/p>\n Training & coaching on the other topics in our portfolio can be found here.\u00a0here<\/a>.<\/strong><\/p>\n Valuable expertise in the field of implementation\/embedded and real-time software development is available.\u00a0here\u00a0<\/strong><\/a>Available for you to download free of charge.<\/p>\nVirtualization types<\/h2>\n
Partitioning<\/h2>\n
Type-2 hypervisor<\/h2>\n
Type-1 Hypervisor<\/h2>\n
Virtualization on ARM<\/h2>\n
ARM Security Extensions (TrustZone)<\/h2>\n
ARM Virtualization Extensions<\/h2>\n
\n
\n
Memory, IRQ and I\/O<\/h2>\n
The Jailhouse Hypervisor<\/h2>\n
concept<\/h2>\n
motivation<\/h2>\n
System requirements<\/h2>\n
Loading and activating the hypervisor<\/h2>\n
How the hypervisor works<\/h2>\n
Successfully tested inmates<\/h2>\n
Shared resources and benchmarks<\/h2>\n
Conclusion<\/h2>\n
\nImplementation \u2013 our training & coaching<\/h2>\n
\nImplementation \u2013 Expertise<\/h2>\n