EN 
DE Methods and implementations
Author: Lubosz Sarnecki, Collabora Ltd.
Contribution – Embedded Software Engineering Congress 2018
Although the field of virtual and augmented reality has existed since the 1990s, we have witnessed rapid, market-driven development in this interdisciplinary area in recent years. While historically dominated by proprietary software, it enjoys an enthusiastic community of open-source hackers and companies developing free drivers and middleware. Open standardization efforts are currently being undertaken by the Khronos group.
Explanation of the term XR
See Fig. 1 (PDF): Milgram's Reality-Virtuality Continuum, 1994
The term XR (Extended Reality) is often used to combine aspects of virtual and augmented reality. It is seen here as a synonym for Mixed Reality, which represents the middle part of the spectrum. Reality-Virtuality Continuum as described by Milgram. The continuum begins with complete real reality, passes through augmented reality, where virtual objects are inserted into reality, and ends in complete virtuality.
Tracking technologies
Besides rendering, tracking is a key component of XR systems. The following section presents some methods commonly used in consumer systems.
Inertial measuring unit
The IMU (Inertial Measurement Unit) is a combination of several inertial sensors and, due to its low cost, is used as the foundation of most tracking systems. The most important sensors are the accelerometer and the gyroscope. Many IMUs also include an additional magnetometer, which, however, is often neglected or omitted from implementations due to its susceptibility to interference from electromagnetic radiation. The use of IMUs in most mobile phones allows projects like Google Cardboard to utilize the smartphone as a complete VR solution, requiring only a cardboard box with plastic lenses as a holder.
To obtain a complete orientation from an IMU, several sensors must be combined.
One implementation for reading the IMU signal from most common consumer VR headsets is OpenHMD[4]. The USB devices are connected to the library hidapi OpenHMD also features sensor fusion and delivers a ready-made head pose via a C interface. OpenHMD is released under the Boost license and has an active developer community.
Although the IMU can in principle be used to determine a position in 3D space, common sensors provide data that is too noisy and drifts too much to be sufficient for this purpose. Therefore, IMUs are usually only used to determine rotation, while the following tracking systems also provide position.
External image-based tracking
A common method for determining a 6DOF (6 Degrees of Freedom) pose, i.e., a pose with 6 degrees of freedom, is external image-based tracking. There are two approaches to this.
In the first approach, the headset contains the actuators, and the external tracker is a camera. Examples of this approach are the Oculus DK2/CV1, PSVR, and OSVR HDK. These devices feature an LED grid on the HMD and controllers that emits infrared light. Blob detection allows the actuators, whose 3D arrangement on the headset is known, to be determined in the 2D camera image. Algorithms such as PnP (Point-N-Point), which are used, for example, in OpenCV Once implemented, the headset's pose in space can be inferred from the 2D positions in the camera image and their 3D arrangement. An open-source implementation of camera-based external tracking can be found in OSVR[12] and the GPLv2 licensed opening[1].
In the other approach, the headset contains the sensors and an external tracker handles the actuators. This is used, for example, in the Lighthouse tracking system of the HTC Vive and other headsets like StarVR.
The trackers emit a signal via two lasers mounted on rotors, similar to laser scanners. This signal is detected by simple light sensors on the headset and controllers. The factory-calibrated sensor array, which can be read by the headset, along with information about the precise moment each sensor was struck by a laser beam, allows for the reconstruction of a pose. Multiple base stations can synchronize using pulses from an LED array and photosensors, without requiring interaction with the tracking software. The HTC Vive's USB protocol was developed by Project LighthouseRedox[2] documented. An approach for an open-source implementation of Lighthouse tracking can be found in my experimental vive-libre[13] Drivers and the MIT-licensed software based on them libsurvive[3].
In both approaches, an increase in the robustness of the system can be achieved by increasing the number of trackers.
See Fig. 2 (PDFLighthouse tracking system
Inside-Out Tracking
Due to the lack of need for external trackers, inside-out tracking is frequently used in modern consumer HMDs. Here, the headset only requires at least one camera, which, using robotics-derived techniques like SLAM (Simultaneous Localization and Mapping), can determine the camera's position in space based on feature detection and map creation. It is the preferred tracking method in modern proprietary AR implementations such as Google's ARCore and Apple's ARKit. The smartphone's or HMD's IMU signal also contributes to accurate location tracking due to its low latency and high frequency. SLAM methods that integrate inertial sensors are called Visual Inertial, or VIO. Powerful open-source implementations include... ORB-SLAM2[6] (GPLv3) and Maplab[5] (Apache 2.0) from ETH Zurich.
For controller tracking, SLAM can be combined with external camera tracking methods, such as those used in Microsoft Mixed Reality headsets. The controller loses position tracking as soon as it leaves the sensors' field of view and reverts to its inertial measurement.
A list of numerous Open Source SLAM implementations can be viewed on OpenSLAM[18].
Hand tracking
With a simple device like the Leap Motion or uSens FINGO, which contains a wide-angle stereo infrared camera, solid hand tracking can be achieved using image processing and artificial intelligence. Unfortunately, it remained limited to open-source drivers such as... OpenLeap[7] only with minimal approaches to implementation. Depth cameras such as the Microsoft Kinect or Intel RealSense offer another alternative to body tracking. ROS provides implementations for hand tracking based on depth cameras[8].
Overview: Standardization approaches and APIs
OpenVR
A very widely used API for VR is Valve's OpenVR[10]. Although its implementation, SteamVR, is not open source, SteamVR offers a driver interface that enables the development or integration of open source tracking drivers. For example, the SteamVR-OpenHMD[9] Project a connection to all headsets supported by OpenHMD.
OSVR
The middleware developed by Sensics OSVR It implements external camera-based tracking for the HDK and HDK2 headsets, as well as a Kalman filter for filtering sensor data and determining pose. As an IPC and foundation of OSVR Core[12] will be vrpn[11] used, which also provides additional device drivers. OSVR It also features connections to proprietary drivers such as SteamVR and Oculus VR.
OpenXR
The Khronos group has published well-known industry standards for graphics interfaces such as OpenGL and Vulkan. They are currently working on the OpenXR API specification, which will enable vendor-independent applications.
See Fig. 3 (PDFAPI fragmentation before and after the use of OpenXR
Actions API
With classic input systems like SDL 1, the application queries the state of predefined keys. For example, the application can be closed when the keyboard's Escape key is pressed, or a jump action in a game can be executed when the second button on the game controller is pressed. This has the disadvantage that remapping keys requires specific implementation in the application or driver. Similarly, the application must know the controller at implementation time. This problem often leads to applications only supporting one controller, such as the Microsoft Xbox controller. Userspace drivers like... xboxdrv[16] help solve this problem by emulating an Xbox controller for various gamepads. However, the API solution is not to call specific buttons, but to define actions that can then be assigned to a button in the driver. Here, the above example becomes an exit and a jump action that are processed in the application. The user can change the button assignment or use predefined profiles. Since there is a high degree of heterogeneity and innovation in input methods in the field of virtual reality, an Actions API is a necessary solution. This allows poses of the head, controllers, or other tracked elements to be integrated and assigned as actions.
A proprietary implementation for an Actions API already exists in SteamVR, which is specified in OpenVR. OpenXR will also have an Actions API, as discussed in the presentation „Standardizing All the Realities: A Look at OpenXR“ was announced[15].
Direct Mode on the Open Source Graphics Stack
XR devices benefit from a special display mode called Direct Mode. In classic Extended Mode, the operating system treats the HMD as a normal monitor – the desktop is extended onto the HMD, but 2D applications displayed directly on the HMD are unusable. A VR application draws in a normal full-screen window displayed on the HMD. This also has the disadvantage that rendering occurs with relatively high latency, since modern window managers, for example, use compositing to display windows, which is not designed for low latency.
The open-source desktop, thanks to the recent introduction of Keith Packard's DRM leasing method, offers the possibility of completely excluding monitors from desktop operation. A list of monitors, i.e., HMDs, that are set to "non-desktop" via their EDID can be found in the Linux kernel. Numerous current components, such as the kernel, X server, and Mesa graphics drivers, are required to use this feature. A detailed list of patches was compiled by Christoph Haag [14].
One way to use DRM leases is through the Vulkan graphics API. This is made possible by Vulkan extensions. VK_EXT_direct_mode_display and VK_EXT_acquire_xlib_display Screen output in Direct Mode can be initialized using Xlib structures. An example of using Direct Mode with Vulkan can be found in my VR graphics demo. xrgears[17], which uses OpenHMD as a tracker.
List of abbreviations
| VR | Virtual Reality |
| AR | Augmented Reality |
| XR | Extended Reality |
| OpenVR | Valve's Open Virtual Reality API |
| OSVR | Sensics Open Source Virtual Reality Middleware |
| OVR | Oculus Virtual Reality API |
| OpenXR | Open Extended Reality Khronos Standard API |
| SLAM | Simultaneous Localization And Mapping |
| IMU | Inertial Measurement Unit |
| OpenCV | Open Computer Vision |
| 6DOF | 6 Degrees of Freedom |
| ROS | Robotic Operating System |
| PnP | Point-N-Point |
| VIO | Visual Inertial Odomentry |
| DRM | Direct Rendering Manager |
| IPC | Inter-process Communication |
| HMD | Head-Mounted Display |
List of illustrations
Fig 1: https://en.wikipedia.org/wiki/Reality–virtuality_continuum
Fig 2: HTC Vive PRE User Guide
Fig 3: OpenXR API Diagram https://www.khronos.org/openxr
Bibliography and list of sources
[1] https://github.com/pH5/ouvrt
[2] https://github.com/nairol/LighthouseRedox
[3] https://github.com/cnlohr/libsurvive
[4] https://github.com/OpenHMD/OpenHMD
[5] https://github.com/ethz-asl/maplab
[6] https://github.com/raulmur/ORB_SLAM2
[7] https://github.com/openleap/OpenLeap
[8] https://wiki.ros.org/hand_interaction
[9] https://github.com/ChristophHaag/SteamVR-OpenHMD
[10] https://github.com/ValveSoftware/openvr
[11] https://github.com/vrpn/vrpn
[12] https://github.com/OSVR/OSVR-Core
[13] https://github.com/lubosz/OSVR-Vive-Libre
[14] https://haagch.frickel.club/#!drmlease.md
[15] Nick Whiting, „Standardizing All the Realities: A Look at OpenXR“, GDC, March 2018
[16] https://gitlab.com/xboxdrv/xboxdrv
[17] https://gitlab.com/lubosz/xrgears
[18] https://openslam-org.github.io
author
Lubosz Sarnecki is a Senior Software Engineer at Collabora and a VR enthusiast since the Oculus DK1 days. He has been working on open-source software projects like GStreamer since 2011. After working on the GStreamer OpenGL plugins, Lubosz focused on graphics and VR. He is the author of the GStreamer VR plugins, numerous open-source VR demos, and is currently working on improving the open-source VR stack. Lubosz holds a B.Sc. in Computer Graphics from the University of Koblenz.
Open Source – our training courses & coaching sessions
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of Open-Source / Embedded Software Engineering.
Training & coaching on the other topics in our portfolio can be found here. here.
Open-source expertise
Valuable expertise in the field of Open-Source / Embedded Software Engineering is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
