EN 
DE A high-performance approach to becoming independent of the end device.
Author: Frank Borchard, XiSys Software GmbH
Contribution – Embedded Software Engineering Congress 2017
The demands placed on the user interfaces of embedded systems are increasing with the introduction of IoT and Industry 4.0. Existing local user interfaces should be able to be displayed as easily and effortlessly as possible on virtually any conceivable device, from smartphones to laptops. To achieve this, the user interface is typically designed as a web application, thus shifting the visualization of the interface logic to the web browser.
Embedded systems in industrial environments often require direct operation at the machine and therefore utilize a stationary user interface. Currently, such requirements are most effectively met by a traditional graphics solution based on a local graphics server and local graphics hardware. However, with the advent of IoT and Industry 4.0, the demand for web interfaces or remote access capabilities via browser-based solutions is becoming increasingly common. The introduction of HTML5 and the ever-improving graphics capabilities now make it possible to fulfill this demand. However, it's important to be aware that web-based solutions represent a significant increase in technical complexity and are also more difficult to secure against external attacks. If a local browser-based solution is to be implemented, the hardware requirements increase considerably and are often no longer commercially viable in industrial environments.
Remote desktop with VNC, RDP, ...
The most common method for forwarding local graphics is the use of remote desktop software. Solutions such as VNC (Virtual Network Computing), RDP (Remote Desktop Protocol), etc., are available for various operating systems. A corresponding service must run on the embedded system, which monitors the contents of the video memory or framebuffer for changes and forwards these to the visualization device using a suitable protocol. A corresponding receiver must always be installed on the display device, which visualizes the data either in a separate window or in a browser. Since neither the location nor the time of changes can be precisely predicted, a relatively high level of effort is required to detect the changes. Smaller systems quickly reach their limits in this regard.
Remote via HTML5
With the capabilities of HTML5, web-based remote visualization can also be implemented. This takes over the tasks of remote desktop software and can also function as a pure web visualization. The application logic always remains on the embedded system. Depending on the use case, the processing power for rendering then shifts from the embedded system to the display system. After a successful connection, the browser receives a set of rules in JavaScript, which interprets commands sent by the web server at runtime. The browser thus gains the basic functionality of an external graphics card. This concept requires no apps, plug-ins, or cookies on the display side.
WebSockets vs. HTTP, or why HTML4 is not a suitable platform
With HTML4, only HTTP-based communication was possible. This meant, as is well known, that a stateless connection had to be simulated using various "hacks" like AJAX. WebSockets enable a permanent full-duplex connection with virtually no overhead compared to HTTP. These protocol characteristics also result in extremely low latency in the millisecond range, thus opening the door to near-real-time user interface behavior. WebSockets-based communication allows for near real-time animation of graphical objects.
concept
With a focus on achieving the highest possible visual performance, unnecessary communication overhead between the graphics server and web server must be avoided. The web server must gain access to the change information as quickly and efficiently as possible. The most effective way to achieve this is to implement the web server as an integral part of the graphics server. In this case, changes can be forwarded from the graphics server to the display device without any significant time loss.
Further efficiency gains are achieved by structuring the website within the browser. As long as only rasterized image information from the video memory is forwarded to the display device, a single canvas element suffices for the entire display. Scrolling, moving, or deleting objects in this case incurs very high data traffic. Efficiency increases significantly as soon as a separate element is created for each object. The graphics server then sends—depending on the requirements—rasterized images or scalable vectors in the form of SVGs that describe the respective object. This shifts more of the drawing workload to the browser. As a welcome side effect, the amount of data to be transmitted is reduced. The browser assembles the overall image of the display or application from the superimposed elements. To move an object, for example, only the corresponding coordinate of the respective element in the DOM tree needs to be adjusted.
(see. PDF, illustration)
Hybrid graphics server
XiSys Software has developed a graphics server that not only features an integrated window and object manager, but also, in its latest version, incorporates web server functionality. The object manager allows the server to build and manage a hierarchically structured object tree. As soon as a user connects via a WebSocket connection, the complete object hierarchy of all objects displayed on the screen is mapped as a DOM tree in the web browser. Depending on the object's properties, one or more canvas or SVG elements are available for each object within the DOM tree. Individual event handlers are automatically assigned to each element, best supporting the object's event properties. For example, when an input element is touched, a soft keyboard is automatically displayed, the element is moved into the visible area of the screen if necessary, and the keyboard input is forwarded to the web server.
If no graphics hardware is available on the local system, or if only an application needs to be visualized in the browser, the browser can request an asynchronous virtual visualization. In this case, a clone of the hybrid graphics server is created. Instead of drawing an object locally, an SVG element is derived from the vector list and sent to the browser. The browser then handles all rendering tasks. The embedded system can thus limit itself to purely administrative tasks – which, of course, greatly reduces resource consumption.
Performance aspects
To achieve high user acceptance, the user interface should respond to input without delay. This isn't a problem with static images. Complex animated graphics, however, pose a challenge. To ensure smooth animations and simultaneously fast responses to user input, the primary focus must be on achieving the highest possible latency-free data throughput. Latency caused by long signal propagation times, poor connections, or overloaded display systems must be avoided at all costs. Therefore, neither the server nor the browser should have to wait for an acknowledgment of a command. As long as the network connection is fast enough and the display system can process incoming commands faster than new ones arrive, everything runs smoothly. Experience shows that performance issues can sometimes occur, particularly with mobile devices. These manifest primarily as slow response times. To counteract this, it's necessary to implement an algorithm in the web server that detects the total delay time to be expected due to a backlog of commands. The processing speed of the graphics server must then be adjusted based on this result. This algorithm is also used to synchronize multiple parallel display devices.
Implementation effort
A compelling advantage of the hybrid graphics server is the reduced effort required for developing mobile applications. The user interface only needs to be developed once; the graphics server automatically handles the adaptation to different end devices. There is no need to create a parallel infrastructure with the associated development tasks.
All advantages at a glance
- No app, plug-ins, or cookies are required for visualization in a browser. The visualization leaves no trace.
- Visualization of multiple systems in iFrames is possible.
- Because changes are predictable and containable, the system being controlled is only minimally burdened.
- The integrated web server can access already rendered data virtually without any time loss and distribute it immediately.
- Organizing a DOM tree for all objects allows the browser to assemble screen layouts. Removing, moving, and scrolling objects is handled by the browser.
- Using SVGs eliminates the need for the embedded system to render objects.
- Asynchronous virtual sessions can be started, which, for example, only allow access to one application.
- The programmer doesn't need to worry about browser incompatibilities. Using a GUI builder, they only need to design their screen layouts once. These are then displayed identically both locally and in the browser.
Conclusion
The presented concept is compelling due to its flexibility. It allows for the implementation of tasks ranging from a local GUI and remote desktop software to visualizations that only support web-based displays. The GUI is identical across all scenarios. The programmer never needs to familiarize themselves with the technologies of different display systems.
author
Frank Borchard studied computer science at the Würzburg University of Applied Sciences. After graduating, he worked in the telecommunications industry at Alcatel-Lucent. Since 2010, he has worked as a freelance IT trainer and consultant in the network and telecommunications sector, as a software design consultant specializing in IT security, and as a freelance employee at XiSys Software GmbH.
IoT / Industry 4.0 – our training courses & coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of IoT/Industry 4.0/system and hardware development.
Training & coaching on the other topics in our portfolio can be found here. here.
IoT / Industry 4.0 – Expertise
Valuable expertise in IoT/Industry 4.0/system and hardware development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
