EN 
DE Author: Manuel Melic, TARA Systems GmbH
Contribution – Embedded Software Engineering Congress 2015
demolition
Balancing low-end hardware with high-end requirements and ensuring future-proof UI software despite short product launch times presents embedded GUI developers with a considerable challenge, especially when they are implementing a visually appealing user interface for the first time.
This paper presents principles that are important for achieving an optimal user experience on embedded platforms: What needs to be considered when GUI development is to be cost-efficient yet flexible enough to quickly respond to changing requirements? How can the UI application be designed to be extensible and scalable so that it can continue to be used in subsequent product generations with new features and new hardware?
The best practices presented in this article are based on over 20 years of professional experience in embedded GUI development.
Main part
Software engineers for embedded systems often stumble into GUI development quite unexpectedly: They need to quickly implement a prototype for the next trade fair, the next (important) customer meeting, or the presentation of a new product idea – a prototype that is more attractive, faster, and easier to use than the current product – and one that doesn't have to hide behind the familiar design and operation of an iPhone or iPad. But the newly appointed HMI developer is inexperienced – it's their first UI project.
In addition to the chronic lack of time and the new challenge ("We're now implementing a graphical user interface"), further questions arise: What hardware will be used? Is newly designed hardware already available, or should the prototype be implemented with existing hardware? Is an evaluation board perhaps the best option after all? What differences can the developer expect later? What performance does the hardware offer for a GUI? What software solutions and support does the chip manufacturer provide? What software solutions are even available? Is there a GUI specification, and if so, what is its current status? What will the look and feel be like, and what should the new application, and thus the new UI, actually accomplish?
Once an overview has been gradually established, there's a pressing need to begin implementation quickly. This means that the necessary time for conceptual work, proof-of-concepts, testing, and the actual implementation of all features isn't available in order to meet the deadline. This is perfectly legitimate, however – it's only a prototype, after all. To reach the goal quickly, it's common practice to use familiar hardware and the corresponding software libraries or GUI tools from the chip manufacturer. Finally, the milestone is reached with great difficulty: the prototype is finished on time, the GUI tool supports development, and management is (initially) satisfied. The trade show, customer visit, or presentation can now take place.
But what happens next? The next evolutionary step is undoubtedly the further development of the prototype into a product. To ensure a good final result, various aspects, as well as lessons learned during the prototype's development, must be considered from a GUI perspective. Specifically, this means overcoming the hurdle of optimal integration with the target hardware. Naturally, the actual scope of functionality must also be clearly defined. The design to be implemented and the expected performance also play a crucial role. If additional developers join the project later on, organizational issues should not be overlooked.
Consequently, numerous problems can arise during development in the five aforementioned subject areas. A few examples:
Hardware
- The hardware initially used is too expensive, too old, too slow, or does not offer the required functionality.
- Should a hardware change become necessary (during the project or with the next product version), it often happens that the software components used, which have too strong a dependency on the previously used hardware, cannot be reused. Consequently, a large part of the software would have to be redeveloped.
- It can also happen that the final hardware is not yet available when software development begins, as it still needs to be developed. However, it is essential to parallelize the development process in order to meet the often tight schedule.
- It may be necessary to supply not just one, but two or more platforms with the same software. For example, the sales representative wouldn't want to place the multi-ton machine on the table when visiting a potential customer – instead, a handy tablet or smartphone should demonstrate the machine's operation and functionality.
Platform-independent software components can prevent the aforementioned problems from the outset. But how can platform independence be achieved? In a tool-based development process, the tool itself should offer the possibility of developing GUIs independently of the eventual target platform. This can be solved with the help of an abstract intermediate layer.
Functionality
- It is obvious that the software version of the implemented prototype only represents a portion of the final product's functionality. If further functionality is to be incorporated that is only specified later in the project, the initially chosen architecture may result in software that is difficult to extend, maintain, and is overly complex.
- Due to time constraints, it's possible the prototype was only implemented as a mock-up or click dummy. Consequently, no connection to the hardware and/or middleware was established – but how exactly should this be implemented now?
- A similar problem becomes apparent to the developer sooner or later when locating the site of the implemented logic: Is there a clear separation between GUI and logic, or is most of the logic located within the graphical user interface?
- Of course, non-functional requirements can also pose obstacles in GUI development. For example, the product might be intended to tap into a new market, which requires an additional language extension. But what is the simplest way to achieve this with the existing software structure?
The aforementioned obstacles that software developers encounter during or after development can be effectively eliminated if sufficient time is allocated to conceptual work at the very beginning of a project. The architecture and structure of the software to be implemented should be clear before the actual implementation begins. This requires identifying and separating elements that can be logically distinct: the appearance, the business logic, the individual UI components (buttons, lists, menus, dialogs, etc.), string literals (for multilingual support), and so on.
Design and Interaction
- Developing good software is an art in itself, and art, as we know, stems from skill. And of course, good design is also an art in itself. Now, there are certainly good software developers who understand something about good design – but that's rather the exception. Consequently, UIs created by good software developers tend to look more like technical control panels – because others might be better at good design.
- Good design isn't defined by being "beautiful" or "appealing." Far more important is good usability. Everyone knows the term. Usability is crucial for making operation safe (i.e., risk-free), efficient (thus increasing productivity), and simple (to shorten training time).
- Finding a good look doesn't necessarily mean the feel will be good too. The transitions between different views – the animations – also need to be designed, implemented, and tested for performance. Only the combination of good design, usability, and smooth operation creates a truly positive user experience.
- Ultimately, management, sales, or the marketing department wants to get a feel for the finished product early in the development process. But how can non-developers quickly get an overview of the current development status without having to create yet another, sometimes costly, preliminary prototype?
Similar to software design, the design of graphical units and interactions—that is, user experience design—should also be approached in terms of components. This is the only way to effectively integrate the two often parallel development processes. While established development models like the waterfall or V-model could theoretically be used, this integration would occur too late in the process. Therefore, it is recommended to employ agile development models such as Scrum or Kanban for both the design and software projects. This allows for faster iterations with more frequent feedback loops between designers, software developers, and product owners. Consequently, the GUI tool used should also support these development methods. For example, this could mean providing the resulting GUI without dependencies on the target platform, directly from the current state of the software. This could be in the form of a PC or Mac application, or as a browser-based version.
Performance and resources
- The familiar usability of smartphones and tablets should of course also be applied to the new HMI to be implemented: It should look similar, behave similarly and, of course, respond just as quickly to user input.
- But how does the software engineer find out if the implemented GUI runs smoothly on the target platform? Because even in the prototype, some animations noticeably stuttered.
- Unlike PC development, embedded systems have limited available memory. Consequently, developers must carefully manage both available RAM and ROM memory from the outset.
To get a handle on performance problems, it's crucial to understand the entire chain. Only then can a potential bottleneck be correctly identified and located: Are the application's performance losses attributable to the GUI tool or library, the graphics driver, or is there perhaps even a hardware problem? It can be quite advantageous if the GUI tool is capable of handling different graphics APIs to help pinpoint the performance issue more precisely.
However, if the goal is to optimize flash memory usage, the GUI tool should offer the ability to include resources—such as images and fonts—both statically (at compile time) and dynamically (at runtime). To minimize RAM usage (after the fact), a simple solution is to reduce the resulting color gamut. Therefore, it is desirable for the GUI tool to support this process automatically.
organization
- It's quite possible that the HMI project underestimated the required developer resources at the outset. To still meet the launch date, it was decided to assign additional developers to the project. What does efficient onboarding look like in this situation?
- As the team grows, it becomes essential to manage and version the source code. Can the software components and tools in use support this?
Proper documentation within a software application helps to record its structure, behavior, and solutions across project boundaries. Even during the project, it can provide new team members with a solid foundation for quickly familiarizing themselves with the architecture. Therefore, the GUI tool should offer inline documentation capabilities for newly developed software, eliminating the need for additional descriptions created with other tools. Larger UI projects often establish processes that enable continuous integration and, consequently, continuous delivery to the customer. Therefore, GUI tools should also integrate with these automated build environments to support such development processes.
The causes and problems listed above can be addressed through targeted measures – using suitable GUI tools and libraries. The following advice is intended to further facilitate the implementation of GUIs for embedded systems:
1) Platform independence
As mentioned above, the GUI tool should be based on an intermediate layer. This layer abstracts the interfaces to the hardware, the operating system, the required resolution and color format, as well as the actual graphics subsystem. This allows for easy hardware replacement by simply porting the abstraction layer to the new platform.
2) Separation of middleware and GUI
A clear separation of business logic from the graphical user interface helps to better structure the business logic. This makes it possible to run parallel development. Furthermore, suitable APIs allow multiple different GUIs to access the same instance of business logic (middleware). This is particularly relevant when implementing companion screens, for example, to add a mobile display to the main, stationary control unit.
3) Leveraging known best practices: OOP with MVC and MVVM
Since GUI projects can quickly become quite complex, it's advisable to choose an object-oriented programming paradigm: Encapsulation, code reusability, and inheritance allow a GUI project to be broken down into individual parts, making the project more manageable. Furthermore, common development patterns—such as the Model-ViewController or Model-View-ViewModel patterns—can help create a clear structure within the project. Therefore, the GUI tool used should also natively support such design patterns.
4) Use asynchronous instead of synchronous APIs
Once the GUI is linked to the middleware, it often happens that animations no longer display smoothly after user input, or even freeze, and further input is no longer possible. This is frequently due to the fact that user input is synchronized with the processing within the middleware: The GUI waits until the middleware has processed the input and cannot perform any further actions – it blocks. To avoid this, all computationally intensive calls to the middleware should be executed asynchronously, i.e., without the GUI actively waiting. The results of the call are then fed back to the GUI at a later time. This is particularly essential for animations within the GUI.
5) User Experience
To achieve a good user experience—the combination of appealing design and good usability—comprehensive planning is essential. To ensure that individual views are easy to grasp and that touch-sensitive elements can be large enough, the user interface functions must be distributed across multiple views. This distribution is based on application scenarios and the logical relationship between functions, with each screen being clear and logically structured. The individual elements must then be weighted according to their importance. This is achieved through size, color, and shape.
Furthermore, the GUI must be internally consistent: The user interface learned in one part of the GUI must behave identically in all other similar areas. To achieve this, it is necessary to structure the GUI concept into components or templates. The latter is an important preliminary step for object-oriented programming. Ultimately, a keen sense of design and experience are required to create a harmonious and attractive whole from a combination of colors, lines, symbols, and fonts. Specialists exist for this area, possessing appropriate methods for requirements gathering, conceptualization, and design.
6) Instant prototyping
If the middleware is separated from the actual UI and uses an abstraction layer to the target platform, prototyping can be achieved with minimal effort. It's helpful to use the host system (for example, a Windows PC) as the prototyping platform. This allows for the rapid deployment of preliminary versions of the user interface to non-developers such as designers, product managers, or marketing staff. Changes to the GUI can then be communicated and implemented more quickly.
7) RAM optimization
The memory consumption of GUI applications is primarily determined by the chosen resolution, the color format used (True Color, High Color, Index 8, etc.), and the number and type of resources used, such as fonts or bitmaps. If the resolution cannot be reduced and the selection of fonts and bitmaps has already been optimized, reducing the color format is often the best way to significantly minimize memory consumption. Therefore, a GUI tool or library should offer the option to change the color format without significantly impacting the design.
8) ROM optimization
Similar to optimizing RAM usage, the size of the resulting application in flash memory is primarily determined by the resources used and the actual program code. Since significant changes to the program code that affect the ROM footprint are rarely possible with a good software architecture, it's worthwhile to optimize the resources used. A crucial factor here is whether these resources are used statically (at compile time) or dynamically (at runtime). For example, switching from static to dynamic usage can have a significant impact. If, for instance, only a few characters need to be displayed in a specific size, the GUI tool used should offer the option to embed only this selection statically in the program code. This avoids having to include the entire font and reduces memory consumption.
9) Performance increase
If the goal is to improve user interface performance rather than memory consumption, the actual drawing operations should be optimized first. Better GUI tools are therefore designed, for example, to render only the changing areas, so-called "dirty areas." This saves processing time and increases the number of frames per second (FPS). Another way to improve FPS with full-screen animations is to use intelligent caching: UI elements such as buttons, sliders, text views, etc., are frozen, duplicated, and combined into a single, unalterable image before the animation begins. The full-screen animation is then performed using this temporary image, which is subsequently removed from the system. This means the animation only needs to be executed on a single cached object instead of many smaller ones. This saves valuable processing time but requires additional memory.
10) Integration
During the project, there are usually several meetings where the newly developed software components are brought together and integrated on the target platform. If it turns out that certain animations or transitions are not displayed satisfactorily, it is advantageous to have a thorough understanding of the entire chain. Consequently, there should be no unknowns in the system: the developer should be familiar with everything from the application itself to the GUI tool or GUI library, right down to the graphical subsystem.
Anyone new to user interface development who finds themselves in the aforementioned situation can save considerable time, money, and frustration by following these best practices. What initially appears to be extra work—precise design, architecture, platform independence, etc.—pays off later, as it often avoids redundant development efforts.
Naturally, marketing and management expect the prototype to reflect the full range of functions of the device being operated, even in its first version. However, the scope of functions should be reduced, especially at the beginning, to allow for greater focus on software quality. The aspects mentioned can thus also serve as arguments for pursuing an effective, sensible, and pragmatic approach. This way, an HMI project can be successfully implemented in a user-friendly, timely, and sustainable manner.
References
Implementation – our training & coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of implementation/embedded and real-time software development.
Training & coaching on the other topics in our portfolio can be found here. here.
Implementation – Expertise
Valuable expertise in the field of implementation/embedded and real-time software development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
