Authors: Gunther Vogel, Robert Bosch GmbH, and Dr. Daniel Simon, Axivion GmbH<\/p>\n
Interrupts are frequently used to control embedded systems, and these require thorough planning and monitoring beyond a certain level of complexity. Simultaneously, multicore processors are becoming increasingly prevalent in otherwise resource-constrained environments, making concurrent programming a new paradigm for many (embedded) programmers ever more important. With the increasing concurrency of software, the risk of race conditions and, consequently, obscure and difficult-to-understand runtime errors in the systems also rises.<\/strong><\/p>\n This paper presents a methodological framework for combating race conditions through constructive measures in design and coding. To ensure that the product is free of race conditions upon delivery, measures supported by analytical tools should also be employed. The constructive and analytical measures are explained, and the paper demonstrates how the results can be efficiently managed throughout a development project.<\/p>\n This article is aimed at development managers and developers of embedded systems in C\/C++ who work with interrupts or multicore hardware. The problem of race conditions is briefly explained. Then, constructive measures for avoiding race conditions are presented, along with their limitations. The article explains which additional analytical measures are necessary and how you can use these measures, supported by tools, in your daily project work to identify and resolve potential software malfunctions early on.<\/p>\n While in purely sequential programming the order in which individual instructions are executed is precisely defined, concurrent software systems consist of multiple threads, such as tasks or interrupts, which are executed logically in parallel. Although the order of instructions within a thread is still defined, this is no longer possible relative to other threads. As long as the threads are executed independently and their results cannot influence each other through the use of shared resources (e.g., CPU or memory), this parallelism is unproblematic. However, a race condition arises as soon as the order of execution between threads has an impact on the system's outcome.<\/p>\n An important and frequently occurring pattern for race conditions is the data race. Here, concurrent processes compete simultaneously for a resource in the form of a shared variable or memory area, and at least one of the processes modifies the state of this resource. If different outcomes result from each overlap of the processes, then this constitutes a race condition.<\/p>\n To illustrate a data race, consider two threads that each increment the same counter variable (see figure below).,\u00a0PDF<\/a>Each thread first loads the value of the variable into a register, increments it, and then saves the calculated value back. If the execution of the two threads overlaps, one overwrites the result of the other. The effect would be that the variable is incremented only once, instead of twice as expected.<\/p>\n Each software system must be guaranteed upon delivery to be free of functional errors in the form of race conditions.<\/p>\n The experiences we have reported are based on the following framework conditions:<\/p>\n Race conditions should be avoided as much as possible from the outset through design. Here are some best practices that can be applied:<\/p>\n Experience shows that even comprehensive rule sets and processes cannot prevent all errors. This is fundamentally due to the ever-increasing complexity of embedded systems and the inherent risk of introducing further and additional errors. For example, composition problems, even with proven and safe components, can lead to new problems arising in combination. Within hybrid software systems, it cannot always be assumed that the specific design and coding rules for in-house developed components have been applied to off-the-shelf and third-party components. Technical and business processes have become so complex that even experienced developers cannot always know or consider all the details. Furthermore, software must be constantly adapted and revised to meet changing needs, which in turn makes the interrelationships even more complex and further increases the risk of additional errors.<\/p>\n Since race conditions still occur despite the described constructive measures, and these cannot be adequately captured by testing, analytical methods must be employed for further verification. A complete and exclusively manual review of today's embedded systems is practically impossible \u2013 not only due to the limited availability of human reviewers, but also, and above all, due to the complexity of these systems. Accordingly, to increase efficiency and effectiveness, static code analysis methods are used, which conservatively overestimate the number of problems. This means, on the one hand, that all possible problems are indeed reported, but on the other hand, that a high number of false positives also occur. As a result, the findings then have to be manually evaluated again. This manual review is still time-consuming, even though it is already "cheaper" than the complete manual review mentioned above.<\/p>\n In a research collaboration with the University of Stuttgart, attempts were therefore made to improve the precision of code analysis methods. The results showed that, despite increasingly complex (and computationally intensive) analysis methods, the precision could not ultimately be increased satisfactorily. This was also due to the complex synchronization patterns of the analyzed embedded systems, which could not be fully captured by the tool developed in the research group.<\/p>\n As a consequence, we now rely more on simpler analysis methods with clearly comprehensible results. Currently, we use a tool from the Axivion Bauhaus Suite, which incorporates the findings from our research collaboration. This tool allows us to analyze a single software version, including reading the source code, in just under an hour.<\/p>\n The aforementioned constructive measures and improvements in the design of the current software generation have also made the subsequent evaluation of the results simpler and more efficient. As a result, the evaluation effort has been reduced to a third of its previous level and now falls to approximately 40 hours for a complete evaluation. Depending on the software's maturity level, between 21 and 301 of the reported variables were revised after the analysis. In about half of the cases, actual functional errors were corrected, ranging from relatively harmless to safety-critical. In the other half, the evaluation revealed design deficiencies that, while not currently critical, were nevertheless improved to eliminate future risks or to facilitate software maintainability.<\/p>\n It was striking, however, that even after the functional changes, the analysis still reported mostly the same variables, as it was still unable to consistently identify the synchronization patterns used. This leads to a decrease in the precision of the analysis results as the analyzed software matures. In general, the biggest challenge for the continued use of the analysis lies in efficiently managing the high number of false positives. Improvements to these aspects are high on the agenda for the further development of code analysis.<\/p>\n When the analysis is used continuously alongside development, it becomes apparent that large portions of the tool's reports remain stable over time. While changes during development can create new race conditions and resolve old ones, our experience shows that resolved race conditions very often persist in the results as false positives, and the number of new race conditions arising within a short period is typically small. This raises the question of how the results can be processed more efficiently without overlooking critical errors. To address this, it is first necessary to determine which of the old assessments for reported variables are still valid and can be transferred to a new software version, and which results require re-examination.<\/p>\n As a compromise, the following heuristics are therefore proposed for selecting the variables to be tested:<\/p>\n Based on our experience comparing complete reviews, these heuristics can address all critical errors discovered so far with significantly less effort. Nevertheless, we still recommend at least randomly re-running previous reviews intended for adoption to mitigate the risk of inaccurate assessments.<\/p>\n In this article, we reported on our experiences with constructive and analytical methods for preventing race conditions. It became clear that only static tools are capable of achieving complete coverage. Despite the significant evaluation effort, the use of this tool has proven beneficial for safeguarding against race conditions and, in our view, is a recommended addition to existing quality measures.<\/p>\n Download the article as a PDF<\/strong><\/a><\/p>\n Do you want to bring yourself up to date with the latest technology?<\/strong><\/p>\n Then find out more\u00a0here<\/strong>\u00a0<\/a>MircoConsult offers training courses\/seminars\/workshops and individual coaching on the topic of multicore\/microcontrollers.<\/p>\n Training & coaching on the other topics in our portfolio can be found here.\u00a0here<\/a>.<\/strong><\/p>\n Valuable expertise on the topic of multicore\/microcontrollers is available.\u00a0here\u00a0<\/a>Available for you to download free of charge.<\/p>\nContent outline<\/h2>\n
\n
Benefits and special features<\/h2>\n
What are Race Conditions?<\/h2>\n
Objective<\/h2>\n
Framework conditions<\/h2>\n
\n
How can race conditions be constructively avoided?<\/h2>\n
\n
Why are design\/coding rules insufficient?<\/h2>\n
Analytical methods and experience with their use<\/h2>\n
Solutions for continuous use<\/h2>\n
\n
Summary<\/h2>\n
\nMulticore \u2013 our training & coaching<\/h2>\n
\nMulticore \u2013 Expertise<\/h2>\n