Health Check for the Development of Real-Time Systems

What's wrong with the development process?

Authors: Ralf Münzenberger, INCHRON; Friedhelm Stappert, Nuremberg Institute of Technology

Contribution – Embedded Software Engineering Congress 2015

Many software projects for embedded systems experience problems with budget or schedule overruns. Reasons include, for example, that important requirements are not considered early enough, leading to errors being detected too late, especially in the dynamic behavior of a system. Based on numerous development projects, a concise questionnaire was created (Real-Time Health Check) developed a framework that uncovers weaknesses in the design process. Based on this, recommendations can be derived on how to better manage the dynamic behavior of real-time systems. Typical problems include: incomplete specification of real-time requirements, insufficiently robust architectures, and incomplete consideration of the overall system in early design phases. Many experienced architects are well aware of the importance of these aspects. Nevertheless, it has been shown that timing errors are often discovered too late in the testing phase.

motivation

The idea for analyzing the design process arose during development projects in the field of real-time systems, where INCHRON GmbH acted as a consultant. The goal is to uncover weaknesses and understand why aspects of a system's dynamic behavior repeatedly cause such significant problems, even though suitable methodologies and tools are available. A concise questionnaire, the "Real-Time Health Check," is used to analyze the extent to which these aspects are considered during the design process.

The questionnaire is aimed at system and software architects, but also at team and project leaders—in general, anyone with insight into the entire design process. Ideally, the test should be conducted at the beginning of a project if past experience has shown recurring problems with dynamic behavior. Furthermore, the test is recommended when a project exhibits high technical or organizational complexity. In such cases, there is a high risk that problems or errors will only be discovered late in the development process, as explained in [1] and [2].

The questionnaire evaluation provides a detailed self-assessment of the development process, i.e., an answer to the question: "Where are we now?". Based on this, a target state and corresponding measures can then be defined, i.e., the question: "Where do we want to go?". The goal here is to better integrate real-time aspects into the design process and thereby minimize the risk for future projects.

The test presented below has already been carried out at several companies, mainly in the automotive sector.

The Real-Time Health Check

The Real-Time Health Check It consists of a concise list of questions on how to handle dynamic aspects during system development. The complete questionnaire can be found online at www.real-time-doctors.com can be viewed and filled out.

A series of questions is used to determine when specific real-time aspects are considered during the development process and when certain errors regarding dynamic system behavior are detected. The questionnaire is optimized using the V-model. The design phases are assumed to be design, implementation, and testing. The specific questions are:

1. In which phase of the development cycle do you sufficiently recognize the dynamic behavior of your system?

2. In which phase do you identify the root causes for dynamic behavior problems?

3. In which phase do you consider the whole system including the environment (sensor, actuator, peripherals, buses, networks, etc.)?

4. In which phase do you find show stoppers that are very expensive to solve?

5. In which phase do you focus on real-time requirements?

6. In which phase do you detect the following real-time errors?

a. Load

b. Response time

c. Data consistency

d. Event chain issues

e. Jitter

7. In which phase are change requests for you most problematic?

8. In which phase do you know that changes will work?

Typical errors in real-time behavior include errors related to processor load, response times of tasks and ISRs (Interrupt Service Routines), data consistency, cascades (e.g., data flow latencies from sensor to actuator), and jitter (e.g., start-to-start jitter of tasks, ISRs, and functions), as shown in question 6. The later such errors are discovered, the more complex and therefore more expensive the correction becomes.

For the test, participants selected a representative development project. Based on this project, they answered each question individually, indicating in which phase (design, implementation, testing) the respective topic was addressed.

For evaluation purposes, the answers are weighted differently and displayed in a "radar diagram" (one radial per question). The larger the enclosed area in the diagram, the higher the risk of encountering problems in the development of real-time systems. The risk assessment is broadly categorized into "Green" ("Excellent Real-Time Health," inner circle), "Yellow" ("Medium Real-Time Health Risk," middle circle), and "Red" ("High Real-Time Health Risk," outer circle).

Participants who reach "green" status already employ a methodical approach and consistently use appropriate tools during the design process. The necessary tasks, artifacts, and roles are an integral part of the development process. These participants typically have a very precise understanding of their design process and know where they stand. (See Fig. 1.) PDF) an evaluation using "Excellent Real-Time Health" can be viewed.

The "Yellow" status indicates that participants still have certain weaknesses in their processes, but they usually don't know exactly where the causes lie. This will be shown in the detailed evaluation in the following section.

Evaluation

The questionnaire has already been completed by 35 participants. Although the results are not representative, they do reveal some interesting trends.

Overall, it can be seen (see Fig. 2, PDF), that only 11% of the surveyed companies achieve "green" status. Almost half (46%), on the other hand, have a high risk.

A methodical approach leads to the goal

Interestingly, all the "green" participants already use a methodical approach, apply appropriate tools, and have integrated them into the development process (Fig. 3, see PDFIn contrast, almost two-thirds (65%) of the participants who have not yet established a structured approach show the "Red" status, as shown in Fig. 4 (see PDF) shown.

A typical example of a „green“ development process is shown in Fig. 1 (see PDFHere, real-time aspects are considered early and adequately, and errors in dynamic behavior are detected and corrected in a timely manner. The processes demonstrate a high level of maturity.

Figure 5 shows a typical representative with medium risk (see PDFProblems still arise here, but the team is fully aware of the issue. With appropriate measures, the processes can be optimized to achieve "green" status.

High real-time risk – there are better ways.

Also representatives with high risk (Fig. 6, see PDFThey are often aware that the issue of timing is considered too late or not sufficiently. When analyzing the reasons why the development process is not changed, the following points emerged particularly frequently:

• No time or budget to adequately consider timing during the design phase
• The necessary tools are not available within the company.
• The necessary know-how is not available.
• The real-time requirements can only be precisely specified during the implementation phase.
• The net execution times of the software can only be determined during the testing phase. A budget estimate is not possible.

Unfortunately, a detailed discussion of these arguments is not possible within the scope of this article. However, the test results from participants with a low real-time risk clearly demonstrate that early consideration of timing in the development process is feasible and avoids time-consuming and expensive redesigns in later development phases. Publications [6] to [9] specifically illustrate how this can be implemented in development processes and what advantages result.

Self-image – External image

For some participants, the self-assessment does not match the actual picture, as shown in Fig. 7 (see PDFThe participant assumes that the dynamic system behavior is taken into account in a timely and sufficient manner (questions 1, 3). Nevertheless, the corresponding errors are only discovered late (question 6). A more detailed analysis would be recommended here to explain the causes of this discrepancy and to uncover the corresponding weaknesses.

Several participants with a medium to high real-time risk completed a second questionnaire to specify more precisely: "Where do we want to go?". Based on this, corresponding process steps were defined and integrated into the development process.

Current research

The fundamentals for addressing timing aspects in system design are well-researched. Examples include the projects TIMMO / TIMMO2USE [4] and Amalthea [3]. In TIMMO and its successor project TIMMO2USE, for instance, the architecture description language "EAST-ADL" was extended to include a component for describing the dynamic behavior of a system. This language is based on events, i.e., observable occurrences during runtime, and specific rules regarding the timing of these events. This formalism allows for the capture of both timing requirements and properties of a system [10].

The AUTOSAR standard of the automotive industry also includes a specification for timing descriptions („AUTOSAR Timing Extensions“) [5].

Summary

The Real-Time Health Check This is an efficient method for identifying weaknesses in the design process when it comes to real-time aspects. Architects and project managers can quickly gain an understanding of their current status and identify any necessary measures to minimize the risk of problems being discovered too late. The test results clearly show that a systematic approach, consisting of a methodology, the necessary process steps, and the use of appropriate tools, leads to a low real-time risk. 651 of the participants who do not consider this have a high real-time risk, while 331 of the participants who do have a low real-time risk.

literature

[1] U. Brodtmann: Return on Investment at Tools – Automobil Elektronik 3/2013.

[2] R. Münzenberger: Dynamic Behavior. Control Unit Development in the Tension Between OEM and Supplier. Paper presented in the proceedings of the 16th International Congress in Automotive Electronics, June 19 and 20, 2012, Ludwigsburg.

[3] AMALTHEA: An Open Platform Project for Embedded Multicore Systems

[4] TIMMO-2-USE (Timing Model – TOOLS, algorithms, languages, methodology, USE cases)

[5] Specification of Timing Extensions: AUTOSAR_TPS_TimingExtensions.pdf

[6] T. Jäger, I. Houben, R. Münzenberger: Model-based architecture development and simulation. Proceedings of the Embedded Software Engineering Congress 2015.

[7] J. Meyer, I. Houben, R. Münzenberger: Mastering communication overhead early in multi-core systems. Proceedings of the Embedded Software Engineering Congress 2014.

[8] A. Wolfram, M. Makarov, T. Kramer, W. Ramisch, R. Münzenberger: Design of Robust System Architectures for Automotive ECUs; Conquest 2009, Nuremberg

[9] R. Münzenberger, M. Dörfel, C. Dietrichs, U. Margull, G. Wirrer: Design of real-time capable control unit software in FlexRay networks. Automotive Developer Forum 2007, Ludwigsburg.

[10] F. Stappert, J. Jonsson, J. Mottok, R. Johansson: A Design Framework for End-To-End Timing Constrained Automotive Applications. Embedded Real-Time Software and Systems (ERTS), Toulouse, France, 2010.

Download the article as a PDF file