EN 
DE Testing from module testing to system testing
Author: Kristian Trenkel, iSyst Intelligente Systeme GmbH
Contribution – Embedded Software Engineering Congress 2016
This article explains the use of debuggers for hardware-in-the-loop (HIL) testing. Two areas of application are presented: firstly, accessing the variables of the software of the electronic control unit (ECU) under test during runtime. This enables the implementation of white-box and grey-box tests in HIL testing. Secondly, measuring the code coverage of test cases without modifying the ECU's software. The implemented abstract interface allows the use of different debuggers without changing the test cases. A generic interface is available for accessing the software variables. Furthermore, the execution of code coverage is separated from the actual test cases.
The development of embedded systems, such as those used in the automotive sector, requires the use of advanced development methods and procedures due to their constantly increasing complexity. As complexity increases, so does the effort required for testing. The necessary test depth can only be achieved in a timely and cost-effective manner through the highest possible level of automation.
Test automation, as used, for example, in hardware-in-the-loop (HIL) testing, is particularly widespread in the automotive sector. To achieve the necessary test depth, especially with regard to safety-critical functions such as driver assistance systems, pure black-box testing is insufficient. It is necessary to also perform white-box and grey-box tests with the actual hardware and software in real time. The XCP protocol [1] is frequently used to access the variables of the control unit under test. However, the use of this protocol affects the software runtime within the control unit.
By using debuggers in conjunction with the functions of modern microcontrollers, it is possible to access ECU variables in real time and thus read and write them during testing. Furthermore, code coverage measurements can be performed without instrumenting the code. In practice, this presents the challenge of integrating debuggers into the test automation environment.
The following describes the integration and application of the BlueBox debugger with the winIDEA software from iSYSTEM [2] and the PowerDebug debugger with the Trace32 software from Lauterbach [3] into the iTestStudio test automation system.
The tool iTestStudio
The iTestStudio test automation tool from iSyst Intelligente Systeme GmbH is used to create and automatically execute tests using test systems, such as HIL test systems. It accesses various test tools, such as CANape, CANalyzer, and ControlDesk, via automation interfaces (e.g., COM).
The iTestStudio software consists of various components, which are shown in Figure 1 (see PDF) are shown.
The interaction of the individual components for test execution is shown in Figure 2 (see PDF) to see.
iTestStudio TA
The iTestStudio TA component (see Figure 3, PDF) represents the actual user interface. This includes the organization of test cases with the compilation of test series and test sequences (see Figure 4, PDF), running/starting the test scripts and creating the test reports.
TestSupport libraries
The second part contains the TestSupport libraries. These libraries handle the execution of the tests with the corresponding Python version, which is not necessarily the same Python version used by iTestStudio TA GUI. They also include functions for saving the results.
TestToolkit
The third component is the TestToolkit. This contains special test functions, interfaces to tools (such as Vector, CANape, or dSPACE ControlDesk), and containers for cal and HIL variables in the ECU, which can be accessed, for example, via XCP using the CANape tool. Appropriate adapters were created between the debugger's automation interfaces and the cal variables to connect the debuggers.
Integration into the HIL test system
In Figure 5 (see PDFFigure 1 shows the schematic setup of a HIL test system as used for the tests described here. A detailed description of the setup can be found in [4].
The central component of the HIL test system is the real-time computer, which performs the environment simulation for the ECU. The control PC is also of central importance. It executes the test cases/test scripts and connects to the real-time computer on one side. The debugger is also connected to it on the other.
Variable access
To implement white-box and grey-box tests within the framework of HIL testing, access to variables in the control unit is necessary. The debug interface (e.g., BDM or Nexus) of modern microcontrollers allows access to the microcontroller's memory and thus to software variables during runtime, without affecting the software's execution time. It is important to note that only access to global variables is possible. Access to local variables is only possible within their respective scopes, which would necessitate halting the software at the relevant point.
From the perspective of the implemented test case, variable access should be abstracted. Within the test case, there should be no dependencies on the tool or interface used. These requirements were achieved through a two-stage abstraction concept.
The first level forms a class, which represents a single variable. There exists a general base class (VarBase), which defines the interface, and one implementation each for the different sources of variables (e.g., CANape, winIDEA, or Trace32). In Figure 6 (see PDF) is an excerpt of the code for the class EcuVar See which one implements the connection to winIDEA. Only the parameter is relevant here. context the method __init__ Tool-specific. This contains the reference to the winIDEA interface. The parameter identifier Contains the variable name. The remaining parameters allow you to pass additional information to the variable, such as a unit of measurement, for example, to use it for output in the test report.
The methods serve as the interface for the test. set and get available which allow writing and reading of the respective variables regardless of the tool used.
The second level of abstraction is a container class that collects all variables for a test project or control unit and handles tool-specific properties (e.g., starting the tool). Figure 7 (see PDF) shows the definition of the variable container.
Within the variable container, an instance of the variable class (e.g., `variable`) is then created for each variable that is to be used in the test. EcuVar) created. Figure 8 shows the application of the two classes. The class EcuVars This is the project-specific implementation of the variable container and contains various variables. In the example in Figure 8 (see PDFThis can be seen for a variable. In the test case, the variable is „"(((TDFM).dummy_module).dummy_structure).State_en"“ of the control unit as state_en accessible.
In the actual test case, the variable can now also be accessed with set or. get Access is possible without knowing which tool is being used. In Figure 9 (see PDFThe following is an example of its application in a test case. First, a reference to the variable container is created and stored in the variable. ecu stored. By means of ecu.state_en The control unit variable can then also be accessed.
Code coverage
To determine test coverage, measuring the processed code on the ECU is crucial. Especially in the context of safety-critical systems (e.g., ISO 26262), determining code coverage is essential. This is usually achieved by instrumenting the ECU software. However, this impacts runtime and memory consumption, and therefore potentially the test results.
Modern microcontrollers allow the executed lines of code (binary code) to be determined in real time at runtime using the debug interface. This information can be recorded during test execution and, after all tests are completed, mapped to the code coverage of the ECU's C code.
Implementing code coverage measurement for iTestStudio has shown that integration is possible without changes to the test cases. The test environment, as shown in Figure 2 (see PDFThe function shown provides standardized functions for switching the control unit on and off. These functions are only called by the test case implementation, while the connection for code coverage measurement can be established in the project-specific part of the functions. The measurement is started when the control unit is switched on, and the code coverage information is saved when the control unit is switched off.
The actual evaluation of the code coverage information takes place after the test has been carried out, using the respective tool of the debugger manufacturer.
Application examples
Testing a steering lock control unit
In a hardware-in-the-loop (HIL) test project for the control unit of a steering lock, XCP support was not available for variable access. Implementing this within the project was also not possible. Since the classification as a safety-critical system required extensive test coverage, it was necessary to find an alternative method for variable access.
Using the BlueBox and winIDEA, access to the ECU software was achieved via winIDEA's Python interface. Integration into the test environment is carried out as described in the chapter. Variable access described.
Code cover measurements on a control unit for roll stabilization
As part of a hardware-in-the-loop (HIL) test project for roll stabilization, measuring the code coverage for the executed HIL tests became necessary. Due to its classification as a safety-critical system, instrumenting the control unit's code was not an option.
As an alternative solution, code coverage was measured using a debugger. Both winIDEA/BlueBox and Trace32/PowerDebug were used. Integration into the test environment was achieved using the automation interfaces of both tools, as described in the chapter. Code coverage described.
This allowed the code coverage of the HIL test cases to be determined without any changes to the test cases or the ECU software. The implementation effort for carrying out the code coverage measurements can be described as low. Furthermore, the developed solution can be integrated into any other HIL test project that uses the iTestStudio.
Summary
Integrating debuggers into HIL testing expands the possibilities of HIL testing to include white-box and grey-box testing, without requiring protocols like XCP and thus avoiding changes to the ECU software's runtime during testing. Furthermore, performing code coverage measurements using debuggers allows for the precise determination of test coverage at the code level, without requiring instrumentation of the ECU software. This measurement therefore has no impact on code size, memory consumption, or runtime. From the outset, a generic interface was prioritized in the implementation of debugger integration into the HIL environment and its connection to test automation. Experience has shown that this allows for the exchange of debuggers without requiring any modifications to the tests.
List of sources
[1] ASAM eV, „ASAM MCD-1 XCP“, ASAM eV, May 27, 2015. [Online] [Accessed on 12 10 2016]
[2] iSYSTEM AG, „iSYSTEM AG“, iSYSTEM AG, [Online] [Accessed on 12 10 2016]
[3] Lauterbach GmbH, „Lauterbach GmbH“, Lauterbach GmbH, 12 10 2016. [Online] [Accessed on 12 10 2016]
[4] K. Trenkel, „From developer's tool to HIL test system“, CAN Newsletter, No. 03/2016, pp. 28-31, 03 2016
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
