EN 
DE Greater efficiency, traceability, and validity for early functional testing
Author: Dr. Hartmut Pohlheim, Model Engineering Solutions
Contribution – Embedded Software Engineering Congress 2015
A method is presented that enables the testing and automated evaluation of requirements across all test sequences. This reduces testing effort and the potential for errors. Furthermore, a method is introduced that supports the required requirement tracing in practice. The combination of both methods in an efficient test tool leads to a significant increase in test coverage and the reliability of test results.
Increasingly complex software, coupled with high quality requirements, makes efficient testing indispensable. Especially in model-based development, ISO 26262 makes it essential to develop efficient testing techniques. The test coverage required by the standard, in conjunction with requirement tracing and the recommendation for requirements-based testing, necessitates a re-evaluation of existing test methods [1].
In this context, we present a method for the automatic verification of requirements-based model tests, which not only ensures higher test coverage but also enables reliable tracing through a framework specifically developed for this purpose.
The combination of automated assessments and automated requirement tracing is available in a test tool for Simulink.®-, Embedded Coder®– and TargetLink®-models integrated, which can meet the requirements of ISO 26262 with considerably less effort [2].
Integrated approach to requirements-based testing
As the name suggests, requirements form the basis of requirements-based testing. Starting with the software's specifications, the tester begins by deriving all the necessary stimulus inputs from the requirements. These serve as inputs for the model under test. After a number of simulations, also called test sequences, the recorded output signals are available. These signals can then be used to evaluate whether the test meets the applicable requirements. The execution coverage can also be measured to determine whether the applied number of stimulus inputs is sufficient for an evaluation (Fig. 1, see [reference]). PDF).
Traditionally, software evaluation was performed by visually inspecting the output signals. This often necessitates keeping the stimulations very simple in order to simplify, or even enable, the complex manual evaluation.
This paper presents an advanced approach to the improved and, above all, automated evaluation of requirements-based test cases. In particular, it breaks with the strict link between test cases and requirements. The automated evaluation is performed by a specially developed framework for the formal validation of requirements. This approach is characterized by the fact that a test formulated once in an assessment can be reused and applied to all conceivable system responses. This underscores the universal nature of requirements and helps to uncover errors that were previously unanticipated.
This method is generally based on three characteristic elements: the requirement, the test specification, and the assessments. To maintain an overview of their strong interdependencies, it is essential, in accordance with the tracing requirements of ISO 26262, to track them at all times. The framework was developed for this purpose. „Requirements, Coverage, and Traceability“ (RCT) developed, which conveniently supports requirements tracking.
The combination of both approaches in one test tool thus enables an integrated approach for a modern and effective test process (Fig. 2, see PDF).
Automated test evaluation with assessments
A fundamental principle of assessments is their reusability. An assessment can be understood as a function that examines the recorded input and output signals as well as the local signals and parameters (Fig. 3, see [reference]). PDF).
This assessment checks the invariance of the system behavior and provides the result for each sequence individually. Three possible result values are available: Passed (the behavior complies with the applicable requirement), Failed (a violation of the requirement was detected during the test sequence) and NotTriggered (The test sequence did not lead to a stimulation of this requirement).
In contrast to a direct examination of n Test sequences with exactly n expected results (Fig. 4, left, see PDF), matrix evaluation using assessments provides significantly higher test coverage through m*n Evaluations, since this is a set of m requirements, through m Assessments are reviewed and this review applies to all n Test sequences are performed (Fig. 4, right, see PDFMatrix evaluation also allows for the aggregation of results at the level of test sequences and assessments. This directly leads to greater reliability of the test results.
To fulfill this series of tasks, an assessment consists of a descriptive component and an evaluative execution component. The first component includes, among other things, traceability information and a descriptive text. In the second component, signal access and its evaluation take place as post-processing of the test sequence (Fig. 5, see [reference]). PDF).
To enable a quick start, a template generator is available. Requirements and needed signals can be selected, and with just a few entries, a suitable template for the assessments is automatically generated.
The name and description of the assessment must be filled in by the user in the description section (Fig. 6, see PDF). For the evaluation, the following must be considered: Triggered The condition under which the requirement is valid must be formulated. Using the provided helper functions, the check can then be implemented, as in the example shown, a classic "if-then" relationship. Permissible response times of the system can also be taken into account (Fig. 7, see...). PDF).
Once all test sequences have contributed to the evaluation of a requirement, their results can be aggregated. The overall result with associated metrics for our example is shown in Fig. 8 (see...). PDF) given. A single violation of the requirement is sufficient to result in a Failed to obtain.
Request tracking and presentation of results data
A characteristic feature of the presented model test is that test sequences, like assessments, always refer to requirements. This results in a multitude of connections (Fig. 9, see below). PDF), the tracking of which often poses a major challenge for the tester in practice.
To support this process, we developed the RCT framework. It automatically follows the links and thus enables clear requirement tracking. The IDs of the requirements serve as the central elements. The tester simply needs to reference these IDs – both within the test specifications and the assessments. This framework provides the necessary tools for this (Fig. 10, see [reference]). PDF). A method for test specification (MTCD) that supports the process presented here is described in [3].
Once all references are entered, a report overview is provided that not only summarizes all applicable requirements but also provides detailed information about the respective test sequences and assessments – as well as information on their processing progress (Fig. 11, see PDF).
The RCT dashboard also contains important information about the degree to which requirements are met and their coverage by test sequences and assessments, thus providing insights into the functional quality of the tested software module. Metrics for evaluating test quality are also provided. These include, among others, the progress of the review of test sequences and assessments, as well as the values for model and code coverage (Fig. 12, see Figure 12). PDF).
Handling changing requirements
The RCT framework report serves two purposes: firstly, it is an important documentation element, and secondly, it is an effective working tool that allows testers to track their workload, especially when the requirements document changes.
The RCT framework allows for the easy loading of the new requirements document into the existing project. RCT's differential analysis identifies the changed requirements and resets the processing status of the elements linked to them. In practice, this initially reduces the test project's key performance indicators (KPIs). Furthermore, the subset of affected test sequences and assessments is displayed, enabling targeted and therefore time-saving revision of these elements – always with the goal of achieving or exceeding the original KPIs (Fig. 13, see Figure 13). PDF).
Summary
This paper demonstrates a methodological approach to achieving higher test coverage. The application of the developed framework for automated requirements verification using assessments provides significant practical support for the user. In the future, test sequences will increasingly take on the task of sufficiently stimulating the system under test to achieve the required high coverage levels. The assessments efficiently verify and ensure compliance with the requirements.
Furthermore, it was demonstrated that the tracing required by the standard can also be supported efficiently using a tool. Applying the corresponding framework allows not only for an evaluation of functional quality during the ongoing test process, but also for statements about the quality of the test itself.
The combination of both approaches shows how a modern application-based model test can already be designed today.
literature
| [1] | ISO, ISO 26262: Road vehicles — Functional safety, International Standard ISO/FDIS 26262, 2011. |
| [2] | Model Engineering Solutions GmbH, 2015. [Online]. Available: https://www.model-engineers.com/de/mtest.html. |
| [3] | Schmidt, Tobias, et al. Efficient Testing Framework for Simulink Models with MTCD and Automated Test Assessments in the Context of ISO 26262. SAE International Journal of Passenger Cars-Electronic and Electrical Systems 7.2014-01-0306 (2014): 166-177. |
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
