EN 
DE A practical report
Author: Kristian Trenkel, iSyst Intelligente Systeme GmbH
Contribution – Embedded Software Engineering Congress 2015
ISO 26262 contains requirements for the qualification of software tools used in the development and testing of safety-critical systems. In practice, however, the interpretations of the various requirements differ considerably. This article describes practical experience with the tool qualification of the test automation tool iTestStudio at a Tier 1 supplier. Furthermore, a possible approach to tool qualification is presented. Ultimately, however, the question remains how the content of the standard should be interpreted.
In the automotive industry, an increasing number of safety-critical functions, such as emergency braking assist, lane keeping assist, and autonomous driving, are being implemented in vehicles. The requirements of ISO 26262 [1] must be observed for the development and testing of these systems. This standard specifies requirements not only for the development process and its tools, but also for testing, including test methods and tools.
For test tools, as for software tools in general, there are detailed requirements that must be evaluated as part of a tool qualification.
Tool qualification according to ISO 26262
The focus of ISO 26262 with regard to tool qualification is described in Chapter 11.1 [2] as follows:
„The first objective of this clause is to provide criteria to determine the required level of confidence in a software tool when applicable.“
For this purpose, an evaluation of the tool is carried out according to the specifications of ISO 26262 in Chapter 11 [2]. This evaluation assesses, firstly, the Tool Impact (TI – TI1 or TI2), which describes the potential introduction or concealment of errors in safety-critical parts. Secondly, the Tool Error Detection (TD – TD1, TD2 or TD3) is examined, which provides information about the tool's reliability. Based on the TI and TD values, the Tool Confidence Level (TCL – TCL1, TCL2 or TCL3) is determined. Depending on the TCL rating, additional measures must be implemented during the development and testing of the tool.
In practice, the primary goal is to achieve a TCL1 rating, as this eliminates the need for further measures. However, experience shows that this rating is not always objective. Often, the focus is on minimizing effort and thus costs during the development phase. For example, even compilers are sometimes rated TCL1. Realistically, it is virtually impossible to completely eliminate errors generated by a compiler.
The tool iTestStudio
The iTestStudio test automation tool from iSyst Intelligente Systeme GmbH, which serves as an example in this article, is used to create and automatically execute tests using test systems such as Hardware-in-the-Loop (HiL) test systems. Various test tools, such as CANape, CANalyzer, and ControlDesk, are accessed via automation interfaces (e.g., COM).
The iTestStudio software consists of various components, as shown in Figure 1 (see PDF) are shown.
The interaction of the individual components for test execution is shown in Figure 2 (see PDF) to see.
iTestStdio TA
The iTestStudio TA component (Figure 3, PDF) represents the actual user interface. This includes the organization of test cases with the compilation of test series and test sequences (Figure 4, PDF), the execution/starting of the test scripts and the creation of the test reports.
TestSupport libraries
Secondly, the TestSupport libraries should be considered. These libraries handle the execution of the tests with the corresponding Python version, which is not necessarily the same Python version used by iTestStudio TA GUI. They also include functions for saving the results.
TestToolkit
Thirdly, the test toolkit needs to be examined. This contains special test functions, interfaces to tools (such as Vector CANape or dSPACE ControlDesk), and containers for cal and hi-fi variables. The interaction of the test automation tool with the test system and various test tools quickly makes the assessment for tool qualification very complex, difficult, and costly and time-consuming.
Qualification at Tier 1
As part of the use of the iTestStudio for testing a chassis control unit classified as ASIL B and at a Tier1 supplier, qualification according to ISO 26262 became necessary.
In the first step, documents relating to the use of iTestStudio and a self-assessment from the person responsible for functional safety at the Tier 1 supplier were requested. Based on ISO 26262 ([2] Chapter 11.4.5 et seq.), a corresponding document was created and submitted. This document includes information on potential errors in the test automation tool that could lead to incorrect ratings in the test reports. It also documents which measures have been taken or should be taken during use to ensure that this misconduct does not go undetected.
In the subsequent meeting, the document (12 pages) was rejected as being too extensive. Further discussion revealed that Tier 1 used a form (Excel spreadsheet) to collect the necessary information for tool qualification. This document, along with an example, was provided. Upon closer inspection, it became apparent that only a single row within the document was designated for data entry. Consequently, it was not possible to generate more comprehensive information within this document. However, this very superficial review was accepted by Tier 1 and resulted in a TCL1 certification for the test automation tool iTestStudio.
In summary, the tool qualification by the Tier 1 supplier was far more superficial than expected according to the ISO 26262 standard. This approach seemed counterproductive to all those primarily involved in testing. Further steps should clarify the necessary scope of tool qualification.
Own approach to tool qualification
To further develop his expertise in functional safety according to ISO 26262, the author participated in the training and examination for the "Functional Safety Professional" certification offered by TÜV Süd. The training also covered the topic of tool qualification and TÜV Süd's perspective on the necessary scope of this qualification.
The following key findings emerged:
- As required by ISO 26262, tool qualification must be carried out anew for each project in the deployment environment.
- In addition to the tool itself, the interfaces to other tools must also be considered.
- All assessments and measures taken must be documented in writing.
As a result, a document was created for iTestStudio which includes the following points:
- Tool structure
- Intended use and integration of the tool
- The development process used
- Implemented security measures (e.g. reviews, software module tests…)
- Consideration of the measure according to ISO 26262 Chapter 11.4.6 – 11.4.9 [2]
For the purposes of considering the measures according to ISO 26262 sections 11.4.6 – 11.4.9 [2], it was assumed that iTestStudio was classified according to TCL3. The necessary measures also depend on the ASIL level of the system under test. With the assumption of TCL3, possible measures from sections 11.4.7 – 11.4.9 [2] could now be considered for all ASIL levels, and their relationship to previously performed activities could be identified. The aforementioned sections provide a wide selection of measures that can increase confidence in the tool and thus make its use for the development and testing of a safety-critical system justifiable.
The procedure described in section d) of chapter 11.4.7.2 serves as an example, stating that "the occurrence of malfunctions and corresponding erroneous outputs of the software tool acquired during previous developments are accumulated in a systematic way." The goal of this section is to strengthen trust by tracing errors and their resolution. In the case of iTestStudio, the tools TRAC and Subversion are used. TRAC is used to record all errors that occur and to document their resolution. The resolved errors, changes, and new features are documented in the release notes (changelog.txt) for each version.
As a result, corresponding activities could be assigned to most of the measures required in ISO 26262. Only point 11.4.6 1d) „Development in accordance with a safety standard“ could not be fully represented. The footnote „QNo safety standard is fully applicable to the development of software tools. Instead, a relevant subset of requirements of the safety standard can be selected.“ in ISO 26262 [2] on page 25 does not make this point very informative.
In summary, ISO 26262 provides a fairly precise set of guidelines for tool qualification. With a sufficiently thorough implementation, a verifiable and reliable classification of a tool can be achieved.
In practice, however, the quality of these considerations unfortunately varies widely. A standardized approach would be desirable, indeed often necessary, to save smaller suppliers the effort of creating new documents for each customer.
Future planning
To simplify tool qualification for customers, a Tool Qualification Kit for iTestStudio is planned. This kit will ideally consist of:
- Documentation of the tool
- Documentation of the operating environment
- Safety assessment according to ISO 26262 (as in chapter Own approach only tool qualification describe)
- List of compatible test tools and test systems
- Executable test project to verify correct installation
During the compilation of the Tool Qualification Kit, it became apparent that most of the components were already available. Internal documents and the environment description for the tool's internal tests can be used with minor adjustments.
Summary
Experience shows that the topic of tool qualification according to ISO 26262 is viewed very differently in different companies. Only rarely are in-depth analyses carried out. Most often, the focus is simply on evaluating a TCL1 with the least possible effort, as described in this article.
From a tester's perspective, this approach is short-sighted or even negligent. The requirements of ISO 26262 are sensible and should be considered. On the one hand, our own evaluation of the iTestStudio tool has shown that an unbiased approach leads to the identification of weaknesses in the development process and an improvement in the tool's quality. On the other hand, a typical development process (e.g., according to the V-model) already covers a large portion of the requirements of ISO 26262.
If the process specifications are adhered to from the beginning of development, then no significant additional effort is required for tool qualification according to ISO 26262. Overall, a standardized approach to tool qualification across different companies (especially OEMs and Tier 1 suppliers) would be desirable.
List of sources
| [1] | ISO, ISO 26262:2011, Geneva: ISO, Road vehicles — Functional safety. |
| [2] | ISO, ISO 26262-8:2011, Geneva: ISO, Road vehicles — Functional safety — Part 8: Supporting processes. |
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
