EN 
DE Tool qualification for safety standards
Author: Erol Simsek, iSYSTEM AG
Contribution – Embedded Software Engineering Congress 2015
Functional safety standards require or recommend a closer preliminary examination of the software tools used in the software development process with regard to their "use risk" and thus their negative impact on system safety. As a software tool manufacturer in this area, we are therefore increasingly confronted with questions such as "Is your tool certified?", "Do you have a TÜV seal of approval?", or "Can you support us in qualifying your tools?" What does this mean in concrete terms, and how have we, as software tool manufacturers and customers, addressed this issue so far?
Why tool qualification at all?
Since the official implementation of ISO 2626 (Road vehicles – Functional safety) in 2011[1], there has been a growing awareness of tool qualification in the embedded market. Tool qualification has been present in other functional safety (FuSi) standards for much longer. The overarching standard is IEC 61508:2010 („Functional safety (Safety-related electrical/electronic/programmable electronic systems," first published in 1998) defines software tools in three criticality classes, T1-T3. Depending on the classification, based on the question "Does the software tool have a negative impact on the safety of the system?", appropriate measures are proposed or required to minimize the operational risk of a software tool. In aviation, the latest version of DO-178 B from 1992 already specifies how to handle in-house developed or purchased software tools within a safety project.
In this context, the term "tool qualification" is used as an umbrella term for a whole range of activities. Essentially, it refers to the closer examination of software tools during the development process of embedded systems, which can be grouped into classification and confidence-building measures. The latter can also lead to the qualification of a software tool. More on this later.
In 2012, more than 20 years after the introduction of DO-178 B, DO-178 C [2] was adopted. After such a long time and the accompanying technological progress, the formulation of the current state of science and technology with regard to the development of safety-critical software had to be adapted. In addition to model-based and object-oriented development as well as formal methods, the RTCA introduced (Radio Technical Commission for AeronauticsThe RTCA also includes a comprehensive document on "Software Tool Qualification Considerations," DO-330. This is primarily a process model that describes the development of software tools (mostly in-house developments within a company) according to a functional safety standard. Furthermore, it systematically outlines how to handle so-called COTS tools and software (commercial-off-the-shelf, i.e., purchased standard tools and software components). The RTCA recommends the document as domain-independent for use in other markets as well [3][4].
What is tool qualification?
DO-330 defines a software tool as follows (freely translated from English):
„… a software tool is a computer program or a functional part thereof that is used to support the development and transformation, testing, analysis and creation of another program in the process, or that causes changes to data or documentation of a program.“
Examples of software tools include code generators, compilers, test tools, self-developed software, scripts, standard office products, etc.
Today, software tools define, characterize, automate, harmonize, standardize, simplify, and implement a software development and testing process. If potential errors in a software tool negatively impact the safety of a system and/or such tool errors are not detected within a process or compensated for by alternative measures within the process, one or more functional safety (FuSi) standards require or recommend a closer examination of software tools prior to safety development.
In principle, the system manufacturer is specifically required to closely examine the historically developed software tool landscape, particularly with regard to the use cases within the process. Additionally, trust in the use of specific software tools must be established and ultimately verifiable. This also includes demonstrating how to avoid misinterpretations by the tool user when using the software tools. This involves, in particular, aligning the expectations of the tool developer and the actual tool user. In the event of demonstrable mistrust in a software tool, a company will implement further tool qualification measures to minimize this mistrust and thus the operational risk associated with the respective software tool. The type of measure depends on the system's safety classification (e.g., SIL1-3 IEC 61508 or ASIL AD ISO 26262) and is described in the respective standards (see below).
To whom must a customer provide proof that they have examined and, if necessary, qualified software tools in the process?
This is a question that can't be answered generally. In aviation, system manufacturers are monitored by the FAA (Federal Aviation Administration) or the EASA (European Aviation Safety Agency). Evidence, both in general and specifically regarding tool qualification, must be provided to these official bodies. Other markets are not as strict about tool qualification. The standard essentially recommends the procedure, and only in the event of a liability claim is corresponding evidence required.
Standards of Functional Safety and Tool Qualification
To determine the operational risk of software tools, the process typically begins with classifying these tools within a project. The various functional safety standards approach this topic differently:
Higher-level standard: IEC 61508 and railway technology: EN50128:2011
- Does the software tool used have an impact on the safety of a system?
- If no, no qualification measures are necessary. Tool Classification Level T1
- If so, what type of tool is it?
a. Constructive, e.g. compiler, code generator at level T2
b. Analysis/Verification, e.g. static code analysis, unit test tool at level T3 - For T2 and T3, measures must always be described to detect or compensate for any potential tool errors in the process. T3 is the strictest classification.
- For T3, there are additional opportunities to build trust through
a. Examination of the operational provenance of a software tool
b. Tool validation
Automotive: ISO26262:2011 Road Vehicles – Functional Safety
- Does the software tool used have an impact on the safety of a system?
- If no, no qualification measures are necessary. Tool Classification Level TCL1
- If so, then a decision must be made regarding the probability of being able to detect or compensate for a potential tool error in the process.
- There are three levels for this.
a. High
b. Mid
c. Low - At High, the tool falls into TCL1
- Mid in TCL2 and Low in TCL3
- In the latter two cases, the standard describes four possible qualification methods:
a. Examination of the operational provenance of a software tool
b. Process audit/assessment at the tool manufacturer
c. Tool validation
d. Testing or proof that a tool has been developed in accordance with a safety standard
Aviation: DO-178C:2012 Software Considerations in Airborne Systems and Equipment Certification, DO-278A Software Integrity Assurance Considerations for Communication, Navigation, Surveillance and Air Traffic Management (CNS/ATM) Systems
- What type of tool is it?
a. Constructive (tool category 1, TK1)
b. Analysis (TK2)
c. Verification (TK3) - Tool Category 1 is the strictest classification. Depending on the category, the respective "Tool Qualification Levels" 1-5 are determined:
a. TK1: TQL1-2
b. TK2: TQL3-4
c. TK3: TQL5 - DO-330 then describes, in the form of tables, the measures necessary for the respective TQL.
- The requirements for the lowest level are already in place.
a. Tool Operational Requirements (TOR) and
b. Tool validation according to these use cases
c. as well as a description of measures that detect or compensate for any potential tool errors in the process - For all higher levels, the measures of TQL 5 and further measures apply, which in principle already correspond to the development and testing of a software tool according to DO-330.
Medical technology: EN ISO 13485 (Quality management), EN ISO 14971 (Risk management), EN 62304 (Life cycle of medical software)
- Medical technology standards do not explicitly mention tool qualification like other standards.
- In principle, the selection of the software tools to be used is required as part of the definition of the software development plan.
- The standard contains guidance such as: „Every tool used should be tried and tested and used thoughtfully. The software development plan should document the tools and their versions,“ from [5], page 112.
Tool Qualification Package
A tool qualification package is essentially a tool validation package. Depending on the software tool, it consists of documentation/instructions for classifying the tool, application scenarios of the tool within the process (tool operational requirements, use cases), and a test execution environment and test cases for validating the software tool. The latter is a kind of reference environment with a corresponding report generator for documenting the validation process. The actual validation of a software tool then takes place at the customer's site, by the customer, and in their development environment. Therefore, a tool qualification package can always be purchased with consulting services for individual customization at the customer's site.
What are the expectations of customers and software tool suppliers?
The decision to purchase a tool qualification package is preceded by a significant portion of the preliminary steps in assessing the operational risks of software tools. Choosing a tool qualification package is directly linked to the decision to validate a software tool. The customer hopes to acquire an automated process that makes the "final" step in the overall assessment of the software toolchain as time- and cost-efficient as possible. The expectation that the validation process is essentially complete upon purchase is, of course, incorrect. The real work begins with the purchase. Therefore, attempts are made to avoid the very complex tool validation process, if possible, through intensive classification of the software tools. The aviation industry, with its predefined categorization, leaves little room for interpretation, whereas other standards immediately engage the system manufacturer in discussions about potential risks posed by a software tool, the probability of tool errors occurring, and so on. These discussions and the resulting decisions are highly individualized and can vary considerably from company to company and from project to project. This process can best be documented by formalizing the classification process using tool support, as described in [6][7].
The effort required for a software tool manufacturer to provide a tool qualification package is considerable and can vary significantly. Corresponding services for commissioning and training must also be offered. Ultimately, the customer is expected to perform validation independently, as required by the standard. A tool qualification package thus becomes a product with consulting services, which is then intended to be marketed accordingly.
How has the market implemented this topic so far?
Today, almost all software tool manufacturers offer a tool qualification package. Simply entering these two words into a search engine yields a wide range of options for this purpose. Furthermore, numerous certificates exist that assess the feasibility of using a software tool in a safety project. While all of these things build trust, they don't relieve the customer of the work of classification and potential qualification/validation.
Status quo and lessons learned
There is much to be learned from aviation, particularly regarding tool qualification. With the DO-330, aviation lives up to its pioneering role in processes, technology, and methods. It remains to be seen whether other markets will follow this very well-written document. After initial hysteria, things have calmed down in the automotive industry regarding tool qualification. They seem to have found an appropriate response to the risks associated with using software tools in the development of a safety system. In any case, preliminary assessment is no longer seen as a nuisance, but rather as beneficial for process efficiency. Other industries consider software tools separately, often adhering to the motto: "Keep your eyes open when selecting tools."
…“in advance as part of the software development planning … selection of the tools used … tools … should be tried and tested and used thoughtfully. The software development plan should document the tools and their versions.“ [5]
And what's happening with software tool vendors? The availability of tool qualification packages is increasing, especially among vendors of verification and requirements management tools, compilers, code generators, and so on. This is simply because the number of requests for such packages is rising. Not because a customer wants to buy the product immediately, but rather because it provides reassurance that the vendor has implemented methods and provides documentation that enable the validation of the software tool. Many software tool vendors obtain additional certification, for example from TÜV, specifically in this regard. This external certification typically confirms the fundamental suitability of a software tool for use in a safety project. This is based not only on an audit or assessment of the software tool vendor's processes, but also on extensive testing of the product itself. Overall, the creation of a tool qualification package and the associated certifications lead to increased transparency in the development process of a software tool.
Summary
The term "tool qualification" is used as an umbrella term for a whole range of activities. Essentially, it refers to the closer examination of software tools in the development process of embedded systems, which can be grouped into classification and confidence-building measures. The latter can also lead to the qualification of a software tool.
The aviation industry's DO-330 provides the most comprehensive description of how software tool development should proceed according to a safety standard. The preliminary evaluation and qualification of commercially available software tools, and software in general, are also very well described there.
Purchasing a tool qualification package alone is insufficient for the preliminary evaluation of software tools in a safety project. A precise classification of the tools through an intensive examination of the application scenarios (use cases, also known as tool operational requirements) within the process and for each individual project is a recommended approach. This allows you to demonstrate, if necessary in the event of liability claims, that you have acted in accordance with the state of the art and scientific knowledge.
As a software tool manufacturer, it is crucial during tool qualification to be authentic, to transparently present development and testing processes to the customer, to implement and maintain our own professional test automation, and ultimately to obtain certification confirming that the software tools are qualifyable according to functional safety standards. This last point is beneficial for the customer because it confirms the fundamental suitability of a software tool for use in a safety context.
Bibliography and list of sources
[1] Stefan Kriso, ISO 26262 – Quo vadis?, Center of Competence „Functional Safety“, Robert Bosch GmbH, Abstatt, 2012
[2] Stephen A. Jacklin, Certification of Safety-Critical Software Under DO-178C and DO-278A, NASA Ames Research Center, Moffett Field, CA, 94035
[3] DO-330, Software Tool Qualification Considerations, RTCA.org, 2011
[4] Frédéric Pothon, DO-330/ED-215 Benefits of the New Tool Qualification Document, ACG Solutions, 2012
[5] Basic Knowledge of Medical Software, dpuntk.verlag, 2nd edition, 2015
[6] Oscar Slotosch, Martin Wildmoser, Jan Philipps, Reinhard Jeschull – Validas AG, Rafael Zahlmann – Infineon, ISO 26262 – Tool Chain Analysis Reduces Tool Qualification Costs, Munich, 2012
[7] Oscar Slotosch, Model-Based Tool Qualification, The Roadmap of Eclipse towards Tool Qualification, Validas AG, Springer-Verlag, 2012
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
