Metrics-driven process development in the context of future connected mobility

A framework for risk assessment of vehicle functions

Authors: Christopher Kugler, Stefan Kowalewski, Chair of Computer Science 11 – Embedded Software, RWTH Aachen University; German Baca Espinoza, Ralf Maquet, Jiju Vadakkepattath, Dirk Macke, Johannes Richenhagen, FEV Europe GmbH

Contribution – Embedded Software Engineering Congress 2017

Mobility is undergoing disruptive change: New areas such as vehicle connectivity and highly intelligent driver assistance systems place high demands on the quality of software products amidst increasing complexity. Consequently, existing development processes must be continuously improved to meet project time and budget constraints. This work defines qualitative, metric-based milestones designed to ensure a smooth transition between development phases. A framework for risk assessment of vehicle functions is proposed, which can be used to control the scope of testing in line with risk-based testing principles. Risk factors are identified that permit application in the context of connected mobility. Previously defined metrics from early development phases are incorporated into the assessment, ensuring a minimum level of objectivity. The framework is evaluated using a case study, and its operational benefits are demonstrated.

1. Introduction

High complexity and quality requirements, coupled with comparatively tight cost and time budgets, pose challenges in the development of automotive control systems. In particular, the increasing number of requirements for advanced driver assistance systems (ADAS) and automated driving (AD) necessitates technological innovation and an agile development process. Especially in the interdisciplinary field of software development, networked control units introduce new areas of concern, such as cybersecurity and robustness against external interference.

Automotive service providers play a special role, as they must develop methods to respond to rapidly changing project parameters. This often necessitates the use of external processes and tools. Consequently, defining and implementing suitable processes within the service provider's organization—processes that can help make the development hub as efficient as possible—becomes a significant challenge.

A proven practice for enabling process flexibility and developing competitive products is the use of quality gates in the development process. The basic idea is comparable to the principle of a factory (Fig. 1 in the P DF), where a subsequent processing step only starts after the defined quality of the workpiece has been ensured.

To define suitable quality thresholds in the context of automotive software development, a variety of metrics are available, but their significance depends on company-specific structures and goals.

The following chapters first introduce exemplary quality constraints for early development phases—requirements definition and architecture development—and discuss their benefits. Subsequently, a framework for risk assessment of vehicle functions is proposed, which builds on these metrics and can make a significant contribution to the practice of risk-based quality assurance.

2. Metrics-driven software processes

Metrics-driven processes are an effective means of managing typical software development challenges. The following process requirements, which contribute to high value, should be highlighted:

1. Agile methods in process execution with well-defined interfaces to customer processes
2. Measurable and assessable process results
3. Risk-aware management of processes (robustness against external influences, flexibility in the process to set (changing) priorities)
4. Appropriate reduction of the complexity of the present task (for example, by dividing it into several steps)

In the context of quality barriers, the main focus here is on point 2), while Chapter 3, with its risk-based framework, essentially addresses point 3).

To ensure high software quality, it is advisable to evaluate artifacts from early development phases according to strict quality criteria in order to prevent potential future problems. This applies particularly to functional and non-functional requirements that serve as the basis for software development. Another key aspect is the structural design of the system within the framework of a software architecture and the associated decomposition into software components, which significantly influences the necessary interfaces and the functional logic to be implemented.

2.1 Quality barriers for requirements

Quality barriers can be implemented using meaningful metrics, so-called Key Performance Indicators (KPIs). Numerous KPIs have been identified and discussed in the literature on the specification and development of requirements (see [SM07], [He10]). Based on these, [No17] considers three essential aspects of requirements that allow for a qualitative assessment of emerging requirements documents and serve as the basis for further discussion in this article.

1. Requirements readiness – for example, rated via:

a. Degree of fulfillment of the SOPHIST requirements template [Ge08]
b. Requirements complexity

2. Requirements coverage – essentially evaluated based on:

a. Degree of coverage of customer requirements (alternatively: convergence rate between specifications and requirements)

3. Requirements test depth – Not the focus here, as its origin lies in later development phases

Detailed definitions and further explanations of the metrics mentioned can be found in [No. 17]. Fig. 2 (see. PDFThis is illustrated graphically. The metrics can be collected regularly for requirements documents and help to assess their quality. Starting further development steps based on these requirements only makes sense if a predefined threshold is exceeded in the defined KPIs.

2.2 Quality barriers for architectural artifacts

A key element of early software development is the software architecture and its related artifacts (interface definition, software components). The challenge here is to minimize the complexity of the architecture and its components to ensure maintainability, testability, and analyzability. In [Ve17], various complexity metrics for software components are discussed and compared in this regard.

The focus is on the number of external and internal interfaces of a component and the number of functions implemented within each component. This information is ultimately aggregated into a KPI called "Software Component Complexity" (see Fig. 3, p. 1). PDFDuring development, a threshold for the maximum tolerated complexity can now be defined based on company-specific parameters. Architectural elements that exceed this threshold must be examined for modularity and further separated.

3 Metric-based framework for risk assessment of vehicle functions in the context of quality assurance

The basic idea of risk-based quality assurance is the recognition that, in reality, under given project conditions, 100% verification and validation of the product is not possible [PR12], [Si11]. Therefore, a prioritization scheme is used that is essentially based on reference-specific risk values. Fig. 4 (see. PDF) describes this procedure.

Approaches to objectifying the risk assessment process through the use of specific key performance indicators (KPIs) and metrics are examined in [No00], [Be99], and [SM07]. Benlarbi et al. focus primarily on code metrics [Be99], whereas Nogueira et al. examine project risks exclusively [No00], which are not the focus of this paper. The risks considered here are product-related and primarily address the technical characteristics of the unit under consideration. An approach similar to that discussed in this paper was presented by Stallbaum et al. in [SM07]: This approach uses requirements metrics to automate the risk assessment of requirements documents, such as use case definitions. There are three key differences compared to the framework presented here:

1. Different unit of analysis: focus on system-wide functions instead of individual use cases.
2. Inclusion of architectural metrics as an additional evaluation factor
3. Inclusion of qualitative evaluation factors (expert assessment)

The aim of the framework described here is to derive function-specific risk values based on the metrics presented in Chapter 2, as well as other ad-hoc defined influencing factors. As a first step, the properties with regard to which a function is to be examined and evaluated are defined. Subsequently, evaluation criteria for these properties are established and appropriate value ranges are defined.

Chapter 3.3 provides an example of an evaluation of the framework using functions from the field of networked mobility (Vehicle-to-Everything, V2X).

3.1 Evaluation scheme for functions

The software quality model of ISO 25010 can be used to define the properties of a function that are relevant for quality assurance (see Fig. 5 in the document). PDF).

Based on these 8 quality attributes, the risk value of a function can be characterized by the following properties:

1. Functional complexity (Functional Suitability)
2. Performance relevance (Performance Efficiency)
3. Usability relevance (Usability)
4. Security relevance (Security, Reliability)
5. Interoperability relevance (Compatibility, Portability)

For these properties, the higher the respective risk value, the higher the risk associated with the function. The methodological approach is therefore to evaluate functions independently of one another with respect to these properties in order to draw conclusions about the risk value of the function as a whole. In the area of quality assurance, the individual assessments can be used to validate the corresponding properties using appropriate testing methods and to set specific priorities. For example, ergonomic tests are only necessary if there is also relevance to usability.

3.2 Derivation of specific risk factors

To evaluate the properties introduced in section 3.1 in a function-specific manner, evaluation criteria and an aggregation scheme must be defined. The metrics introduced in Chapter 2 are used, among other things, for the evaluation criteria. Table 1 illustrates the composition based on the essential evaluation criteria.

The rating scales for the individual criteria are chosen to be binary where possible; in individual cases, key figure-specific scales are used, which are subsequently normalized to a value range of 0 to 1. The weighted arithmetic mean is used as the aggregation scheme.

Table 1: Exemplary evaluation criteria for functional properties

3.3 Evaluation based on V2X functions

To evaluate the risk assessment scheme for a function, the necessary framework conditions for two V2X functions are defined as examples, and the results for each criterion are compared with regard to plausibility. A detailed description of the functions under consideration follows:

Function A: Firmware over the Air (FOTA)
To enable software updates of vehicle control units without a workshop visit, selected control units can be updated remotely outside the workshop via a communication protocol after suitable authentication by the customer.
Key requirements for this function are:

1. Robustness against interruptions (e.g. communication or power failure) and external manipulation during the flashing process.
2. Verification of the authenticity of the software to be installed before the flashing process starts.
3. Verification of a successful flash operation via a checksum (or similar method) and feedback to the customer. In case of a failed flash operation, a rollback to the old software is possible.
4. The flashing process must be compatible with different types of control units. This requires the implementation of a standardized flashing protocol.

Function B: Cooperative Collision Avoidance (CoCA)
To improve collision avoidance, vehicles can exchange lateral and longitudinal control parameters with each other via a communication standard, such as 5G mobile communications, and evaluate them in real time. A warning message is sent to the driver, and depending on the situation, braking or steering intervention occurs when hazardous situations are detected. Key requirements for this function are:

1. Vehicle-to-vehicle (V2V) communication in real time; unidirectional latency of less than 10ms [3G16].
2. Robustness against external interference, including data manipulation (implementable, for example, via encrypted communication).
3. Clear identification of the vehicles against each other.
4. 99,999%ige reliability in safety-relevant driving maneuvers [3G16].

Table 2 shows the evaluation of both functions based on the defined evaluation criteria.

Table 2: Exemplary evaluation for V2X functions

When calculating the overall risk value across all properties for the entire function, a value of ( is obtained.0.58+0+1+0.83+1)/5=0.68 for function A, and accordingly (0.75+1+0+0.92+1)/5=0.73 for function B.

However, at least in the context of quality assurance, it makes more sense to consider the individual properties separately. For example, the evaluations show that ergonomic tests are important for functionality. FOTA are useful – due to the necessary authentication by the customer and subsequent feedback upon completion of the flashing process – in the function CoCA However, their scope may be significantly limited. Furthermore, the higher values of CoCA Given the functional complexity and safety relevance of the function, it is evident that the test intensity and therefore the test scope for this function must be comparatively higher.

4 Summary and Outlook

This paper first discussed the need for metrics-driven processes and then defined suitable quality thresholds for requirements and architecture development. Subsequently, a framework for the risk assessment of vehicle functions was proposed, which incorporates the previously defined metrics into the risk assessment. This results in greater explanatory power and a degree of objectivity in the risk assessment process. Based on the risk values, the scope and focus of quality assurance measures can then be meaningfully estimated and managed.

The initial results are promising, and the methodology outlined here will be further evaluated in the future. To this end, company-specific risk thresholds will be defined as part of quality assurance, which will directly influence the scope and intensity of testing.

Furthermore, plans are in place to incorporate metrics from later software development phases, such as system verification, into the risk assessment, thereby enabling an iterative approach throughout the project. Potential influencing factors include, for example, function-specific error rates and the stability of requirements – measured via incoming change requests.

bibliography

[SM07] Stallbaum, H.; Metzger, A.: Employing Requirements Metrics for Automating Early Risk Assessment. Proceedings of MeReP07, 2007.

[He10] Heidenreich, M.: Metrics and tool support for requirements verification. OBJEKTspektrum, issue RE, 2010

[No17] Nowack, J. et. al.: Continuous Process for the Validation of Transmission Controls, CTI Symposium Shanghai, To appear

[Ge08] Geltinger, G. et al.: The SOPHIST Rulebook, DHBW Ravensburg, 2008

[Ve17] Venkitachalam, H. et. al.: Metric-based Evaluation of Powertrain Software Architecture, SAE Int. J. Passeng. Cars – Electron. Electr. System., pp. 194-208, 2017

[PR12] Veenendaal, Erik van: The PRISMA Approach, Uitgeverij Tutein Nolthenius, 2012

[Be99] Benlarbi, S. et. al.: Issues in Validating Object-Oriented Metrics for Early Risk Prediction. Proceedings of 10^th ISSRE, International Symposium on Software Reliability Engineering, Flordia, USA, 1999

[No00] Nogueira, JC et. al. : A Formal Risk Assessment Model for Software Evolution. Proceedings of the 22nd ICSE, International Conference on Software Engineering, 2000

[Si11] Siller, A. et al.: Systematically to test cases – Requirements-based strategy for testing and validation of electrical/electronic systems. Automotive 2011

[IS11] ISO/IEC 25010:2011: System and software quality models, https://www.iso.org/iso [October 2017]

[3G16] 3GPP TR 22.886 V15.0.0, Study on enhancement of 3GPP Support for 5G V2X Services, Release 15, 2016

Download the article as a PDF

---

Software Engineering Management – our training courses & coaching sessions

Do you want to bring yourself up to date with the latest technology?

Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of Software Engineering Management / process, project and product management.

Training & coaching on the other topics in our portfolio can be found here. here.

---

Software Engineering Management – Expertise

Valuable expertise in software engineering management / process, project and product management is available. here Available for you to download free of charge.

To the specialist information

You can find expertise on other topics in our portfolio here. here.