EN 
DE Reuse and inheritance of test cases
Author: Michael Wittner, Razorcat Development
Contribution – Embedded Software Engineering Congress 2017
The challenge of testing software variants lies in the fact that each variant must be fully tested. The following presents a method for reusing and inheriting variant tests. By defining base tests that are inherited by variant tests, redundant work can be avoided. With each application change, the tests only need to be maintained in one place.
Problem statement
Safety-critical standards in various industries, such as ISO 26262 for automotive engineering, IEC 62304 for medical technology, and IEC 61511 (a more general standard applicable to all industries), require complete code coverage during testing. This means that every software variant must be fully tested. In practice, this is often done by copying the tests of one variant and adapting them to the other. New requirements or software changes increase the cost of such variant testing due to the redundant changes that need to be made in all variants. Besides the significant effort involved in maintaining and expanding such tests, errors can easily creep in, for example, through copy and paste, which can ultimately lead to safety-critical errors in the application going undetected.
What is a variant?
There are various ways to create software variants (example C/C++ source code):
- Enabling/disabling code segments using defines
- Generation of code variants using tools (e.g., from MATLAB)
- Copying, renaming, and modifying source files
- Running the same sources on different hardware platforms (for high security requirements)
- Same code with different application values
A software variant is defined by a specific configuration of a software module (for example, a C source file). This variant does not necessarily have to be functional; it can also be an abstract variant that will never function in a finished device in this form. Only through specific settings (usually via defines) do the various actually usable software variants emerge from an abstract base variant.
Test objective: Code coverage
To achieve complete code coverage for each variant, the measurement result of the variant-specific code coverage could be added to the measurement result of the shared code. The simple example in Figure 1 (see below) illustrates this. PDFThis is intended to illustrate why such a simple tally of code coverage cannot work and why programming errors might go undetected. The function shown has a common part in lines 19-23 and an additional part for variant 1 in lines 15-17. In variant 1, another value is added before the variable "level" is checked.
The common section in lines 19-23 could be adequately tested with two test cases. Variant 1 does not add an additional program branch, so no further test case is necessary for complete code coverage. A simple test case with "supplementary_level > MINIMUM" would suffice as a functional test for this variant. Summing the coverage measurement results would yield complete code coverage. However, this illustrates that a simple programming error, such as the missing addition operator in line 16 of Figure 2 (see...), can lead to... PDF) depicted, would not be discovered.
It is therefore not enough to test individual parts of a variant code assembled using defines and to add up the measurement results of the code coverage: Each code variant should be seen as an independent program, since hidden or added parts can influence the common parts.
High maintenance effort required for changes
It's in the nature of software variants that they possess similar functionalities. Therefore, the necessary tests will also be relatively similar. It's worthwhile considering variant testing, especially when the differences between variants are relatively minor. The testing effort can be significantly reduced through the targeted reuse of test cases.
The simplest approach would be to first develop a set of basic tests that apply more or less to all variants. These tests are then copied for each variant and modified as needed. This avoids creating new tests, but at the same time results in a huge number of almost identical tests that require significant effort to maintain whenever the software is changed.
Solution approach
The following example presents a method for creating and maintaining variant tests. The example deals with a function for displaying the fuel level of different vehicle variants (cars and trucks). A further complication is that the "truck" variants may be equipped with an auxiliary tank, the level of which also needs to be taken into account.
Example function
As an example, consider a function that returns a status based on the fuel level of a vehicle's tank. The specification is shown in Figure 3 (see below). PDFThe function, presented graphically, should return a warning or alarm when a defined level is undershot. Otherwise, it returns the value "Normal".
A simple implementation of this function could be as shown in Figure 4 (see below). PDF) as displayed. The level of an auxiliary tank is optionally included in the calculation via the variant configuration (#define TRUCK).
Analysis of the software hierarchy
The number of software variants to be tested is automatically determined by all possible configurations of the software. For testing purposes, it's crucial to consider whether a variant (for example, the base configuration) can actually be tested: whether it's executable software or merely a base configuration that doesn't yet constitute a functional unit on its own. Abstract tests can sometimes be defined for such abstract variants. However, executable tests only emerge through the further implementation of these tests for a specific functional variant.
On the other hand, defining a variant hierarchy primarily serves to improve clarity in highly nested software variants. In the presented method, the variants are created in a variant tree, which can be multi-level and reflects the variant structure of the software. This tree serves as a guide for determining which tests should be created at which variant level.
The example (see Figure 5) PDFThe diagram shows a hierarchy of variants for different vehicles, with trucks available in versions with either a permanently attached or an optionally switchable auxiliary fuel tank. The variant "Truck" in this case could be an abstract configuration or a specific version of a vehicle.
Definition of variant tests
Tests can be divided into two different types:
- Basic tests
- Variant tests
The basic tests relate to abstract functionalities, such as the basic configuration of a software module. At this level, all potential test cases for the variants derived from this basic module can already be defined. Initial test data can also be defined for basic tests, but it doesn't need to be complete (because you don't intend to execute it yet anyway). At the top level of a variant tree, the possible test cases can be defined, for example, using a classification tree. The test specification below takes into account all possible variant configurations in our example.
The test specification describes the necessary tests that are now inherited in the variant tree: The inherited tests can be modified, hidden, or supplemented with specific tests in each variant. Figure 7 (see below) illustrates this. PDFFor example, all tests that do not relate to the "passenger car" variant are hidden.
Each overarching variant test therefore only needs to be created once and only needs to be maintained in one place when there is a new requirement or change to the application.
Rules for the inheritance of test cases
In our example, all possible test cases were defined at the top level. However, for some variants (for example, "car"), only a few of these test cases are relevant: The remaining test cases must be hidden for this variant. Additionally, it might be useful to add further tests at the level of a subordinate variant. The following operations are therefore necessary for test case inheritance:
- Modifying inherited test data
- Deleting or hiding inherited test cases
- Adding additional test cases
At the test data level, a distinction must also be made between inherited and locally defined values. During variant synchronization, all inherited values are updated. This results in the following states for the value of a variable in a variant test:
- Value was inherited
- The value was inherited and transferred.
- The value was defined locally for this variant test.
These values can be represented by a color code as shown in Figure 8 (see Figure 8). PDF) shown to be easily distinguishable: The light blue values were inherited, the purple value was overwritten in the "Truck" variant.
In the basic tests, most values were already assigned in the classification tree within the test case specification; therefore, they appear grayed out. In test case 2.1, a value of "40" was already entered for the basic tests, which corresponds to a normal fill level for an assumed 80-liter tank. This value must be significantly increased in the "Truck" variant, assuming a 1000-liter tank. Therefore, the value of "level" in this variant was overwritten with "500".
Unambiguous identification of test cases
For unambiguous identification, each base test case is assigned a Universal Unique Identifier (UUID). These UUIDs are generated globally and temporally uniquely. When a test case is inherited by a variant, the inherited and adapted test case for the variant is ultimately the same (base) test case. During a test review, the inherited test cases can therefore be clearly compared with the base tests or the tests of a different variant.
The UUID becomes particularly important when synchronizing test cases: If base test cases are moved, deleted, or modified, the corresponding inherited test case must be located and updated during synchronization. Inherited test cases that no longer exist must be deleted. Assigning UUIDs also enables geographically distributed work on variant tests, as all test cases are uniquely defined and the tests can therefore be easily merged and updated.
Results
For the variant example shown, the following strategy was used to define tests:
- Adoption of variant definitions from the application design
- Definition of all possible top-level test cases for all variants
- Hiding the unnecessary test cases in each variant
- Supplementing or implementing the tests separately in each variant.
The advantage of this approach lies in the centralized specification of test cases within a single classification tree. This improves clarity and provides a complete overview of all tests during a review. Hiding unnecessary test cases in variants is relatively simple and simultaneously forces the test engineer to consider which test case is actually required for their variant. Alternative approaches, as described below, are also possible.
„Natural“ approach
For a test engineer, it is generally easier to create tests for a specific variant than to think about all variants simultaneously at an abstract level. Depending on the type of software being tested, it can therefore be useful to first test one variant completely and then transfer the generated test cases to another variant.
To utilize test case inheritance, a hierarchical definition of variants is required. Therefore, it would be necessary to transfer the test cases of the fully tested variant to a higher-level (base) variant and, if necessary, generalize them. Naturally, transferring the test data requires that at least the majority of the variant function's interface is also available at this higher level.
Variants in the classification tree
It would also be conceivable to filter parts of the classification tree according to variant. This would then generate a separate test case specification for each variant. The entire classification tree with all variants would still be managed and edited at the top level of the variant hierarchy. For test review, either the entire tree or the respective subtree of the variant could be viewed.
Sources
https://www.razorcat.com
author
Michael Wittner, a graduate computer scientist, has many years of experience in software development and testing. After studying computer science at the Technical University of Berlin, he worked as a research assistant at Daimler AG, developing test methods and tools. Since 1997, he has been the managing partner of Razorcat Development GmbH, the manufacturer of the unit test tools TESSY and CTE, the test management tool ITE, and the test specification language CCDL.
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
