EN 
DE From testable requirement to oracle
Authors: Ralf Gerlich, Rainer Gerlich, Dr. Rainer Gerlich BSSE System and Software Engineering
Contribution – Embedded Software Engineering Congress 2018
Manual Requirements tests are complex: The input data must cover the requirements, and observed output data must be checked for compatibility with the requirements. Test cases can also be automatically generated from test models, but these must first be created manually. In contrast, the approach presented here uses simpler forms of requirements formalization to simplify the test data generated during automatic processes. Robustness tests with massive stimulation to generate, map to requirements, and check the results for correctness.
Status of requirements verification through functional tests
Image 1 (see PDFThis illustrates the classic approach to verifying requirements through functional tests (unit tests) and manually created test cases. Based on requirements, source code is written, and a relationship between requirements and functions is manually established. Test cases for the functional test are derived from the requirements. Input data and expected results are transferred into test scripts that execute the test and evaluate it as successful or failed.
Model-driven approaches for requirements testing
In recent years, various approaches have been developed to derive requirements tests from models. This involves modeling the desired behavior and annotating it to determine requirement coverage and verify requirement fulfillment [1]. Test sequences and data are then derived from these models using random walks or similar methods and manually or automatically converted into test scripts.
A specification model that can be used as a test model for such a method with minimal modification is ideal for this approach. If this is not the case, such a model must first be created and manually compared with the specification – which is usually only available in text form.
Automated robustness tests
Automated robustness tests – sometimes also referred to as "fuzzing" – are used to test the robustness of a component against unwanted inputs [2]. For this purpose, a component is stimulated with random data, and anomalies during execution are observed. This can sometimes also identify functional weaknesses in the component. The simple method of test data generation enables a high test throughput – a massive Stimulation.
However, the results of these tests do not allow for any conclusions to be drawn about requirement fulfillment or coverage. A mapping of the test inputs to the requirements is lacking.
Automated requirements-based testing
The goal of automated requirements-based testing at the code level is to automatically establish a correlation between requirements and test cases, and simultaneously determine which functions in the code are affected by which requirement. The test data should be generated and applied automatically, thus drastically reducing manual effort.
The chosen approach uses three illustrations[3]:
- from input data to requirements (requirements coverage),
- from requirements to functions, and
- of requirements to Oracles.
This principle is shown in Figure 2 (see PDF) is shown. Machine-readable requirements are integrated into the functional test and evaluated during massive stimulation. This makes it possible to determine both the requirement coverage and the fulfillment of the requirements for the executed test cases. In case of non-fulfillment, the corresponding test vector, consisting of input and output data, describes a counterexample.
The latter is achieved through oracles. An oracle is an executable function that, given an input and an observed output, provides a prediction. pass/fail/don't know-Verdict is generated. It is possible that individual oracles only make decisions for specific real subsets of the input space. Ideally, however, the oracles together cover the entire input space.
Image 3 (see PDFThe diagram shows further details of the automated process. Through massive stimulation, the input space is scanned using a predefined number of test vectors. The oracle is applied to each test vector, and the result is recorded. In this way, counterexamples can be found.
There is generally a many-to-many relationship between requirements and functions. For example, certain requirements, such as those relating to computational accuracy, can apply to multiple functions. Likewise, a single function can implement multiple requirements.
To map requirements and functions, the relevant elements—data objects, structures, parameters—must be extracted from the requirement text and assigned to their corresponding code elements. This can be achieved using naming conventions that translate requirements names into code names or use them directly. Oracle-requirement and requirement-function relationships allow for mapping oracles to the functions to which they must be applied, thus avoiding time-consuming manual mapping.
In Figure 4 (see PDFThe logical flow is depicted within the context of hierarchical requirements. Requirements are detailed top-down to a level where they can finally be implemented in code. Only at this level does it make sense to define oracles. Requirements at higher levels of abstraction are typically not suitable for verification through code-level tests or unit tests.
Massive stimulation and derivation of test vectors
For each function, a test environment is automatically created, which is then used to stimulate the function with data from the range of values of the function parameters – including global variables [2]. Special cases such as -1 or 0 for integer parameters are also taken into account. Other methods for targeted coverage of specific program parts are also used. Furthermore, invalid values can be injected and parts of the code can be selectively added, for example to simulate memory shortages, etc.
From the massive set of test inputs and observed outputs thus analyzed, interesting cases are selected for generating regression test suites. These are inputs that contribute to coverage, trigger exceptions, or meet or violate the specifications of an oracle. These suites can then be re-executed using external test management tools – such as Cantata or VectorCAST – and their results evaluated.
Further analysis is possible: Input data that covers code but cannot be assigned to any requirement or oracle indicates missing requirements or unnecessary code. If the precondition for an oracle is never met in the input space of all functions to be tested, it is an uncovered requirement.
The oracle approach
The oracles in this approach are represented as temporal implication relationships: If Condition A must be true for the input before the call is made., then Condition B must hold true for input and output after the call (Figure 5, see PDFIf condition A is not met, condition B is not evaluated and the oracle delivers a don't know as a result.
This structure also allows the use of a tautology – an expression that is always true – as a precondition. In this case, condition B must be satisfied for every test vector. For example, this allows for the implementation of requirements regarding value range restrictions for the results.
Known dependencies between requirements allow the result to be extrapolated to higher-level requirements. At each level, it can therefore be determined which functions contribute to a positive or negative outcome.
Verification via massive stimulation relies on a very large, but still finite, number of test vectors; that is, a complete analysis of the state space is generally not possible. However, the number of automatically generated stimuli far exceeds the number of manually generated cases, which is particularly relevant for finding counterexamples.
Examples
See PDF.
Notation of requirements
Currently, most requirements are described as free text, which cannot be processed by machines. An analysis of such requirements revealed that they are often incomplete, ambiguous, or inconsistent, and therefore unsuitable for this method. During manual verification, these weaknesses then have to be compensated for through creativity.
Therefore, the use of a suitable notation is required. This may also have a format more suitable for the user than the oracle notation. However, it must be possible to convert it – automatically – into the oracle notation.
Requirements that are already in a formalized table format are well suited for this.
Quality assurance of the oracles
Like any code, oracles are subject to quality assurance. Errors in the oracles can otherwise lead to errors in the software being overlooked. Since an oracle can be applied to large subsets of the input set, it is even possible that errors with significant consequences will go undetected if the oracle is incorrect. However, this risk also exists when manually creating test scripts or test code. In contrast, measures to prevent errors can be implemented when automatically extracting oracles.
outlook
Since no machine-readable requirements were provided, the current implementation is based on manually implemented oracles in C code to demonstrate feasibility and benefits.
Future work aims to identify a more abstract, yet user-friendly notation. This will involve analyzing text-based requirements and translating them into a suitable notation that can be automatically translated into oracles. Close collaboration with users will be essential for this process.
thanksgiving
The content of this article is a result of the project “Automated Source-code-based Testing, Continued” for the European Space Agency (ESA) (ESA Contract No. 4000116014) within the framework of the General Support Technology Programme (GSTP). The budget was provided by the German Federal Ministry for Economic Affairs and Energy (BMWi) through the Space Administration of the German Aerospace Center (DLR). We thank our Technical Officer, Maria Hernek (ESA), for her support of our work and for suggesting the use of massive stimulation for requirements-based testing.
Bibliography and list of sources
[1] H.-J. Herpel, G. Willich, J. Li, J. Xie, B. Johansen, K. Kvinnesland, S. Krueger, P. Barrios: “MATTS – A step towards Model Based Testing”, Eurospace Symposium DASIA'2016 ’Data Systems in Aerospace“, May 10-12, 2016, Tallinn, Estonia
[2] R. Gerlich, R. Gerlich, M. Prochazka, K. Kvinnesland, B. Johansen: “A Case Study on Automated Source-Code-Based Testing Methods”, Eurospace Symposium DASIA'2013 ’DAta Systems in Aerospace“, May 14-16, 2013, Porto, Portugal
[3] R. Gerlich, R. Gerlich, M. Hernek, J. Ramachandran, A. Pascoe, G. Johnson: “Challenges Regarding Automation of Requirements-based Testing”, Eurospace Symposium DASIA'2017 ’DAta Systems in Aerospace“, May 30 – June 1, 2017, Gothenburg, Sweden
authors
Dr. rer. nat. Dipl.-Inf. Ralf Gerlich He began his professional software development career in the aerospace industry 20 years ago, initially focusing on systems programming. Now, his work primarily involves software verification methods at all levels of software development (constructive, organizational, and analytical), both manually and using tools, as well as fully automated methods. His interests encompass both the theoretical-mathematical and the practical-pragmatic aspects. He has been with BSSE System and Software Engineering since the company's founding.
Dr. rer. nat. Dipl.-Phys. Rainer Gerlich He has over 35 years of experience in industrial software engineering, including more than 30 years in the aerospace industry, with a focus on automation in the software lifecycle, requirements engineering, verification, automated testing, embedded systems, safety-critical applications, project management, and tool development. He has been the owner of BSSE System and Software Engineering since 1996.
Testing, Quality & Debugging – Our Training & Coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topics of testing, quality & debugging.
Training & coaching on the other topics in our portfolio can be found here. here.
Testing, Quality & Debug – Expertise
Valuable expertise on the topics of testing, quality & debugging is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
