EN 
DE Measuring code coverage during penetration tests
Author: Dr. Sabine Poehler, Verifysoft Technology GmbH
Contribution – Embedded Software Engineering Congress 2018
In the field of safety-critical software development, measuring code coverage as part of testing has long been a standard tool. It is required by common safety standards. A more recent application for coverage analysis is its use during penetration testing.
As part of a bachelor's thesis, we are investigating how the evaluation of penetration tests can be simplified by measuring code coverage. At the same time, this parallel analysis allows us to monitor the quality of the penetration test itself.
Code Coverage
To demonstrate that software has been adequately tested, various code coverage measures are used, e.g.:
- Function Coverage: every function was called,
- Statement Coverage: every instruction was executed,
- Decision Coverage: Every decision, e.g. in an if-statement, was evaluated as true and as false.
In the field of safety-critical software development, there are other relevant coverage measures, the highest level being Modified Condition/Decision Coverage (MC/DC), which, however, are not the focus of our investigation.
See Fig. 1 (PDFTestwell CTC++ Coverage Report – Decision and Statement Coverage per File
Penetration tests
Software that is accessible "from the outside" in any way is at risk from criminal attackers.
Software companies that develop business-critical web applications therefore typically conduct penetration tests as part of their quality assurance – or have them conducted by third parties. This allows them to demonstrate to their own customers or auditors that the software blocks defined attacks.
When security vulnerabilities are discovered, the subsequent analysis is not always easy for the development department. With a self-conducted penetration test, for example using suitable tools, there is still a certain degree of control over the process – although even here a high level of detailed understanding of how penetration tests work in general and of the specific tool used is required.
Another aspect comes into play when third parties conduct the penetration test: There is initially little control over exactly what was tested and when. Furthermore, if the testing company is not commissioned by the client itself, but by another party, then direct collaboration is not necessarily constructive or based on trust. In such cases, analyzing the submitted report creates a significant workload for the development department. If actual or perceived critical vulnerabilities are found, then the pressure to address them, both in terms of time and content, is considerable.
Technical, organizational, and human challenges are therefore closely intertwined in this security issue. Since embedded software is now exposed to numerous external attack vectors, penetration testing will foreseeably play a similar role in the development and quality control of such software as it has long done for web applications.
Measuring code coverage during a penetration test
How can one help the developer or development department tasked with analyzing a penetration test?
Our idea is to test (or have tested) a version of the software that has been instrumented for measuring code coverage from the outset.
In addition to the penetration test report, a report on the accessed parts of the tested software is also automatically generated. This can be helpful in several ways:
Areas of the software that should absolutely not be accessed during the penetration test can be defined in advance. For example, if a penetration test is performed on an application requiring user login with an unlogged-in user, then this essentially includes all components of the software that have nothing to do with verifying the username and password. For such areas of the software, the classic coverage target is reversed: Instead of 100% coverage, 0% coverage is ideal.
On the other hand, if a vulnerability is discovered, the coverage report helps to trace the attacker's path through the software.
And last but not least, the quality of the penetration test itself is also analyzed: Coverage measurement reveals which parts of the software were actually targeted for attacks. This is helpful in each of the described implementation scenarios: When using a penetration tool in-house, its scope can be expanded – a contracted service provider is monitored to ensure the complete fulfillment of their task.
Exemplary project
This approach is currently being investigated practically (as of October 2018) within the framework of a bachelor's thesis in the following setup:
- The Domoticz home automation system
Numerous devices and sensors can be monitored and controlled with this open source system.
Technically, Domoticz is implemented in C++ at its core and has a web frontend as its user interface.
Domoticz serves as a test object for penetration tests and runs on a Raspberry Pi for this purpose. - Arachni
„The "Web Application Security Scanner Framework" is used to perform penetration tests via the Domoticz web frontend. - Further penetration tests
The plan includes the use of additional tools as well as in-house developed penetration tests. - Testwell CTC++
The code coverage is determined using the Code Coverage Analyzer Testwell CTC++.
In this setup, the basic ideas of combining penetration tests and coverage measurements are explored and their applicability is examined.
author
Sabine Poehler works at Verifysoft Technology GmbH as a product manager for the Testwell product line, particularly for the Testwell CTC++ code coverage analyzer. She is responsible for the strategic development of the Testwell tools and heads the support and development department.
Our training courses & coaching sessions
Do you want to bring yourself up to date with the latest technology?
Then find out more here Regarding training courses/seminars/workshops and individual coaching sessions offered by MircoConsult on the topic Quality, Safety & Security.
Training & coaching on the other topics in our portfolio can be found here. here.
Quality, Safety & Security – Expertise
Valuable expertise on the topics of quality, safety & security is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
