EN 
DE How does one deal with emergent system properties?
Author: Dr. Thomas Liedtke, ICS AG
Contribution – Embedded Software Engineering Congress 2016
The increasing proportion of software-implemented requirements leads to a growing number of vulnerabilities and potential attack vectors to exploit weaknesses.
Safety risks are often determined using deductive (frequently mathematical) approaches, employing established methods and probabilities. Highly improbable risks, whose impact is therefore difficult to assess, can be disregarded due to their low probability of occurrence. Only rationally conceivable causes must be considered in risk analyses. In the worst-case scenario, an attempt is made to achieve a fail-safe state.
When considering security risks, even highly improbable threats and complex vulnerabilities must be included as potential threats. Due to the conflict of interest inherent in attacks, risk analysis must also consider inductive and random scenarios. Furthermore, potential security gaps arising from emergent system properties must be taken into account. Antifragility becomes a necessary property for preparedness.
Introduction
„"It's not about predicting the future, but about being prepared for the future."“ Pericles (490 – 429 BC)
The term "security" has multiple meanings in the German language:
- Safety, also known as functional safety, ensures that a system behaves in accordance with its expected functionality. It is about human safety.
- Security refers to maintaining functionality in the event of an attack, protecting a technical system, and upholding fundamental values from its environment (e.g., attackers).
Both the complexity of the requirements for technical systems (e.g., decentralization of controls, a wide variety of end devices and operating concepts to be supported, increasingly intelligent actuators and sensors, an increased number of interfaces, and ever more sophisticated services) and the complexity of the development processes (changing infrastructures, a wide variety of standards to be supported) are increasing rapidly.
While system architectures have traditionally been rather simple, consisting of a central control unit monitored by an independent control unit [IEC61508], components of a modern system are themselves responsible for safety (e.g., driver assistance systems in automobiles) [ISO26262]. Furthermore, products must be able to configure themselves openly and dynamically during operation and are no longer solely statically closed (keywords: Industry 4.0, batch size 1).
Safety vs. Security
Safety is defined as protection against (unintentional) malfunction (during intended use) and the absence of unacceptable risks. The actual functionality corresponds to the specified target functionality.
The standards to be applied require proven procedures for the development of safety-related systems:
- The V-model is specified as the procedural model.
- Depending on the criticality of the function to be implemented, a more or less strict role concept and/or separation of personnel must be fulfilled.
- The procedures for testing, validation and documentation are strictly hierarchical and adapted to the horizontal levels of the V-model.
- Test criteria are defined as structural or functional coverage criteria.
Procedures and methods in the safety process are often top-down oriented, mathematical, and require a great deal of discipline and diligence. The aim is to transform fragile systems into robust ones that, if necessary, will bring the system to a safe state in the event of an incident.
Security is defined as protection against (intentional) attacks (and accidents/disasters). Security means functional correctness in the face of active attacks.
The Stuxnet attack marked a turning point in the security world. The immense motivation, the scale of the attack, its high complexity, and the enormous effort involved were unprecedented. The ability to penetrate so deeply into supposedly proprietary systems was also unforeseen.
Security risks can be reduced, avoided (the system is restructured to eliminate the threat), accepted, or delegated. Unlike safety, security does not initially have a defined secure state. A secure system should always be kept running, and a denial of service should be avoided at all costs.
Since the primary protection goal for safety is always life and limb, security requires defining system-dependent protection goals. Depending on the protection goal, different design principles (such as "visibility and transparency") and design strategies (such as "data minimization") must be applied.
Conflicting objectives
Due to the conflict of interest between attackers and operators (which is absent in safety-related analyses), a whole range of additional issues need to be addressed in security risk and threat analysis. For example, implementation attacks and side-channel attacks (re-engineering) must be given much greater consideration. Security attacks are also frequently characterized by the use of significant resources and are carried out remotely (e.g., industrial espionage). Communication channels and external system interfaces represent particularly vulnerable security gaps.
When developing safety-related systems with safety and security requirements, a prioritization between orthogonal target values must be established [LiHo16]. Privacy (data protection) and KRITIS (security of critical infrastructures) can be considered as further dimensions (see Figure 1, PDF).
Methodologically, there are similar approaches, so security is also understood as a parallel management process to development. The "interweaving" of security and safety is the subject of many recent reports (e.g., [GGH+15]).
Traditional hazard and risk analysis starts with an identified hazard to determine its impact and probability of occurrence (e.g., [Sto09]). Corresponding risk acceptance criteria then determine further development measures. The next chapter will demonstrate that a different approach is necessary for security.
Antifragility and emergence
The IEC 62443 standard [IEC 62443-3] specifies a top-down decomposition for the risk assessment of a system. For this purpose, the system is divided into components (zones) and channels (conduits). The components and channels are then analyzed individually.
Emergent properties of complex systems require dealing with unknown system properties. Systems must be designed to be more than robust: they must be antifragile [Tal14].
Antifragility is defined as the opposite of fragility. Robustness, in the sense of a triad, is positioned between the two. Possible characteristics are listed in Table 1 (see PDF).
A number of examples demonstrate the advantages of antifragility and the desirability of this property for secure systems. Antifragility is more than simply hardening systems; it's about overcompensating and preparing them to withstand future challenges (attacks). Not all emerging vulnerabilities are known during development. They are often discovered over time (possibly through trial and error). This means that systems with security measures must be continuously updated and hardened.
When considering security analyses, the following fallacies, among others, should be avoided:
- What you can't see/understand doesn't exist (see Stuxnet, 911, …)
- Because rare events can be explained in retrospect, they give the impression that they were predicted (illusion of predictability).
Unlike fragility, antifragility is not measurable. The susceptibility of a porcelain plate to breakage compared to a plastic plate, or the health of an elderly person compared to a young person, can be assessed. However, how well a system is protected against accidents and rare (previously unknown) events is much more difficult to evaluate.
Since it is easier to deal with harmful consequences than to predict rare events, predictions and risk management need to be turned on their head.
A possible example of a risk analysis from the transportation sector is shown in Figure 2 (see PDF) shown.
In the final step, a detailed risk analysis is conducted against potential, already known threats. The sources used for vulnerabilities, weaknesses, and known attacks play an important role in this analysis.
Only antifragile systems can withstand the randomness and trial-and-error nature of attacks in the long run. A number of possible considerations regarding emergence from a security perspective therefore include, for example...
- Definition of possible countermeasures in the event that an attack has already been partially successful (e.g., a supposedly secure "outer" wall has been breached)
- Honeypots provide experience with attack scenarios; attackers often operate by trial and error.
- Utilize experience databases on known threats and vulnerabilities
- Use random-based testing methods: fuzzy logic tests, penetration tests, etc.
- Anticipating attacks (Intrusion Detection, Prevention)
- Sufficiently consider conflicts of interest (e.g., in implementation attacks, side-channel attacks, social engineering, etc.) in security analyses.
- The "dark side of the force" is organized. What is possible will eventually happen (existing security vulnerabilities are exploited, the motivation of attackers is high).
- Repeat risk and threat analyses, regularly harden systems
Summary
Established safety considerations and methods are insufficient to guarantee security. The factors of intent, conflict of interest, chance, trial and error, and the high motivation of attackers must be factored in much more heavily than in pure safety threat analyses.
To be prepared for the future, rigid boundaries (e.g., virus scanners on PCs) are insufficient. Dynamic defense strategies that constantly renew and update themselves must be implemented.
literature
- [GGH+15] Benjamin Glas, Carsten Gebauer, Jochen Hänger, Andreas Heyl, Jürgen Klarmann, Stefan Kriso, Priyamvadha Vembar, Philipp Wörz, „Integration Challenges“, Automotive Safety and Security Conference 2015
- [IEC-61508] Standard „Functional safety of electrical/electronic/programmable electronic safety-related systems“
- [IEC62443-3] „Security for industrial process measurement and control – Network and system security“
- [ISO26262] „Road vehicles – Functional safety“
- [LiHo16] Thomas Liedtke and Bernhard Hohlfeld, „How do safety and security fit together? The example of eCall.“ „Forum Safety & Security“; Conference Proceedings, July 6–7, 2016, Munich. ISBN 978-3-645-50168-2
- [Sto09] Ketil Stolen, SINTEF & UiO, „Security Analysis: The CORAS Approach“
- [Stuxnet] Wikipedia link (accessed 20.10.2016)
- [Tal14] Nassim Nicholas Taleb, "Antifragility: A Guide to a World We Don't Understand." Btb, 2014
Our training courses & coaching sessions
Do you want to bring yourself up to date with the latest technology?
Then find out more here for training courses/seminars/trainings/workshops and individual coaching sessions by
Training & coaching on the other topics in our portfolio can be found here. here.
Quality, Safety & Security – Expertise
Valuable expertise on the topics of quality, safety & security is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
