EN 
DE Use cases for machine learning in vehicles
Author: Dr. Eduard Kromer, iNTENCE automotive electronics GmbH
Contribution – Embedded Software Engineering Congress 2018
Terms like artificial intelligence, machine learning, and deep learning are on everyone's lips, especially because of concrete applications such as autonomous driving, speech recognition, and gesture recognition. What exactly are these technologies, and why is there so much hype? How does machine learning actually work? What are some interesting applications in vehicles? Are the possibilities being overestimated? Are the risks being exaggerated?
What is Artificial Intelligence / Machine Learning / Deep Learning?
„AI is the science and engineering of making intelligent machines.“
John McCarthy [1]
See Fig. 1 (PDF)
Artificial intelligence (AI) is a very broad field and also an umbrella term for technical approaches that deal with perception, reasoning, learning, and action—that is, intelligent actions attributed to humans [2]. Machine learning (ML) is a subdiscipline of AI and describes methods that enable machines to independently generate knowledge from experience, i.e., to learn from data, without requiring a formal set of rules. Classic applications of ML include recommendation systems from Amazon/Netflix or email spam filters. Deep learning, in turn, is a subfield of ML characterized by its ability to learn complex concepts using deep neural networks by building them up from simpler concepts [3].
How does machine learning / deep learning work?
Machine learning (ML) is the science and art of programming computers to learn from data. At its core, ML systems can be divided into the following non-mutually exclusive categories, see [4]:
- Supervised, unsupervised, semi-supervised, and reinforcement learning – Did monitoring take place during training?
- Online vs. Batch Learning – Can models learn incrementally and spontaneously?
- Instance-based vs. model-based learning – Is a predictive model being built?
In the following, we will focus solely on supervised and unsupervised learning. In supervised learning, the machine learning algorithm is provided with training data that is labeled, meaning a solution already exists for it. The algorithm then attempts to learn a classification (e.g., spam or not spam based on email text and sender) or a regression (e.g., rental price based on location and age of an apartment). In unsupervised learning, the training data is not labeled, and typical algorithms in this area attempt to find clusters within the data (e.g., similar users on a shopping portal) or to embed high-dimensional problems in a lower-dimensional space.
See Fig. 2 (PDF)
Neural networks (NNs) learn complex concepts by building them from simpler concepts through hidden layers and with the help of large training datasets.
See Fig. 3 (PDF)
In so-called Convolutional Neural Networks (CNNs), this process can be visualized using the example of facial recognition as follows:
See Fig. 4 (PDF)
When training neural networks with labeled data, information initially flows as input from the input layer through the hidden layers to the output layer, resulting in an output (forward propagation). A cost function determines the distance between this output and the actual label using a specific metric. The backpropagation algorithm then allows the information from the cost function to flow back through the network to calculate its gradient. Finally, an optimization algorithm, such as stochastic gradient descent, can use this gradient to determine the next optimization step. This process is (ideally) repeated until no further numerical improvement is achieved.
AI use cases in vehicles and reasons for automation
Vehicle automation is necessary and unavoidable for many different reasons. Among other things, it increases road safety, makes shared mobility more efficient and affordable, and thus reduces traffic congestion. Furthermore, it increases comfort for passengers and allows them to use their travel time more effectively.
Modern vehicles are equipped with numerous sensors, and this number will continue to grow as we move towards full automation. Artificial intelligence is key to understanding sensor data from vehicles and, consequently, real-world traffic. Traffic is simply too complex to be captured by a formal set of rules. Therefore, AI is the key to autonomous driving.
On the path to autonomy levels 4 (high automation) and 5 (full automation), obstacles still need to be overcome, but even now, for autonomy levels 2 (partial automation) and 3 (conditional automation), deep learning is demonstrating its great potential. For outward-facing sensors in cars, neural networks can recognize traffic objects (road users, traffic signs, etc.) and lane markings, and even perform complete instance segmentations.
See Figures 5 and 6 (PDF)
Furthermore, they are able to detect movements and directions of movement (e.g., of pedestrians) and thus assist in maneuver planning.
In addition to information about what's happening outside the vehicle, information about the driver is also collected, thus increasing both safety and driver comfort. By recognizing the driver even before they enter the vehicle (e.g., via facial recognition using CNNs), the vehicle can make individual, driver-specific adjustments. For example, seat, mirror, and climate control settings, among many others, can be adjusted as soon as the driver is detected and approaches the car. If the sensors detect that the driver is approaching the trunk and carrying something, it could be opened automatically.
Inside the vehicle, neural networks ensure that head movements, gaze direction and specifically the driver's eyes can be detected and tracked in order to monitor the driver's fatigue and attention, thus increasing the safety of all road users.
See Fig. 7 (PDF)
Neural networks in the form of speech and/or gesture recognition ensure communication with the vehicle, providing both an improved user experience and additional safety, as the driver can concentrate better on the traffic with high-quality solutions in this area than with classic operating concepts.
Finally, the car's sensors and object and motion detection via neural networks ensure a safe exit for the driver at their destination. Cyclists or other vehicles that could endanger the driver during exit, or that would themselves be at risk, are detected in time, and the driver is warned accordingly.
Obstacles on the road to full automation
Despite the many breakthroughs in computer vision through deep learning, there are still many hurdles to overcome on the path to artificial intelligence that can enable full vehicle automation. Neural networks are very problem-specific, still make mistakes that humans wouldn't, and can sometimes be easily tricked.
See Fig. 8 (PDF)
Furthermore, neural networks require significantly more data than humans to learn a specific concept. Creating labeled datasets for training these networks is still sometimes very time-consuming and hinders progress.
List of abbreviations
AI: Artificial Intelligence
AI: Artificial Intelligence
ML: Machine Learning
DL: Deep Learning
NN: Neural network
CNN: Convolutional Neural Network
List of illustrations
Figure 1: Differentiation of the terms Artificial Intelligence / Machine Learning / Deep Learning; Source: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
Figure 2: Supervised vs. unsupervised learning; Source: https://beta.cambridgespark.com/courses/jpm/01-module.html
Figure 3: Neural network with three hidden layers; Source: https://www.rsipvision.com/exploring-deep-learning/
Figure 4: Neural network learns what makes human faces distinctive; Source: https://wangjieshu.com/2017/12/23/symbol-vs-connectionism-a-closing-gap-in-artificial-intelligence/
Figure 5: Object recognition using neural networks; Source: Fraunhofer IAIS
Figure 6: Instance segmentation; Source: https://bair.berkeley.edu/blog/2018/05/30/bdd/
Fig. 7: Gaze direction and attention detection using neural networks; Source: https://www.i6.in.tum.de/pub/Main/TeachingWs2017ContextPredictionSeminar/Introduction_Session.pdf
Fig. 8: Toaster problem; Source: https://arxiv.org/pdf/1712.09665.pdf
Bibliography and list of sources
[1] https://www-formal.stanford.edu/jmc/whatisai.pdf
[2] Stuart Russel and Peter Norvig, Artificial Intelligence – A Modern Approach, Prentice Hall, 2010
[3] https://www.deeplearningbook.org
[4] Aurélien Géron, Hands-On Machine Learning with Scikit-Learn & TensorFlow, O'Reilly, 2017
author
Dr. Eduard Kromer is the technical project manager for deep learning and artificial intelligence at iNTENCE automotive electronics GmbH. He holds a doctorate in mathematics from Justus Liebig University Giessen, conducted research at Princeton University, and was a postdoctoral researcher at the Department of Statistics at UC Berkeley. He brings many years of experience in machine learning, and his areas of interest include artificial intelligence, deep learning, and computer vision.
Email: e.kromer@intence.de
Our training courses & coaching sessions
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching for system and hardware development.
Training & coaching on the other topics in our portfolio can be found here. here.
Expertise
Valuable expertise in system and hardware development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
