EN 
DE Why the clock is ticking ever louder
Author: Rainer Grimm, Modern
Contribution – Embedded Software Engineering Congress 2018
Python 3 was released in 2008. Since Python 3 is not backward compatible with Python 2, existing code was typically developed further in Python 2, and new code was written directly in Python 3. This coexistence of two parallel systems will come to an abrupt end on January 1, 2020. Support for Python 2 will cease on that date. Therefore, migrating the Python 2 codebase to Python 3 is unavoidable.
New features in Python 3
print is a function
The new syntax for `print` is generally `print(*args, sep = " ", end = " ", file = sys.stdout, flush = True)`, where `args` denotes the arguments, `sep` the separator between the arguments, `end` the end-of-line character, `file` the output medium, and `flush` the buffer. The following table compares the syntactic changes of the `print` function, including its default values.
| Examples | Python 2.* | Python 3.* |
| General form | print "x= ",5 | print("x= ",5) |
| Line break | print() | |
| Suppress line breaks | print x, | print(x, end = „") |
| Suppressing line breaks without spaces | print(1, 2, 3, 4, 5, sep = „“) | |
| Output redirection | print >> sys.stderr, „error“ | print(„error“, file = sys.stderr) |
The advantage of the new version only becomes apparent upon closer inspection, as the print function can now be overloaded: The listing shows a print function that writes to both standard output and a log file. It achieves this by using the built-in function `__builtins__.print`. (See figure in the...) PDF)
Lazy Evaluation
Doing only what's necessary is certainly a virtue in programming languages. In Python 3, lazy evaluation is given significantly more weight. Lists, dictionaries, and Python's functional building blocks no longer generate the entire list, but only as much as is necessary for evaluating the expression. This on-demand evaluation saves valuable memory and time. The Python interpreter achieves this by returning only a generator that creates the values on demand. This was already the subtle difference between the `range()` and `xrange()` functions in Python 2. With Python 3, `range()` now behaves like `xrange()`, making the second function redundant.
The same applies to the functional building blocks `map()`, `filter()`, and `zip()`. These functions have been replaced by their equivalents from the `itertools` library. For dictionaries, the resulting generators are called `Views`. If, however, the programmer needs the fully expanded list, simply encapsulating the generator in a `list()` constructor helps, as the following example shows: `list(range(11))` yields [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10].
True Division
Python beginners are often surprised that 1/2 equals 0. Python 3 eliminates this and distinguishes between true division and floor division. While true division results in 1/2 = 0.5, floor division behaves like division in Python 2. Its notation uses two forward slashes: 1//2 == 0.
Unicode strings and byte strings
While in Python 2 programmers had to explicitly declare strings as Unicode strings, these strings are now implicitly Unicode strings. Python 3 only recognizes text and data. Text (str [1]) are strings and correspond to the Unicode string from Python 2. Data (bytes [2]) are 8-bit strings and correspond to the Python 2 strings. Python 3 developers must declare data as: b“8-bit string“. The table provides an overview.
| type | Python 2.* | Python 3.* |
| 8-bit string | „"string"“ | b“string“ |
| Unicode string | u“string“ | „"string"“ |
To convert between data types, the functions `str.encode()` and `bytes.decode()` are available. This conversion is necessary in Python 3 when using both data types, as implicit type conversion no longer occurs. (See figure in the diagram.) PDF)
Function Annotations
With Function Annotations, Python 3 offers the possibility of binding metadata to a function. In a second step, the function can also be decorated with decorators [3] that automatically generate documentation from the metadata or check the types at runtime. The equivalent functions `sumOrig()` and `sumMeta()` show the function declaration with and without metadata. The second function is extended with metadata about the function's signature and return value. The metadata can be referenced using the function attribute `__annotations__`. (see figure in the diagram.) PDF)
Cleanup work in Python 3
Where there are changes, legacy issues also need to be resolved and disposed of. This includes libraries that have been removed, that are now written in lowercase according to the Python Style Guide [4], that have been repackaged, or that coexist in a C and a Python implementation.
Import Idiom
The familiar Python idiom of first importing the fast C implementation of a module and falling back to the Python implementation in case of errors is no longer necessary. Python handles this automatically. (see figure in the diagram) PDF)
More details about the changes to the standard library can be found in [5].
Cooperative supercalls and old-style classes
There are other points that make life easier for Python programmers. For example, in cooperative `super` calls, they no longer need to specify the class instance and class name. Old-style classes no longer exist in Python 3, so the cumbersome inheriting of `object` is no longer necessary to access Python's newer features.
Input removed
Direct evaluation of the input using the `input()` command is no longer possible, as the input is now available as an input string. This closes a major security vulnerability. (see figure in the diagram) PDF)
Consequently, the function raw_input() was renamed to input(), and raw_input() was removed.
Backporting Python Features
The purpose of Python 2.7 is to make the transition to version 3 as easy as possible. For this reason, the project has backported many features from Python 3.0 to Python 2.7.
Context Manager
The context manager with `with` is an important new feature available with Python 2.6. Python automatically binds a resource (file, socket, mutex, etc.) upon entering the `with` block and releases it upon exiting. C++ programmers will find this idiom reminiscent of Resource Acquisition Is Initialization (RAII) [6].
From the user's perspective, the `with` statement behaves like a `try...finally` block, since both the `try` and `finally` blocks are always executed. However, all of this happens without explicit exception handling.
So how does it all work? Any object that offers the context management protocol, meaning it has the internal methods `__enter()__` and `__exit()__`, can be used within a `with` block. Upon entering the `with` block, Python automatically calls the `__enter()__` method, and upon exiting, it automatically calls the `__exit()__` method. The `file` object includes these methods by default. (See figure in the diagram.) PDF)
Resource management can also be quickly implemented using the `__enter()__` and `__exit()__` methods. For those who find this too much work, the `contextmanager` decorator from the `contextlib` library [6] can be used to benefit from context management. Further use cases can be found in Python Enhancement Proposal (PEP) 0343 [8].
Abstract Base Classes
Perhaps the most significant syntactic extension occurs in Python 2.6 with the introduction of abstract base classes. Previously, whether an object could be used in a context depended on the object's characteristics, not on its formal interface specification.
This idiom is called duck typing, based on the poem by James Whitcomb Riley: „When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.“
Once a class has an abstract method, it becomes an abstract base class and cannot be instantiated. Classes derived from it can only be created if they implement these abstract methods. Abstract base classes in Python behave similarly to abstract base classes in C++; in particular, abstract methods may contain an implementation.
In addition to abstract methods, Python also has abstract properties. Python version 3 uses abstract base classes in the modules numbers [9] and collections [10].
The crucial question remains: How does a class become an abstract class? The class requires the metaclass ABCMeta. Subsequently, the corresponding methods can be declared as `@abstractmethod` or properties as `@abstractproperty` using the appropriate decorator. Furthermore, the use of abstract base classes means that static typing is introduced into the dynamically typed language.
The example cannot instantiate the Cygnus class because it does not implement the abstract method quack().
(see figure in the PDF)
Multiple processors
Python's answer to multiprocessor architectures is the new multiprocessing library [11]. This module mimics the well-known Python threading module, but instead of a thread, it creates a process, and it does so in a platform-independent manner. The multiprocessing module was necessary because in CPython, the standard implementation of Python, only one thread can run in the interpreter. This behavior is due to the so-called Global Interpreter Lock, or GIL for short [12].
Migration to Python 3
A clearly defined path is emerging for porting Python 2 code to Python 3, whereby the developer must test the code and fix problems after each step. (see figure in the PDF)
The four lines of code are intended as an example for migrating from Python 2 to 3. All four lines of the example use functional components of Python because some changes have been made to these built-in functions. (see figure in the diagram.) PDF)
The first function calculates the sum of the three numbers 2, 3, and 4 by applying these arguments to the lambda function. The built-in `reduce()` function successively reduces the list of all numbers from 1 to 10 inclusive by multiplying the result of the last multiplication by the next number in the sequence. The last two functions filter out all words from the string that begin with a capital letter. The code already works under Python 2.6, so only steps 3 and 4 need to be completed for porting.
Calling the Python 2.7 interpreter with the -3 option reveals the incompatibilities with version 3: Both the `apply()` and `reduce()` functions are no longer built-in in Python 3. (See figure in the...) PDF)
The code is quickly repaired and the deprecation warnings no longer appear. (see figure in the diagram) PDF)
The script 2to3.py proves very helpful when correcting Python 2 code, as it automatically generates Python 3 code in the final step. The tool offers several options for this. (see figure in the...) PDF)
The direct way is to overwrite the original file: python `port.py -w`. The result is the source code ported to Python 3. Interestingly, the code generator has replaced the `filter()` expression with an equivalent list comprehension. (See figure in the diagram.) PDF)
Python 2 and Python 3 are supported
If the Python code needs to be rewritten to be supported by both Python 2 and Python 3, the scripts `futurize` [13] and `modernize` [14] can be used. The `modernize` script is more conservative in modifying the Python code than the `futurize` script. As in the previous section, there are two prerequisites for modifying the Python code: having test coverage and having already migrated the Python code to Python 2.7.
The cheat sheet „Cheat Sheet: Writing Python 2-3 compatible code“ [15] describes in detail the syntactic differences between Python 2 and Python 3 and how to write Python code that is supported by both Python 2 and Python 3.
Further information
[1]: Data type str: https://docs.python.org/3/library/functions.html#str
[2]: Data type bytes: https://docs.python.org/3/library/functions.html#bytes
[3]: Rainer Grimm, „Decorators in Python“: Linux Magazine 06/09, p. 96
[4]: PEP 8 – Style Guide for Python Code: https://www.python.org/dev/peps/pep-0008/
[5]: What's New in Python 3: https://docs.python.org/3/whatsnew/3.0.html#library-changes
[6]: RAII: https://de.wikipedia.org/wiki/RESOURCESBELEUNG_IST_INITIALIZATION
[7]: Library contextlib: https://docs.python.org/3/library/contextlib.html#module-contextlib
[8]: PEP 0343: https://www.python.org/dev/peps/pep-0343
[9]: Library numbers: https://docs.python.org/3/library/numbers.html#module-numbers
[10]: Library collections: https://docs.python.org/3/library/collections.html#module-collection
[11]: Library multiprocessing: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing
[12]: The Global Interpreter Lock (GIL): https://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
[13]: Futurize: https://python-future.org/automatic_conversion.html
[14]: Modernize: https://python-modernize.readthedocs.io/en/latest
[15]: Cheat Sheet: Writing Python 2-3 compatible code: https://python-future.org/compatible_idioms.html
author
Rainer Grimm has worked for many years as a software architect, team leader, and trainer. In his free time, he enjoys writing articles about the programming languages C++, Python, and Haskell, and also speaks at industry conferences. On his blog, Modernes C++ (Heise Developer), he delves deeply into his passion for C++. He has been self-employed since 2016. Sharing knowledge about modern C++ is particularly important to him. His books "C++11 für Programmierer" (C++11 for Programmers), "C++", and "C++-Standardbibliothek" (C++ Standard Library) for the "kurz und gut" (short and sweet) series are published by O'Reilly. His English-language works, "The C++ Standard Library" and "Concurrency with Modern C++", have been translated into several languages.
Implementation – our training & coaching
Do you want to bring yourself up to date with the latest technology?
Then find out more here MircoConsult offers training courses/seminars/workshops and individual coaching on the topic of implementation/embedded and real-time software development.
Training & coaching on the other topics in our portfolio can be found here. here.
Implementation – Expertise
Valuable expertise in the field of implementation/embedded and real-time software development is available. here Available for you to download free of charge.
You can find expertise on other topics in our portfolio here. here.
