Behavior-driven testing and automatic unit test generation

In many development organizations, a gap exists between the business department and test automation. Business departments often specify tests functionally, lacking details about behavior. Conversely, business testers often don't understand the implementation of automated tests due to a lack of developer expertise. Very often, too few tests are specified and automated, and the resulting test coverage is sometimes negligently low. Behavior-Driven Testing (BDT) is a technique from agile development that closes precisely this gap between business testers and automation specialists. As a complement to a structured test automation approach (such as BDT), it is often also beneficial to automatically fill the test gaps with generated unit tests.

Test Driven Development (TDD) as a precursor to BDT

Michael Feathers is one of the "gurus" in software development and coined the phrase a long time ago:

„"I don't care how good you think your design is. If I can't walk in and write a test for an arbitrary method of yours in five minutes it's not as good as you think it is, …

… and whether you know it or not, you're paying a price for it!„

This quote emphasizes that understanding a program (e.g., for a tester) becomes easier and better the better it is structured. This, of course, also positively influences the overall quality of the software.

To ensure understanding and software quality as early as possible, it is therefore advisable to involve testers at a very early stage of software development. Ideally, even before the first line of code is written. This approach is called "test-driven software development".

Here, a developer or a tester/test automation specialist with the appropriate skills specifies and ideally automates the test cases necessary for testing the desired software BEFORE programming begins. Theoretically, test-driven software development (TDD) is also possible without test automation. However, this approach is very inefficient. Therefore, TDD almost exclusively uses automated tests.

The described and automated test cases can also be considered a (detailed) requirements specification. Test cases are essentially just another form of specified requirements for the software.

TDD can be applied at several levels:

• Small-scale testing (through unit tests) and
• Large-scale testing (through integration and system tests)

See Fig. 1 in the PDF – Basic process of Test Driven Development
The goal of TDD is to create testable and maintainable code from the outset and to increase test coverage. Another advantage is that in (agile) projects, where explicit requirements specifications are often lacking, the software requirements are well-defined and even automatically verifiable through the use of test cases.The advantages of TDD are:

• The quality of development will be improved.
• Test coverage is increased
• There is a better basis for sprint acceptance (test = requirements).
• Efficiency in the sprint is increased.
• A "live" (test) specification is created.

However, there are also some challenges:

• TDD alone does not create good requirements and code. Some things cannot be automated as tests in advance (usability, system context, process modeling, architecture, etc.).
• A very high level of discipline is required!

Behavior Driven Development (BDD) or Behavior Driven Testing BDT

BDD is a variant of TDD. This method was developed by Dan North in 2003 as a response to Test-Driven Development. The reason was that unit tests are difficult to read for business users, who often lack developer expertise.

The motivation behind BDD is that user stories, as a widely used form of specification in agile projects, very often only describe the requirements for software functionally. Therefore, Behavior Driven Development – as the name suggests – aims to introduce a stronger behavioral and process-oriented perspective.

The idea behind BDT is that the user stories should now also be supplemented from the testers' point of view by behavior-oriented test descriptions in a simple but structured form, so that the tests can then be easily automated.

The description should be written in natural language, be easily understandable and readable. This makes it possible for non-programmers to create "code" for automated testing.

See Fig. 2 in the PDF –Structure of the specification in Behaviour Driven Development (BDD)

The process at BDD is as follows:

The department or product owner creates user stories as before.

• In addition to a user story, the behavior is now also described in the form of a "scenario".
• A scenario begins with a precondition, which can consist of one or more "steps".
• A "step" is a related element that can be used repeatedly and can contain part of a test case (e.g., an input or output, or an action).
• If the prerequisites are met, the next step is to initiate the "action" (or trigger).
• This then leads in the third step to the "postcondition" (or result) of the action.

These steps describe a part of a software program both functionally and in terms of its behavior.

This makes it very easy to derive the test cases. From the example shown above, the tests can be derived as follows:

Trigger	precondition	precondition	Postcondition
	(Input)	(Input)	(Expected result)
	Step 1	Step 2
Press Add	50	70	120

By parameterizing the steps, any combination of test data can be used for test execution, thus enabling standard cases, limit values, and error cases to be tested with an abstract test case.

To enable the automation of tests specified in prose, there are specific BDD frameworks and BDD tools that support this approach. The following example uses the tool "Cucumber":

See Fig. 3 in the PDF – Behaviour Driven Development (BDD) using Cucumber as an example

BDD is an interesting approach that attempts to achieve better coupling between the stakeholders (subject matter experts, testers and developers).

The automated regression tests created in this process provide rapid feedback, leading to a "living" specification and documentation of the system.

However, there are also some challenges here:

• The subject matter experts typically work with other tools (e.g., test management tools).
• There is usually no tool-supported direct link between test specifications and automation code, which makes it difficult to ensure consistency between test cases and test code. The code calls (steps) used in BDD tools are often created text-based, without directly linking them to the generated test automation code in the specification environment. As a result, any typo (whether made by the tester or the automation engineer) creates a gap in the logic and breaks the automation. There is clearly room for improvement in these tools.
• The tools currently available on the market are, without additions, primarily suitable for development-related tests (unit/component level) and hardly suitable for system tests.

In general, however, BDD/BDT can be considered an interesting approach that is becoming increasingly widespread in practice. The gap between business testers and test automation engineers can be significantly reduced, and with appropriate integration of available frameworks (test management tool with automation environment and BDD tool), this gap can also be closed technically within the tools themselves.

Automatic Unit Test Generation

The increasing complexity of systems means that more and more tests are necessary to ensure that these systems meet the state of the art and to reduce liability towards clients in the event of errors.

In parallel, the development of these systems requires an increasing number of software developers, who are currently not available on the job market to the required extent and may have to be purchased externally, which makes development more expensive.

Due to increasing system complexity, developers must also dedicate more and more time to securing the system through appropriate unit tests (approximately 20-301 TP3T of development time is spent on unit test creation). This slows down the actual development process. If this results in delayed product launches, it leads to significant revenue losses.

The functional coverage achieved through testing is often much lower than necessary.

A simple branching example (see graphic). PDFHere, the developer typically writes two tests, one following the left path and one following the right. This achieves 100% code coverage, and most developers are satisfied with that.

However, to properly test the branching in the graphic, at least 14 test cases are necessary:

• Each (individual) condition must be tested separately (4 normal test cases)
• Threshold values (upper/lower) of the two conditions (4 further test cases)
• Error tests outside the upper and lower bounds, as well as within the bounds with error values (6 additional test cases)

Creating (unit) tests is therefore becoming increasingly time-consuming and expensive! It is therefore necessary to automate the test creation process wherever possible.

The following presents an approach that allows a large portion of the unit tests, which are often still created manually today, to be created automatically.

Technical implementation:

See figure in the PDF