let’s dev GmbH & Co. KG - The brand for groundbreaking custom software

Blog

Top!

let’s dev | Scroll to top
let’s dev | Scroll to next content item

Hello

let’s dev | Scroll to previous content item
let’s dev Blog | 100% code coverage in software testing - a reasonable goal?
by Raphael
06. März 2020

100% code coverage in software testing - a reasonable goal?

Unlike other manufacturing industries, such as the automotive industry, you can't take physical measurements of software and its components to ensure quality. However, the software industry also needs tools for quality assurance. This is where the so-called software tests come into play.

What are software tests?

Glenford J. Myers and Corey Sandler define software testing in their book "The Art of Software Testing" as follows:

“Testing is the process of executing a program with the intent of finding errors”

Tests are therefore the process of running a program multiple times to find errors. This can be done manually by testers or automatically by small programs that run and test the text program. In the following, the automatic tests are considered.

Automatic software testing can basically be divided into two categories: Functional tests and Non-functional tests. Functional tests are tests in which the software or parts of the software are checked for correctness. For example, whether the software behaves in a certain situation as defined in the specifications. Non-functional tests relate to software properties such as performance or compliance with guidelines.

What are the types of functional tests?

There are many different types of functional tests, but only three will be discussed here. These are unit tests, integration tests and system tests.

A unit test is a test of a component. For a test to be considered a unit test, the input must be controllable and the tested component must have no dependency on any other component. An example would be a method that divides the first passing parameter by the second passing parameter. Here, for example, it can be tested whether the calculation is performed correctly and whether the division by 0 is handled.

An integration test takes place as soon as more than one component is tested simultaneously. For example, if our method for splitting two numbers would call another method.

System tests, or end-to-end tests, are tests that sample a process defined by program specifications. UI tests are an example of such tests.

This could be written in a specification for an Android program: When the floating action button is pressed, a new window opens in which a new plane can be added. A UI test would then press the button and check whether this specified window actually opens. You can read how to write UI tests in Android with Espresso in the zugehörigen blog post

What is code coverage?

Code coverage is a metric used to measure which parts of the software are covered by tests. More precisely, it means which parts of the code are passed through by the execution of the tests. It should be noted that there are two terms that are often used interchangeably, but have different meanings. These terms are code coverage and test coverage.

While code coverage is calculated exclusively by the coverage of the code by execution of the tests, test coverage is understood as the complete coverage of the program by all automatic and manual tests. Code coverage is usually calculated and visualized by tools such as Cobertura or Jacoco.

Screenshot JaCoCo

Code coverage can be measured in several ways:

These different readings can vary greatly in some cases. Here is a small example based on one method:

public int divide(int numerator, int denominator) throws ArithmeticException{
    if (denominator == 0) {
        System.out.println("Cannot divide by zero");
        throw new ArithmeticException();
    } else {
        return (numerator / denominator);
    }
}

For this method, we would write the following unit test:

@Test
public void test_divide() {
    int result = divide(4, 2);
    Assert.assertEquals(2, result);
}

If we then calculate the coverage of this test, we get the following results:

Here you can already see how different the measurements can be for different types of code coverage. While the function coverage is at complete 100%, the coverage of the branches is only at half. Code coverage can therefore be used to find out which parts of the code have already been covered with tests. However, it cannot say anything about the quality of the tests.

The test in the above example does execute the function and check the result, but only one possible case is covered. For example, what if 0 is passed as a divisor? If we add the following test for the behavior when 0 is passed, we also achieve 100% line and branch coverage:

@Test(expected = ArithmeticException.class)
public void test_divide_zeroAsDenominator_ShouldThrowException(){
    divide(2,0);
}

Has this method now been fully tested? The answer to this question is no, because the behavior of the function is not tested for passing values that do not lead to a round result. Nevertheless, the coverage here showed us a hundred percent coverage in all categories. Here you can already see that even with complete coverage, it is not guaranteed that the covered code itself has been fully tested.

I would also like to mention here that it is relatively easy to achieve high code coverage by writing tests that do not contain assert statements. These have little value to the project and should not be written. Nevertheless, such a test would reveal runtime errors in the code passed through, but this is no excuse for me to write assert free tests.

100% code coverage a reasonable goal?

The topic of 100% code coverage is relatively controversial in the industry. Basically, there are two different views on this topic:

1.) 100% code coverage should always be aimed for and is an important code quality feature. There are different views on how the 100% can be achieved. For example, 100% coverage can be achieved by making sure from the outset that testable code is writing and trivial functions, as for example getter/setter either from the Coverage to exclude or not at all to write in the first place. Here, some also argue that non-testable code should not exist. In addition, not mainly unit tests, but also integration tests should be written.

2.) 100% code coverage should not be aimed for, since it says nothing about the quality and completeness of the tests, and achieving higher coverage is associated with too much effort above a certain threshold. In addition, with a fixed coverage, the temptation is great to write tests for coverage instead of for quality assurance, which contradicts the original purpose of tests.

However, there is a general consensus that a program should have high test coverage. Moreover, even supporters of the first view say that code coverage says nothing about the quality of the tests and 100% coverage does not mean 100% bug-free code.

Does it now make sense to aim for 100% code coverage in a project?

In my opinion, it is not useful to set the 100% or even any other number as a target for code coverage. However, this is not to say that code coverage is a useless metric. I think every project should aim for high code coverage without setting a fixed percentage.

Code coverage is a useful metric to use in any case. Through it, you can see which modules could use more testing. I also believe that high coverage leads to fewer bugs getting through to the production release. This impression is confirmed by Microsoft's analysis of their testing efforts. In addition, unit tests should not be used exclusively to achieve high code coverage.

For me, the tendency to write tests according to maximum code coverage speaks against specifying the code coverage to be achieved. Even if high-quality tests are written, more important modules may not be tested. Overall, I think it is more important to cover critical modules with tests and to achieve high coverage for these modules than to write tests for overall coverage.

More articles from our blog

let’s dev Blog | Sketch, Figma & Co. - We take a look at the most popular UI und Prototyping Tools in 2021

Corporate

Sketch, Figma & Co. - We take a look at the most popular UI und Prototyping Tools in 2021

by Ellen

2021-07-15

Read more
let’s dev Blog | Tailwind: An innovative project for the future use of old wind turbines

Corporate

Tailwind: An innovative project for the future use of old wind turbines

by Karl

2021-06-24

Read more
let’s dev Blog |

Corporate

by Julian

2021-06-10

Read more
let’s dev Blog | Smart Prognosis of Energy with Allocation of Resources

Corporate

Smart Prognosis of Energy with Allocation of Resources

by Karl

2021-02-18

Read more
let’s dev Blog | Dasoman - Data-Sovereignty-Manager

Corporate

Dasoman - Data-Sovereignty-Manager

by Karl

2021-01-11

Read more
let’s dev Blog | We look back on the past months - And wish all the best for the coming year 2021!

Corporate

We look back on the past months - And wish all the best for the coming year 2021!

by Julian

2020-12-17

Read more
let’s dev Blog | iOS User Interface Tests

Technical

iOS User Interface Tests

by Nicolas

2020-11-12

Read more
let’s dev Blog | Adobe Max - Online for the first time

Corporate

Adobe Max - Online for the first time

by Julia

2020-10-29

Read more
let’s dev Blog | CAN2BLE

Technical

CAN2BLE

by Raphael

2020-09-24

Read more
let’s dev Blog | Mensch und Computer 2020 - Digital Change in the Flow of Time

Corporate

Mensch und Computer 2020 - Digital Change in the Flow of Time

by UX Team

2020-09-18

Read more
let’s dev Blog | Neumorphism – A new era of user interface design?

Technical

Neumorphism – A new era of user interface design?

by Julian

2020-08-13

Read more
let’s dev Blog | UX Research Part 3 - UX Methods

Technical

UX Research Part 3 - UX Methods

by Elena

2020-05-28

Read more
let’s dev Blog | UX Research Part 2 - What is UCD and what does User Research have to do with it?

Technical

UX Research Part 2 - What is UCD and what does User Research have to do with it?

by Elena

2020-04-23

Read more
let’s dev Blog | go-digital promotes establishment of home office workstations

Corporate

go-digital promotes establishment of home office workstations

by Karl

2020-03-19

Read more
let’s dev Blog | Google Passes - Card Management on Android Devices

Technical

Google Passes - Card Management on Android Devices

by Michelle

2020-03-12

Read more
let’s dev Blog | Swift UI - Simple and fast implementation of user interfaces

Technical

Swift UI - Simple and fast implementation of user interfaces

by Tobias

2020-03-02

Read more
let’s dev Blog | In dialog with the business juniors - Exciting insights into business start-ups and digital transformation

Corporate

In dialog with the business juniors - Exciting insights into business start-ups and digital transformation

by Julian

2020-02-27

Read more
let’s dev Blog | Simplified testing of iOS push notifications in the simulator with Xcode 11.4

Technical

Simplified testing of iOS push notifications in the simulator with Xcode 11.4

by Manuel

2020-02-26

Read more
let’s dev Blog | National meeting of the consortium of the SPEAR research project at let's dev in Karlsruhe

Corporate

National meeting of the consortium of the SPEAR research project at let's dev in Karlsruhe

by Karl

2020-01-27

Read more
let’s dev Blog | UX Research Part 1 - Why User Research is so important

Technical

UX Research Part 1 - Why User Research is so important

by Elena

2020-01-23

Read more
let’s dev Blog | Dark Mode

Technical

Dark Mode

by Elisa

2020-01-09

Read more
let’s dev Blog | We wish you a Merry Christmas - And a Happy New Year!

Corporate

We wish you a Merry Christmas - And a Happy New Year!

by Julian

2019-12-20

Read more
let’s dev Blog | Exchange on the topic of digitization with the Business Club Luxembourg at the Embassy of Luxembourg in Berlin

Corporate

Exchange on the topic of digitization with the Business Club Luxembourg at the Embassy of Luxembourg in Berlin

by Karl

2019-12-17

Read more
let’s dev Blog | DaSoMan at the Internet+ Expo in Foshan (China)

Corporate

DaSoMan at the Internet+ Expo in Foshan (China)

by Karl

2019-12-13

Read more
let’s dev Blog | Google Play Console: Pre-Launch Reports

Technical

Google Play Console: Pre-Launch Reports

by Fabian

2019-12-11

Read more
let’s dev Blog | DevFest 2019 in Hamburg

Technical

DevFest 2019 in Hamburg

by Julian

2019-12-05

Read more
let’s dev Blog | Vernissage digital art in the media theater of the Humboldt University Berlin

Corporate

Vernissage digital art in the media theater of the Humboldt University Berlin

by Karl

2019-11-21

Read more
let’s dev Blog | World Usability Day 2019 in Karlsruhe - let's dev supports as main sponsor

Corporate

World Usability Day 2019 in Karlsruhe - let's dev supports as main sponsor

by Aileen

2019-11-11

Read more
let’s dev Blog | Gutted - Open Day at the Alter Schlachthof Karlsruhe 2019

Corporate

Gutted - Open Day at the Alter Schlachthof Karlsruhe 2019

by Julian

2019-09-26

Read more
let’s dev Blog | Mensch und Computer 2019 - Conference on User Experience and Usability in Hamburg

Corporate

Mensch und Computer 2019 - Conference on User Experience and Usability in Hamburg

by Elena

2019-09-17

Read more
let’s dev Blog | Business and Enterprise App Distribution on iOS

Technical

Business and Enterprise App Distribution on iOS

by Aileen

2019-08-05

Read more
let’s dev Blog | Digital Transformation - Chances and Challenges in the Automotive Industry, Agriculture and New Technologies

Corporate

Digital Transformation - Chances and Challenges in the Automotive Industry, Agriculture and New Technologies

by Karl

2019-07-17

Read more
let’s dev Blog | let's dev supports runners at the 7th KIT Championship

Corporate

let's dev supports runners at the 7th KIT Championship

by Karl

2019-07-05

Read more
let’s dev Blog | Automated testing of C++ code with Google Test and Google Mock - Part 2

Technical

Automated testing of C++ code with Google Test and Google Mock - Part 2

by Arne

2019-06-13

Read more
let’s dev Blog | Apple WWDC 2019: These are the highlights of the keynote

Technical

Apple WWDC 2019: These are the highlights of the keynote

by Nicolas

2019-06-05

Read more
let’s dev Blog | App Builders 2019

Technical

App Builders 2019

by Nicolas

2019-05-23

Read more
let’s dev Blog | Official opening of the Consolidation and Expansion Center (FUX)

Corporate

Official opening of the Consolidation and Expansion Center (FUX)

by Helena

2019-04-15

Read more
let’s dev Blog | Delegation from Nottingham to visit the Alter Schlachthof in Karlsruhe

Corporate

Delegation from Nottingham to visit the Alter Schlachthof in Karlsruhe

by Helena

2019-04-14

Read more
let’s dev Blog | The time has come: We are moving!

Corporate

The time has come: We are moving!

by Helena

2019-03-26

Read more
let’s dev Blog | Automated testing of C++ code with frameworks - part 1

Technical

Automated testing of C++ code with frameworks - part 1

by Arne

2019-02-20

Read more
let’s dev Blog | The app in the Google Play Store

Technical

The app in the Google Play Store

by Elisa

2019-01-24

Read more
let’s dev Blog | „UX Day“ 2018

Corporate

„UX Day“ 2018

by Aileen

2018-12-17

Read more
let’s dev Blog | let's dev supports SG Siemens volleyball players from Karlsruhe

Corporate

let's dev supports SG Siemens volleyball players from Karlsruhe

by Helena

2018-12-04

Read more
let’s dev Blog | SMEs shape digitalization - SME Conference 2018

Corporate

SMEs shape digitalization - SME Conference 2018

by Helena

2018-11-12

Read more
let’s dev Blog | Apple Wallet

Technical

Apple Wallet

by Maik

2018-10-26

Read more
let’s dev Blog | „Mensch und Computer“ 2018

Corporate

„Mensch und Computer“ 2018

by Judith

2018-09-24

Read more
let’s dev Blog | State Design Pattern in Android

Technical

State Design Pattern in Android

by Thomas

2018-09-17

Read more
let’s dev Blog | let's dev is an authorized consulting company in the „go-digital“ funding program

Corporate

let's dev is an authorized consulting company in the „go-digital“ funding program

by Helena

2018-09-01

Read more
let’s dev Blog | App Design & Development Conference 2018

Corporate

App Design & Development Conference 2018

by Helena

2018-08-14

Read more
let’s dev Blog | iOS 12: The top new features at a glance

Technical

iOS 12: The top new features at a glance

by Nicolas

2018-07-17

Read more
let’s dev Blog | let's dev at CEBIT

Corporate

let's dev at CEBIT

by Karl

2018-06-11

Read more
let’s dev Blog | Introduction to User Interface (UI) Testing with Espresso

Technical

Introduction to User Interface (UI) Testing with Espresso

by Raphael

2018-06-07

Read more
let’s dev Blog | The app in the Apple App Store: what information is needed?

Technical

The app in the Apple App Store: what information is needed?

by Aileen

2018-04-27

Read more
let’s dev Blog | Smart Pointer in C++

Technical

Smart Pointer in C++

by Matthias

2018-04-01

Read more
let’s dev Blog | User interface design for iPhone X: all innovations at a glance

Technical

User interface design for iPhone X: all innovations at a glance

by Helena

2018-02-07

Read more
let’s dev Blog | WebVR - Virtual Reality Experience in the Browser with the A-Frame Framework

Technical

WebVR - Virtual Reality Experience in the Browser with the A-Frame Framework

by Judith

2018-01-10

Read more
let’s dev Blog | Deutsche Bahn Open Data Hackathon

Corporate

Deutsche Bahn Open Data Hackathon

by Karl

2015-03-31

Read more
let’s dev Blog | Blur effects under iOS 7

Technical

Blur effects under iOS 7

by Katja

2014-04-24

Read more
let’s dev Blog | Beyond App Store - iOS application distribution

Technical

Beyond App Store - iOS application distribution

by Karl

2012-08-27

Read more
let’s dev Blog | Front-end architecture - Model View Presenter and Message Bus

Technical

Front-end architecture - Model View Presenter and Message Bus

by Karl

2011-03-08

Read more