Top!
Hello
Unlike other manufacturing industries, such as the automotive industry, you can't take physical measurements of software and its components to ensure quality. However, the software industry also needs tools for quality assurance. This is where the so-called software tests come into play.
Glenford J. Myers and Corey Sandler define software testing in their book "The Art of Software Testing" as follows:
“Testing is the process of executing a program with the intent of finding errors”
Tests are therefore the process of running a program multiple times to find errors. This can be done manually by testers or automatically by small programs that run and test the text program. In the following, the automatic tests are considered.
Automatic software testing can basically be divided into two categories: Functional tests and Non-functional tests. Functional tests are tests in which the software or parts of the software are checked for correctness. For example, whether the software behaves in a certain situation as defined in the specifications. Non-functional tests relate to software properties such as performance or compliance with guidelines.
There are many different types of functional tests, but only three will be discussed here. These are unit tests, integration tests and system tests.
A unit test is a test of a component. For a test to be considered a unit test, the input must be controllable and the tested component must have no dependency on any other component. An example would be a method that divides the first passing parameter by the second passing parameter. Here, for example, it can be tested whether the calculation is performed correctly and whether the division by 0 is handled.
An integration test takes place as soon as more than one component is tested simultaneously. For example, if our method for splitting two numbers would call another method.
System tests, or end-to-end tests, are tests that sample a process defined by program specifications. UI tests are an example of such tests.
This could be written in a specification for an Android program: When the floating action button is pressed, a new window opens in which a new plane can be added. A UI test would then press the button and check whether this specified window actually opens. You can read how to write UI tests in Android with Espresso in the zugehörigen blog post
Code coverage is a metric used to measure which parts of the software are covered by tests. More precisely, it means which parts of the code are passed through by the execution of the tests. It should be noted that there are two terms that are often used interchangeably, but have different meanings. These terms are code coverage and test coverage.
While code coverage is calculated exclusively by the coverage of the code by execution of the tests, test coverage is understood as the complete coverage of the program by all automatic and manual tests. Code coverage is usually calculated and visualized by tools such as Cobertura or Jacoco.
Code coverage can be measured in several ways:
These different readings can vary greatly in some cases. Here is a small example based on one method:
public int divide(int numerator, int denominator) throws ArithmeticException{
if (denominator == 0) {
System.out.println("Cannot divide by zero");
throw new ArithmeticException();
} else {
return (numerator / denominator);
}
}
For this method, we would write the following unit test:
@Test
public void test_divide() {
int result = divide(4, 2);
Assert.assertEquals(2, result);
}
If we then calculate the coverage of this test, we get the following results:
Here you can already see how different the measurements can be for different types of code coverage. While the function coverage is at complete 100%, the coverage of the branches is only at half. Code coverage can therefore be used to find out which parts of the code have already been covered with tests. However, it cannot say anything about the quality of the tests.
The test in the above example does execute the function and check the result, but only one possible case is covered. For example, what if 0 is passed as a divisor? If we add the following test for the behavior when 0 is passed, we also achieve 100% line and branch coverage:
@Test(expected = ArithmeticException.class)
public void test_divide_zeroAsDenominator_ShouldThrowException(){
divide(2,0);
}
Has this method now been fully tested? The answer to this question is no, because the behavior of the function is not tested for passing values that do not lead to a round result. Nevertheless, the coverage here showed us a hundred percent coverage in all categories. Here you can already see that even with complete coverage, it is not guaranteed that the covered code itself has been fully tested.
I would also like to mention here that it is relatively easy to achieve high code coverage by writing tests that do not contain assert statements. These have little value to the project and should not be written. Nevertheless, such a test would reveal runtime errors in the code passed through, but this is no excuse for me to write assert free tests.
The topic of 100% code coverage is relatively controversial in the industry. Basically, there are two different views on this topic:
1.) 100% code coverage should always be aimed for and is an important code quality feature. There are different views on how the 100% can be achieved. For example, 100% coverage can be achieved by making sure from the outset that testable code is writing and trivial functions, as for example getter/setter either from the Coverage to exclude or not at all to write in the first place. Here, some also argue that non-testable code should not exist. In addition, not mainly unit tests, but also integration tests should be written.
2.) 100% code coverage should not be aimed for, since it says nothing about the quality and completeness of the tests, and achieving higher coverage is associated with too much effort above a certain threshold. In addition, with a fixed coverage, the temptation is great to write tests for coverage instead of for quality assurance, which contradicts the original purpose of tests.
However, there is a general consensus that a program should have high test coverage. Moreover, even supporters of the first view say that code coverage says nothing about the quality of the tests and 100% coverage does not mean 100% bug-free code.
In my opinion, it is not useful to set the 100% or even any other number as a target for code coverage. However, this is not to say that code coverage is a useless metric. I think every project should aim for high code coverage without setting a fixed percentage.
Code coverage is a useful metric to use in any case. Through it, you can see which modules could use more testing. I also believe that high coverage leads to fewer bugs getting through to the production release. This impression is confirmed by Microsoft's analysis of their testing efforts. In addition, unit tests should not be used exclusively to achieve high code coverage.
For me, the tendency to write tests according to maximum code coverage speaks against specifying the code coverage to be achieved. Even if high-quality tests are written, more important modules may not be tested. Overall, I think it is more important to cover critical modules with tests and to achieve high coverage for these modules than to write tests for overall coverage.
