Top!
Hello
Testing applications is also an important part of the development process in the mobile environment. For libraries written in C++, this is made possible by gtest and gmock. We present the use of these with simple examples.
Automated tests have become an indispensable part of modern development processes. They not only help to ensure the quality of the software, but also enable the developer to detect and eliminate errors and problems at an early stage.
Different types of tests are to be distinguished here. As examples unit tests for testing a component and integration tests, which examine the interaction of several components, are mentioned. For most programming languages there are in addition frameworks, which facilitate the production and execution significantly, so also for C++.
The Google frameworks gtest and gmock were initially developed in separate projects, but have since been merged. gtest is a testing framework for C++, which is intended to facilitate the testing and assessing of applications. Included functionalities are e.g. the organization of tests in test cases and the creation of fixtures for reuse. With gmock, on the other hand, mock objects can be created, e.g. to test whether a function can be called with the expected parameters. In the first part of this article, we will first take a closer look at the gtest framework.
The large number of C++ unit testing frameworks on the market makes the right choice considerably more difficult. It therefore makes sense to compare the frameworks with each other in advance from various points of view. Already in 2004 some of these frameworks like CppUnit or Boost.Test were compared. Even though gtest did not exist at that time, it is still interesting to take a look at the metrics used:
These questions will now be answered in the course of this article. It should be said in advance that gtest fulfills all the requirements mentioned. Nevertheless, the choice of framework always depends on the project in which it is to be used and must be determined individually.
This post is about gtest, but all the general topics are valid for gmock as well, since the functionality has been merged into one library.
A unique nomenclature must be defined for the description. gtest uses the designation TEST() or Test for individual tests, the grouping of tests for a component is called Test Case. A term that will appear more frequently in the further course is that of the Test Fixture. This is a class that contains functions and data that are reused in multiple tests.
The framework works on a variety of platforms as well as with different compilers. Supported systems include Windows, Mac OS X, and Linux, among others. It is possible to build and link gtest as a static and dynamic library. The use of cmake simplifies integration. Thus, gtest can be built as a stand-alone or integrated into other projects.
add_subdirectory (${ CMAKE_CURRENT_BINARY_DIR }/ googletest -src ${
CMAKE_CURRENT_BINARY_DIR }/ googletest - build EXCLUDE_FROM_ALL )
Before the actual implementation, as shown above, the download from GitHub must take place. Here you have the option of integrating the code directly or always loading it automatically as a dependency.
In principle, it makes sense to set up tests similar to the project structure. This makes it easier for other developers to find tests for a particular class. It is also quicker to find out whether tests exist for a class. Tests that require the same resources should be written in one file. This makes it very easy to use the fixtures and prepare the resources only in one place.
A simple test is written in a macro:
TEST ( TestCaseName , TestName ) {
... test body ...
}
The TestCaseName as well as the TestName are freely selectable. The names of the tests may also be repeated in different TestCases, only within a TestCase the TestName must be unique.
Fixture tests are used to access the same resources in different test cases. This can be a dummy object, which is filled with test data, but it can also be a complex initialization. To be able to use this functionality, a class must be created beforehand, which then serves as a fixture.
# include <gtest / gtest .h>
class MyFixture : public testing :: Test {
protected :
void SetUp () override ;
bool myFunction ();
};
void BaseTest :: SetUp () {
Test :: SetUp ();
... my additional setup ...
}
TEST_F ( MyFixture , TestName ) {
bool myBool = myFunction ();
... my aditional testing ...
}
In the header the function "myFunction" has the visibility "protected". All functions and properties called in the test macro must be visible "protected" or "public". The SetUp function can be used to perform complex initializations and is executed before each test. Analogously, a "TearDown" can also be implemented. The macro in (2) has changed slightly compared to (1). TEST has become TEST_F. The F stands for Fixture. The name of the fixture can be freely chosen, it only has to be identical in the fixture class and in the macro.
Assertions are one of the most important functions of any test framework. gtest provides a number of different checks that can be used to compare values or objects.
Assertions are available as fatal or non-fatal checks. A fatal check terminates the currently running test if the comparison is judged to be false. A non-fatal check allows the test to finish, but subsequently marks it as failed.
Fatal comparisons start with "ASSERT_", non-fatal ones with "EXPECT_". If possible, non-fatal checks should always be made.
The following examples show the use of some checks. However, it is only a small section, of the function provided by Google.
EXPECT_TRUE ( MyClass :: myBoolFunc ());
std :: string expectedResult (" this result ");
EXPECT_STREQ ( expectedResult . c_str () , MyClass :: myStringFunc (). c_str ());
EXPECT_EQ (42 , MyClass :: myIntFunc ());
EXPECT_GT (0, MyClass :: myLongFunc ());The various macros in (1) test multiple conditions. The macro with _STREQ checks two strings for equality, via _EQ the equality of numbers is matched and _GT checks whether the first number is greater than the second.
User * user = MyClass :: getMyObject ();
ASSERT_NE ( nullptr , user );
EXPECT_EQ (25 , user ->age);
EXPECT_EQ (female , user -> gender );
In (2) the existence of the object is checked with ASSERT_NE. Of course, it does not make sense to let the test continue if the object does not exist. The check of the age, on the other hand, is done via EXPECT_EQ, since other properties can also be checked here, even if the age is not correct.
In the following, both variants will not always be discussed. Whenever Assert or Expect is mentioned, both variants are meant, if not explicitly mentioned.
It is possible to give an individual output to a check. This is also output in the log in case of a failure and makes it easier to identify the cause. For this purpose, the message is appended to the assertion using the stream operator «.
TEST_F ( MyTestFixture , checkitemCount ) {
int itemCount = _sut -> getItemCount ();
EXPECT_EQ (0, itemCount ) << " Item count was not the expected size .";
}
Death tests can be used to check whether the program crashes on a particular execution. But why would one want to test this? In C++ it is common to secure preconditions by checks with abort(). Especially within the st1 or other libraries this kind of backup is used.
To use Death Tests, the death or crash of the program must be expected within the test:
EXPECT_DEATH (_sut -> myDeadlyFunction () , ".* this did happen .*")
When using the macro, two parameters are used. The first parameter must define the call that will cause the program to crash. The second argument is a regex that describes the expected error message. Thus the test fails under two conditions: First, if the program does not crash when myDeadlyFunction is called, and second, if the error message does not satisfy the passed regex.
Why is the error message crucial? If the software did not terminate with the expected message, the logic that was supposed to be tested was not addressed. A condition was therefore not fulfilled or a system error occurred before or after. Thus it is unclear whether the condition to be tested is fulfilled. The regex makes it possible to terminate the program exactly at the expected point.
Basically, this is a group of asserts and not a test type of its own. However, since exceptions are also becoming more and more important for C++, it should be noted that a check with gtest is also possible for these. The assertions also exist as fatal and non-fatal variants.
EXPECT_THROW (_sut -> myThrowingFunc () , std :: exception );
ASSERT_THROW (_sut -> myThrowingFunc () , std :: exception );
EXPECT_NO_THROW (_sut -> myNotThrowingFunc ());
ASSERT_NO_THROW (_sut -> myNotThrowingFunc ());
Floating Point Due to internal representations and rounding problems with floating point numbers, a comparison of these is usually difficult, because two floating point numbers will almost never have the exact same value. Therefore the comparison with ASSERT_EQ will fail. To solve this problem there is a specific macro for comparing floating point numbers. These match on four units in the last place (ULP). There are specific versions for float and double available:
ASSERT_FLOAT_EQ (0.0f, _sut -> myFloat ());
ASSERT_DOUBLE_EQ (0.0 , _sut -> myDouble ());
EXPECT_FLOAT_EQ (0.0f, _sut -> myFloat ());
EXPECT_DOUBLE_EQ (0.0 , _sut -> myDouble ());
gmock brings own matchers to gtest. Since it is possible to write your own matchers, individual comparisons can also be created. So complex comparisons can be outsourced to matchers and provided with proper error messages. Multiple use of matchers is also conceivable.
For example, if you want to compare the content of a vector of pointers, there is no standard comparison for this. So the user has to iterate the vector manually and compare the content. If one has several tests that compare these vectors, a lot of duplicate code quickly results. If the code is outsourced to functions, you get cluttered tests. Matchers can be used to create a comparison operation for the vectors that works with gtest. A simple example for this looks as follows:
MATCHER_P ( vectorContentEq , expectedVector , " Expected vector content does
not match .") {
bool result = (arg. size () == expectedVector . size ());
for ( int i = 0; i < std :: min(arg . size () , expectedVector . size ()); ++i) {
auto firstItem = arg.at(i);
auto secondItem = expectedVector .at(i);
if( firstItem != secondItem ){
result = false ;
break ;
}
}
return result ;
}
MATCHER_P defines a new matcher named vectorContentEq. The P stands for a matcher that expects a paramater. In this case the parameter is expectedVector. The length of the vectors and the content of the elements are compared.
There are also more matchers than assertion macros. So there are situations where testing with the matchers provided by Google makes more sense than with macros. For example, some matchers can compare parts of strings:
EXPECT_THAT ( myString , StartsWith ("My string Starts with "));
To use matchers, the EXPECT_THAT macro is used and the matcher is passed in the second argument. However, matchers are not a unique implementation of gmock. A similar concept can be found in newer jUnit versions. There, too, the expression matcher is used.
Testing functions with input parameters is one of the everyday tasks of a developer. If the function should behave the same with different parameters, the same test must be written several times. With gtest you pass parameters to a test and then specify with which values in the parameter the tests are to be executed.
TEST_P ( MyParamTest , TestName ) {
std :: string myParam = GetParam ();
... normal testing with usage of myParam ...
}
There are some small differences to be seen. The macro ends on P for parameterised test. In the implementation the call GetParam() was used to access the parameter. For readability reasons the parameter is assigned to a local variable. However, this procedure is not mandatory necessary. If the test class is now executed, no result is to be expected. The reason for this is that it must first be defined with which values the tests are to be executed. Several tests can be executed with the same parameters without any problems.
INSTANTIATE_TEST_CASE_P ( myParamTestInstance , MyParamTest , :: testing ::
Values (" value1 ", " value2 ", " value3 "));
Now all tests in the test case "MyParamTest" are executed three times, once with each parameter.
In contrast to other frameworks, assertions in gtest can be written in almost any function. Only two conditions must be met. First, the include command of gtest must be included, and second, fatal assertions must be handled separately. This applies to all assertions beginning with "ASSERT_" as well as the direct macro "Fail", which can only be used in functions with the return type "void".
It is good practice to outsource repetitive blocks of assertions to auxiliary classes. An example of this is a user object with multiple properties. If you want to check this, a whole series of assertions are available. It is obvious to move this block to an auxiliary class or auxiliary function. However the problem arises that the possible output of a failed assertion is no longer very meaningful. Because it is no longer at first sight from which test the assertion failed.
gtest has introduced the macro "SCOPED_TRACE" for this case, which appends a message with file and line number to the log in the current context. If the current scope is left, the message is also removed. For the example from just now this looks as follows:
TEST_F ( MyTestFixture , testUserWithSomething ) {
if( something ()){
SCOPED_TRACE (" Something ");
User * user = sut -> getUser ();
testUser ( user );
}
User * user = sut -> getUser ();
testUser ( user );
}
Without the "SCOPED_TRACE" macros it would be very difficult to find out in case of assertion errors in the "testUser" function which of the two cases has led to the failure. With the macro, however, a log of this type would be displayed in case of a failure in the "if":
path / MyTestFixture .cpp :198: Something
However, if the test fails only on the second call, the message is not present in the log.
The standard output of gtest looks like this:
... a lot mor tests and logs ...
[ OK ] MyTestCase . myTest (4 ms)
[----------] 4 tests from DeviceOnboardingManagerTest (18 ms total )
[----------] Global test environment tear - down
[==========] 452 tests from 40 test cases ran. (16823 ms total )
[ PASSED ] 452 tests .
However, the fact that the gtest outputs can be mixed with the actual outputs on "std::count" is problematic here. However, it is possible to modify these outputs. The following adjustments can be made:
It should also be mentioned here that gtest is supported by integrated development environments. For example, it is possible to execute only individual tests or test cases. In the CLion development environment, this looks like this:
Figure 1: gtest in CLion
gtest is considered a threadsafe on systems that provide "pthreads". The attribute provides information about whether it is safe to call program code from different threads. However, on other systems, such as Windows, it is not currently safe to execute assertions from two threads. However, this is not important for most applications.
In the second part of the blog series, we introduce the mock framework gmock and give our take on automated testing with the two featured frameworks. Stay tuned!
