Analyzing Code Coverage for Linux Test Projects

This tutorial shows how to use VisualGDB to quickly analyze the code coverage of your Linux Unit Test projects. We will create a basic test project based on the GoogleTest framework and show how to quickly find functions with incomplete coverage. We will also show how to use the call information displayed by VisualGDB to quickly find potential causes for bugs.

Before you begin, install VisualGDB 5.3 or later.

  1. Start Visual Studio and open the VisualGDB Linux Project Wizard:01-prj
  2. On the first page of the wizard select “Unit test project -> Use MSBuild” and pick your test framework. In this example we will use GoogleTest:02-test
  3. On the next page select the remote computer you want to target. In this tutorial we will build the code directly on Linux and VisualGDB will automatically collect and analyze code coverage reports and transfer them to Windows. Once you have selected the remote computer, click “Next”:03-machine
  4. Proceed with the default source access settings (upload modified sources via SSH) and click “Finish” to create the project:04-source
  5. VisualGDB will create a basic project including 3 unit tests based on the GoogleTest framework. Locate and open the <Project Name>Tests.cpp file that contains the generated tests:05-created
  6. We will now create a very basic queue class capable of enqueuing and dequeuing elements one-by-one and will then add a unit test validating it. Replace the contents of the “tests” file with the following code:

    Then add a basic test that will enqueue 3 elements and then dequeue them:

  7. Now you can use the Test Explorer window to run the test and verify that it succeeds:t1
  8. The fact that the test succeeded means that the scenarios covered by the test work as expected, however it doesn’t indicate how much of the programs’ code was actually tested. Code coverage reports fill this gap by providing detailed statistics on functions, blocks and lines executed during the test run. Open VisualGDB Project Properties, go to the Code Coverage page and enable the “Build code coverage reports” checkbox. VisualGDB will suggest automatically enabling code coverage instrumentation:07-coverage
  9. Instrumenting your program for code coverage will add significant run-time overhead, making your program slower than the non-instrumented version. You can then temporarily disable it via Visual Studio Project Properties -> Instrumentation -> Generate Code Coverage Reports (the option in VisualGDB Project Properties will stop VisualGDB from downloading and parsing the report files, but won’t disable the instrumentation):08-coverAs the instrumentation settings are stored separately for each configuration, you can create a separate configuration via VisualGDB Project Properties and use it solely for checking code coverage so that your regular test performance won’t be affected.
  10. It is also recommended to disable code coverage instrumentation for the files from the test framework itself. This will slightly improve the performance and remove unnecessary information from the coverage reports. You can do this by selecting those files in Solution Explorer, opening Visual Studio Properties and disabling report generation under Configuration Properties -> C/C++ -> Instrumentation:09-nocoverNote that the settings under the C/C++->Instrumentation only affect the compiler (but not the linker). If you are editing global project settings manually, use the Configuration Properties->Instrumentation page instead:10-run
  11. Run the test again. When it is completed, open the coverage report viewer via Test->VisualGDB Code Coverage Reports:11-showcover
  12. VisualGDB will display the list of functions along with their invocation count and coverage statistics. It will also highlight the covered lines directly in the source code:12-report
  13. Click on the “lines” column to sort the functions by the fraction of covered lines, then enter “file:tests.cpp” in the “filter” field to limit the displayed functions to your main source file:13-sortYou can hover the mouse over the ‘filter’ field to view help on supported filter expressions.
  14. The report will show that the Queue<int>::Enqueue() method has only 45% line. Double-click on it to navigate to its definition in the code:14-missingThe coverage report quickly shows that the following branches are not covered:
    • The branch checking whether the queue is empty before dequeuing
    • The branch checking whether enough memory is allocated before enqueuing more elements
    • The branch moving the enqueued elements to the beginning of the buffer before adding more (the current implementation is not optimal, but clearly demonstrates 2 different conditions)
  15. The easiest branch to cover is the empty queue check. Adjust the test method as follows, build the project and re-run the test:

  16. Covering both reallocation and moving requires interleaving enqueuing and dequeuing operations and increasing the batch sizes. To do this, replace the test code with the following:

    Then build and run the tests to ensure that all the code is now covered:16-coveredNote that the test coverage reports always show the coverage for the tests scheduled during a test run. I.e. if you only run 1 test out of 2, the functions checked by the second tests will be shown as not covered.
  17. Now we will show how VisualGDB manages the coverage reports internally. Open the project folder in Windows Explorer and go to the CoverageReports subdirectory:17-reportsEach .scovreport file corresponds to one coverage report from one test run.
  18. You can delete the old report files either manually, or in the Coverage Report viewer. VisualGDB will also automatically remove the old report files when their amount exceeds the limit specified via global settings: cleanup
  19. If the coverage reports appear inconsistent, open VisualGDB settings (from Tools->Options) and enable the “Keep raw coverage reports” setting:19-raw
  20. VisualGDB will then keep the original gcov report files downloaded for each session:20-rawfiles
  21. You can check the consistency of the reports by looking at the job.txt and the .gcov files referenced from it:21-txtfilesThe job.txt file contains lines starting with “>” that specify the base source directories followed by paths of .gcov files listing line and function coverage (VisualGDB automatically runs the gcov tool to generate .gcov files from .gcda and .gcno files). If the .gcov files appear inconsistent, check that you are using a recent version of the compiler.
  22. Finally we will show how to use the code coverage to quickly spot inconsistencies in the program behavior. We will add a very basic smart pointer class and try to use it with our custom queue:

    If you run the test now, VisualGDB will trigger an “invalid memory access” error in the assignment operator:22-crash
  23. Try commenting it out and run the test again. Note how the code coverage shows 10 invocations of the operator=(), but only one invocation of the constructor:23-cov
  24. As a result, operator=() is trying to free a pointer that was never initialized and contains garbage. You can address this by modifying the Queue() constructor to call the placement new each time it allocated an object, acting similarly to the new[] operator:

    The reallocation logic should also contain a call to placement new for each allocated object:
  25. Build and run the project now to see that the tests pass. Note that now the Enqueue() method declaration corresponds to 2 physical methods: Queue<int>::Enqueue() and Queue<SmartPointer>::Enqueue(). VisualGDB can distinguish between them and will show you the detailed statistics once you click on the “2/2 variants covered” link above the method declaration:24-popup

You can find the source code for the program shown in this tutorial in our Github repository.