Embedded Integration Tests vs. Unit Tests

VisualGDB supports 2 types of tests for embedded projects: Unit Tests and Integration Tests. This page explains the differences between them and the recommended use cases.

Unit Tests

Unit Tests are essentially functions inside your program that check for certain conditions. They can be defined using one of the 3 test frameworks: CppUTest, GoogleTest and TinyEmbeddedTest (the only supported framework for IAR and Keil compilers).

The unit test functions run on the target device, so they need to be a part of the embedded binary, affecting the memory footprint. When you select a subset of unit tests in Test Explorer and run them, VisualGDB does the following:

1. Programs the entire binary into the FLASH memory (including the tests that were not selected).
2. Passes the list of selected tests to the test framework.
3. Captures the information about any failed tests (error messages, call stacks) and shows it in Test Explorer.

The target program can automatically distinguish between regular debugging sessions (started with F5) and unit test sessions (launched via Test Explorer) using the IsRunningUnitTests() function. Below is a typical contents of main() for a project containing unit tests:In order to physically exclude the unit tests from the release binary, simply create separate test and non-test configurations via VisualGDB Project Properties:Then define a macro (e.g. HAS_TESTS) in the test configurations, and check it in the code.

Alternatively, you can set the “Excluded from Build” flag on the source files with unit tests for the non-test configurations:

Running via Command Line

You can use VisualGDB to run embedded unit tests via command line without having to launch Visual Studio. For Advanced CMake projects you can use the command line below:

VisualGDB.exe /runtests <project.vgdbcmake> /TargetPath:<relative path to the application> /output:<output file>

The /TargetPath is needed because the final executable path is computed by CMake after evaluating the CMakeLists.txt files and is not stored anywhere in the VisualGDB settings.

For MSBuild projects you can use the command line below:

VisualGDB.exe /runtests <container.vgdbtestcontainer> /output:<output file>

Test container files (.vgdbtestcontainer) are created during the project build and contain various information computed by MSBuild. You can read more about them in this tutorial.

You can replace the /output argument with /vsoutput to produce test reports in the Visual Studio Test Result format (.trx).

If you would like to produce tracing reports (e.g.containing all function calls) when running tests, use the Tracepoints window to export your tracepoints into an XML file, and then add the following parts to the test running command line:

/traceconfig:<tracepoints.xml> /tracereport:<regular report> /xmltracereport:<XML trace report>

You can then open the trace report file in Visual Studio and step through it to view the function calls, variable values, and other data gathered via tracing.

Unit Tests for Large Projects

Large embedded projects can be partitioned into multiple static libraries linked together into the final binary. Each library can contain both the regular logic and the unit tests that will be conditionally compiled when building a test-enabled configuration:You can create static libraries via the regular VisualGDB Embedded Project Wizard, however adding unit tests to them will require extra care:

1. Open VisualGDB Project Properties for the libraries and unreference all embedded frameworks. Also set the “exclude startup file” flag on the Embedded Project page. This will make sure no code between the main project and the static libraries gets duplicated.
2. If you reference the test framework from the libraries, VisualGDB will include an extra copy of the .cpp files from the framework in each of them. To avoid this, locate the <mcu>.props file in the main project’s directory and fork it into a test.props file that only contains includes and defines for the test framework, e.g.:

   <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <ItemDefinitionGroup> <ClCompile> <PreprocessorDefinitions>SYSPROGS_TEST_PLATFORM_EMBEDDED;TINY_EMBEDDED_TEST_CONTEXT_RESTORE_MODE=0;TINY_EMBEDDED_TEST_HOOK_STDIO=0;%(ClCompile.PreprocessorDefinitions)</PreprocessorDefinitions> <AdditionalIncludeDirectories>%(ClCompile.AdditionalIncludeDirectories);$(TESTFW_BASE)/com.sysprogs.unittest.tinyembtest</AdditionalIncludeDirectories> </ClCompile> </ItemDefinitionGroup> </Project>

   <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <ItemDefinitionGroup> <ClCompile> <PreprocessorDefinitions>SYSPROGS_TEST_PLATFORM_EMBEDDED;TINY_EMBEDDED_TEST_CONTEXT_RESTORE_MODE=0;TINY_EMBEDDED_TEST_HOOK_STDIO=0;%(ClCompile.PreprocessorDefinitions)</PreprocessorDefinitions> <AdditionalIncludeDirectories>%(ClCompile.AdditionalIncludeDirectories);$(TESTFW_BASE)/com.sysprogs.unittest.tinyembtest</AdditionalIncludeDirectories> </ClCompile> </ItemDefinitionGroup> </Project>

3. Reference test.props from the library projects after the Microsoft.Cpp.Targets reference:

   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <Import Project="test.props" />

     <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <Import Project="test.props" />

4. Using VisualGDB project properties, create test and non-test configurations in the library projects. Make sure the unit test code is only built in test configurations.
5. Enable the whole archive mode for libraries. Otherwise, the test code will be deemed unused during linking and will get discarded:
6. Make sure the main project also contains test and non-test configurations. Reference the libraries from the main project.

You can also go for a more complicated layout having multiple test binaries additionally to the main binary:This will reduce the memory footprint of test binaries, since only one of them will need to be loaded at the same time. VisualGDB will automatically show the tests from multiple binaries in Test Explorer and will handle the memory loading for you.

Embedded Integration Tests

The Embedded Integration Tests are automatically recorded scripts that can replay common debugging steps and verify that the program responds to them as expected. The following steps can be recorded:

• Setting breakpoints
• Stepping through code
• Changing variable values
• Using ‘Set Next Statement’

You can create the following assertions for the integration tests:

• Check that a certain function is present at the call stack
• Check that a specified variable has the specified value
• Check that a custom GDB command produces expected output

Embedded Integration Tests are stored in .vgdbtests files and will be shown in Test Explorer as long as the project includes the .vgdbtest file in Solution Explorer.

The Integration Test logic runs on the debugger side, so they do not introduce any overhead to the tested binary. You can use them in the production code to simulate common conditions (e.g. modify variables to simulate pressed buttons) and verify that the program responds as expected.