Improving your Embedded Firmware Quality with Unit Tests

This tutorial shows how to use VisualGDB to create unit tests for embedded projects that will run directly on the embedded device. Unit tests are supported on the ARM Cortex devices using Segger J-Link or OpenOCD and can be supported on other devices using VisualGDB extension SDK. Before you begin, install VisualGDB 5.2 or later.

In this tutorial we will create a simple test project for the STM32F4Discovery board that will verify that the execution time of the sinf() function matches a certain constraint. We will show how switching the FPU mode from hardware to software would instantly cause the test to fail. If your program contains performance-critical code, you can use this approach to separately test the performance of each critical part and catch slowdowns caused by recent changes before they start causing trouble.

  1. Begin with starting Visual Studio and selecting File->New Project->VisualGDB->Embedded Project Wizard:01-embproj
  2. On the first page select Create a new project -> Unit test project -> TinyEmbeddedTest:02-tinyembtestVisualGDB also supports CppUTest and GoogleTest frameworks out-of-the-box, so you can use one of those if your device has enough FLASH memory. The TinyEmbeddedTest framework provided by Sysprogs focuses on using as little FLASH space as possible and providing basic CppUTest-compatible API:
  3. On the next page select the ARM toolchain and choose your device from the device list:03-deviceTo minimize the FLASH memory usage you can select “newlib-nano” as the C library type (requires the “provide default stubs for system calls” flag). We will show the difference it makes in the end of this tutorial.
  4. Proceed with the default sample for the test project:04-sample
  5. Finally select your debug method. Unit tests work with OpenOCD and Segger J-Link, so we select OpenOCD here:05-debug
  6. Press “Finish” to create your project. VisualGDB will create a basic project with 3 tests. Check the SuccessfulTest1 test for a quick overview of various checks supported by the test framework:06-checks
  7. Build your project and select Test->Run->All Tests to verify that the tests can be started properly. Alternatively you can select the tests you want to run in the Test Explorer, right-click there and select “Run”: 07-deftests
  8. Now we will measure the time taken by the sinf() function in soft-FP and hard-FP modes. On many ARM Cortex devices (including most STM32F4 devices) this can be done by enabling the debug cycle counter, setting the CYCCNT register to 0 before running a function and then reading the CYCCNT register immediately after. The following function demonstrates this:
  9. VisualGDB test projects are normal executable projects with a main() function. Hence you can quickly prototype things by calling functions like this one directly from main() after the HAL_Init() call. Then set a breakpoint at the “nop” line and start debugging. Once the breakpoint is hit, note down the value of the “cycles” variable:08-swcycles
  10. Now we will switch the project to the hardware floating point mode:09-hwmode
  11. Build it and check the new time:10-hwcyclesSwitching to hardware FP mode makes more than 7.5x difference! So if a toolchain update or an accidental change in settings breaks this, the performance impact could be quite dramatic.
  12. Now let’s make a simple test that will automatically verify that the sinf() function completes in time. Declaring tests using TinyEmbeddedTest and CppUTest consists of 2 steps:
    • Defining a test group that will share common initialization and cleanup steps
    • Defining the actual tests

    The test group can be easily defined using the TEST_GROUP() macro. We will put the code enabling the debug cycle counter there so that we can run multiple timing-related tests without enabling it each time:

    The actual test is very straight-forward as well:

    Also don’t forget to remove the MeasureSinTime() function as we don’t need it anymore.
  13. If you build your project now and run the newly added TimingTests::sinf() test, VisualGDB should show that it passes:11-hard-pass
  14. If you switch your project back to software FP mode, build it and run the test, the timing change should get caught immediately:12-soft-fail
  15. You can debug failed tests by right-clicking at them and selecting “Debug Selected Tests”. Once a test failure is reported, VisualGDB will treat it as an exception and stop:13-break
  16. Now we will add 2 small improvements to our test. First of all, when a failure is detected, we can output something more meaningful than “unexpected boolean value”. Second of all, if the timing is OK, we can still report it to keep a track of it in the test logs. Both tasks can be accomplished with a simple helper class:

    It will automatically reset the cycle counter when created and will automatically check the timing when leaving the scope. This will reduce the actual test method to this:

    Note that the ‘volatile’ statements are needed to prevent the optimizer from throwing away the call to ‘sinf()’ as we are calling it with a constant input and not doing anything with the value.
  17. If you run the test in software FP mode now, you will see a much more meaningful message:14-logfail
  18. Similarly a successful test will now output the actual timing into the Output->Tests window:15-logpass
  19. You can easily use VisualGDB tests with a continuous integration system. Simply configure it to run the following command line:

  20. For non-MSBuild projects, the test container file is the .vgdbsettings file corresponding to the debug or release configuration. For MSBuild projects where the binary name is computed in Visual Studio on-the-fly, a test container is a simple text file containing the .vgdbsettings file path on the first line and the full binary path on the next one:17-testcontainerVisualGDB automatically generates those files when you start your tests from Visual Studio.
  21. A test run started from command line will generate an XML file with the details on the test outcome, output produced by it and any encountered errors:
  22. If you ever start running out of the FLASH memory on your device with unit tests, consider switching the C library to Newlib-nano and enabling the “Provide default stubs for system calls” flag: 18-setnano
  23. Combined with the TinyEmbeddedTest framework, this will dramatically reduce the size of the binary (96KB to 19KB in this example):19-nano