{"id":8134,"date":"2023-07-12T16:57:40","date_gmt":"2023-07-12T23:57:40","guid":{"rendered":"https:\/\/visualgdb.com\/w\/?p=8134"},"modified":"2023-10-03T13:01:04","modified_gmt":"2023-10-03T20:01:04","slug":"using-live-tracing-to-record-firmware-behavior","status":"publish","type":"post","link":"https:\/\/visualgdb.com\/tutorials\/tracing\/embedded\/","title":{"rendered":"Using Live Tracing to Record Firmware Behavior"},"content":{"rendered":"<p>This tutorial shows how to use VisualGDB Live Tracing to record the behavior of a simple embedded project. We will create a basic demo application that randomly adds and removes items from 4 simple queues, will let it run until an overflow occurs, and will then use Live Tracing to:<\/p>\n<ul>\n<li>Create a table of exact enqueue\/dequeue actions leading to the overflow<\/li>\n<li>Plot the queue utilization over time, seeing which item caused each change<\/li>\n<li>Quickly find the time when a particular item was inserted in the queue<\/li>\n<\/ul>\n<p>Live Tracing works by patching the firmware to record the selected data in a circular buffer. It does not require changing the source code, or using any special debugging hardware. It runs without stopping the target, is not affected by JTAG latency, and has very minor overhead. It can be used to reconstruct the events leading to a crash, create reference recordings of data-heavy scenarios (e.g. USB packets), or see what the program is doing in real time without ever having to stop it in the debugger.<\/p>\n<p>Live tracing is supported starting from VisualGDB 6.0 and requires the Custom edition or higher.<\/p>\n<ol>\n<li>Launch Visual Studio and begin creating a new Embedded Project:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/01-newprj.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8135\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/01-newprj.png\" alt=\"\" width=\"1014\" height=\"675\" \/><\/a><\/li>\n<li>Enter the name and location for your project: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/02-prjname.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8136\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/02-prjname.png\" alt=\"\" width=\"1014\" height=\"675\" \/><\/a><\/li>\n<li>In this tutorial we will use the most basic setup possible, so proceed with creating a regular Embedded Application using Advanced CMake: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/03-cmake.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8137\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/03-cmake.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Select the toolchain and device you would like to target: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/04-dev.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8138\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/04-dev.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Proceed with the most basic &#8220;LEDBlink (HAL)&#8221; sample:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/05-sample.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8139\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/05-sample.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Finally, select the debugging settings that work with your device and ensure the &#8220;<strong>Enable software tracing<\/strong>&#8221; checkbox at the bottom of the page is checked. Then click &#8220;Finish&#8221; to create the project: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/06-debug-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8454\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/06-debug-1.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Replace the contents of the main source file with <a href=\"https:\/\/github.com\/sysprogs\/tutorials\/blob\/master\/visualgdb\/tracing\/BasicTracingDemo\/BasicTracingDemo.cpp\">this code<\/a>. It creates 4 global queues (<strong>s_Queues<\/strong>) and keeps randomly inserting and removing items from them (<strong>EnqueueTestItem()<\/strong> and <strong>DequeueItem()<\/strong>) until the total number of allocated items exceeds 16 (simulating an overflow). Build the project, start debugging it and make sure the &#8220;bkpt 255&#8221; line triggers a breakpoint: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/08-bkpt-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8162\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/08-bkpt-1.png\" alt=\"\" width=\"1367\" height=\"873\" \/><\/a>If you use the classical debugging functionality (call stack, watch), you can view the current state of the queues (e.g. how many items have been inserted in each one), however it doesn&#8217;t show how we arrived at this state.<\/li>\n<li>Live Tracing will do exactly that &#8211; show what the program was doing before the breakpoint was hit. Open the VisualGDB Project Properties, go to the <strong>Software Tracing<\/strong> page and ensure the &#8220;Enable software tracing&#8221; checkbox is checked:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8457\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout1.png\" alt=\"\" width=\"1053\" height=\"814\" \/><\/a><\/li>\n<li>You can switch to the Advanced view to fine-tune various tracing parameters (e.g. where exactly to place the traced data and how to hook the vector table), although the default settings should work for most cases:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8458\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout2.png\" alt=\"\" width=\"1143\" height=\"950\" \/><\/a> We advise enabling the relocation record output (doesn&#8217;t affect the actual firmware) so that VisualGDB can double-check that the generated tracing code won&#8217;t overlap anything in the original image.<\/li>\n<li>Once the tracing has been enabled, VisualGDB will show 3 additional views:\n<ul>\n<li>Tracepoint bar <strong>(1)<\/strong><\/li>\n<li>Tracepoint list <strong>(2) <\/strong>that can be opened via <strong>Debug-&gt;Windows-&gt;Tracepoints<\/strong><\/li>\n<li>Trace data view <strong>(3)<\/strong> that opens when you select a tracepoint in the tracepoint list<\/li>\n<\/ul>\n<p><a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/10-panels.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8144\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/10-panels.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>You can use the tracepoint list to quickly trace arbitrary functions, however in this tutorial we will focus on using the tracepoint bar. Click on the last line of the <strong>EnqueueTestItem()<\/strong> function and select to trace <strong>value<\/strong> and <strong>queue<\/strong>:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/11-tp1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8145\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/11-tp1.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>Similarly, put one more tracepoint at the return statement in <strong>DequeueItem()<\/strong> and another one at the call to <strong>free()<\/strong>. Trace <strong>queueNumber<\/strong>, <strong>queue<\/strong> and <strong>item<\/strong> variables from each of them. You can always use the <strong>Tracepoints<\/strong> window to see the list of current tracepoints and change the data they collect:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/12-list.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8146\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/12-list.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>Double-click in the &#8220;Tracepoint&#8221; column to give the tracepoints more meaningful names than the file and line number:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/13-renamed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8147\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/13-renamed.png\" alt=\"\" width=\"1367\" height=\"871\" \/><\/a><\/li>\n<li>Select all 3 tracepoints and enable tracing of <strong>g_Queues[*].ItemCount<\/strong> in the Trace Data window:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/14-globals.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8148\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/14-globals.png\" alt=\"\" width=\"1367\" height=\"873\" \/><\/a><\/li>\n<li>Now press F5 to begin the debug session. Once the hardcoded breakpoint triggers, open the <strong>Events<\/strong> view in the <strong>Tracepoints<\/strong> window: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/15-recorded.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8149\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/15-recorded.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>You can double-click on each event and VisualGDB will show the code location associated with it, as well as the data that was captured when it happened: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/16-evt.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8150\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/16-evt.png\" alt=\"\" width=\"1367\" height=\"873\" \/><\/a>In this example the target is actually stopped at a breakpoint, however tracepoints would work the same way if the target was running. In each case the data was captured, stored in the circular buffer, and the program continued running without any interruption.<\/li>\n<li>Software tracing can quickly produce large quantities of data, so VisualGDB provides a few mechanisms for navigating it easily. The one we will demonstrate first is the table views. Click &#8220;<strong>Create a new table view<\/strong>&#8221; in the <strong>Trace Data<\/strong> window:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/17-table.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8151\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/17-table.png\" alt=\"\" width=\"1367\" height=\"871\" \/><\/a><\/li>\n<li>Rename the view to &#8220;Queue Events&#8221;, select one of the &#8220;<strong>Enqueued item<\/strong>&#8221; events in the event list, and drag the <strong>queue<\/strong> and <strong>value<\/strong> nodes into the column list:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/18-drag.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8152\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/18-drag.png\" alt=\"\" width=\"1367\" height=\"871\" \/><\/a><\/li>\n<li>Select a &#8220;<strong>Dequeued item<\/strong>&#8221; event and drag the <strong>queue<\/strong> and <strong>item.Value<\/strong> nodes into the same columns:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/19-drag2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8153\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/19-drag2.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a>Similarly, drag <strong>queue.ItemCount<\/strong>. It is important to drag the new items exactly into the same columns, so VisualGDB knows they are interchangeable.<\/li>\n<li>Note how the table view now shows both &#8220;<strong>enqueued<\/strong>&#8221; and &#8220;<strong>dequeued<\/strong>&#8221; events, displaying the queue pointer and the value that was added or removed from the queue. You can sort the rows in different ways, or filter them to show a specific queue.<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/20-singleq-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8165\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/20-singleq-2.png\" alt=\"\" width=\"1367\" height=\"871\" \/><\/a>Note how filtering the events clearly shows how ItemCount keeps increasing after each enqueueing, and decreases after each dequeueing.<\/li>\n<li>Now we will use a table view to plot how the queue utilization changed over time. Click &#8220;<strong>Create a new graph view<\/strong>&#8221; in the <strong>Trace Data<\/strong> window:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/21-graph-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8166\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/21-graph-1.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>Drag the <strong>ItemCount<\/strong> nodes for each of the queues into 4 separate columns and rename the columns accordingly. VisualGDB will now plot them so you can see how the utilization was slowly growing over time: <a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/22-plotted.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8156\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/22-plotted.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a>You can double-click at any point in the graph to quickly jump to that event in the <strong>Tracepoints<\/strong> window where you can flag it as favorite, or give it a meaningful name.<\/li>\n<li>Finally, we will show how to easily find related events. Click &#8220;<strong>Edit quick search indexes<\/strong>&#8221; in the <strong>Trace Data<\/strong> window:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/23-indexes.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8157\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/23-indexes.png\" alt=\"\" width=\"1367\" height=\"873\" \/><\/a><\/li>\n<li>Drag the <strong>item.Value<\/strong> from the &#8220;<strong>Dequeued<\/strong>&#8221; event and &#8220;<strong>value<\/strong>&#8221; from the &#8220;<strong>Enqueued<\/strong>&#8221; one into the same column, so that VisualGDB knows they are interchangeable. Now each time you view either of the 2 event types, VisualGDB will display the lightning symbol near the &#8220;<strong>value<\/strong>&#8221; field:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/25-list.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8158\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/25-list.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>If you click it, VisualGDB will show all events with the same value. In this example, it shows when this specific item was enqueued, however you can also use it to show all operations on a certain mutex (by indexing the mutex handle), or all events that happened when a particular thread was active (by indexing the current thread pointer):<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/26-filtered-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8167\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/26-filtered-1.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>Press Shift-F5 to end the debug session. VisualGDB will create a trace report storing all the tracepoints and events captured during the session. You can replay it an any time later to compare how the firmware operates under different circumstances:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/27-replay.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8160\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/27-replay.png\" alt=\"\" width=\"1367\" height=\"872\" \/><\/a><\/li>\n<li>Tracing everything all the time is not practical, and setting the tracepoints from scratch could be inconvenient. So, VisualGDB provides 2 mechanisms for managing the tracepoints. You can export them to a tracepoint list file (including the exact data you selected for tracing) and import it later, or organize them into groups and keep enabling\/disabling the groups as you need them:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/28-export.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8161\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/28-export.png\" alt=\"\" width=\"1367\" height=\"873\" \/><\/a><\/li>\n<li>If the tracepoints generate the data faster than VisualGDB can read it, the program will continue running without recording these events and VisualGDB will reflect it in the status line of the Tracepoints window. If this happens too often, consider increasing the trace buffer size via <strong>VisualGDB Project Properties<\/strong>:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8459\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/layout3.png\" alt=\"\" width=\"1143\" height=\"718\" \/><\/a><\/li>\n<li>If you are using a device supported by our <a href=\"https:\/\/visualgdb.com\/documentation\/flashpatchers\/\">FLASH patching plugins<\/a> (or are using Segger J-Link that handles FLASH patching on its own), you can add\/remove tracepoints at any point during debugging. If not, you would need to restart debugging after adding\/removing FLASH tracepoints, however it can enable\/disable tracepoints and change which data is being collected. If you would like to add the tracepoints dynamically and cannot run the entire firmware from RAM, consider enabling the &#8220;<strong>Patch code for initially disabled tracepoints<\/strong>&#8221; checkbox and reserving some memory for runtime trace program changes. You can then create disabled tracepoints at all the relevant locations when the program is not running, and enable them as you need later:<a href=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/advanced.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-8462\" src=\"https:\/\/visualgdb.com\/w\/wp-content\/uploads\/2023\/07\/advanced.png\" alt=\"\" width=\"1143\" height=\"718\" \/><\/a>If you are not planning to do that, consider disabling the &#8220;<strong>patch code<\/strong>&#8221; setting so that the disabled tracepoints don&#8217;t introduce any overhead.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial shows how to use VisualGDB Live Tracing to record the behavior of a simple embedded project. We will<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[247],"tags":[248,61],"_links":{"self":[{"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/posts\/8134"}],"collection":[{"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/comments?post=8134"}],"version-history":[{"count":6,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/posts\/8134\/revisions"}],"predecessor-version":[{"id":8463,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/posts\/8134\/revisions\/8463"}],"wp:attachment":[{"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/media?parent=8134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/categories?post=8134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/visualgdb.com\/w\/wp-json\/wp\/v2\/tags?post=8134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}