11.1. Source-Based Code Coverage in tiarmclang

The TI Arm Clang Compiler Tools (tiarmclang) support Source-Based Code Coverage that is particularly suited for embedded applications. In addition to being generally useful for thorough application development, code coverage is required by internal and external developers in the Industrial and Automotive markets for Functional Safety.

The following forms of Code Coverage are supported:

• Function coverage is the percentage of functions which have been executed at least once. A function is considered to be executed if any of its instantiations are executed. * Instantiation coverage is the percentage of function instantiations which have been executed at least once. Template functions and static inline functions from headers are two kinds of functions which may have multiple instantiations.
• Line coverage is the percentage of code lines which have been executed at least once. Only executable lines within function bodies are considered to be code lines.
• Region coverage is the percentage of code regions which have been executed at least once. A code region may span multiple lines (e.g in a large function body with no control flow). However, it is also possible for a single line to contain multiple code regions (e.g in “return x || y && z”). Region coverage is equivalent to Statement coverage provided by other vendors.
• Branch coverage is the percentage of source-condition-based branches that have been taken at least once. The new tiarmclang tools’ support for Branch coverage (also known as Branch Condition coverage) provides a finer level of coverage than that which is provided by other vendors, allowing users to track “True/False” execution coverage across leaf-level boolean expressions used in conditional statements. This makes it much more informative and useful than Decision coverage* that some other vendors support, which only tracks execution counts for a single control flow decision point, which may be a boolean expression comprised of conditions and zero or more boolean logical operators.

Note

Coverage across Function Instantiations

If a function has multiple instantiations, as in the case of C++ function templates, the instantiation reflecting the maximum coverage of lines, regions, or branches is used for the final coverage tally. In other words, a function definition is considered fully covered if any one of its instantiations is fully covered with respect to lines, regions, or branches.

11.1.1. Support for Embedded Use Cases

The tiarmclang compiler tools minimize the data size requirements of Code Coverage by allocating memory space for only the counters and keeping all other coverage related information in non-allocatable sections preserved in the object file itself. This ensures that target memory is only utilized for incrementing counters. In addition, the runtime support only supports writing counters to a file as part of a “baremetal” profiling model and nothing else. Support for writing a full raw profile file, merging counters, etc., is not included as part of tiarmclang.

Note that instrumentation that is inserted to track the counters will introduce cycle performance and codesize overhead, depending on the size of the program. This is due to the additional instructions needed, number of counters needed, and impact to existing code optimization. Reducing the size of counters will be addressed as a future enhancement for the compiler to decrease the memory footprint introduced by code coverage to better mitigate the codesize overhead.

11.1.2. Effects of Code Optimization

The tiarmclang compiler derives instruction-to-source mappings through the Abstract Syntax Trees during the compilation’s Code Generation (CodeGen) phase that are eventually lowered to an intermediate representation, which is where counter instrumentation occurs. Counter instrumentation is completed prior to optimization passes that operate on the intermediate representation, so this means that coverage data is very accurate with respect to the source code. Counter increments that would have occured in an unoptimized program occur in the optimized variant. For example, counter mapping regions for an inlined function are created with instrumention prior to inlining. If inlining is performed, the instrumentation is inlined along with it. The resulting execution counts map back to the original source as though the function had never been inlined.

While counter instrumentation is not obstructed by optimization, the presence of counter instrumentation may inhibit certain optimizations

11.1.3. Tool Usage

In addition to the tiarmclang compiler, which is used to produce counter instrumentation, the tools used to produce and visualize code coverage data are tiarmprofdata and tiarmcov.

11.1.3.1. Useful Coverage Profile Merging Options (tiarmprofdata)

USAGE: tiarmprofdata merge [options] <filename ...>

Format of output profile

• –binary - binary encoding (default)
• –text - test encoding

Profile kind

• –instr - instrumentation profile (default)
• –obj-file=<string> - object file
• –output=<file> - output file
• –remapping-file=<file> - symbol remapping file
• –sparse - generate a sparse profile (only meaningful for –instr)

11.1.3.2. Useful Coverage Visualization Options (tiarmcov)

USAGE: tiarmcov {subcommand} [OPTION]...

Subcommands

• export - Export instrprof file to structured format either as text (JSON) or as LCOV. * JSON: tiarmcov export –format=json * LCOV: tiarmcov export –format=lcov
• report - Summarize instrprof style coverage information.
• show - Annotate source files using instrprof style coverage.

Function Filtering Options

• –ignore-filename-regex=<string> - Skip source code files with file paths that match the given regular expression.
• –line-coverage-gt=<number> - Show code coverage only for functions with line coverage greater than the given threshold.
• –line-coverage-lt=<number> - Show code coverage only for functions with line coverage less than the given threshold.
• –name=<string> - Show code coverage only for functions with the given name.
• –name-regex=<string> - Show code coverage only for functions that match the given regular expression.
• –name-whitelist=<string> - Show code coverage only for functions listed in the given file.
• –region-coverage-gt=<number> - Show code coverage only for functions with region coverage greater than the given threshold.
• –region-coverage-lt=<number> - Show code coverage only for functions with region coverage less than the given threshold.

General Options

• –instr-profile=<string> - File with the profile data obtained after an instrumented run.
• –num-threads=<uint> - Number of merge threads to use (default: autodetect).
• –object=<string> - Coverage executable or object file.
• –output-dir=<string> - Directory in which coverage information is written out.
• –path-equivalence=<string> - <from>,<to> Map coverage data paths to local source file paths.
• –project-title=<string> - Set project title for the coverage report.
• –show-branch-summary - Show branch condition statistics in summary table.
• –show-instantiation-summary - Show instantiation statistics in summary table.
• –show-region-summary - Show region statistics in summary table.
• –summary-only - Export only summary information for each source file.

Source-Based Viewing Options (for “tiarmcov show”)

• –show-branches=<value> - Show coverage for branch conditions. * =count - show True/False counts * =percent - show True/False percent
• –show-expansions - Show expanded source regions.
• –show-instantiations - Show function instantiations.
• –show-branches=<value> - Show coverage for branch conditions.
• –show-line-counts-or-regions - Show the execution counts for each line, or the execution counts for each region on lines that have multiple regions.
• –show-regions - Show the execution counts for each region.

11.1.4. Generating Instrumented Binaries

Source code must be built using tiarmclang with -fprofile-instr-generate -fcoverage-mapping options. For example:

tiarmclang -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo

11.1.5. Retrieving the Counters From Memory

Once the executable has been loaded and executed one or more times, the counters should be retrieved from memory and written to a raw profile data file on the host. Counters are stored in an allocated memory section named __llvm_prf_cnts, and this section is demarcated with the start and stop symbols, __start__llvm_prf_cnts and __stop__llvm_prf_cnts, which can allow the target memory to be read from the host. The data retrieved in memory should be saved to a file, and this file is the raw profile counter file.

11.1.5.1. Retrieving Counters Using the CCS Scripting Console

Retrieving counters from memory can be done in Code Composer Studio (CCS) using the following example script, which can be pasted into the CCS scripting console:

var scriptEnv = Packages.com.ti.ccstudio.scripting.environment.ScriptingEnvironment.instance();
var server = scriptEnv.getServer("DebugServer.1");
var session = server.openSession("Texas Instruments XDS110 USB DebugProbe_0/CORTEX_M4_0");

var cntStart = session.symbol.getAddress("__start___llvm_prf_cnts");
var cntStop = session.symbol.getAddress("__stop___llvm_prf_cnts");

var cntContent = session.memory.readData(0, cntStart, 8, cntStop - cntStart);

var executable = session.symbol.getSymbolFileName();
var outFile = new Packages.java.io.RandomAccessFile(executable + ".cnt" , "rw");

outFile.setLength(0);
for each (var val in cntContent) {
    outFile.writeByte(Number(val));
}
outFile.close();

This example script will produce a raw profile counter file named after the executable using the “.cnt” suffix.

11.1.5.2. Retrieving Counters Using Compiler Runtime Support

Alternatively, the counter data can also be retrieved from memory using a function that is provided as part of the compiler runtime support, __llvm_profile_write_file(). This function will write the counters from the target to the host using runtime routines (fwrite()). Any other means of downloading the data may also be used. This will produce a raw profile counter file using the default filename default.profraw.

int test_main(int argc, const char *argv[]) {
  // Call into an important routine
  important_func1();

  // Call into an important routine
  important_func2();

  // Write out counter details to file
  __llvm_profile_write_file();

  // Exit
  return 0;                                                                     }

11.1.6. Processing the Raw Profile Counter Data Into an Indexed Profile Data File

An indexed profile data file should be produced for each executable that is run; it is produced based on a raw profile counter file that has the runtime counter data retrieved from memory (see Retrieving the Counters From Memory section above).

This is done by invoking the tiarmprofdata utility and indicating the raw profile counter file as well as the executable used to produce it. This is required since in order to support embedded use cases, pertinent code coverage information must be extracted from non-allocatable sections in the executable. The result is an indexed profile data file. In the example below, the raw profile counter files used as input are app1.profcnts, app2.profcnts, and app3.profcnts. The resulting indexed profile data file produced for each is app1.profdata, app2.profdata, and app3.profdata, respectively.

tiarmprofdata merge -sparse -obj-file=app1.out app1.profcnts -o app1.profdata
tiarmprofdata merge -sparse -obj-file=app2.out app2.profcnts -o app2.profdata
tiarmprofdata merge -sparse -obj-file=app3.out app3.profcnts -o app3.profdata

11.1.6.1. Merging Multiple Indexed Profile Data Files from Multiple Executables

An indexed profile data file for each executable must be produced before any profile data from multiple executables can be merged. If multiple executables have been run based on the same source code base, the corresponding indexed profile data files for each of the executables can then be merged into a single indexed profile data file.

tiarmprofdata merge -sparse app1.profdata app2.profdata app3.profdata -o app_merged.profdata

Wildcards can be used to identify the range of indexed profile data files used as input.

11.1.7. Visualization

In order to visualize the code coverage, the single merged indexed profile data file along with each of the corresponding executables must be given as input to the tiarmcov visualization tool. The visualization tool can be used to generate a dump of the source file along with a summary report in either HTML or Text format. The names of each executable must be specified individually by name using the –object=<executable option.

11.1.7.1. HTML Format

When generating HTML output, a summary coverage report is also generated at the root of a directory tree that contains coverage data for each of the files. For the source-based coverage views, it is recommended to use –show-expansions and –show-instantiations options to see the full view of all macro expansions and function template instantiations, respectively. In addition, branch coverage information can be included in the source-based view, and it can be represented in terms of execution count or percentage.

The following example will visualize coverage in HTML with macros and templates expanded; it will also include detailed branch coverage in terms of execution count.

tiarmcov show --format=html --show-expansions --show-instantiations --show-branches=count --object=./app1.out --object=./app2.out --object=./app3.out -instr-profile=app-merged.profdata --output-dir=/example/directory

11.1.7.2. Text Format

When generating Text output, the summary coverage report is generated using a separate tiarmcov report option. For example, to view the source-based coverage view:

tiarmcov show --show-expansions --show-branches=count --object=./app1.out --object=./app2.out --object=./app3.out -instr-profile=app-merged.profdata

To view the report:

tiarmcov report --object=./app1.out --object=./app2.out --object=./app3.out -instr-profile=app-merged.profdata

Text Format Output Example

    2|       |
    2|       |#include <stdio.h>
    3|       |#include <stdlib.h>
    4|       |
    5|       |#ifdef __clang__
    6|       |extern void __llvm_profile_write_file(void);
    7|       |#endif
    8|       |
    9|      2|#define BRANCH_MACRO(x, y) (x == y)
   10|       |
   11|       |int main(int argc, char *argv[])
   12|      3|{
   13|      3|    if (argc == 1)
   ------------------
   |  Branch (13:9): [True: 1, False: 2]
   ------------------
   14|      1|    {
   15|      1|#ifdef __clang__
   16|      1|        __llvm_profile_write_file();
   17|      1|#endif
   18|      1|        return 0;
   19|      1|    }
   20|      2|
   21|      2|    int arg1 = atoi(argv[1]);
   22|      2|    int arg2 = atoi(argv[2]);
   23|      2|    int cnt  = atoi(argv[3]);
   24|      2|
   25|      2|    int x = arg2 == 0 || arg1 == 0;
   ------------------
   |  Branch (25:13): [True: 0, False: 2]
   |  Branch (25:26): [True: 1, False: 1]
   ------------------
   26|      2|
   27|      2|    printf("Hello, World! %u\n", x);
   28|      2|
   29|      2|    int i;
   30|     22|    for (i = 0; i < cnt; i++)
   ------------------
   |  Branch (30:17): [True: 20, False: 2]
   ------------------
   31|     20|    {
   32|     20|        if (arg1 == 0 || arg2 == 2 || arg2 == 34)
   ------------------
   |  Branch (32:13): [True: 10, False: 10]
   |  Branch (32:26): [True: 10, False: 0]
   |  Branch (32:39): [True: 0, False: 0]
   ------------------
   33|     20|        {
   34|     20|            printf("Hello from the loop!\n");
   35|     20|        }
   36|     20|    }
   37|      2|
   38|      2|    if ((arg1 == 3) && 1)
   ------------------
   |  Branch (38:9): [True: 0, False: 2]
   |  Branch (38:24): [Folded - Ignored]
   ------------------
   39|      0|      printf("This never executes\n");
   40|      2|
   41|      2|    if (BRANCH_MACRO(arg1, arg1))
   ------------------
   |  |    9|      2|#define BRANCH_MACRO(x, y) (x == y)
   |  |  ------------------
   |  |  |  Branch (9:28): [True: 2, False: 0]
   |  |  ------------------
   ------------------
   42|      2|      printf("This executes on a macro expansion\n");
   43|      2|
   44|      2|    // Explicit Default Case
   45|      2|    switch (arg2) {
   46|      1|      case 1: printf("Case 1\n");
   ------------------
   |  Branch (46:7): [True: 1, False: 1]
   ------------------
   47|      1|              break;
   48|      1|      case 2: printf("Case 2\n");
   ------------------
   |  Branch (48:7): [True: 1, False: 1]
   ------------------
   49|      1|              break;
   50|      0|      default: break;
   ------------------
   |  Branch (50:7): [True: 0, False: 2]
   ------------------
   51|      2|    }
   52|      2|
   53|      2|    // Implicit Default Case
   54|      2|    switch (arg2) {
   ------------------
   |  Branch (54:13): [True: 0, False: 2]
   ------------------
   55|      1|      case 1: printf("Case 1\n");
   ------------------
   |  Branch (55:7): [True: 1, False: 1]
   ------------------
   56|      1|              break;
   57|      1|      case 2: printf("Case 2\n");
   ------------------
   |  Branch (57:7): [True: 1, False: 1]
   ------------------
   58|      1|              break;
   59|      2|    }
   60|      2|
   61|      2|#ifdef __clang__
   62|      2|    __llvm_profile_write_file();
   63|      2|#endif
   64|      2|
   65|      2|    return 0;
   66|      2|}

File '/scratch/aphipps/llvmtest/cov/demo/demo.c':
Name                        Regions    Miss   Cover     Lines    Miss   Cover  Branches    Miss   Cover
-------------------------------------------------------------------------------------------------------
main                             28       4  85.71%        55       2  96.36%        30       8  73.33%
-------------------------------------------------------------------------------------------------------
TOTAL                            28       4  85.71%        55       2  96.36%        30       8  73.33%

11.1.8. Important Considerations for Branch Coverage

Branch Coverage is a new feature added to Source-Based Code Coverage supported by the TI Arm Clang Compiler Tools.

• Some other vendors define Branch Coverage as only covering Decisions that may include one or more logical operators. However, Branch Coverage in the tiarmclang compiler supports coverage for all leaf-level boolean expressions (expressions that cannot be broken down into simpler boolean expressions). For example, “x = (y == 2) || (z < 10)” is a boolean expression that is comprised of two conditions, each of which evaluates to either TRUE or FALSE. This support is functionally closer to GCC GCOV/LCOV support.
• When showing branch coverage, each TRUE and FALSE condition represents a branch that is tied to how many times its corresponding condition evaluated to TRUE or FALSE. This can also be shown in terms of percentage.

44|      3|    if ((VAR1 == 0 && VAR2 == 2) || VAR3 == 34 || VAR1 == VAR3)
------------------
|  Branch (44:10): [True: 1, False: 2]
|  Branch (44:20): [True: 0, False: 1]
|  Branch (44:31): [True: 0, False: 3]
|  Branch (44:42): [True: 0, False: 3]
------------------

• When viewing branch coverage details in a source-based visualization, it is recommended that users show all macro expansions (using option –show-expansions), particularly since macros may contain hidden boolean expressions. In addition, macro expansions can be nested (macros are often defined in terms of other macros), as demonstrated in the following example. The coverage summary report will always include these macro-based boolean expressions in the overall branch coverage count for a function or source file.

58|      3|        MACRO2;
------------------
|  |    7|      5|#define MACRO2( MACRO)
|  |  ------------------
|  |  |  |    6|      2|#define MACRO (MACRO_CONDITION ? VAR2 : VAR1)
|  |  |  |  ------------------
|  |  |  |  |  |    5|      2|#define MACRO_CONDITION (VAR1 != 9)
|  |  |  |  |  |  ------------------
|  |  |  |  |  |  |  Branch (5:16): [True: 2, False: 0]
|  |  |  |  |  |  ------------------
|  |  |  |  ------------------
|  |  ------------------
|  |  |  Branch (7:17): [True: 2, False: 0]
|  |  ------------------
------------------

• Coverage is not tracked for branch conditions that the compiler can fold to TRUE or FALSE since for these cases, branches are not generated. This matches the behavior of other code coverage vendors. In the source-based visualization, these branches will be displayed as [Folded - Ignored] so that users are informed about what happened.

38|      2|    if ((VAR1 == 3) && TRUE)
------------------
|  Branch (38:9): [True: 0, False: 2]
|  Branch (38:24): [Folded - Ignored]
------------------

• Branch coverage is tied directly to branch-generating conditions in the source code. As such (unlike with GCOV), users should not see hidden branches that aren’t actually tied to the source code.
• For switch statements, a branch region is generated for each switch case, including the default case. If there is no explicitly defined default case, a branch region is generated to correspond to the implicit default case that is generated by the compiler. The implicit branch region is tied to the line and column number of the switch statement condition (since no source code for the implicit case exists). In the example below, no explicit default case exists, and so a corresponding branch region for the implicit default case is created and tied to the switch condition on line 65.

65|      3|    switch (condition)
------------------
|  Branch (65:13): [True: 2, False: 1]
------------------
66|      3|    {
67|      1|        case 0:
------------------
|  Branch (67:9): [True: 1, False: 2]
------------------
68|      1|            printf("case0\n"); // fallthrough
69|      1|        case 2:
------------------
|  Branch (69:9): [True: 0, False: 3]
------------------
70|      1|                               // fallthrough
71|      1|
72|      1|        case 3:
------------------
|  Branch (72:9): [True: 0, False: 3]
------------------
73|      1|            printf("case3\n"); // fallthrough
74|      3|
75|      3|    }

11.1.9. Known Limitations

• Counter Initialization After Some Startup Routines

  • For functions that are part of special boot/reset routines that get called prior to C runtime initialization (e.g. for SYS/BIOS), counter information for these functions will be clobbered. If code coverage data is needed for functions like these, a special startup sequence may be required in your system to ensure the counters are properly initialized to zero and not re-initialized later unless the counter data can be extracted first.

• Code Composer Studio Integration

  • Presently, CCS doesn’t have direct support for tiarmclang compiler Code Coverage, though support will be added soon. This support will make it very straightforward for users to build projects for code coverage, download counter data from memory, and visualize the data.

• Counter Size

  • Counters are presently 64bits in size, which may be too large for some embedded use cases
  • Counters that have large counts may overflow either during execution or when counter data is merged together by the tiarmprofdata tool. When the counter data is merged, tiarmprofdata uses saturating addition, so the final value will reflect the largest possible value. This will affect the accuracy of the visualization.

• Unexpected Function Instantiations of the Same Function

  • The tiarmcov tool uses a function hash to distinguish between functions. This hash is based on the function name, source filename, as well as all included header filenames as well as their filepaths. For functions that have the same name across multiple binaries, if any of the filepaths are different, then a different function hash will be used, and functions that have the same name will be treated by tiarmcov as separate function instantiations of the same function. In the source-based visualization, these instantiations will show up as subviews preceded by a general summary view of the function.
  • If a build system regenerates the constituent header files for a source file across different builds such that the header filepaths end up being different from build to build, then even if the header files are identical across builds, the function will be represented as multiple instantiations of the same function. If these functions are actually identical, then there will only exist one final set of merged counters for the function, and the coverage will be identical across all instantiations. This will not negatively impact the final coverage summary of covered lines, regions, or branches.

• Function Differences

  • Different function definitions across multiple executables that have the same function name will likely be reported as having “mismatched data”. This is a known issue in code coverage for common function names like main(). Care should be taken to filter out cases like this using tiarmcov’s filtering mechanism since each instance clearly represents a different function.
  • Two or more functions that have the same code base but built different such that they contain different macro expansions will be visualized as multiple instantiations of the same function. This doesn’t impede coverage.

• Source Filtering

  • The source filtering facility implemented by tiarmcov isn’t as fully featured as it is for other vendors, like LCOV. Specifically, embedded filter tags aren’t supported (e.g. LCOV_EXCL_[START|STOP]). Please see the filtering options for more information (tiarmcov –help).

• Branch Coverage

  • Future compiler enhancements will likely be implemented to minimize the number of counters actually used in nested boolean expressions like “((A || B) && C)”, for example.
  • Modified Condition/Decision Coverage is not supported.