Introduction

TI provides a suite of LLVM/Clang-based and proprietary optimizing C/C++ compilers for our programmable embedded processors. Both toolchains leverage 30+ years of compiler development expertise to provide toolchains which generate highly optimized code for these processors. Over the years, TI compilers have been used to develop software across varied applications such as automotive, communications equipment, enterprise systems, industrial equipment and personal electronics.

ISO/IEC/IEEE 24765:2017 defines verification and validation as the process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements.

Compiler validation and verification is challenging due the complexity of the software and a very large input domain. To overcome these challenges, compiler development at TI follows a robust, well defined, documented software development process. TI uses a collection of commercial, open source and in-house compiler test suites along with extensive infrastructure for automated verification and validation.

Compiler Test Suites

Commercial Test Suites

PlumHall C and C++

  • The Plum Hall Validation Suite for C™ is a set of C programs for testing and evaluating C language compilers, with both positive and negative tests for all requirements of the C standard.
  • Suite++ ©: The Plum Hall Validation Suite for C++ is a set of C++ source code test cases comparing compiler behavior to the ISO/IEC C++ Standard. Suite++ includes both positive tests (for required behavior) and negative tests (for production of diagnostic messages).

Perennial C and C++

  • CVSA™ is a comprehensive C compiler conformance validation suite covering all of the ISO/IEC C standard, including C 2011.
  • C++VS™ is a comprehensive C++ compiler validation Suite covering all of the ISO/IEC C++ standing, including C++ 2014.

Solid Sands C and C++ SuperTest

  • SuperTest is a test and validation suite for C and C++ compilers with a huge library of hand-written and generated tests.

Dinkumware

  • “Proofer” to test library adherence to C++ standards.

CodeSourcery

  • C++ ABI Test Suite is a comprehensive set of tests for conformance to the C++ ABI for the Arm Architecture, including tests for name-mangling, object layout, virtual function table layout, and runtime library support.

Open source Test Suites

  • GNU Compiler Collection C and C++ test suites These are C language test suites used to test GCC and include torture tests, GCC extension tests and Binutils tests
  • LLVM tests Unit and regression tests from the LLVM testing infrastructure.
  • STLport test suite Extensive tests of the Standard Template Library functionality.

In-house Test Suites

The following test suites were developed in house and run during the compiler validations.

  • Kernels - key loops that are checked for performance and correctness.
  • Applications - common embedded applications, industry performance benchmarks etc.
  • Regression Tests The code for each bug report is added to a regression test suite.
  • Unit Tests These are black and white box tests of specific functionality.

Compiler validation infrastructure

The compiler build and validation process leverages automation to run a large number of tests with different compiler options across processor ISAs and generate reports summarizing pass/fail status. Tests are run on all platforms (Windows, Linux, Mac OSX) supported by a compiler release. Compiler validation is done during multiple stages of development:

  • Precommit validation This validation is run before every commit to the code base. All tests in the precommit test plan must pass before a commit is merged.

  • Nightly validations This is nightly integration testing, run every 24 hours. Approximately ~ 5 million binary executables are generated and run across ISAs and compiler option variations.

  • Release validations This validation is run before every release. This run exercises all test suites with approximately 39 million binary executables generated and run across ISAs and compiler option variations. All tests included in the release test plan must pass before a release is made available externally.