From RTSC-Pedia

[printable version]

offline version generated on 04-Aug-2010 21:08 UTC

RTSC Packaging Primer/Lesson 8

Package Test — managing a charlie.sqrtlib test suite

Though not obvious at first, package producers will often become consumers of their own packages—such as when managing a suite of application programs that test the target content delivered in packages like charlie.sqrtlib. Following a simple organizational pattern used frequently amongst package producers, we'll construct a companion package named charlie.sqrtlib.test which builds and executes a collection of test programs that in general would exercise client-visible ("black-box") and supplier-proprietary ("white-box") features of charlie.sqrtlib per se.

Generalized test packages

To exercise charlie.sqrtlib in a wide-range of potential settings, the charlie.sqrtlib.test package and the programs it contains introduce a level of generality not always seen within individual client applications. To illustrate, we'll study one program named isqrtTest1 that effectively generalizes the client application from Lesson 3—enabling us to invoke the isqrt function with different inputs as well as to configure the Settings.optimize parameter with different values, all while building and executing the same program across multiple targets.

The target-program.  Using prog.c as a starting point, isqrtTest1.c allows input values for the isqrt function to enter the program via command-line arguments available to main—nothing unusual here from the perspective of C.

#include <acme/utils/Bench.h>
#include <xdc/runtime/System.h>
#include <charlie/sqrtlib/isqrt.h>
 
#include <stdlib.h>
 
int main(int argc, char* argv[])
{
    unsigned int x, y;
 
    x = argc > 0 ? atoi(argv[1]) : 0;
    System_printf("==> isqrt(%d)\n, x");
 
    Bench_begin("cycle count:");
    y = isqrt(x);
    Bench_end();
 
    System_printf("<== %d\n", y);
    return 0;
}

Guided by the argc and argv arguments to main declared at line 2, the statement at line 3 conditionally converts the program's first argument (a string, by definition) into the integer it presumably represents using the built-in atoi function declared in the <stdlib.h> header included earlier at line 1. Beyond line 3, the remainder of this test program mirrors prog.c exactly—echoing the input x, benchmarking an isqrt call, and then displaying the output y.

As you'll see shortly, the package.bld script for charlie.sqrtlib.test not only prescribes "what to build" when manufacturing the isqrt program for multiple targets with alternative configurations, but also defines "what to test" when executing this program on a particular platform.

The meta-program.  Unlike isqrtTest1.c, the corresponding isqrtTest1.cfg meta-program represents a more significant departure from the original prog.cfg script covered in Lesson 3.

var Program = xdc.useModule('xdc.cfg.Program');
 
var Bench = xdc.useModule('acme.utils.Bench');
var Settings = xdc.useModule('charlie.sqrtlib.Settings');
var System = xdc.useModule('xdc.runtime.System');
 
Bench.enable = (Program.build.target.isa != "x86");
 
switch (Program.build.cfgArgs.optMode) {
    case "SPACE":
        Settings.optimize = Settings.OPTIMIZE_FOR_SPACE;
        break;
    case "TIME":
        Settings.optimize = Settings.OPTIMIZE_FOR_TIME;
        break;
}

Leveraging the special xdc.cfg.Program meta-module imported at line 1, the isqrtTest.cfg script accesses information originating earlier in the flow, when first running the package.bld script for charlie.sqrtlib.test.

2  Program.build.target

This property returns an XDCscript object whose properties in turn characterize the RTSC target selected for building this program. Here, the target's isa property ultimately determines whether we enable the acme.utils.Bench module at runtime (which, as it turns out, proves woefully inaccurate when used on an "x86" host PC).

3  Program.build.cfgArgs

This property returns an XDCscript object defined within the package.bld script itself. Here, we've used a string previously assigned to this object's optMode property to drive initialization of the Settings.optimize configuration parameter with either OPTIMIZE_FOR_SPACE or OPTIMIZE_FOR_TIME.

The package.bld script introduced momentarily will fill in some important missing details, specifically the idiom used to formulate configuration-time arguments for meta-programs such as isqrtTest1.cfg.

The package specification.  Needless to say, charlie.sqrtlib.test requires a package.xdc specification file found in the corresponding package directory—just like any other RTSC package.

requires acme.utils;
requires charlie.sqrtlib;
 
/*! Companion tests for charlie.sqrtlib */
package charlie.sqrtlib.test {
    /* no module declarations */
}

Similar to the specification of charlie.sqrtlib.samples from Lesson 4, the package declaration at line 2 introduces no additional programmatic elements into the charlie.sqrtlib.test namespace—in contrast with the charlie.sqrtlib package from Lesson 5, which introduces the Settings module at this point.

But unlike any of the package.xdc files we've already encountered, the specification of charlie.sqrtlib.test explicitly declares its dependency on other RTSC packages via some requires statements beginning at line 1. As a rule, these statements reflect references within the current package's content to programmatic artifacts found in other RTSC packages—such as the reference to acme/utils/Bench.h within isqrtTest1.c or the reference to charlie.sqrtlib.Settings within isqrtTest1.cfg.

Prescribing a set of tests

In the spirit of the charlie/sqrtlib/package.bld script introduced back in Lesson 5—a portable prescription for building a pair of function libraries across multiple targets, expressed through a single for loop—the package.bld script for charlie.sqrtlib.test takes this idiom to the next level. Structured as a series of nested for loops, the script concisely prescribes multiple test programs built for multiple targets, configured in multiple ways, and executed multiple times with different inputs.

var Build = xdc.useModule('xdc.bld.BuildEnvironment');
var Pkg = xdc.useModule('xdc.bld.PackageContents');
 
var TEST_INFO = [
    {testName: "isqrtTest1", testValues: ["100", "1000", "65536"]}
];
 
for each (var testInfo in TEST_INFO) {
 
    for each (var targ in Build.targets) {
        for each (var optMode in ["SPACE", "TIME"]) {
 
            var progName = testInfo.testName + "-" + optMode;
            var exe = Pkg.addExecutable(progName, targ, targ.platform)
 
            exe.addObjects([testInfo.testName + ".c"]);
            exe.attrs.cfgScript = testInfo.testName + ".cfg";
            exe.attrs.cfgArgs = "{optMode: '" + optMode + "'}"; 
 
            for each (var v in testInfo.testValues) {
                exe.addTest({groupName: optMode, args: v});
            }
        }
    }
}

The outer for loop beginning at line 2 successively binds the local variable testInfo to each element of the TEST_INFO array defined earlier at line 1—limited here to just one element in the interest of brevity. In general, though, each element of the TEST_INFO array comprises an XDCscript object featuring a testName and testValues property respectively bound to a string and an array of strings. Each element of the latter array in turn defines a distinct command-line argument list passed via argv to this program—which would run three times using this example.

The next pair of nested for loops first assigns targ at line 3 to each successive element of the Build.targets array—initialized in the current config.bld script—and then assigns optMode at line 4 to each string in the two-element array defined in place. Moving forward, we can now repeatedly invoke addExecutable at line 5 with a unique set of arguments that characterize each test program (of many!) configured, compiled, and linked downstream:

• the program's name, formed here by mangling testInfo.testName with optName;
• the program's target, an element of the Build.targets array currently bound to targ; and
• the program's platform, taken here from a property of targets often assigned within config.bld.

Similar to the addLibrary call seen back in Lesson 5, the XDCscript object returned by addExecutable—bound here to the local variable exe—also supports an addObject function whose input at line 6 consists of an array of C source-file names (eg. isqrtTest1.c). Continuing onward, the assignment at line 7 names this executable's corresponding meta-program (eg. isqrtTest1.cfg); the assignment at line 8 formulates a string that later becomes an XDCscript object with an initialized optMode property, available to the meta-program during configuration.

Finally, the addTest function call at line 9—performed once for each command-line argument list in the current testInfo.testValues array—associates a unique test run with this executable program. The XDCscript object passed to addTest defines some attributes of the newly-created test (specifically, its groupName and args) whose meaning will become clear once we build and execute all of the programs prescribed by this package.bld script.

Building the programs

Similar to the way we've already built the charlie.sqrtlib libraries back in Lesson 5, we'll now clean-and-rebuild the charlie.sqrtlib.test programs by invoking xdc clean followed by xdc all from within the package directory. Output from the latter command (abbreviated to improve readability) traces the configuration, compilation, and linking of four distinct programs—all leveraging the same isqrtTest1.c and isqrtTest1.cfg source files.

%> xdc all
making package.mak (because of package.bld) ...
generating interfaces for package charlie.sqrtlib.test (because ...) ...
 
configuring isqrtTest1-SPACE.x64P from package/cfg/isqrtTest1-SPACE_x64P.cfg ...
cl64P package/cfg/isqrtTest1-SPACE_x64P.c ...
cl64P isqrtTest1.c ...
lnk64P isqrtTest1-SPACE.x64P ...
 
configuring isqrtTest1-TIME.x64P from package/cfg/isqrtTest1-TIME_x64P.cfg ...
    ...
 
configuring isqrtTest1-SPACE.x86GW from package/cfg/isqrtTest1-SPACE_x86GW.cfg ...
    ...
 
configuring isqrtTest1-TIME.x86GW from package/cfg/isqrtTest1-TIME_x86GW.cfg ...
    ...
 
all files complete.

Focusing on the steps involved in building isqrtTest1-SPACE.x64P at line 1—configure, compile, link—these mirror the basic program configuration flow introduced back in Lesson 3. Looking closer, the output before and after line 1 in fact follows the same pattern also seen in Lesson 3 when we eventually built our first client application using the configuro command from within a GNU makefile.

Recalling that the xdc command itself generates a special package.mak file after first running the package.bld script—and then forwards this generated file to gmake which actually dispatches the individual build steps—only the minimal amount of re-work takes place when package producers modify sources files like isqrtTest1.c and isqrtTest1.cfg during development and then invoke xdc all.

In situations where producers find themselves working across a set of inter-dependent packages—simultaneously modifying library sources in charlie.sqrtlib as well as test programs in charlie.sqrtlib.test—we recommend using an extended form of the xdc command that operates on a set of packages designated using the -P option and its variants.

Finally, the xdc command enables you to build for just one target; for example, invoking xdc all,64P—with or without the -P option—would only manufacture programs (or libraries) using ti.targets.C64P. For even finer control, simply name the individual program (or library) as the xdc command's goal—for example, xdc isqrt1-SPACE.x64P would build only this one program.

Running the tests

Here too, we'll rely upon the xdc command to run the test programs contained within the charlie.sqrtlib.test package—from executing all programs with all pre-defined inputs to selecting a subset of the package's programs based on test-group name or target.

xdc test

execute all programs for all targets, with all inputs

xdc test,«target-suffix»

execute all programs only for the designated target, with all inputs

xdc «test-group-name».test

execute all programs in the designated test-group, for all targets and with all inputs

xdc «test-group-name».test,«target-suffix»

execute all programs in the designated test-group and only for the designated target, with all inputs

To illustrate, the following form of the xdc command would run all programs in the TIME test-group built for the 64P target, using all pre-defined inputs. Note that each test run has a distinct command-line input, resulting in distinct output from executing the underlying program. Recall that line 9 of the package.bld script examined earlier defines both the test-group name and command-line argument list associated with each of the test programs contained within the charlie.sqrtlib.test package.

%> xdc TIME.test,64P
running isqrtTest1-TIME.x64P 100 ...
==> isqrt(100)
cycle count: [38]
<== 10
running isqrtTest1-TIME.x64P 1000 ...
==> isqrt(1000)
cycle count: [63]
<== 31
running isqrtTest1-TIME.x64P 65536 ...
==> isqrt(65536)
cycle count: [172]
<== 256

While the xdc command can run individual programs (xdc isqrtTest1-TIME.x64P.test exemplifies the pattern), you can't vary the input passed to the program. For more ad hoc testing of individual programs—composing argument lists from the command-line—consider the xdc.tools.loader utility bundled with the XDCtools product as an alternative.

%> xs xdc.tools.loader isqrtTest1-TIME.x64P 256
==> isqrt(256)
cycle count: [63]
<== 16
%> xs xdc.tools.loader isqrtTest1-SPACE.x64P 256
==> isqrt(256)
cycle count: [190]
<== 16

See also

TODO:  write it

[printable version]

offline version generated on 04-Aug-2010 21:08 UTC