			      ==================
			      REMOTE TEST DRIVER
			      ==================

This program runs tests on a remote board by execution of a state machine. The state machine is
provided in the configuration and control file provided to the program when it is run.

The state machine's transitions are driven by:

 (*) Pattern matching on input acquired from the board's console port.
 (*) GDB remote protocol packets coming from the board's gdbstub port (optional).
 (*) Timeouts.

The state machine's states can:

 (*) Specify commands to be sent to the board.
 (*) Specify a new timeout.
 (*) Send GDB remote protocol commands to gdbstub.
 (*) Issue shell commands (to do things like resetting the board).
 (*) Select next test from test list and dispatch it.
 (*) Log results.

The program needs to be provided with a number of resources:

 (*) The serial port to which board console attached.
 (*) The serial port to which board gdbstub attached.
 (*) The serial port lock file directory.
 (*) The board communication state machine.
 (*) A list of tests to be run.
 (*) The names of two log files: one for a test result summary, and a communications log.

The program iterates through the list of tests at the state machines behest until all are done. The
program does no work to upload the test to the board. The test should be made available by means of
NFS or attached disc.


==================
CONFIGURATION FILE
==================

The configuration file provides a set of general options to the program and the description of the
state machine used to communicate with the board.

All leading and trailing spaces are trimmed. Any line effectively beginning with a hash is a
comment and is ignored. Blank lines are also ignored. Lines ending in backslashes are merged with
the following lines.

The general options are:

 (*) condev /dev/ttyS1
 (*) condev /dev/ttyS1,115200
 (*) condev /dev/ttyS1,115200,2N6r
 (*) gdbdev /dev/ttyS0
 (*) gdbdev /dev/ttyS0,,1N8x
 (*) gdbdev serial.farm.ges.redhat.com:8001

     Specify the ports to which the board's console and gdb stub are attached. condev is mandatory,
     gdbdev is optional.

     If the first character is a slash, then it is assumed to be a device file name and optionally
     parameters. Parameters can be added with commas to control the baud rate and some extra serial
     control parameters specified as a list of single characters:

	12	Stop bits (default 1)
	NEO	Parity (default none)
	5678	Data bits
	r	CTS/RTS flow control
	x	XON/XOFF flow control

     Otherwise the name is assumed to be a network address and TCP port number behind which a
     serial port server resides.

 (*) lockdir /var/lock

     Specifies the directory in which serial port locking files are to be deposited. Defaults to
     /var/lock. Lock files are only created for devices in /dev/.

 (*) testlist /data/inst-frv/testdriver/testdriver/warthog/frv.testlist

     Specifies the list of tests which are to be run on the board. Mandatory.

 (*) testlog "test-%Y%m%d-%H%M%S.log"
 (*) commlog "comm-%Y%m%d-%H%M%S.log"

     Specify the filenames to which the test result summary log and the communication log will be
     dumped. Note the filename is passed through strftime before use.

 (*) start [DEAD]

     Specifies the state in the machine at which execution should start. Mandatory.

All global options must precede the state definitions.


The state machine is specified as a collection of state definitions, each of which has a
description associated with it. State names are always surrounded by square brackets. A state
definition is introduced by a state name by itself on a line, eg:

	[REDBOOT_PROMPT0]
		cmd ""

A state can be one of three types: stop, input or action.


========================
STOP STATE CONFIGURATION
========================

A stop state specifies that the driver program should abandon the testing procedure and exit. The
board will be left in an uncertain state:

	[SETUP_PROBLEM]
		stop


=========================
INPUT STATE CONFIGURATION
=========================

An input state can only move on in response to an external or timeout event, and may not perform an
action:

	[TEST_CHECKING]
		match "RedBoot> " [TEST_CRASHED]
		match "Data Access Err" [TEST_CRASHED]
		match "Error" [TEST_FAILED]
		match "Success" [TEST_SUCCEEDED]
		gdbmatch [TEST_CRASHED]


Input states support the following options:

 (*) match "Success" [TEST_SUCCEEDED]

     Searches the character stream coming from the board's console for text matching the specified
     regular expression. The pattern cannot span lines, and the current line is rechecked against
     all the patterns attached to a state after each character is added. If a match is made, all
     data up to and including that match are excluded from further matching.

     If a match is made, the state machine transitions to the named state.

     The string holding the regular expression can be bounded by any character, as long as there's
     one at each end.

     Several match statements can be specified on each input state.

 (*) gdbmatch [TEST_CRASHED]

     If a valid GDB remote protocol packet comes in, a transition is taken to the named state.

     No more than one gdbmatch statement can be specified per input state.


==========================
ACTION STATE CONFIGURATION
==========================

An action state has a mandatory transition that will take place no matter what external events have
taken place, though this can be overridden by a timeout event.

	[DEAD]
		testlog reboot
		shell "ssh passion heyu2 off k5"
		shell "ssh passion heyu2 on k5"
		set-timeout 45 [SETUP_PROBLEM]
		goto [RESETTING]


An action state is described by a goto statement, an optional action statement and one or more
option statements. The goto statement indicates the transition that the machine should go to next
after executing the action (if there is one).

Action states support the following actions:

 (*) shell "ssh passion heyu2 off k5"

     The specified command string will be passed to the system() call to be executed. The program
     will exit if the command fails. Several shell statements can be added to an action state, and
     will be executed one after the other.

 (*) cmd "load -r -b 0x100000 /tftpboot/frv"

     This statement will send the specified command to the board via its console port. The test
     driver will watch to make sure that the command characters are echoed by the board. If not,
     the driver program will abort the test procedure and exit.

 (*) gdbchat "?"

     This statement causes a GDB remote protocol packet to be sent through the gdbstub port. The
     leading dollar character and the trailing hash character and checksum characters are added by
     the driver program. The program will wait for a response until a timeout occurs, and the
     packet will be resent if NAK'd.

 (*) run

     This causes the driver program to read the next test from the test list file and send the test
     command as for the "cmd" statement.

     If there are no more tests, the driver program will exit.

Note that action statements are mutually exclusive with regard to one another, and apart from the
shell statement may only be used once per action state.

Command issuing procedures in the state machine must wait for prompts between sending commands, or
else commands will get overridden before being completely issued.

Certain option statements can also be attached to action states:

 (*) testlog reboot

     This statement causes a reboot marker to be written to the test summary log.

 (*) testlog success
 (*) testlog failure
 (*) testlog timeout
 (*) testlog crash

     These statements cause test completion records to be written to the test summary log against
     the name of the test last selected by the "run" statement.


=====================
GENERAL STATE OPTIONS
=====================

Certain option statements can also be attached to input or action states:

 (*) set-timeout 60 [SETUP_PROBLEM]

     This statement requests that an alarm be set for the given amount of seconds, clearing the old
     timeout alarm. Should the alarm go off, a transition will be forced to the state specified.

 (*) clear-timeout

     This statement disables the timeout alarm.


==============
TEST LIST FILE
==============

The driver program requires a list of test commands in addition to the configuration and state
machine file. This list is referenced by a general option in the configuration file.

This file is processed line-by-line. Leading spaces are removed from each line. Any subsequently
blank line or line beginning with a hash is ignored.

Each remaining line is then a test specification. Each of these consist of a test name and a shell
command to execute that test. For instance, to run the chdir syscall tests from LTP:

	chdir01 chdir01
	chdir01A symlink01 -T chdir01
	chdir02 chdir02
	chdir03 chdir03
	chdir04 chdir04


=====================
TEST SUMMARY LOG FILE
=====================

One of the log files produced is a summary of the tests done and in what manner they completed:

	###############################################################################
	#
	# Test results for run beginning Wed Mar 17 15:52:20 2010
	#
	###############################################################################
	-- reboot --
	accept01: success
	access01: success
	access02: failure
	access03: timeout
	access04: crash

It has a header inserted to record the time and date at which the test was run. It records the
status of each test and reboot markers. A quick summary of this file can be be generated with the
summarise.pl program included in this directory:

	dhowells>./testdriver/summarise.pl test-20100317-155220.log 
	Successes: 547/706
	Failures : 142/706
	Timeouts : 14/706
	Crashes  : 3/706


=======================
COMMUNICATIONS LOG FILE
=======================

All communications with the board are logged in the communications log file:

	###############################################################################
	#
	# Communication log for run beginning Wed Mar 17 15:52:20 2010
	#
	###############################################################################
	Int - dev: 19, req: 1, vector: 38
	+FLASH configuration checksum error or invalid key
	<<FLASH configuration checksum error or invalid key
	>>RTL8029 at 0x24000000, interrupt: 25

This has the output from the gdbstub port interpolated into the console output inside of << >>
markers.

	<<+>><<$#00>>{{+}}{{$?#3f}}<<+>><<$S02#b5>>{{+}}{{$c#63}}<<+>>Linux version 2.4
	.24-uc0 (dhowells@warthog.cambridge.redhat.com)

GDB remote protocol packets sent to the board are encapsulated within {{ }} markers.

Each test is begun with a banner to make it easy to find:

	Success
	# 
	###############################################################################
	#
	# -- BEGINNING TEST alarm04 --
	# 
	###############################################################################
	alarm04
	sig_rev     1  PASS  :  Functionality of alarm(3) successful
	# if [ $? = 0 ]; then echo Success; else echo Error; fi
	Success
	# 

