--------------------------------------------------------------------------------

                       TCI648X/C647X RAPIDIO LINUX HOWTO

                           A. Jacquiot - October 2010
                            (c) Texas Instruments 

--------------------------------------------------------------------------------

 I - RapidIO package content
 ===========================

   The RapidIO tgz package includes the following files and directories:

    Makefile -> top heading Makefile
    misc     -> examples and prebuilt BIOS images
    utils    -> user-land utility

   We suppose that the tgz package is extracted in the $RIO_SRC_DIR 
   directory.

 II - Building the RapidIO package sources
 =========================================
 
   In order to build the RapidIO utilities, you must have setup the build C6x
   environment (with source setenv) as explained in the linux-c6x.org WiKi.
   In particular the SDK_DIR variable must have been set.
   
   To build and install the binaries:

    $ cd $RIO_SRC_DIR
    $ make
    $ make install

   The binaires are installed in $RIO_SRC_DIR/rootfs. You must copy the content
   of this directory in the Linux root filesystem to use them on the target.

 III - Configuring the kernel RapidIO driver
 ===========================================

   You must check that the Linux kernel is built with the following features:
   
     CONFIG_RAPIDIO_TCI648X=y
     CONFIG_RAPIDIO_DEV=y
     CONFIG_RAPIDIO=y
     CONFIG_RAPIDIO_DISC_TIMEOUT=2
     CONFIG_RIONET=y
     CONFIG_RIONET_TX_SIZE=128
     CONFIG_RIONET_RX_SIZE=128


   The TCI648x sRIO hardware configuration is defined in the following file:
     linux-c6x/arch/c6x/platforms/board-<board>.c

   You can override this configuration wih Linux kernel command line paramaters

        riohdid=        Host device id number. If setting -1, the driver will
                        use the base Id set in the host RapidIO configuration space.

        rioports=       List of sRIO ports used by the driver
                        Format: <port_1>[,<port_2>[,..<port_n>]]

        rioinit=        RapidIO initialization method
                        Format: { enum | discov | discov_wait }
			'enum': perform RapidIO bus enumeration
                        'discov': perform RapidIO bus discovery
                        'discov_wait': perform RapidIO bus discovery after waiting 
			for beeing enumerated by remote host 

        riomode=        sRIO serdes configuration predefined mode
                        0 is for the sRIO boot config 0 (SERDES ref clock 125 MHz,
			Link rate 1.25 Gbps)
			1 is for the sRIO boot config 1 (SERDES ref clock 125 MHz,
			Link rate 3.125 Gbps)
			2 is for the sRIO boot config 2 (SERDES ref clock 156.25 MHz,
			Link rate 1.25 Gbps)
			3 is for the sRIO boot config 3 (SERDES ref clock 156.25 MHz,
			Link rate 3.125 Gbps)

	rioedma-threshold=  Threshold value in bytes that indicates from which size
                        of DirectIO transfer the driver uses the EDMA instead of the CPU
			to load LSU. By default this value is set to -1 meaning that 
                        EDMA is never used. In practice a value lower than 32768 will
                        make transfer slower with EDMA than with CPU due to EDMA
                        programming overhead. Typical value is 32768 if you want to 
                        use EDMA to relax CPU for big transfers.

   example: riohdid=0 rioports=1 rioinit=enum riomode=0 rioedma-threshold=32768

            This will enumerate the sRIO bus on port 1 with the sRIO serdes 
            configuration of the RIO bootmode 0. The EDMA transfer is used for 
            transfer greater than 32KB.

  IV - Using the kernel RIO driver
  ================================

   Boot the board with the standard Linux kernel then log on the target.

   During the kernel boot, you must get the following traces:
  
     RIO: register sRIO controller for hostid 0
     RIO: enumerate master port 0, RIO0 mport
     TCI648x RapidIO driver v2.1
     RIO: setting EDMA threshold to 0x8000

   To check that the RIO driver is running:

    #  ls -la /dev/ | grep rio
    crw-rw----    1 root     root      154,   0 Nov 30 00:00 /dev/rio1.1	
  
   This example indicates that a remote RapidIO device is found with device id 1
   on the port 1.

   The syntax is /dev/rio<port>.<device_id>

   This file is a character special device. It allows to perform Direct I/O
   and doorbell operations to a remote RapidIO site.

   You can read and write to remote memory address space by reading/writing
   in this file. The offset 0 of this file is mapped to the remote Direct I/O
   base (by default at the DDR start, i.e. 0x80000000 on C6474). You can change
   this base address by performing ioctl() commands.

   For other operations, the following IOCTL (in include/linux/rio.h) are available:


     ioctl(fd, RIO_DIO_BASE_SET, &base); /* Set the base offset for a site */
     ioctl(fd, RIO_DIO_BASE_GET, &base); /* Get the base offset of a site */
     ioctl(fd, RIO_DIO_MODE_SET, &mode); /* Set the Direct I/O mode for a site */
     ioctl(fd, RIO_DIO_MODE_GET, &mode); /* Get the Direct I/O mode of a site */
     ioctl(fd, RIO_DBELL_TX, &info);     /* Sent a doorbell to a site */
     ioctl(fd, RIO_DBELL_RX, &info);     /* Wait for a doorbell from a site */

  'base' is an address value, 'mode' is the Direct I/O mode (RIO_DIO_MODE_WRITER,
   RIO_DIO_MODE_WRITE, RIO_DIO_MODE_SWRITE), 'info' is the doorbell info data.

  V - Using the RIONET driver
  ===========================

  The kernel RapidIO driver also offers the capabilities to use Ethernet over the 
  RapidIO message passing interface.

  During the kernel boot, you must get the following traces:

    eth1: rionet Ethernet over RapidIO Version 0.2, MAC 00:01:00:01:00:00
    Using 00:e:0001 (vid 0030 did 0092)
    ...

  Log on the target and configure the eth1 interface:

    # ifconfig eth1 192.168.0.1

  Boot another Linux kernel on another C647x device connected by sRIO, log on
  the second target and configure the eth1 interface too:

    # ifconfig eth1 129.168.0.2

  You can now communicate between the two CI647x devices with TCP/IP like with a
  standard Ethernet device (e.g. by using telnet).

 VI - The rioconfig utility
 ===========================

   To admistrate and test the RapidIO driver, a 'rioconfig' utility is provided
   in the package:

   # rioconfig
     Usage:  rioconfig {-b ADDR|--base=ADDR} DEV             Changing DirectIO base offset to ADDR
	     rioconfig {-m MODE|--mode=MODE} DEV             Changing DirectIO mode to MODE
	     riocongig {-d DBELL|--dbell=DBELL} DEV          Sending doorbell number DBELL
	     rioconfig {-w DBELL|--dbell-wait=DBELL} DEV     Waiting doorbell number DBELL
	     rioconfig {-t DIO_MODE |--bench=DIO_MODE} [-l LOOP|--loop=LOOP]
	               [-B BUF_SIZE|--buf-size=BUF_SIZE] [-b BASE|--base=BASE] [--no-check]
                                                             Launch DirectIO performance benchmark


  - rioconfig -b <addr> /dev/rio<port>.<num> will change the Direct I/O base address of
    site <num> to <addr>.

  - rioconfig -m <mode> /dev/rio<port>.<num> will change the Direct I/O default mode
    of site <num> to <mode>. (<mode> can be 0 for NWRITE_R, 1 for NWRITE and 2 for SWRITE).
 
  - rioconfig -d <dbell> /dev/rio<port>.<num> will post a doorbell with <dbell> info to a 
    site <num>.

  - rioconfig -w <dbell> /dev/rio<port>.<num> will wait a doorbell with <dbell> info 
    from a site <num>.

  - rioconfig -t <mode>  /dev/rio<port>/.num> will bench Direct I/O read and
    write operations with <mode> mode on site <num>. <mode> can by 'n' for NWRITE_R, 'a'
    for NWRITE and 's' for STREAM.
    The --no-check flag will do the test without performing data verification.
    Base offset of the test can be changed with the -b flag, default buffer size 
    can be changed with the -B flag and the test can be performed in loop using the 
    -l flag.

 VII - Examples
 ==============

    1) READ/WRITE WITH DIO
    ----------------------

       - Boot Linux on Faraday 1

       - Connect with CCS the Faraday 2 in order to set up the DDR timing.

       - On the target, do the following commands:
  
        # echo 'test' > /dev/rio1.1
        # od -a /dev/rio1.1 | more

          You must get the 'test' string.

       - On the Faraday 2 with CCS, view memory at address 0x80000000, you must seen
         the 'test' string too in CCS.

    2) BOOT OF REMOTE FARADAY AND DOORBELL TX/RX
    --------------------------------------------

       - Set the Jumper of the board to BOOTMODE_0 (BOOTM0 on, BOOTM1 on, BOOTM2 on 
         and BOOTM3 off).

       - Boot Linux on the Faraday 1

       - Log on the target

       - Do the following commands:

         # loadremotecore /dev/rio1.1 0 /opt/images/SRIO_boot.out

          It will load the SRIO_boot.out COFF image to the core 0 of the second 
          Faraday

         # rioconfig -w 1 /dev/rio1.1

          To wait a doorbell with info=1 from Faraday 2

       - Open a second terminal or telnet session and do the following commands:

         # rioconfig -d 0 /dev/rio1.1 

         It will send a doorbell to the second Faraday, which will boot with the
         SRIO_boot.out DSP/BIOS application that will send a doorbell 1 to the first
         Faraday.

         The 'rioconfig -w 1 /dev/rio1.1' command must exit without error.

   3) DIRECTIO PERFORMANCE BENCHMARK
   ---------------------------------

       - Boot Linux on Faraday 1

       - Connect with CCS the Faraday 2 in order to set up the DDR timing.

       - On the target, do the following command:
          
         # rioconfig -t s --buf-size=8192 --no-check --loop=10 /dev/rio1.1
