4.13.1. Introduction¶

PCIe module supports dual operation mode: End Point (EP or Type0) or Root Complex (RC or Type1). This driver focuses on EP mode but it also provides access to some basic RC configuration/functionality. For RC this is the lowest level; additional software is needed to perform generic enumeration of third party devices.

The PCIe subsystem has two address spaces. The first (Address Space 0) is dedicated for local application registers, local configuration accesses and remote configuration accesses. The second (Address Space 1) is dedicated for data transfer. This PCIe driver focuses on configuration of the interface and sending/receiving interrupts via Address Space 0. Data is transferred outside the scope of the LLD using CPU or EDMA through the data area.

The LLD abstacts the configuration of standards-based PCIe registers (Type 0, Type 1 and Capabilities registers) so same user code can be used on both device types. However, there are different interfaces for the registers not based on PCIe standards (port logic and ti-conf which generally covers interrupts and address translation). That portion of user code needs to differ between C66x/K2x, AM57xx, and AM65xx devices.

Modes of Operation

The LLD is intended to bring up the PCIe peripheral, open memory mappings, and send/receive interrupts.

Root Complex (RC)

The PCIe peripheral can be used as a root complex. One or more other endpoints can be connected (more than one requires a PCIe switch on the board). The LLD configures the peripheral in RC mode. It doesn’t attempt to emulate the full enumeration capability of a Linux driver. Instead the user will need to supply code specific to each endpoint they intend to support.

Endpoint (EP)

The PCIe peripheral can be used as an endpoint. This is the more intended usecase for the LLD. Once the link is initialized, the LLD can provide data addresses and send interrupts to the RC.

Interrupts

The PCIe Sample example for AM65xx provides code to send interrupts from an endpoint, and the LLD/example together contain code to receive/demux the interrupts on an RC.

4.14.1. Driver Configuration¶

Board-specific configurations

PCIe’s board specific configuration is in the example in PDK_INSTALL_DIR/packages/ti/drv/pcie/example/sample/am65xx/src/pcie_sample_board.c. Calling sequence is in example and repeated below.

PCIe configuration structure

The pcie_soc.c binds the driver to the hardware on the board. It is passed into the driver via the call to Pcie_init().

4.14.2. API Call Flow¶

The API call flow is covered in pcie_sample.c.

The overall components are:

1. Initialize the driver
2. Initialize the SERDES and Power the peripheral (see example code for order for each device)
3. Configure RC or EP symmetrically to talk to another similar device
4. Perform data IO and interrupt.

4.15.1. Examples¶

Detailed instructions to run example

PCIE_sample_ExampleProject

Ensure 2 Like EVMs are connected with a x1 PCIe male/male cross cable.

Build binaries appropriate for your board (evm or idk).

Load the binaries to A53 or R5 core on each board. Select one board as RC and other as EP by putting values from serial console. Wait for linkup and data teansfer. See table above to determine whether output is expected on serial console or CCS console.

Sample example output

Note that output will vary slightly based on device type. The following is from AM65xx. The output from the RC and EP are shown below.

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
R
*                RC mode                     *
*                RC mode                     *
**********************************************

Version #: 0x02030005; string PCIE LLD Revision: 02.03.00.05:Jun 23 2021:09:54:15

PCIe Power Up.
Serdes Init Complete (0)
Serdes Init Complete (1)
PLL configured.
PowerUP linkCap gen=3 change to 2
PowerUP linkCtrl2 gen=3 change to 2
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 3
Starting link training...
Link is up.
Checking link speed and # of lanes
Expect 2 lanes, found 2 lanes Pass
Expect gen 2 speed, found gen 2 speed Pass
Root Complex received data.
UDMA memcpy application started...
Speed: 1344 Mbps
UDMA memcpy using TR15 block copy Passed!!
Test passed.

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
E
*                EP mode                     *
*                EP mode                     *
**********************************************

Version #: 0x02030005; string PCIE LLD Revision: 02.03.00.05:Jun 23 2021:09:54:15

PCIe Power Up.
Serdes Init Complete (0)
Serdes Init Complete (1)
PLL configured.
PowerUP linkCap gen=3 change to 2
PowerUP linkCtrl2 gen=3 change to 2
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 3
Starting link training...
Link is up.
Checking link speed and # of lanes
Expect 2 lanes, found 2 lanes Pass
Expect gen 2 speed, found gen 2 speed Pass
End Point received data.
End Point sent data to Root Complex, completing the loopback.
End of Interrupt Test.
UDMA memcpy application started...

PCIE_ssd_ExampleProject

Format the ssd with a FAT partition with the below steps:

1. Connect ssd to a linux system
2. Create a partition using below command sudo fdisk /dev/nvme0n1 chose “n” to create partition
3. Create filesystem on the partition using below command mkfs -t vfat /dev/nvme0n1p1

Build binaries for A53 core. Load and run the binariy.

Once the link up is successful, console interface will come up if valid FAT partition is detected.

Run the commands to interact with the file system. Supported console commands include ls (to list files of directory), cd (change directory), pwd (present working directory) and cat (text file read operation).

SSD example output

**********************************************
*             PCIe Test Start                *
**********************************************

Version #: 0x02030005;
string PCIE LLD Revision: 02.03.00.05:Jul 13 2021:15:44:28

Serdes Init Complete (0)
Serdes Init Complete (1)
Configuring pcie_RC_MODE.
PowerUP linkCap gen=3 change to 2
PowerUP linkCtrl2 gen=3 change to 2
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 3
Starting link training...
Link is up.

Configuring pcie_EP_MODE.
Raw 0xa808144d
Vendor ID 0x144d
Device ID 0xa808
Size of admin_subq = 128 bytes
Size of command structure is 64 bytes
Size of admin_compq = 32 bytes
Size of command structure is 16 bytes
Value of cap_reg is 0x303c033fff
NVME_REG_CSTS Value : 0x0
NVME_REG_CSTS Value : 0x1

Admin queue creation complete
 == NVME IDENTIFY CONTROLLER DATA ==
    PCI VID   : 0x144d
    PCI SSVID : 0x144d
    SN        : S4EVNX0R156445E
    MN        : Samsung SSD 970 EVO Plus 500GB
    RAB       : 0x2
    AERL      : 0x3
    SQES      : 0x66
    CQES      : 0x44
    NN        : 0x1
Identified NVMe model Samsung SSD 970 EVO Plus 500GB
Maximum IO submission queues allowed by controller: 32
Maximum IO completion queues allowed by controller: 32
NVME INIT Complete
FATFS INIT Done
FATFS OPEN DONE
All tests have passed.
0:>