3.2.2.10. PCIe End Point

Introduction

PCIe controller IPs integrated in AM64x are capable of operating either in Root Complex mode (host) or End Point mode (device). When operating in End Point (EP) mode, the controller can be configured to be used as any function depending on the use case (‘Test endpoint’ and ‘NTB’ are the only PCIe EP functions supported in Linux kernel right now).

Block Diagram

Following is the block diagram of framework for endpoint mode:

../../../../../_images/ep_framework.png

Features of AM64

There is one instance of PCIe subsystem. Following are some of the main features:

  • The instance can be configured to operate in Root Complex mode or End Point mode

  • One lane configuration, capable up to 5.0 Gbps/lane (Gen2)

  • One Physical Function (PF)

  • Support for Legacy, MSI and MSI-X Interrupt

  • There can be 32 different address mappings in outbound address translation unit. The mappings can be from regions reserved for the PCIe instance.

    • For instance PCIE0, there are two regions in SoC Memory Map:

      • 128 MB region with address in lower 32 bits

      • 4 GB region with address above 32 bits

Capabilities of AM64 EVM

There is one instance of the PCIe subsystem on the EVM. Following are some of the details for that instance:

Instance

Supported lanes

Supported Connector

PCIE0

1 lane

Standard female connector

Hardware Setup Details

AM64x is, by default, intended to be operated in Root Complex mode. So in order to connect two boards, a specialized cable like below is required.

../../../../../_images/Pcie_ep_cable.jpg

This cable can be obtained from Adex Electronics (https://www.adexelec.com).

Modify the cable to remove resistors in CK+ and CK- in order to avoid ground loops (power) and smoking clock drivers (clk+/-).

Remove the RST resistors to avoid reset (PERST) being propagated from Root Complex to End Point. Also in Root Complex to End Point loopback connection, End Point running Linux should be initialized before Root Complex comes up. Propagating reset from Root Complex to End Point will do PORZ of End Point, which should be avoided.

The ends of the modified cable should look like below:

  • A side

    ../../../../../_images/PCIE_A_both_sides.jpg
  • B side

    ../../../../../_images/PCIE_B_both_sides.jpg

For End Point mode, PCIE_1L_MODE_SEL (switch 5) and PCIE_2L_MODE_SEL (switch 6) in sw3 should be set to ‘1’.

../../../../../_images/dip-switch.png

Following is an image of two AM64 EVMs connected back to back. There is no restriction on which end of the cable should be connected to host and device.

../../../../../_images/am64-evm-back-to-back.jpg

Refer to the following image to toggle between Root Complex mode and End Point mode.

../../../../../_images/am64-pcie-rc-ep-sel.png

End Point (EP) Device Configuration

DTS Modification

The default dts is configured to be used in root complex mode. In order to use it in endpoint mode, the following changes have to be made in dts file.

To configure AM64 EVM in EP mode, apply the following patch:

diff --git a/arch/arm64/boot/dts/ti/k3-am642-evm.dts b/arch/arm64/boot/dts/ti/k3-am642-evm.dts
index 96f90ebf28aa..86703b1dd7f3 100644
--- a/arch/arm64/boot/dts/ti/k3-am642-evm.dts
+++ b/arch/arm64/boot/dts/ti/k3-am642-evm.dts
@@ -711,13 +711,13 @@ &pcie0_rc {
        phys = <&serdes0_pcie_link>;
        phy-names = "pcie-phy";
        num-lanes = <1>;
+       status = "disabled";
};

&pcie0_ep {
        phys = <&serdes0_pcie_link>;
        phy-names = "pcie-phy";
        num-lanes = <1>;
-       status = "disabled";
};

Linux Driver Configuration

The following config options have to be enabled in order to configure the PCI controller to be used as a “Endpoint Test” function driver.

CONFIG_PCI_ENDPOINT=y
CONFIG_PCI_ENDPOINT_CONFIGFS=y
CONFIG_PCI_EPF_TEST=y
CONFIG_PCI_J721E=y
CONFIG_PCIE_CADENCE_EP=y

Endpoint Controller devices and Function drivers

To find the list of endpoint controller devices in the system:

root@evm:~# ls /sys/class/pci_epc/
f102000.pcie-ep

To find the list of endpoint function drivers in the system:

root@evm:~# ls /sys/bus/pci-epf/drivers
pci_epf_test  pci_epf_ntb

Using the pci-epf-test function driver

The pci-epf-test function driver can be used to test the endpoint functionality of the PCI controller. Some of the tests that are currently supported are:

Test

Description

BAR

A known pattern is written and read back from BAR

Interrupt (legacy/MSI/MSI-X)

Raise an interrupt (legacy/MSI/MSI-X) from EP

Read

Read data from a buffer in RC, and perform a cyclic redundancy check (CRC) for that data

Write

Write data to a buffer in RC, and perform a cyclic redundancy check (CRC) for that data

Copy

Copy data from one RC buffer to another RC buffer, and perform a cyclic redundancy check (CRC) for that data

../../../../../_images/pci-epf-test.png

Creating pci-epf-test device

PCI endpoint function device can be created using the configfs. To create pci-epf-test function, the following commands can be used:

mount -t configfs none /sys/kernel/config
cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_test/func1

The above commands create the pci-epf-test function device.

The PCI endpoint framework populates the directory with configurable fields.

root@evm:/sys/kernel/config/pci_ep# ls functions/pci_epf_test/func1
baseclass_code  cache_line_size  deviceid  interrupt_pin  msi_interrupts  msix_interrupts  progif_code  revid  subclass_code  subsys_id  subsys_vendor_id  vendorid

The driver populates these entries with default values when the device is bound to the driver. The pci-epf-test driver populates vendorid with 0xffff and interrupt_pin with 0x0001.

root@evm:/sys/kernel/config/pci_ep# cat functions/pci_epf_test/func1/vendorid
0xffff
root@evm:/sys/kernel/config/pci_ep# cat functions/pci_epf_test/func1/interrupt_pin
0x0001

Configuring pci-epf-test device

The user can configure the pci-epf-test device using the configfs. In order to change the vendorid and the number of MSI interrupts used by the function device, the following commands can be used:

root@evm:/sys/kernel/config/pci_ep# echo 0x104c > functions/pci_epf_test/func1/vendorid

The above command configures Texas Instruments as the vendor.

root@evm:/sys/kernel/config/pci_ep# echo 0xb010 > functions/pci_epf_test/func1/deviceid

The above command configures the deviceid.

root@evm:/sys/kernel/config/pci_ep# echo 2 > functions/pci_epf_test/func1/msi_interrupts
root@evm:/sys/kernel/config/pci_ep# echo 2 > functions/pci_epf_test/func1/msix_interrupts

The above command configures the number of interrupts. 2 is the number of MSI and MSI-X interrupts being configured. The number of interrupts configured should be between 1 to 32 for MSI and 1 to 2048 for MSI-X.

Binding pci-epf-test device to a EP controller

In order for the endpoint function device to be useful, it has to be bound to a PCI endpoint controller driver. Use the configfs to bind the function device to one of the controller drivers present in the system.

root@evm:/sys/kernel/config/pci_ep# ln -s functions/pci_epf_test/func1 controllers/f102000.pcie-ep/

Starting the EP device

In order for the EP device to be ready to establish the link, the following command should be given:

root@evm:/sys/kernel/config/pci_ep# echo 1 > controllers/f102000.pcie-ep/start

The complete sequence when using one physical function will look like the following:

mount -t configfs none /sys/kernel/config
cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_test/func1
echo 0x104c > functions/pci_epf_test/func1/vendorid
echo 0xb010 > functions/pci_epf_test/func1/deviceid
echo 2 > functions/pci_epf_test/func1/msi_interrupts
echo 2 > functions/pci_epf_test/func1/msix_interrupts
ln -s functions/pci_epf_test/func1 controllers/f102000.pcie-ep/
echo 1 > controllers/f102000.pcie-ep/start

HOST Device Configuration

The PCI EP device must be powered-on and configured before the PCI HOST device. This restriction is because the PCI HOST doesn’t have hot plug support.

Linux Driver Configuration

The following config options have to be enabled in order to use the “Endpoint Test” PCI device.

CONFIG_PCI=y
CONFIG_PCI_ENDPOINT_TEST=y
CONFIG_PCIE_CADENCE_HOST=y

lspci output

0000:00:00.0 PCI bridge: Texas Instruments Device b010
0000:01:00.0 Unassigned class [ff00]: Texas Instruments Device b010

Using the Endpoint Test function device

pci_endpoint_test driver creates the Endpoint Test function device which will be used by the following pcitest utility. pci_endpoint_test can either be built-in to the kernel or built as a module. For testing legacy interrupt, MSI interrupt has to be disabled in the host.

pcitest.sh added in tools/pci/ can be used to run all the default PCI endpoint tests. Before pcitest.sh can be used, pcitest.c should be compiled using following steps:

cd <kernel-dir>
make headers_install ARCH=arm64
aarch64-linux-gnu-gcc -Iusr/include tools/pci/pcitest.c -o pcitest
cp pcitest  <rootfs>/usr/sbin/
cp tools/pci/pcitest.sh <rootfs>

pcitest output

pcitest can be used as follows:

root@evm:~# ./pcitest -h
usage:  -h                      Print this help message
[options]
Options:
        -D <dev>                PCI endpoint test device {default: /dev/pci-endpoint-test.0}
        -b <bar num>            BAR test (bar number between 0..5)
        -m <msi num>            MSI test (msi number between 1..32)
        -x <msix num>           MSI-X test (msix number between 1..2048)
        -i <irq type>           Set IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X)
        -e                      Clear IRQ
        -I                      Get current IRQ type configured
        -l                      Legacy IRQ test
        -r                      Read buffer test
        -w                      Write buffer test
        -c                      Copy buffer test
        -s <size>               Size of buffer {default: 100KB}

Sample usage

root@evm:~# ./pcitest -i 1 -D /dev/pci-endpoint-test.0
SET IRQ TYPE TO MSI:            OKAY
root@evm:~# ./pcitest -m 1 -D /dev/pci-endpoint-test.0
MSI1:           OKAY
root@evm:~# ./pcitest -e -D /dev/pci-endpoint-test.0
CLEAR IRQ:              OKAY
root@evm:~# ./pcitest -i 2 -D /dev/pci-endpoint-test.0
SET IRQ TYPE TO MSI-X:          OKAY
root@evm:~# ./pcitest -x 1 -D /dev/pci-endpoint-test.0
MSI-X1:         OKAY
root@evm:~# ./pcitest -e -D /dev/pci-endpoint-test.0
CLEAR IRQ:              OKAY

The script pcitest.sh runs all the bar tests, interrupt tests, read tests, write tests and copy tests.

Files

Serial No

Location

Description

1

drivers/pci/endpoint/pci-epc-core.c

PCI Endpoint Framework

drivers/pci/endpoint/pci-ep-cfs.c

drivers/pci/endpoint/pci-epc-mem.c

drivers/pci/endpoint/pci-epf-core.c

2

drivers/pci/endpoint/functions/pci-epf-test.c

PCI Endpoint Function Driver

3

drivers/misc/pci_endpoint_test.c

PCI Driver

4

tools/pci/pcitest.c

PCI Userspace Tools

tools/pci/pcitest.sh

5

drivers/pci/controller/pci-j721e.c

PCI Controller Driver

drivers/pci/controller/pcie-cadence.c

drivers/pci/controller/pcie-cadence-ep.c

drivers/pci/endpoint/pcie-cadence-host.c