4.2. HSR_PRP¶

4.2.2.1. Firmware Features Supported¶

• HSR/PRP
  • The implementation is as per IEC 62439-3 Ed 2.0
  • Operates as a DAN (Dual Attached Node)
• HSR
  • Operates as a DANH
  • As per clause 5 of the standard
  • Support for modes – H, T, U and N as per standard
  • Modes can be changed at run time
• PRP
  • Operates as a DANP
  • As per clause 6 of the standard
  • Two ports as per standard, Port A and Port B
• Support for Node Table
  • Total 128 entries on AMICx, AM335x, AM437x
  • Total 256 entries on AM57xx & K2G
  • Hash Table for faster lookup
  • O(1) complexity
  • Node Table statistics
• Support for Multicast Filtering
  • Supported on all SoCs
  • Hash Table for faster lookup
  • O(1) complexity
• Duplicate Discard Table
  • Duplicate discard on Port to Host path (HSR and PRP)
  • Duplicate discard table on Port to Port path (HSR)
  • Data integrity (CRC) check during port to port forwarding, except cut through (HSR)
• QoS scheme : 3-bit VLAN PCP
  • No of levels supported : 8
  • Number of host queues : 2 | 4 QoS levels per host queue
  • Number of port queues : 4 | 2 QoS levels per port queue
  • Number of host queues are configurable
• Statistics
  • Supports all MIB statistics as per standard
  • Node Table statistics for debugging
• PTP/1588
  • PTP Slave and Master mode
  • Supports P2P clock.
  • PTP over 802.3 (Annex F)
  • Transparent Clock supported
  • Ordinary Clock supported
  • Single and Two step clock supported
  • Peer delay Response is always sent as two-step
• Storm Prevention : Yes. Configurable per port

4.2.2.2. Duplicate algorithm and table¶

Handling duplicate frames is one of the main tasks in HSR and PRP. The LRE must not provide the duplicate of a frame to its upper layer in order to offload the processor. The algorithm for discarding duplicates is not specified in IEC 62439-3. However, the standard mandates that the algorithm must be designed such that it never rejects a legitimate frame, while occasional acceptance of a duplicate can be tolerated.

In HSR, additionally to discarding duplicates destined to the host, a discard operation should also be used to prevent frames from looping in the network. A looping frame is a frame that is never discarded in any node of the network and therefore it keeps looping indefinitely. Normally, such a frame should be dropped by its source node. The function handles the case when a defective HSR device does not filter correctly according to source/destination address. This operation is less critical than the host duplicate discard. It is also specified that any duplicate discard method shall be able to forget an entry identified by the source MAC address and the sequence number after a time EntryForgetTime (default 400ms).

A table holds entries representing frames that have been registered in the LRE, each entry is represented bya frame signature. The signature consists of the source MAC address and the 16 bit sequence number. The sequence number value is incremented by 1 when a frame and its duplicate are sent by a source in an HSR network. This signature allows identifying a frame uniquely. When a frame is received, the list is searched by comparing the frame signature with all entries in the table. If such an entry is found, the frame is a duplicate and shall be discarded. If no entry exists, the frame is the first one received and needs to be forwarded to the application and/or – in case of HSR - to the other port. With a successful forwarding process (frame received with no error), a new entry with the signature of that frame is created in the table.

A linear lookup process is too time consuming; thus Hash algorithms are used to decrease the search time.

4.2.2.3. Port to Host Duplicate Table¶

It is used to prevent sending duplicate frames to the upper layers. This table is common for both PRUs. Resource sharing implies that collision (two PRUs accessing the same data) must be handled. Since the table is common to both PRUs, the host duplicate rejection must be done after the EOF is detected in order to avoid rejecting legitimate frames.

4.2.2.4. Port to Port Duplicate Table¶

A PRU forwards the frames received from one HSR port to the other HSR port, unless the frame was sent already. Each PRU has its own port duplicate table. A search operation can start before the EOF is detected since the table is not shared between the PRU and is not subject to any modification during the reception of a frame. In order to minimize the load of the PRU after EOF reception, the search operation in the port duplicate table is executed during frame reception. Moreover, this allows to quickly detecting if the frame should be discarded early in the reception process. An insertion in the table is still made only after the EOF is detected, but the operation is much faster since the result of the search operation can used.

4.2.2.5. Node Table¶

The node table is a central element of the HSR/PRP mechanism although the standard declares node tables optional. This firmware implements node tables for HSR and PRP. Node table handling is implemented in the PRU and comprises of registering incoming supervision and non-supervision-frames as well as ageing and deleting of old entries. The standard requires updating the node table when a supervision frame is received. Accessing the node table is thus not critical since supervision frames are sent only every 2 seconds. For HSR, to maintain statistics for each source node of the network and to facilitate debugging, the implementation updates the node table for each frame received by the host from a source that has previously sent a supervision frame. A further advantage of this approach is having a similar implementation for HSR and PRP. Indeed, the PRP Standard requires updating the node table, if implemented, for all the incoming frames in order to keep track of the traffic contented to each redundant LAN. The node table is accessible by the host for management purposes. Update- and register operations in the table should be done when the frame is completely received in order to avoid registering erroneous frames.

The node table search/update operation is very critical since it is resource/time consuming (maximum NODE_TABLE_SIZE is 256 entries). In the worst case scenario (node table full), a linear lookup operation in the receiving task could load the PRU too much and in consequence could cause Rx overflow or TX underflow. Moreover the node table is common to both PRUs and since it is accessed often, the resource-sharing management has additional impact on the latency. Optimized linear tables were considered, but the timing remained too critical in the worst case scenario. Hash tables are not considered because it is not allowed to lose entries due to collisions. The implemented solution is a sorted table with a two layer table structure. This allows to significantly optimize the lookup time especially when the table has lots of valid entries.

It is important to notice that in a normal HSR case scenario there are - in average - significantly more successful lookup operations (MAC address found in the table) than insertion and deletion operations. Indeed, insertion operations are only made with a supervision frame. This assumption cannot be made for PRP since all frames can create an entry in the node table. The design is therefore optimized according to this assumption.

4.2.2.6. Frame duplication from the host¶

Frames sent by the host must be duplicated and should be sent to both ports nearly at the same time for HSR and PRP. In a simple approach, the host stores the frame in the queue of one PRU, the PRU then waits until both ports are available and finally sends the frame at the same time via both ports. However, this solution is not resource efficient and it increases the latency of one PRU when waiting for the PORT to be ready. Therefore, a solution that off-loads the PRU from the duplication task is implemented: the host driver duplicates the frames and puts a copy to each PRU queue.

The implementation is however not straightforward:

Since the frames are not “linked” to each other, there is no guarantee that the frames are sent by both PRUs within a short interval. The host could control the queue status and only push the frames in the highest priority queue of both PRU when the queues are free. This would force each PRU to send the frame and its duplicate next. Nevertheless, for HSR this solution removes the prioritization of the frame circulating in the HSR ring as a PRU could buffer a high priority frame from the ring due to a lower priority frame sent by the host. Frames circulating in the ring are to be treated with higher priority than frames sent by the host. There is a compromise to be found.

The solution is the following: The host driver duplicates the frame and puts the duplicates in the queues of each PRU. The queue is selected based on the priority of the frame. Each PRU handles the frame independently and we only have the assurance that each frame will be sent out within an unknown interval. The drawback of this solution is the lack of control on when each frame will be sent out. The interval can be bigger than one maximum sized Ethernet frame in this case.

4.2.2.7. Supervision Frame¶

Incoming supervision frames are received and processed by the PRU. On reception of a frame, the PRU updates the node table and the statistic counters. For HSR, if the received supervision frame originates from the receiver (i.e. it traversed the whole ring) it is discarded. Otherwise it is sent to the next node in store and forward mode. The reason for using store and forward mode is the following: A supervision frame might trigger the creation of an entry in the node table and this task is time consuming. By sending the frame in store and forward mode, more time is available for this operation while receiving the frame. Each device in an HSR/PRP network sends supervision frames at a constant time interval. Outgoing supervision frames are composed and sent by the host CPU.

4.2.2.8. Cut-Through for HSR¶

Cut through happens when switch firmware bypasses the transmit queues and directly copies data from Rx FIFO to Tx FIFO. The concept is explained here

The concept of forwarding frames from HSR port to HSR port when operating in cut-through mode deserves more explanation since the following boundary conditions require a carefully balanced implementation: •Cut-through of a frame shall start as early as possible in order to minimize the propagation delay in each node. The ICSS switch supports starting the cut-through operation after a configurable amount of bytes received. Theoretically, for HSR the minimum number of bytes required is 22 (12 bytes Source-/Destination MAC, 4 bytes VLAN Tag, 6 bytes HSR Tag) in order to have all the necessary data to take the routing decision. The receiving process is optimized to allow cut through as soon as possible depending on the type of the frame ( see start receiving process in 3.2.2 Micro Scheduler). •Duplicates shall not be cut through (i.e. the detection of a duplicate shall be finished before the next node in the ring starts processing the frame) •Detection of duplicates (computing the hash index and scanning a bin in the hash table) is time consuming

Waiting for the duplicate detection before taking the cut through decision would delay the frame too much. Starting to cut-through a frame without port duplicate rejection minimizes the delay and allows starting the receiving process after 14 bytes. Once the cut through of a frame has started, the port duplication rejection process starts. If the frame is identified as duplicate or corrupted, the transmission of the frame is aborted by asserting the TX_RESET which will reset the transmit FIFO and clear all its contents and therefore corrupt the frame being cut-though.

4.2.2.9. HSR/PRP Memory Map¶

Shared RAM Memory Map

Table: Shared RAM Memory Map

Table: LRE Interface Stats

Table: PRU0 RAM Memory Map

Table: PRU1 RAM Memory Map

4.2.3.1. Overview¶

In both HSR and PRP protocols, the DAN (Dual Attached Node) sends an identical frame to both the interfaces and uses a sequence number in the tag to allow drop duplicates at the Rx node to achieve redundancy. This section describes the different software components of a Linux based HSR/PRP solution to support redundancy.

4.2.3.2. Features supported¶

• HSR/PRP

  • 100 Mbits/s Full Duplex Ethernet Interface
  • The implementation is as per IEC 62439-3 Ed 2.0
  • Operates as a DAN (Dual Attached Node)

• HSR

  • Operates as a DANH
  • As per clause 5 of the standard
  • Support for modes – H, T, U and N as per standard
  • Modes can be changed at run time
  • Cut-through switching

• PRP

  • Operates as a DANP
  • As per clause 6 of the standard
  • Two ports as per standard, Port A and Port B

• Support for Node Table

  • Total 128 entries on AMICx, AM335x, AM437x
  • Total 256 entries on AM57xx & K2G
  • Hash Table for faster lookup
  • O(1) complexity
  • Node Table statistics

• Support for Multicast Filtering

  • Supported on all SoCs
  • Hash Table of 256 entries for faster lookup
  • O(1) complexity
  • User configurable mask to select bits for hashing

• Support for VLAN over HSR/PRP interface

• Support for VLAN Filtering

  • VLAN filter table of 4096 entries for perfect match
  • O(1) complexity

• Duplicate Discard Table

  • Duplicate discard on Port to Host path (HSR and PRP)
  • Duplicate discard table on Port to Port path (HSR)
  • Data integrity (CRC) check during port to port forwarding, except cut through (HSR)

• QoS scheme : 3-bit VLAN PCP

  • No of levels supported : 2
  • Number of host queues : 2 | 2 QoS levels per host queue
  • Number of port queues : 2 | 2 QoS levels per port queue

• Statistics

  • Supports all MIB statistics as per standard
  • Node Table statistics for debugging

• Storm Prevention : Yes. Configurable per port

• Dual instance of HSR/PRP using two PRU-ICSS on AM571x

• VLAN tag support in Supervision frames

• Run time change of Protocol at PRU-ICSS without reboot

• SAN PTP/1588

  • Supported using PRP firmware in EMAC mode
  • PTP Slave and Master mode
  • Supports P2P clock.
  • PTP over 802.3 (Annex F)
  • Ordinary Clock (GMAC/PRU port)
  • Boundary Clock supported
  • Two step clock supported

• Redundant PTP/1588

  • Transparent clock as per IEC62439-3 Annex C

4.2.3.3. Software Architecture¶

The diagram below highlights the software components that are modified or developed specifically for HSR/PRP. These along with other standard Linux OS components are used to implement the Linux HSR/PRP DAN (Dual Attached Node).

• Linux Kernel drivers
  • HSR/PRP driver
  • Ethernet driver
• User space
  • Net-SNMP
  • iproute2

4.2.3.3.2. Linux HSR/PRP Driver¶

The Linux networking subsystem in upstream kernel has added support for HSR driver starting in v3.12 (HSRv0 - IEC 62439-3:2010) and enhanced the same to support HSRv1 (IEC 62439-3:2012) in v4.6. This driver allows user to create an HSR network device with a pair of slave network interfaces (A and B) that are standard Ethernet interfaces. For example on TI’s AM572x IDK EVM, there are two Ethernet interfaces (10/100) per each PRU ICSS. The PRU Emac driver available for ICSS PRU Ethernet h/w supports two Ethernet ports. Using ip link command, user will be able to setup a HSR interface on this EVM that pair the two PRU Ethernet interfaces to create a HSR node or DAN-H.

TI has enhanced the Linux HSR driver and ip link command in iproute2 package to support PRP defined in IEC 62439-3 clause 4. With this change, user will be able to create a Doubly Attached Node with PRP (DAN-P) using standard Ethernet interfaces. As in the case of HSR, using ip link command, user will be able to setup a prp interface using two Ethernet interfaces such as the one provided by PRU EMAC driver.

Linux HSR/PRP driver provides a standard Ethernet interface to application layer and hide the details of the redundancy protocol under the driver layer. Linux HSR/PRP driver implements the Link Redundancy Entity (LRE) that is central to implementing redundancy using respective protocol.

HSR LRE functions and L2 routing can be offloaded to the firmware running on the PRU cores of ICSS. The diagram below shows the architecture of an Offloaded HSR/PRP driver. The upper layer driver is existing Linux HSR/PRP driver described in the previous section. The PRU Ethernet driver exports HSR/PRP capabilities in the NetDev feature flags. The green blocks in the Redundancy layer shows the software blocks that gets offloaded to PRUs in ICSS which is shown green as well. The upper layer HSR/PRP driver uses the feature flags exported by the PRU Ethernet driver to disable corresponding functions in its layer. The firmware is a re-use from the TI RTOS implementation that is discussed above. Re-use means the data structures and API should be re-used across both TI RTOS and Linux driver implementations. A common PRU Ethernet driver is developed to support existing EMAC device as well HSR/PRP

byte 0 - To host from PRU 1, PCP 0-3 => Q3
byte 1 - To host from PRU 1, PCP 4-7 => Q2
byte 2 - To host from PRU 0, PCP 0-3 => Q1
byte 3 - To host from PRU 0, PCP 4-7 => Q0

PRU0/PRU1
---------
byte 0 - from Host to PRU, PCP 0-3 => Q3
byte 1 - from Host to PRU, PCP 4-7 => Q2

PRU0
-----
byte 2 - from PRU1 to PRU0, PCP 0-3 => Q1
byte 3 - from PRU1 to PRU0, PCP 4-7 => Q0

PRU1
-----
byte 2 - from PRU0 to PRU1, PCP 0-3 => Q1
byte 3 - from PRU0 to PRU1, PCP 4-7 => Q0

From PRU0 to Host, PCP 0-3 => Q1
From PRU0 to Host, PCP 4-7 => Q0

From PRU1 to Host, PCP 0-3 => Q3
From PRU1 to Host, PCP 4-7 => Q2

PRU0/PRU1
-- ---------
From Host to PRU0/PRU1, PCP 0-1 => Q3
From Host to PRU0/PRU1, PCP 2-3 => Q2
From Host to PRU0/PRU1, PCP 4-5 => Q1
From Host to PRU0/PRU1, PCP 6-7 => Q0

ifconfig eth2 down 0.0.0.0
ifconfig eth3 down 0.0.0.0

ethtool -K eth2 hsr-rx-offload off
ethtool -K eth3 hsr-rx-offload off

ethtool -K eth2 prp-rx-offload off
ethtool -K eth3 prp-rx-offload off

ethtool -K eth2 hsr-rx-offload on
ethtool -K eth3 hsr-rx-offload on

ethtool -K eth2 prp-rx-offload on
ethtool -K eth3 prp-rx-offload on

ethtool -k eth2

ifconfig eth2 0.0.0.0 down
ifconfig eth3 0.0.0.0 down
ifconfig eth2 hw ether 70:FF:76:1C:0E:8C
ifconfig eth3 hw ether 70:FF:76:1C:0E:8C
ethtool -K eth2 hsr-rx-offload on
ethtool -K eth3 hsr-rx-offload on
ifconfig eth2 up
ifconfig eth3 up
ip link add name hsr0 type hsr slave1 eth2 slave2 eth3 supervision 45 version 1
ifconfig hsr0 192.168.2.20

ip link delete hsr0
ifconfig eth2 down
ifconfig eth3 down
ethtool -K eth2 hsr-rx-offload off
ethtool -K eth3 hsr-rx-offload off
ethtool -K eth2 prp-rx-offload on
ethtool -K eth3 prp-rx-offload on
ifconfig eth2 up
ifconfig eth3 up
ip link add name prp0 type prp slave1 eth2 slave2 eth3 supervision 45
ifconfig prp0 192.168.2.20

ip link delete prp0
ifconfig eth2 down
ifconfig eth3 down
ethtool -K eth2 prp-rx-offload off
ethtool -K eth3 prp-rx-offload off
ethtool -K eth2 hsr-rx-offload on
ethtool -K eth3 hsr-rx-offload on
ifconfig eth2 up
ifconfig eth3 up
ip link add name hsr0 type hsr slave1 eth2 slave2 eth3 supervision 45 version 1
ifconfig hsr0 192.168.2.20

ip link delete hsr0
ifconfig eth2 down
ifconfig eth3 down
#Restore eth3 MAC address
ifconfig eth3 hw ether 70:FF:76:1C:0E:8E
ethtool -K eth2 hsr-rx-offload off
ethtool -K eth3 hsr-rx-offload off
ifconfig eth2 192.168.2.20
ifconfig eth3 192.168.3.20

ip link delete prp0
ifconfig eth2 down
ifconfig eth3 down
ethtool -K eth2 prp-rx-offload off
ethtool -K eth3 prp-rx-offload off
#Restore eth3 MAC address
ifconfig eth3 hw ether 70:FF:76:1C:0E:8E
ifconfig eth2 192.168.2.20
ifconfig eth3 192.168.3.20

echo 1 > /sys/devices/platform/pruss<ID>_eth/net/eth<if number>/prp_emac_mode

4.2.3.4. Multicast filtering¶

Multicast filtering is an Ethernet frame filtering feature in PRU firmware based on the destination MAC address of the received frame. The PRU provides a Multicast filter table in Data RAM1 of PRU with a size of 256 entries of 1 byte each. Firmware implements an imperfect match for filtering the frames based on a hash calculated using the destination MAC address of the frame and a configurable mask if the destination address is a multicast MAC address.

Hash = (MAC_ADDR[0] AND MASK[0]) XOR
       (MAC_ADDR[1] AND MASK[1]) XOR
       (MAC_ADDR[2] AND MASK[2]) XOR
       (MAC_ADDR[3] AND MASK[3]) XOR
       (MAC_ADDR[4] AND MASK[4]) XOR
       (MAC_ADDR[5] AND MASK[5])

Where MASK is a user configurable value provided at boot time and MAC_ADDR is the multicast MAC address which is extracted from the destination address of the Ethernet frame and AND is the bitwise AND operation. In other words, a bitwise AND operation is performed between each byte of MAC address and a corresponding MASK byte and the resulting bytes are XOR-ed together to get the hash value. The hash is used to index into the Multicast filter table to check if it is set (1) or reset (0). If set, the frame is forwarded to ARM core (a.k.a Host). If reset, the frame is dropped at the PRU. This is an imperfect match since there can be multiple MAC addresses that produces the same hash value. So these frames will get forwarded to the ARM core.

ti_prueth.pruss<ID>_port<PORT>_mc_mask is the module parameter for MASK where <ID> is described in section PRU-ICSS Subsystem Driver and <PORT> is ‘0’ in HSR/PRP case since both ports share the same MASK (port 1 does not need to be set and has no effect), and ‘0’ or ‘1’ in Dual EMAC case since both ports can be configured independently.

Sample bootargs with MASK configured (HSR/PRP)

setenv pruss1_port0_mc_mask "FF:FF:FF:FF:00:00"
setenv pruss2_port0_mc_mask "FF:FF:FF:FF:00:00"
setenv args_mmc 'run finduuid;setenv bootargs console=${console} ${optargs} root=PARTUUID=${uuid} rw rootfstype=${mmcrootfstype}  ti_prueth.pruss1_ethtype=${pruss1_ethtype} ti_prueth.pruss2_ethtype=${pruss2_ethtype} ti_prueth.pruss1_port0_mc_mask=${pruss1_port0_mc_mask} ti_prueth.pruss2_port0_mc_mask=${pruss2_port0_mc_mask}'

A comparable sample bootargs with MASK configured for Dual EMAC

setenv pruss1_port0_mc_mask "FF:FF:FF:FF:00:00"
setenv pruss1_port1_mc_mask "FF:FF:FF:FF:00:00"
setenv pruss2_port0_mc_mask "FF:FF:FF:FF:00:00"
setenv pruss2_port1_mc_mask "FF:FF:FF:FF:00:00"
setenv args_mmc 'run finduuid;setenv bootargs console=${console} ${optargs} root=PARTUUID=${uuid} rw rootfstype=${mmcrootfstype}  ti_prueth.pruss1_ethtype=${pruss1_ethtype} ti_prueth.pruss2_ethtype=${pruss2_ethtype} ti_prueth.pruss1_port0_mc_mask=${pruss1_port0_mc_mask} ti_prueth.pruss1_port1_mc_mask=${pruss1_port1_mc_mask} ti_prueth.pruss2_port0_mc_mask=${pruss2_port0_mc_mask} ti_prueth.pruss2_port1_mc_mask=${pruss2_port1_mc_mask}'

Typically, an application joins the multicast group either using a raw socket of type SOCK_DGRAM and use setsockopt() API to Join or leave the multicast group. An interesting article on this is available at

https://www.tenouk.com/Module41c.html

This causes the Multicast MAC address to be added to the mc_list of the socket and the associated network device in kernel and finally get passed to the Ethernet device driver (in our case, it is PRU Ethernet device driver). The relevant API is ndo_set_rx_mode() of the net_device_ops structure in Linux kernel associated with the network device. The PRU Ethernet device driver calculates the hash as described above and then writes 1 to MC filter table using the hash value as index. The PRU Ethernet device driver also supports allmulti which is used to enable receieve of all multicast frames at an interface. This is an option passed to the ifconfig command.

Example
>ifconfig eth2 192.168.2.20 allmulti

To remove the option
>ifconfig eth2 192.168.2.20 -allmulti

To display the Multicast address list of an interface, say eth2, user types
>ip maddr show dev eth2

sample display

root@am57xx-evm:~# ip maddr show dev eth2
6:      eth2
        link  33:33:00:00:00:01 users 2
        link  01:00:5e:00:00:01 users 2
        link  33:33:ff:1c:16:e0 users 2
        link  01:00:5e:00:00:fb
        link  01:00:5e:00:00:fc
        link  33:33:00:01:00:03 users 2
        link  33:33:00:00:00:fb users 2
        inet  224.0.0.1
        inet6 ff02::fb
        inet6 ff02::1:3
        inet6 ff02::1:ff1c:16e0
        inet6 ff02::1
        inet6 ff01::1

PRU Ethernet driver also provides a debugfs file mc_filter to display MC filter table in the memory

/sys/kernel/debug/prueth-prp-<ID>/mc_filter

or

/sys/kernel/debug/prueth-hsr-<ID>/mc_filter

or (for Dual EMAC)

/sys/kernel/debug/prueth-eth<if number>/mc_filter

where ID/’if number’ is described in section PRU-ICSS Subsystem Driver

Sample display

root@am57xx-evm:~# cat /sys/kernel/debug/prueth-prp-2/mc_filter
MC Filter : enabled
MC Mask : ff:ff:ff:ff:0:0
MC Filter table below 1 - Allowed, 0 - Dropped

  0: 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 30: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 40: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 50: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
 60: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 80: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 90: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 a0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 b0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 c0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 d0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 e0: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
 f0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

4.2.3.5. VLAN over PRU Ethernet¶

Virtual LAN (VLAN) is a standard Linux feature that can be enabled over PRU Ethernet devices. There are many websites that describes how Linux VLAN works. Some of them are at [[2]] or at [[3]] and some configuration details are also discussed at [[4]].

User may use ip link command to setup vlan interface over prueth interfaces. For example, over a hsr interface, if user would like to setup two vlan interfaces say, hsr0.2 and hsr0.3, user execute the following commands assuming the hsr0 interface is already setup:-

Node-1

ifconfig hsr0 0.0.0.0
ip link add link hsr0 name hsr0.2 type vlan id 2
ip link add link hsr0 name hsr0.3 type vlan id 3
ifconfig hsr0.2 192.168.2.40
ifconfig hsr0.3 192.168.3.40
ip link set hsr0.2 type vlan egress 0:0
ip link set hsr0.3 type vlan egress 0:7

Node-2

ifconfig hsr0 0.0.0.0
ip link add link hsr0 name hsr0.2 type vlan id 2
ip link add link hsr0 name hsr0.3 type vlan id 3
ifconfig hsr0.2 192.168.2.50
ifconfig hsr0.3 192.168.3.50
ip link set hsr0.2 type vlan egress 0:0
ip link set hsr0.3 type vlan egress 0:7

The above assume we have a network with two nodes connected back to back. The procedure is same if more than two nodes are present in the network. Using similar commands, user would be able to overlay multiple virtual LANs over the physical LAN created. The egress option of ip link command allow mapping packet priority to VLAN PCP values. In the example the packets on VLAN ID 2 is mapped to PCP value of 0 and VLAN ID 3 is mapped to PCP value of 7. At the Egress PRU Ethernet device looks at the PCP value and places it into one of the 4 queues. PRU scan the high priority queue first for packets and transmits them before transmitting packets from the lower priority queues. At the ingress, the PRU checks the PCP value of the packet in the VLAN header and places the frame into one of the ingress queues. More details on Queue usage is available at Queue Usage and VLAN PCP to Queue Map

In the above example, after passing some UDP traffic over hsr0.2 and hsr0.3, the user may check following statistics to verify if the packets are going over the right VLAN interface

On Node-1

cat /proc/net/vlan/hsr0.3
hsr0.3  VID: 3   REORDER_HDR: 1  dev->priv_flags: 1001
         total frames received        68090
          total bytes received     52598480
      Broadcast/Multicast Rcvd           52

      total frames transmitted        68090
       total bytes transmitted     52598459
Device: hsr0
INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
 EGRESS priority mappings: 0:7
root@am57xx-evm:~# cat /proc/net/vlan/hsr0.2
hsr0.2  VID: 2   REORDER_HDR: 1  dev->priv_flags: 1001
        total frames received        68105
         total bytes received     52604702
      Broadcast/Multicast Rcvd           53

      total frames transmitted        68074
       total bytes transmitted     52583138
Device: hsr0
INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
 EGRESS priority mappings: 0:0
root@am57xx-evm:~# cat /sys/kernel/debug/prueth-eth2/stats
   TxQ-0    TxQ-1    TxQ-2    TxQ-3    RxQ-0    RxQ-1
=====================================================
   68041        0        0    68908    67809    68925
root@am57xx-evm:~# cat /sys/kernel/debug/prueth-eth3/stats
   TxQ-0    TxQ-1    TxQ-2    TxQ-3    RxQ-2    RxQ-3
=====================================================
   68041        0        0    68912      234      695

Similar procedure can be used for setting up VLAN interfaces over PRU EMAC and HSR Ethernet types.

4.2.3.6. VLAN Filtering¶

The PRU has a 4096 entry VLAN filter table that allows filtering out unwanted VLAN traffic to the host. As soon a VLAN interface is created, the 802.1q Linux kernel module sends the VID information down to the lower layer HSR or PRP Linux device which in turn pass it down to the slave Ethernet devices below it (or directly to the Ethernet device in the case of Dual EMAC). The PRU Ethernet driver gets the VID information via net_device_ops:ndo_vlan_rx_add_vid(). On receiving this, PRU Ethernet driver sets the entry at the VID index in VLAN filter table to 1. When the VLAN interface is deleted, the driver receives the same information through ndo_vlan_rx_kill_vid() and reset the entry at the VID index.

PRU firmware on receiving a VLAN frame, extracts the VID and look up the VLAN filter table for an entry at the VID if VLAN filtering is enabled in firmware. If the entry is 1, it forwards the frame to the Host. Otherwise the frame is dropped. There are additional controls to allow priority tagged frames to Host if the corrsponding bit is set in firmware shared memory. PRU Ethernet driver always enables Priority tagged frames to the Host. User may setup a VLAN interface with VID 0 to send or receive priority tagged frames. See section VLAN for details on how to assign egress priority mapping for the priority tagged VLAN interface.

Useful commands

User can use the following command to view the VLAN filter table at PRU. (Similar commands apply for HSR and Dual EMAC)

cat /sys/kernel/debug/prueth-prp-<ID>/vlan_filter

Where ID is described at PRU-ICSS Subsystem Driver

root@am57xx-evm:~# cat /sys/kernel/debug/prueth-prp-2/vlan_filter
VLAN Filter : enabled
VLAN Filter untagged : allowed to Host
VLAN Filter priority tagged: allowed to Host

   0: 0011000000000000000000000000000000000000000000000000000000000000
  64: 0000000000000000000000000000000000000000000000000000000000000000
 128: 0000000000000000000000000000000000000000000000000000000000000000
 192: 0000000000000000000000000000000000000000000000000000000000000000
 256: 0000000000000000000000000000000000000000000000000000000000000000
 320: 0000000000000000000000000000000000000000000000000000000000000000
 384: 0000000000000000000000000000000000000000000000000000000000000000
 448: 0000000000000000000000000000000000000000000000000000000000000000
 512: 0000000000000000000000000000000000000000000000000000000000000000
 576: 0000000000000000000000000000000000000000000000000000000000000000
 640: 0000000000000000000000000000000000000000000000000000000000000000
 704: 0000000000000000000000000000000000000000000000000000000000000000
 768: 0000000000000000000000000000000000000000000000000000000000000000
 832: 0000000000000000000000000000000000000000000000000000000000000000
 896: 0000000000000000000000000000000000000000000000000000000000000000
 960: 0000000000000000000000000000000000000000000000000000000000000000
1024: 0000000000000000000000000000000000000000000000000000000000000000
1088: 0000000000000000000000000000000000000000000000000000000000000000
1152: 0000000000000000000000000000000000000000000000000000000000000000
1216: 0000000000000000000000000000000000000000000000000000000000000000
1280: 0000000000000000000000000000000000000000000000000000000000000000
1344: 0000000000000000000000000000000000000000000000000000000000000000
1408: 0000000000000000000000000000000000000000000000000000000000000000
1472: 0000000000000000000000000000000000000000000000000000000000000000
1536: 0000000000000000000000000000000000000000000000000000000000000000
1600: 0000000000000000000000000000000000000000000000000000000000000000
1664: 0000000000000000000000000000000000000000000000000000000000000000
1728: 0000000000000000000000000000000000000000000000000000000000000000
1792: 0000000000000000000000000000000000000000000000000000000000000000
1856: 0000000000000000000000000000000000000000000000000000000000000000
1920: 0000000000000000000000000000000000000000000000000000000000000000
1984: 0000000000000000000000000000000000000000000000000000000000000000
2048: 0000000000000000000000000000000000000000000000000000000000000000
2112: 0000000000000000000000000000000000000000000000000000000000000000
2176: 0000000000000000000000000000000000000000000000000000000000000000
2240: 0000000000000000000000000000000000000000000000000000000000000000
2304: 0000000000000000000000000000000000000000000000000000000000000000
2368: 0000000000000000000000000000000000000000000000000000000000000000
2432: 0000000000000000000000000000000000000000000000000000000000000000
2496: 0000000000000000000000000000000000000000000000000000000000000000
2560: 0000000000000000000000000000000000000000000000000000000000000000
2624: 0000000000000000000000000000000000000000000000000000000000000000
2688: 0000000000000000000000000000000000000000000000000000000000000000
2752: 0000000000000000000000000000000000000000000000000000000000000000
2816: 0000000000000000000000000000000000000000000000000000000000000000
2880: 0000000000000000000000000000000000000000000000000000000000000000
2944: 0000000000000000000000000000000000000000000000000000000000000000
3008: 0000000000000000000000000000000000000000000000000000000000000000
3072: 0000000000000000000000000000000000000000000000000000000000000000
3136: 0000000000000000000000000000000000000000000000000000000000000000
3200: 0000000000000000000000000000000000000000000000000000000000000000
3264: 0000000000000000000000000000000000000000000000000000000000000000
3328: 0000000000000000000000000000000000000000000000000000000000000000
3392: 0000000000000000000000000000000000000000000000000000000000000000
3456: 0000000000000000000000000000000000000000000000000000000000000000
3520: 0000000000000000000000000000000000000000000000000000000000000000
3584: 0000000000000000000000000000000000000000000000000000000000000000
3648: 0000000000000000000000000000000000000000000000000000000000000000
3712: 0000000000000000000000000000000000000000000000000000000000000000
3776: 0000000000000000000000000000000000000000000000000000000000000000
3840: 0000000000000000000000000000000000000000000000000000000000000000
3904: 0000000000000000000000000000000000000000000000000000000000000000
3968: 0000000000000000000000000000000000000000000000000000000000000000
4032: 0000000000000000000000000000000000000000000000000000000000000000

/sys/kernel/debug/prueth-eth<if number>/vlan_filter

4.2.3.7. Broadcast Storm Prevention¶

Broadcast storm is defined as an excessive amount of Layer 2 broadcast Ethernet frames at the ingress of a network interface causing resources being wasted in the device for processing these frames and thereby affecting the device performance.

PRUETH driver implements a control mechanism that user can enable to drop these frames if the number of frames received during a window (100 msec) reaches a threshold. PRUETH uses a credit value as the threshold which is user configurable on a per interface basis. There is a sysfs entry

/sys/devices/platform/pruss<ID>_eth/net/eth<if number>/nsp_credit

Where ID and if number are obtained from the table under the section PRU-ICSS Subsystem Driver. The driver writes the credit value in DRAM based on the sysfs entry from user. The PRU firmware uses the credit value as a counter. The driver refreshes the counter every 100 msec. i.e write the credit value to memory every 100 msec. PRU Firmware upon receiving a broadcast frame decrements this counter until it becomes zero. Firmware forwards the broadcast frames to Host or ARM until the counter is non zero. PRU Firmware drops the broadcast frames if the counter is zero. Since driver writes the credit value to DRAM every 100 msec, the number of broadcast frames sent to host or ARM during a 100 msec window is limited to the credit value and anything beyond that is dropped by the firmware. Firmware maintains a statistics counter, stormPrevCounter, for the number of frames dropped due to storm prevention control and can be seen using ethtool -S command.

Note that this feature is disabled by default and user needs to configure it explicitely to enable the feature. A write of non zero value to the sysfs file enables the feature and zero disables the feature. The value to be used may be experimented in a particular network situation and configured by the user. In a typical network, there will be few of these per 100 msec window such as ARP broadcast, DHCP broadcast, router advetisement etc. So user may observe the broadcast traffic in the network using network tools such as wireshark and set the credit value accordingly.

Example

To enable do

>echo 10 > /sys/devices/platform/pruss2_eth/net/eth2/nsp_credit

for setting credit value to 10 for eth2. Similarly for other interfaces.

To disable do

>echo 0 > /sys/devices/platform/pruss2_eth/net/eth2/nsp_credit

4.2.3.8. Receive Interrupt Pacing¶

Currently PRUETH driver receives one interrupt per frame from each port. This results in lesser performance due to interrupt handling overhead in the Linux kernel especially for small size frames such as that used in grid infrastructure networks.

A well-known technique used in rx packet processing of a network driver is to use rx interrupt pacing to accumulate the Ethernet frames at the hardware or underlying network device’s buffer so as to generate less number of interrupts per packet to the host. This reduces the overhead in Linux kernel associated with interrupt handling by distributing the same over muliple packets, thereby reducing CPU usage. In this case PRUETH firmware accumulates the frames in the ingress ring buffer between host and PRU for a configurable period of time and raise an interrupt on timeout expiry (timeout equals to the configured period) there by resulting in lesser number of interrupts per frame. PRUETH firmware implements a Rx Pacing timer and the same gets configured by the PRUETH driver for HSR/PRP Ethernet types when the network device is opened and stopped when the device is closed. This timer is implemented per interface since driver has separate interrupts per PRU or interface.

The user may show the timer value using the ethtool –c option which is a standard Linux command to show different coalesce parameters at the network device. –C option allows user to override the default value at the device. The driver uses a default timeout value of 123 usec (Duration of an MTU frame). User is not expected to change this as there are limited number of buffers in the receive path between Host and PRU and using a bigger value can result in packet loss. However ethtool –c option is implemented in the PRUETH driver to show current setting in the driver and -C to change it if needed. Driver supports rx-usecs and adaptive-rx parameters from the available list of parameters and can be changed using –C option. When adaptive-rx is turned ON, every time a frame is received at the PRU and the buffer is empty, the PRU firmware raises an interrupt. However if the buffer is already occupied with some frames and the timer is already running, it will wait until timer expiry to raise the next interrupt. This would be useful to improve latency when using interrupts pacing. Currently this is disabled by default as enabling the same result in a lower throughput. However user may enable the same through ethtool –C command if needed.

Example

To show the current value of pacing timer, use

>ethtool –c <Ethernet interface>

To change the current value, use

>ethtool –C <Ethernet interface> <param> <value> <param> <value>

For example for eth2, user type

>ethtool –c eth2

Note that this changes the timer at both PRUs. So the command may be issued to one of the pair of Ethernet interfaces used for the HSR or PRP interface. i.e for ICSS2 (assuming eth2 and eth3), user may use either of the below command to set the timer

>ethtool –C eth2 rx-usecs 123

or

>ethtool –C eth3 rx-usecs 123

Similarly to enable adaptive-rx, user types

>ethtool –C eth2 adaptive-rx on

or

>ethtool –C eth3 adaptive-rx on

Both parameters may be updated in one go as

>ethtool –C eth2 rx-usecs 123 adaptive-rx on

More details on the command syntax are available by typing man ethtool on a Linux PC as this is a standard Linux command.

Limitation

Currently this feature is not supported for Dual EMAC ethernet type (i.e when EMAC firmware is running on PRU ICSS).

4.2.3.9. VLAN Tag in Supervision frame¶

HSR/PRP protocol standard IEC62439-3 defines that the PRP_Supervision frame as well as HSR_Supervision frames optionally include a VLAN tag header. Starting with Processor SDK 5.1.x, VLAN tag is supprted in Linux HSR/PRP implementation.

In Linux, user use the ip link add command to create an HSR/PRP interface. ip link command is supported by the iproute2 package which is an open source software. hsr or prp type is currently a supported link type by ip link command. To support VLAN tag in SV frames, the command now accepts 3 more parameters from user. They are:-

• VLAN Identifier (VID) - sv_vid
• Priority code point (PCP) - sv_pcp
• Drop eligible indicator (DEI) or formerly known as Canonical Format Indicator (CFI) - sv_dei

User provides following values in the ip link add command for this purpose.

sv_vid <vid value> sv_pcp <pcp value> sv_dei <dei value>

These are all optional parameters. At least one of the value to be provided to use vlan. A default value of zero is used if not supplied by the user.

4.2.3.10. Node Table¶

HSR/PRP offloaded LRE supports a Node Table (NT) of 256 entries on AM5/K2G devices and 128 entries on AM3/4 devices. NT entries are maintained in shared RAM of the PRU ICSS. For HSR, an entry containing the MAC address of the remote node is inserted to the Node table when a supervison(SV) frame is received from that node. For PRP, this gets added for any frame received from the node. Firmware first looks up the address in the NT upon receiving a SV frame. The look up status is indicated in a bit in the buffer descriptor. The same is checked by the driver and if the look up status indicates no success, the MAC entry is inserted to the NT by driver. For subsequent frames received from the node, the firmware updates the NT and the insertion process is skipped in the driver.

The driver also runs an ageing timer to forget the node if there was no frame received from the remote node for a period of time as specified by the standard. During the timeout, if the entry is aged, then the same is deleted from the NT.

In Summary, search/update of an NT entry is done in firmware and insert/ delete of an entry is done by the driver.

The NT entries can be displayed by the user at the console using the proc file /proc/<if_name>/node-table.

Example: If the interface name is prp0, a sample command is shown below to display the NT entries.

root@am57xx-evm:~# cat /proc/prp0/node-table

Remote nodes in network: 1

Node[0]:
MAC ADDR: 70:ff:76:1c:16:ef
DANP
Time Last Seen: RxA=0 RxB=0

4.2.3.11. Net-SNMP¶

The TI SDK release ships tisdk-rootfs-image-am57xx-evm.tar.xz with Net-SNMP 5.7.3 binaries pre-installed and snmpd is started as part of the user space initialization. TI has developed a Net-SNMP module to support IEC-62439-3 related MIB access at the DAN using snmp commands such as snmpwalk, snmpget etc. NOTE: IEC-62439-3 MIB is supported only in the offloaded case. So user is expected to create HSR/PRP interface with offload. For details on how to setup HSR/PRP interface with offload, please refer Testing HSR/PRP Firmware Offload

Command examples

An example snmpwalk command execution is shown below where 192.168.2.20 is the IP address of the remote DAN-P or DAN-H prp/hsr interface.

root@am57xx-evm:/etc/snmp# snmpwalk -v 2c -c public 192.168.2.20 iec62439
IEC-62439-3-MIB::lreManufacturerName.0 = STRING: TI LRE
IEC-62439-3-MIB::lreInterfaceCount.0 = INTEGER: 1
IEC-62439-3-MIB::lreRowStatus.1 = INTEGER: active(1)
IEC-62439-3-MIB::lreNodeType.1 = INTEGER: hsr(2)
IEC-62439-3-MIB::lreNodeName.1 = STRING: hsr0
IEC-62439-3-MIB::lreVersionName.1 = ""
IEC-62439-3-MIB::lreMacAddress.1 = STRING: d2:ef:e6:2a:1f:5b
IEC-62439-3-MIB::lrePortAdminStateA.1 = INTEGER: active(2)
IEC-62439-3-MIB::lrePortAdminStateB.1 = INTEGER: active(2)
IEC-62439-3-MIB::lreLinkStatusA.1 = INTEGER: up(1)
IEC-62439-3-MIB::lreLinkStatusB.1 = INTEGER: up(1)
IEC-62439-3-MIB::lreDuplicateDiscard.1 = INTEGER: discard(2)
IEC-62439-3-MIB::lreTransparentReception.1 = INTEGER: 0
IEC-62439-3-MIB::lreHsrLREMode.1 = INTEGER: modeh(1)
IEC-62439-3-MIB::lreSwitchingEndNode.1 = INTEGER: hsrnode(5)
IEC-62439-3-MIB::lreRedBoxIdentity.1 = INTEGER: id1a(2)
IEC-62439-3-MIB::lreEvaluateSupervision.1 = INTEGER: true(1)
IEC-62439-3-MIB::lreNodesTableClear.1 = INTEGER: noOp(0)
IEC-62439-3-MIB::lreProxyNodeTableClear.1 = INTEGER: noOp(0)
IEC-62439-3-MIB::lreDupListResideMaxTime.1 = INTEGER: 26214 binaryFractionOfSecond
IEC-62439-3-MIB::lreCntTxA.1 = Counter32: 7384
IEC-62439-3-MIB::lreCntTxB.1 = Counter32: 7385
IEC-62439-3-MIB::lreCntTxC.1 = Counter32: 4032
IEC-62439-3-MIB::lreCntErrWrongLanA.1 = Counter32: 0
IEC-62439-3-MIB::lreCntErrWrongLanB.1 = Counter32: 0
IEC-62439-3-MIB::lreCntErrWrongLanC.1 = Counter32: 0
IEC-62439-3-MIB::lreCntRxA.1 = Counter32: 4024
IEC-62439-3-MIB::lreCntRxB.1 = Counter32: 4025
IEC-62439-3-MIB::lreCntRxC.1 = Counter32: 0
IEC-62439-3-MIB::lreCntErrorsA.1 = Counter32: 3351
IEC-62439-3-MIB::lreCntErrorsB.1 = Counter32: 3351
IEC-62439-3-MIB::lreCntErrorsC.1 = Counter32: 0
IEC-62439-3-MIB::lreCntNodes.1 = INTEGER: 1
IEC-62439-3-MIB::lreCntProxyNodes.1 = INTEGER: 0
IEC-62439-3-MIB::lreCntUniqueA.1 = Counter32: 4
IEC-62439-3-MIB::lreCntUniqueB.1 = Counter32: 4
IEC-62439-3-MIB::lreCntUniqueC.1 = Counter32: 0
IEC-62439-3-MIB::lreCntDuplicateA.1 = Counter32: 0
IEC-62439-3-MIB::lreCntDuplicateB.1 = Counter32: 0
IEC-62439-3-MIB::lreCntDuplicateC.1 = Counter32: 3992
IEC-62439-3-MIB::lreCntMultiA.1 = Counter32: 0
IEC-62439-3-MIB::lreCntMultiB.1 = Counter32: 0
IEC-62439-3-MIB::lreCntMultiC.1 = Counter32: 0
IEC-62439-3-MIB::lreCntOwnRxA.1 = Counter32: 0
IEC-62439-3-MIB::lreCntOwnRxB.1 = Counter32: 0
IEC-62439-3-MIB::lreNodesMacAddress.1.1 = STRING: 70:ff:76:1c:f:8d
IEC-62439-3-MIB::lreTimeLastSeenA.1.1 = Timeticks: (0) 0:00:00.00
IEC-62439-3-MIB::lreTimeLastSeenB.1.1 = Timeticks: (1) 0:00:00.01
IEC-62439-3-MIB::lreRemNodeType.1.1 = INTEGER: danh(3)

Individual MIB variable may be queried or set using snmpset or snmpget commands. Below are some examples:-

snmpset

snmpset -v 2c -c private 192.168.2.20 IEC-62439-3-MIB::lreHsrLREMode.1 i 3

snmpget

snmpget -v 2c -c public 192.168.2.20 IEC-62439-3-MIB::lreManufacturerName.0

MIB Support

Note NS: Not supported, NA: Not Applicable

4.2.3.12. iproute2¶

iproute2 is enhanced to allow creating a prp interface similar to hsr interface using two slave interfaces.

4.2.3.13. Test Procedure¶

 => setenv netargs 'setenv bootargs console=${console} ${optargs} root=/dev/nfs nfsroot=${serverip}:${rootpath},${nfsopts} rw ip=dhcp
ti_prueth.pruss1_ethtype=${pruss1_ethtype} ti_prueth.pruss2_ethtype=${pruss2_ethtype}'
 => setenv pruss1_ethtype 1
 => setenv pruss2_ethtype 2
 => saveenv

 => setenv args_mmc 'run finduuid;setenv bootargs console=${console} ${optargs} root=PARTUUID=${uuid} rw rootfstype=${mmcrootfstype}
ti_prueth.pruss1_ethtype=${pruss1_ethtype} ti_prueth.pruss2_ethtype=${pruss2_ethtype}'

dmesg | grep "TI PRU ethernet (type"

prueth pruss2_eth: TI PRU ethernet (type 0) driver initialized

prueth pruss2_eth: TI PRU ethernet (type 1) driver initialized

prueth pruss2_eth: TI PRU ethernet (type 2) driver initialized

root@am57xx-evm:~# cat /sys/kernel/debug/prueth-hsr-2/node_table

Remote nodes in network: 1

Node[1]:
MAC ADDR: 70:ff:76:1c:0f:99
state: valid
DANH
RxA=2992 SupRxA=148
RxB=2992 SupRxB=148
Time Last Seen: Sup=0 RxA=0 RxB=0

root@am57xx-evm:~# cat /sys/kernel/debug/prueth-prp-2/node_table

Remote nodes in network: 1

Node[1]:
MAC ADDR: 70:ff:76:1c:0f:99
state: valid
DupDiscard (0x80)
DANP
RxA=17 SupRxA=17
RxB=17 SupRxB=17
Time Last Seen: Sup=0 RxA=0 RxB=0
PRP LineID Err: A=0 B=0

root@am57xx-evm:~# ethtool -S eth2
NIC statistics:
     txBcast: 1
     txMcast: 11635
     txUcast: 3
     txOctets: 816642
     rxBcast: 0
     rxMcast: 36
     rxUcast: 17
     rxOctets: 4841
     tx64byte: 6
     tx65_127byte: 11633
     tx128_255byte: 0
     tx256_511byte: 0
     tx512_1023byte: 0
     tx1024byte: 0
     rx64byte: 4
     rx65_127byte: 49
     rx128_255byte: 0
     rx256_511byte: 0
     rx512_1023byte: 0
     rx1024byte: 0
     lateColl: 0
     singleColl: 0
     multiColl: 0
     excessColl: 0
     rxMisAlignmentFrames: 0
     stormPrevCounter: 0
     macRxError: 0
     SFDError: 0
     defTx: 0
     macTxError: 0
     rxOverSizedFrames: 0
     rxUnderSizedFrames: 0
     rxCRCFrames: 0
     droppedPackets: 0
     txHWQOverFlow: 0
     txHWQUnderFlow: 0
     lreTxA: 11636
     lreTxB: 11639
     lreTxC: 66
     lreErrWrongLanA: 0
     lreErrWrongLanB: 0
     lreErrWrongLanC: 0
     lreRxA: 40
     lreRxB: 40
     lreRxC: 0
     lreErrorsA: 11560
     lreErrorsB: 11559
     lreErrorsC: 0
     lreNodes: 0
     lreProxyNodes: 0
     lreUniqueRxA: 0
     lreUniqueRxB: 0
     lreUniqueRxC: 0
     lreDuplicateRxA: 0
     lreDuplicateRxB: 0
     lreDuplicateRxC: 0
     lreMultiRxA: 0
     lreMultiRxB: 0
     lreMultiRxC: 0
     lreOwnRxA: 0
     lreOwnRxB: 0
     lreDuplicateDiscard: 2
     lreTransRecept: 1
     lreNtLookupErrA: 0
     lreNtLookupErrB: 0
     lreNodeTableFull: 0

4.2.3.13.4. HSR Testing¶

To test HSR, user would need two AM572x EVMs.

1. Setup HSR Ring network as per diagram below. Connect the PRU2ETH0 (See the marking on the EVM) Ethernet ports of the two EVMs together (corresponds to Linux interface eth2) as shown below. Similarly, connect the PRU2ETH1 ports (Linux interface eth3) as well.
2. Configure the bootargs to boot PRU with HSR firmware as described above at Testing HSR/PRP Firmware Offload
3. Boot the EVMs using the pre-built images from the Processor SDK release. These images are built with Linux HSR/PRP driver enabled. Login to the console using root user name.
4. Note the MAC Address of eth2 at DAN-H-1(Say MAC-A) using the ifconfig command. Do ifconfig eth2 at the Linux console of DAN-H-1
5. Note the MAC Address of eth2 at DAN-H-2(Say MAC-B) using the ifconfig command. Do ifconfig eth2 at the Linux console of DAN-H-2
   1. ifconfig eth2 down
   2. ifconfig eth3 down
   3. ifconfig eth3 hw ether <MAC-A>
   4. ifconfig eth2 up
   5. ifconfig eth3 up
   6. ip link add name hsr0 type hsr slave1 eth2 slave2 eth3 supervision 45 version 1
   7. ifconfig hsr0 <IP Address of hsr interface at DAN-H-1> up
6. Repeat the above steps for DAN-H-2, but this time use MAC-B in step 4.3 above. And use IP Address from the same subnet as that of DAN-H-1 hsr interface hsr0.

For example use Ip Address 192.168.2.20 for DAN-H-1 and 192.168.2.30 for DAN-H-2. Assume CPSW ports at the EVM are on a different Subnet than the HSR interface.

Once both hsr0 interfaces are created, user should be able to do a ping from DAN-H-1 to DAN-H-2 or vice-versa. Disconnect Ethernet cable at eth2 or eth3. The Ping should continue to go through. User could run iperf between the two HSR interfaces and test the iperf is not affected when one of the cable is disconnected. This verifies redundancy.

A Sample script that automates the steps 4.1 to 4.7, say setup.sh, is below

#!/bin/sh
if [ $# -lt 3 ]
then
        echo "setup.sh <hsr/prp> <MAC-Address for both slaves> <ip address for hsr/prp interface>"
        exit
fi

if [ "$1" != "hsr" ] && [ "$1" != "prp" ]
then
    echo "use hsr or prp as first argument"
    exit
fi

echo "Setting up $10 interface with MAC address $2 for slaves and IP address $3"

ifconfig eth2 down
ifconfig eth3 down
ifconfig eth2 hw ether $2
ifconfig eth3 hw ether $2
ifconfig eth2 up
ifconfig eth3 up

if [ "$1" == "hsr" ]
then
        ip link add name hsr0 type hsr slave1 eth2 slave2 eth3 supervision 45 version 1
        ifconfig hsr0 $3 up
else
        ip link add name prp0 type prp slave1 eth2 slave2 eth3 supervision 45
        ifconfig prp0 $3 up
fi

Using this script, user can setup hsr interface as

/setup.sh hsr <MAC-A> <IP Address of the interface>

where it is assumed that the setup.sh is copied to the root directory of the target file system and is made executable

Below script can be used to teardown the hsr interface (say teardown.sh)

#!/bin/sh
if [ $# -lt 1 ]
then
        echo "teardown.sh <hsr/prp>"
        exit
fi

if [ "$1" != "hsr" ] && [ "$1" != "prp" ]
then
    echo "use hsr or prp as first argument"
    exit
fi

if [ "$1" == "hsr" ]
then
        ip link del hsr0
else
        ip link del prp0
fi

Using the above script, user teardown the hsr interface as

/teardown.sh hsr

Sample script below can be used to create two instances of HSR/PRP at PRU-ICSS1 and PRU-ICSS2 on AM571x. For other platform where applicable, this can be customized by changing the slave eth interface names to match with that on the specific platform. The script also allows configuring the LRE to use VLAN tag for Supervision frames by providing an optional vlan string in the command line of the script as documented in the script.

On PRU-ICSS1 of AM571x

#!/bin/sh
if [ $# -lt 4 ]
then
       echo "setup-pruss1.sh <hsr/prp> <MAC-Address of slave-A>"
       echo "   <ip address for hsr/prp interface>"
       echo "  <if_name of hsr/prp interface>  [vlan]"
       echo " vlan is optional and if present use vlan tag"
       echo " in the SV frame. Use values of vid, pcp and"
       echo " and dei hardcoded in the script"
       exit
fi

if [ "$1" != "hsr" ] && [ "$1" != "prp" ]
then
       echo "use hsr or prp as first argument"
       exit
fi

if_a=eth4
if_b=eth5
if_name=$4
vl=$5
#change this as needed
vid=10
pcp=5
dei=0

ifconfig $if_a down
ifconfig $if_b down
ifconfig $if_a hw ether $2
ifconfig $if_b hw ether $2
ifconfig $if_a up
ifconfig $if_b up

echo "Setting up $if_name with MAC address $2 for slaves and IP address $3"
echo "          using $if_a and $if_b"

if [ "$1" = "hsr" ]; then
       if [ "$vl" = "vlan" ]; then
       options="version 1 sv_vid $vid sv_pcp $pcp sv_dei $dei"
               echo $options
       else
               options="version 1"
       fi
else
       if [ "$vl" = "vlan" ]; then
               options="sv_vid $vid sv_pcp $pcp sv_dei $dei"
       else
               options=""
       fi
fi

ip link add name $if_name type $1 slave1 $if_a slave2 $if_b supervision 0 $options
ifconfig $if_name $3 up

On PRU-ICSS2 of AM571x

#!/bin/sh
if [ $# -lt 4 ]
then
       echo "setup-pruss2.sh <hsr/prp> <MAC-Address of slave-A>"
       echo "   <ip address for hsr/prp interface>"
       echo "  <if_name of hsr/prp interface>  [vlan]"
       echo " vlan is optional and if present use vlan tag"
       echo " in the SV frame. Use values of vid, pcp and"
       echo " and dei hardcoded in the script"
       exit
fi

if [ "$1" != "hsr" ] && [ "$1" != "prp" ]
then
       echo "use hsr or prp as first argument"
       exit
fi

if_a=eth2
if_b=eth3
if_name=$4
vl=$5
#change this as needed
vid=10
pcp=5
dei=0

ifconfig $if_a down
ifconfig $if_b down
ifconfig $if_a hw ether $2
ifconfig $if_b hw ether $2
ifconfig $if_a up
ifconfig $if_b up

echo "Setting up $if_name with MAC address $2 for slaves and IP address $3"
echo "          using $if_a and $if_b"

if [ "$1" = "hsr" ]; then
       if [ "$vl" = "vlan" ]; then
               options="version 1 sv_vid $vid sv_pcp $pcp sv_dei $dei"
               echo $options
       else
               options="version 1"
       fi
else
       if [ "$vl" = "vlan" ]; then
               options="sv_vid $vid sv_pcp $pcp sv_dei $dei"
       else
               options=""
       fi
fi

ip link add name $if_name type $1 slave1 $if_a slave2 $if_b supervision 0 $options
ifconfig $if_name $3 up

Sample logs are shown below:

DAN-H-1

root@am57xx-evm:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 70:FF:76:1C:0F:8D  [   38.944687] random: nonblocking pool is initialized
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@am57xx-evm:~# /setup.sh hsr 70:FF:76:1C:0F:8D 192.168.2.20
Setting up hsr0 interface with MAC address 70:FF:76:1C:0F:8D for slaves and IP address 192.168.2.20
[   62.319328]  remoteproc6: powering up 4b2b4000.pru0
[   62.325015]  remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 4350
[   62.335303] ti-pruss 4b280000.pruss: configured system_events = 0x0000060000500000 intr_channels = 0x00000095 host_intr = 0x00000115
[   62.348724]  remoteproc6: remote processor 4b2b4000.pru0 is now up
[   62.354947] net eth2: started
[   62.360842] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   62.408479]  remoteproc7: powering up 4b2b8000.pru1
[   62.413790]  remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 4382
[   62.422088] ti-pruss 4b280000.pruss: configured system_events = 0x0060000000a00000 intr_channels = 0x0000012a host_intr = 0x0000022a
[   62.434059]  remoteproc7: remote processor 4b2b8000.pru1 is now up
[   62.440281] net eth3: started
[   62.444625] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[   62.509276] device eth2 entered promiscuous mode
[   62.549502] device eth3 entered promiscuous mode
[   62.593878] hsr0: Slave A (eth2) is not up; please bring it up to get a fully working HSR network
[   62.604214] IPv6: ADDRCONF(NETDEV_UP): hsr0: link is not ready
[   62.614722] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am57xx-evm:~#
root@am57xx-evm:~# [   63.648569] eth2: Link is Up - 100Mbps/Full - flow control off
[   63.654446] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   63.748602] eth3: Link is Up - 100Mbps/Full - flow control off
[   63.754477] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

=========Setup hsr0 on DAN-H-2=============================================

root@am57xx-evm:~# ping 192.168.2.30
PING 192.168.2.30 (192.168.2.30): 56 data bytes
64 bytes from 192.168.2.30: seq=0 ttl=64 time=0.400 ms
64 bytes from 192.168.2.30: seq=1 ttl=64 time=0.190 ms
64 bytes from 192.168.2.30: seq=2 ttl=64 time=0.200 ms

--- 192.168.2.30 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.190/0.263/0.400 ms
root@am57xx-evm:~#
root@am57xx-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 33546
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   108 MBytes  90.7 Mbits/sec
root@am57xx-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 47913
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   106 MBytes  89.0 Mbits/sec   0.078 ms  587/76246 (0.77%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
root@am57xx-evm:~#
root@am57xx-evm:~# cat /proc/hsr0/node-table

Remote nodes in network: 1

Node[0]:
MAC ADDR: d6:ab:09:0d:f0:e5
DANH
Time Last Seen: RxA=0 RxB=0

root@am57xx-evm:~# cat /proc/hsr0/lre-stats
LRE statistics:
Rx Offloaded: 1
    lreTxA: 15915369
    lreTxB: 15915369
    lreTxC: 673
    lreErrWrongLanA: 207816
    lreErrWrongLanB: 207789
    lreErrWrongLanC: 0
    lreRxA: 0
    lreRxB: 27
    lreRxC: 15707590
    lreErrorsA: 0
    lreErrorsB: 0
    lreErrorsC: 0
    lreNodes: 1
    lreProxyNodes: 0
    lreUniqueRxA: 0
    lreUniqueRxB: 0
    lreUniqueRxC: 0
    lreDuplicateRxA: 0
    lreDuplicateRxB: 0
    lreDuplicateRxC: 0
    lreMultiRxA: 0
    lreMultiRxB: 0
    lreMultiRxC: 0
    lreOwnRxA: 0
    lreOwnRxB: 0

DAN-H-2

root@am57xx-evm:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr D6:AB:09:0D:F0:E5
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@am57xx-evm:~# /setup.sh hsr D6:AB:09:0D:F0:E5 192.168.2.30
Setting up hsr0 interface with MAC address D6:AB:09:0D:F0:E5 for slaves and IP address
192.168.2.30
[   66.015629]  remoteproc6: powering up 4b2b4000.pru0
[   66.021263]  remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 4350
[   66.031502] ti-pruss 4b280000.pruss: configured system_events = 0x0000060000500000
intr_channels = 0x00000095 host_intr = 0x00000115
[   66.044965]  remoteproc6: remote processor 4b2b4000.pru0 is now up
[   66.052043] net eth2: started
[   66.055419] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   66.108268]  remoteproc7: powering up 4b2b8000.pru1
[   66.113562]  remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 4382
[   66.123585] ti-pruss 4b280000.pruss: configured system_events = 0x0060000000a00000
intr_channels = 0x0000012a host_intr = 0x0000022a
[   66.136973]  remoteproc7: remote processor 4b2b8000.pru1 is now up
[   66.143998] net eth3: started
[   66.147284] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[   66.218855] device eth2 entered promiscuous mode
[   66.258015] device eth3 entered promiscuous mode
[   66.303911] hsr0: Slave A (eth2) is not up; please bring it up to get a fully working HSR
network
[   66.313621] IPv6: ADDRCONF(NETDEV_UP): hsr0: link is not ready
[   66.323162] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am57xx-evm:~# [   67.358460] eth2: Link is Up - 100Mbps/Full - flow control off
[   67.364336] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   67.518489] eth3: Link is Up - 100Mbps/Full - flow control off
[   67.524363] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

root@am57xx-evm:~# iperf -c 192.168.2.20
------------------------------------------------------------
Client connecting to 192.168.2.20, TCP port 5001
TCP window size: 70.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 33546 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   108 MBytes  90.9 Mbits/sec
root@am57xx-evm:~# iperf -c 192.168.2.20 -u -b 90M
------------------------------------------------------------
Client connecting to 192.168.2.20, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 47913 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.7 Mbits/sec
[  3] Sent 76247 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec   106 MBytes  89.0 Mbits/sec   0.078 ms  587/76246 (0.77%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

root@am57xx-evm:~# cat /proc/hsr0/node-table

Remote nodes in network: 1

Node[0]:
MAC ADDR: 70:ff:76:1c:0f:8d
DANH
Time Last Seen: RxA=0 RxB=0

root@am57xx-evm:/# cat /proc/prp0/lre-stats
LRE statistics:
Rx Offloaded: 1

    lreTxA: 208128
    lreTxB: 208128
    lreTxC: 31305063
    lreErrWrongLanA: 0
    lreErrWrongLanB: 0
    lreErrWrongLanC: 0
    lreRxA: 0
    lreRxB: 0
    lreRxC: 208
    lreErrorsA: 0
    lreErrorsB: 0
    lreErrorsC: 0
    lreNodes: 1
    lreProxyNodes: 0
    lreUniqueRxA: 0
    lreUniqueRxB: 0
    lreUniqueRxC: 0
    lreDuplicateRxA: 0
    lreDuplicateRxB: 0
    lreDuplicateRxC: 0
    lreMultiRxA: 0
    lreMultiRxB: 0
    lreMultiRxC: 0
    lreOwnRxA: 0
    lreOwnRxB: 0

4.2.3.13.5. PRP Testing¶

For Testing PRP, user is required to use the setup shown in the above diagram. A DAN-P is attached to two independent networks (A and B) each of which is shown as a separate Ethernet switch. Do the following steps to setup the prp interface. Note that in the below example Linux interface eth2 is Link-A (slave1 in ip link command) and eth3 is Link-B (slave2 in ip link command). Link-A is the Ethernet cable that connects DUT to Switch A and Link-B is the Ethernet cable that connects to Switch-B. This is very important for PRP as swapping them is an incorrect connection and can produce undesirable results.

1. Connect the DAN-P-1 EVM, PRU2ETH0 (marking on EVM) to a port at Ethernet switch A and PRU2ETH1 to a port at Ethernet switch B
2. Similarly, connect DAN-P-2 EVM, PRU2ETH0 to a port at Ethernet switch A and PRU2ETH1 to a port at Ethernet switch B
3. Configure the PRU to load PRP firmware by setting env variable as discussed at Testing HSR/PRP Firmware Offload
4. Boot the EVMs using the pre-built images from the Processor SDK release. These images are built with Linux HSR/PRP driver enabled. Login to the console using root user name.
5. This time user may use the script (setup.sh) to create the prp interface. Note the MAC-A and MAC-B as in the case of HSR (MAC address of eth2)
6. setup the prp interface at DAN-P-1 as follows
   1. /setup.sh prp <MAC-A> <IP Address of prp interface>
7. setup the prp interface at DAN-P-2 as follows:-
   1. /setup.sh prp <MAC-B> <IP Address of prp interface>

If PRP interface uses IP address 192.168.2.20 and eth2 MAC address is 70:ff:76:1c:0f:8d, following example command may be used at DAN-P-1

/setup.sh prp 70:ff:76:1c:0f:8d 192.168.2.20

If PRP interface uses IP address 192.168.2.30 and eth2 MAC address is 70:ff:76:1c:0f:8e following example command may be use at DAN-P-2

/setup.sh prp 70:ff:76:1c:0f:8e 192.168.2.30

To teardown the prp interface do

/teardown.sh prp

Sample logs are shown below:

DAN-P-1

root@am57xx-evm:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 70:FF:76:1C:0F:8D
          inet6 addr: fe80::72ff:76ff:fe1c:f8d%3068183320/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:80429 errors:0 dropped:0 overruns:0 frame:593
          TX packets:40905 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:121704175 (116.0 MiB)  TX bytes:2716445 (2.5 MiB)

root@am57xx-evm:~# /setup.sh prp 70:FF:76:1C:0F:8D 192.168.2.20
Setting up prp0 interface with MAC address 70:FF:76:1C:0F:8D for slaves and IP address
192.168.2.20
[21649.978346] ti-pruss 4b280000.pruss: unconfigured system_events = 0x0000060000500000
host_intr = 0x00000115
[21649.988138]  remoteproc6: stopped remote processor 4b2b4000.pru0
[21649.994196] net eth2: stopped
[21650.048408] ti-pruss 4b280000.pruss: unconfigured system_events = 0x0060000000a00000
host_intr = 0x0000022a
[21650.058199]  remoteproc7: stopped remote processor 4b2b8000.pru1
[21650.064258] net eth3: stopped
[21650.084733]  remoteproc6: powering up 4b2b4000.pru0
[21650.090419]  remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 4350
[21650.098711] ti-pruss 4b280000.pruss: configured system_events = 0x0000060000500000
intr_channels = 0x00000095 host_intr = 0x00000115
[21650.110746]  remoteproc6: remote processor 4b2b4000.pru0 is now up
[21650.117709] net eth2: started
[21650.121113] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[21650.131268]  remoteproc7: powering up 4b2b8000.pru1
[21650.136538]  remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 4382
[21650.144871] ti-pruss 4b280000.pruss: configured system_events = 0x0060000000a00000
intr_channels = 0x0000012a host_intr = 0x0000022a
[21650.156838]  remoteproc7: remote processor 4b2b8000.pru1 is now up
[21650.163795] net eth3: started
[21650.167169] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[21650.188520] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[21650.248510] device eth2 entered promiscuous mode
[21650.287995] device eth3 entered promiscuous mode
[21650.328127] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

root@am57xx-evm:~# ping 192.168.2.30
PING 192.168.2.30 (192.168.2.30): 56 data bytes
64 bytes from 192.168.2.30: seq=0 ttl=64 time=0.401 ms
64 bytes from 192.168.2.30: seq=1 ttl=64 time=0.189 ms
64 bytes from 192.168.2.30: seq=2 ttl=64 time=0.186 ms
^C
--- 192.168.2.30 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.186/0.258/0.401 ms
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 38900
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   109 MBytes  91.5 Mbits/sec
^Croot@am57xx-evm:~# iperf -s  -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 43497
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   106 MBytes  88.8 Mbits/sec   0.070 ms  0/76109 (0.0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
^Croot@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:/# cat /proc/prp0/node-table

Remote nodes in network: 1

Node[0]:
MAC ADDR: 2a:da:8c:50:1b:86
DANP
Time Last Seen: RxA=0 RxB=0

root@am57xx-evm:/# cat /proc/prp0/lre-stats
LRE statistics:
Rx Offloaded: 1

    lreTxA: 208128
    lreTxB: 208128
    lreTxC: 31305063
    lreErrWrongLanA: 0
    lreErrWrongLanB: 0
    lreErrWrongLanC: 0
    lreRxA: 0
    lreRxB: 0
    lreRxC: 208
    lreErrorsA: 0
    lreErrorsB: 0
    lreErrorsC: 0
    lreNodes: 1
    lreProxyNodes: 0
    lreUniqueRxA: 0
    lreUniqueRxB: 0
    lreUniqueRxC: 0
    lreDuplicateRxA: 0
    lreDuplicateRxB: 0
    lreDuplicateRxC: 0
    lreMultiRxA: 0
    lreMultiRxB: 0
    lreMultiRxC: 0
    lreOwnRxA: 0
    lreOwnRxB: 0

DAN-P-2

root@am57xx-evm:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 2A:DA:8C:50:1B:86
         inet6 addr: fe80::28da:8cff:fe50:1b86%3068203800/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:40881 errors:0 dropped:10 overruns:0 frame:64
         TX packets:81022 errors:0 dropped:3528075 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:2718950 (2.5 MiB)  TX bytes:122601929 (116.9 MiB)

root@am57xx-evm:~# /setup.sh prp 2A:DA:8C:50:1B:86 192.168.2.30
Setting up prp0 interface with MAC address 2A:DA:8C:50:1B:86 for slaves and IP address
192.168.2.30
[21684.048303] ti-pruss 4b280000.pruss: unconfigured system_events = 0x0000060000500000
host_intr = 0x00000115
[21684.058095]  remoteproc6: stopped remote processor 4b2b4000.pru0
[21684.064152] net eth2: stopped
[21684.118368] ti-pruss 4b280000.pruss: unconfigured system_events = 0x0060000000a00000
host_intr = 0x0000022a
[21684.128160]  remoteproc7: stopped remote processor 4b2b8000.pru1
[21684.134217] net eth3: stopped
[21684.158347]  remoteproc6: powering up 4b2b4000.pru0
[21684.164069]  remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 4350
[21684.172524] ti-pruss 4b280000.pruss: configured system_events = 0x0000060000500000
intr_channels = 0x00000095 host_intr = 0x00000115
[21684.184563]  remoteproc6: remote processor 4b2b4000.pru0 is now up
[21684.191761] net eth2: started
[21684.195184] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[21684.205638]  remoteproc7: powering up 4b2b8000.pru1
[21684.210960]  remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 4382
[21684.219252] ti-pruss 4b280000.pruss: configured system_events = 0x0060000000a00000
intr_channels = 0x0000012a host_intr = 0x0000022a
[21684.231220]  remoteproc7: remote processor 4b2b8000.pru1 is now up
[21684.238235] net eth3: started
[21684.241684] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[21684.339123] device eth2 entered promiscuous mode
[21684.377962] device eth3 entered promiscuous mode
[21684.425802] prp0: Slave A (eth2) is not up; please bring it up to get a fully working HSR
network
[21684.436772] IPv6: ADDRCONF(NETDEV_UP): prp0: link is not ready
[21684.443615] IPv6: ADDRCONF(NETDEV_CHANGE): prp0: link becomes ready
root@am57xx-evm:~# [21684.488601] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[21684.518615] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready

root@am57xx-evm:~# iperf -c 192.168.2.20
------------------------------------------------------------
Client connecting to 192.168.2.20, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 38900 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   109 MBytes  91.5 Mbits/sec
root@am57xx-evm:~# iperf -c 192.168.2.20 -u -b 88M
------------------------------------------------------------
Client connecting to 192.168.2.20, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 43497 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.5 Mbits/sec
[  3] Sent 76110 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec   106 MBytes  88.8 Mbits/sec   0.069 ms  0/76109 (0.0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
Remote nodes in network: 1

Node[0]:
MAC ADDR: 70:ff:76:1c:0f:8d
DANP
Time Last Seen: RxA=0 RxB=0

root@am57xx-evm:/# cat /proc/prp0/lre-stats
LRE statistics:
Rx Offloaded: 1

    lreTxA: 208128
    lreTxB: 208128
    lreTxC: 31305063
    lreErrWrongLanA: 0
    lreErrWrongLanB: 0
    lreErrWrongLanC: 0
    lreRxA: 0
    lreRxB: 0
    lreRxC: 208
    lreErrorsA: 0
    lreErrorsB: 0
    lreErrorsC: 0
    lreNodes: 1
    lreProxyNodes: 0
    lreUniqueRxA: 0
    lreUniqueRxB: 0
    lreUniqueRxC: 0
    lreDuplicateRxA: 0
    lreDuplicateRxB: 0
    lreDuplicateRxC: 0
    lreMultiRxA: 0
    lreMultiRxB: 0
    lreMultiRxC: 0
    lreOwnRxA: 0
    lreOwnRxB: 0

4.2.3.14. Performance Test and Logs¶

Boot up two AM571x/572x EVM to configure PRU Ethernet in HSR or PRP mode as described at Testing HSR/PRP Firmware Offload They are connected over eth2 and eth3 as described in HSR Testing or PRP Testing

UDP

Sample iperf UDP test showing no packet loss for MTU sized packets. For HSR/PRP, there are 6 bytes of Tag effectively reducing pay load size to 1466 (1500 - 6 - 20 - 8)

Server Side

root@am57xx-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 34255
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-60.0 sec   630 MBytes  88.1 Mbits/sec   0.002 ms    0/450887 (0%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Client side

root@am57xx-evm:~# iperf -c 192.168.2.20 -u -b88M -l1466 -t60
------------------------------------------------------------
Client connecting to 192.168.2.20, UDP port 5001
Sending 1466 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 34255 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   630 MBytes  88.1 Mbits/sec
[  3] Sent 450888 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec   630 MBytes  88.1 Mbits/sec   0.002 ms    0/450887 (0%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

NOTE: When doing UDP test, user may observe out of order packets sometimes. This is normal as the driver doesn’t use NAPI and packets may get queued to backlog queues of each CPU which may get delivered out of sequence to the iperf application. Out of order delivery of UDP packets are normal and kernel doesn’t guarantee in sequence delivery of UDP packets to application. So iperf may show some packets lost during the test. User may use some of the tools available to migrate the rx irq threads and iperf server to same CPU, example CPU 0, to mitigate out of order issue and get a no loss iperf stats for udp tests.

For example, here is a snapshot of the test that shows packet lost during the test. The test also shows 106 datagram received out of order.

root@am57xx-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 5001 connected with 192.168.2.20 port 41976
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-600.0 sec  6.15 GBytes  88.0 Mbits/sec   0.004 ms    5/4501411 (0.00011%)
[  3]  0.0-600.0 sec  106 datagrams received out-of-order

Run the iperf -s -u & Do ps command and find the PID of the iperf and migrate the process to CPU 0 as follows

root@am57xx-evm:~# ps
 PID TTY          TIME CMD
 964 ttyS2    00:00:00 login
1063 ttyS2    00:00:00 sh
1135 ttyS2    00:00:00 iperf
1141 ttyS2    00:00:00 ps
root@am57xx-evm:~# taskset -cp 0 1135
pid 1135's current affinity list: 0,1
pid 1135's new affinity list: 0

Now find the irq thread for eth2, eth3 devices using

root@am57xx-evm:~# ps -ef  | grep "eth3"
root      1464     2  0 06:01 ?        00:00:00 [irq/498-eth3]
root      1495  1065  0 06:04 ttyS2    00:00:00 grep irq/498-eth3
root@am57xx-evm:~# ps -ef  | grep "eth2"
root      1462     2  0 06:01 ?        00:00:00 [irq/496-eth2]
root      1497  1065  0 06:04 ttyS2    00:00:00 grep irq/496-eth2

In the above example 1462 is the process id of irq/496-eth2 thread and 1464 is the process ID of irq/498-eth3 . Do following to migrate them to CPU 0.

root@am57xx-evm:~# taskset -cp 0 1464
pid 1464's current affinity list: 0,1
pid 1464's new affinity list: 0
root@am57xx-evm:~# taskset -cp 0 1462
pid 1462's current affinity list: 0,1
pid 1462's new affinity list: 0
root@am57xx-evm:~#
root@am57xx-evm:~#

Now start iperf client on the other side. The server side log below with these tuning shows no packet loss.

[  4] local 192.168.2.30 port 5001 connected with 192.168.2.20 port 36415
[  4]  0.0-600.0 sec  6.15 GBytes  88.0 Mbits/sec   0.007 ms    0/4501355 (0%)
[  4]  0.0-600.0 sec  1 datagrams received out-of-order

Table: UDP Performance for HSR

Table: UDP Performance for PRP

NOTE-1:Test ran for 1 minute with single direction traffic (Client to Server). Both ports were individually disconnected and then reconnected during the test. Reduced the traffic from 88Mbits/sec until there is no out of order or lost packets in the iperf stats i.e. iperf command client side

iperf3 -c <IP> -u -l1472 -b<Rate>M -t60 -i5

TCP

HSR

Server side

root@am57xx-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 54148
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec   663 MBytes  92.7 Mbits/sec

Client side

root@am57xx-evm:~# iperf -c 192.168.2.20  -t60
------------------------------------------------------------
Client connecting to 192.168.2.20, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 54148 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   663 MBytes  92.7 Mbits/sec

PRP

Server side

root@am57xx-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.20 port 5001 connected with 192.168.2.30 port 54150
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec   660 MBytes  92.3 Mbits/sec

Client side

root@am57xx-evm:~# iperf -c 192.168.2.20 -t60
------------------------------------------------------------
Client connecting to 192.168.2.20, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 54150 connected with 192.168.2.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   660 MBytes  92.3 Mbits/sec

4.2.3.15. Redundancy Test and Logs¶

For HSR and PRP redundancy is implemented using duplicate links and sending frame over both links and discarding the duplicate at the receive side. The test requires two DAN-P or DAN-H nodes connected over eth2 and eth3 Ethernet Links. Boot up the EVM with bootargs set to configure PRU Ethernet in HSR or PRP mode as described at Processor_SDK_Linux_HSR_PRP#Testing_HSR.2FPRP_Firmware_Offload Do an iperf UDP test at 88Mbits/sec for MTU sized packets (size = 1466) and then pull out either eth2 or eth3 Ethernet cable. Make sure iperf stats doesn’t show packet loss during the test period. Here is the log for HSR that show link failed during the test and there was no packet loss.

eth3 link failed

root@am57xx-evm:~# iperf -c 192.168.2.20 -u -b88M -l1466 -t60
------------------------------------------------------------
Client connecting to 192.168.2.20, UDP port 5001
Sending 1466 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 50443 connected with 192.168.2.20 port 5001
[  108.740402] prueth pruss2_eth eth3: Link is Down
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   631 MBytes  88.2 Mbits/sec
[  3] Sent 451002 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec   631 MBytes  88.2 Mbits/sec   0.003 ms    0/451001 (0%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order
root@am57xx-evm:~# [  181.444988] prueth pruss2_eth eth3: Link is Up - 100Mbps/Full - flow control off

eth2 link failed

root@am57xx-evm:~# iperf -c 192.168.2.20 -u -b88M -l1466 -t60
------------------------------------------------------------
Client connecting to 192.168.2.20, UDP port 5001
Sending 1466 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.30 port 49260 connected with 192.168.2.20 port 5001
[  204.253286] prueth pruss2_eth eth2: Link is Down
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   631 MBytes  88.2 Mbits/sec
[  3] Sent 451028 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec   631 MBytes  88.2 Mbits/sec   0.005 ms    0/451027 (0%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order
root@am57xx-evm:~# uname -a
Linux am57xx-evm 4.9.41-rt23-gc038d21a22 #2 SMP PREEMPT RT Wed Sep 27 06:34:09 EDT 2017 armv7l GNU/Linux

4.2.3.16. Useful Commands¶

Linux HSR/PRP driver help commands

To dump lre_info such as sv_vid, sv_pcp, sv_dei, protocol etc used at the LRE of Linux HSR/PRP driver

• cat /sys/kernel/debug/<hsr/prp-if_name>/lre_info example cat /sys/kernel/debug/hsr0/lre_info

To dump Node_Table

• cat /proc/<hsr/prp-if_name>/node-table

To set hsr-mode - echo 1 > /proc/<hsr/prp-if_name>/hsr-mode

where 1 is for Mode-H Other values are 2 (Mode-N), 3 (Mode-T), 4 (Mode-U), 5 (Mode-M)

To clear node table - echo 1 > /proc/<hsr/prp-if_name>/clear-nt

To change duplicate detection mode - echo 1 > /proc/<hsr/prp-if_name>/dd-mode

To change duplicate list max resident time - echo 1000 > /proc/<hsr/prp-if_name>/dlrmt

To change prp transparent reception mode - echo 1 > /proc/<hsr/prp-if_name>/prp-tr

To disable SV frame transmission - echo 1 > /proc/<hsr/prp-if_name>/disable-sv_frame

PRU Ethernet driver help commands

To see queue stats

• cat /sys/kernel/debug/prueth-eth<if_number>/stats

To see lre_config

• cat /sys/kernel/debug/prueth-<hsr/prp>-<ID>>/lre_config

To dump node table

• cat /sys/kernel/debug/prueth-<hsr/prp>-<ID>>/node_table

Also

• cat /sys/kernel/debug/prueth-<hsr/prp-<ID>/new_nt_bins

In the above commands ID and if_nubers are chosen as described in section PRU-ICSS Subsystem Driver

4.2.3.17. FAQ¶

• How to disable udhcpc from sending DHCP request over prueth Ethernet interfaces (eth2-eth3) when it is configured to run HSR/PRP protocol in firmware?

Linux HSR/PRP driver uses the eth2/eth3 interface as slave interfaces to implement the protocol on top of it. Application is expected to send or receive packets over the hsr or prp lre interface also known as C port. However prueth also support plain Ethernet and the file system /etc/systemd/network/10-eth.network has a general rule to enable DHCP over all of the interface matching eth*. Modify the file to match unique name instead of generic name wild cards. For example to do DHCP over only eth0 interface, 10-eth.network may be modified as follows:-

root@am57xx-evm:~# cat /etc/systemd/network/10-eth.network

[Match]
Name=eth0
KernelCommandLine=!root=/dev/nfs

[Network]
DHCP=yes

To do DHCP over hsr0 or prp0 interfaces, user may add additional files to /etc/systemd/network/ one for each prp0 or hsr0 similar to eth0, by using a Match name string of hsr0 or prp0. More details on this is at [[6]]

• Why do I see duplicate frames displayed by ping when I use PRP interface?

In ICSS firmware, PRU-0 is assumed to be for Link-A and PRU-1 for Link-B of a LRE in the case DANP. So in PRP case, Link-A should be connected network A and Link-B should be connected to Network B of a PRP network since net-id has to match as per duplicate detect alogrithm. If this convention is not followed, duplicate detection logic fails and duplicates are sent to upper layer causing ping application to complain about duplicates. So when creating prp interface, make sure ip link command parameter slave 1 use linux interface associated with PRU-0 and slave 2 use interface associated with PRU-1. Also when making network connection make sure that PRU-0 port is connected to Network A and PRU-1 port to Network B.