3.2.2.8.3.1. CPSW3g¶
3.2.2.8.3.1.1. Introduction¶
The TI AM64x family of devices have multi port Gigabit Ethernet Switch subsystem. SoCs may have more than 2 external port with selectable RGMII and RMII interfaces and an internal Communications Port Programming Interface (CPPI) port (Host port 0). Host Port 0 CPPI Packet Streaming Interface interface supports 8 TX channels and one RX channel DMA channel.
Multi port Gigabit Ethernet Switch subsystem support both independent MAC mode of operation supporting multiple independent Ethernet interfaces with complete isolation (no packet switching between external ports), or can be configured in Switch mode with IEEE 802.1Q HW bridge in managed or learning mode between each of the individual MAC ports.
The driver follows the standard Linux network interface architecture and supports the following features:
Mulitple indpendent MAC mode
Switch mode as per IEEE 802.1Q
Cut Through forwarding in Switch mode
10/100/1000 Mbps mode of operation.
Auto negotiation.
Linux NAPI support
VLAN filtering
Ethertool
CPTS/PTP as per 802.1AS-2011 (TSN)
EST/TAS offload as per 802.1Q-2018 (TSN)
IET/preemption offload as per 802.1Q-2018 (TSN)
Forwarding and Queuing Enhancements for Time-Sensitive Streams (FQTSS) as per 802.1Q-2018 previously referred to as CBS or 802.1Qav
Note
Default mode of operation is multiple indpendent MAC ports.
3.2.2.8.3.1.1.1. Supported platforms¶
SoC |
Number of external ports |
|---|---|
AM62AX |
2 external ports (CPSW3g) |
AM62X |
2 external ports (CPSW3g) |
AM64X |
2 external ports (CPSW3g) |
3.2.2.8.3.1.2. Driver Configuration¶
Default SDK build will have these configurations enabled. In case of custom builds, please ensure following configs are enabled.
CONFIG_TI_DAVINCI_MDIO
CONFIG_TI_K3_AM65_CPSW_NUSS
CONFIG_TI_K3_AM65_CPSW_SWITCHDEV
CONFIG_TI_K3_AM65_CPTS
CONFIG_TI_AM65_CPSW_TAS
CONFIG_PHY_TI_GMII_SEL
For further details regarding the above configs, refer:
drivers/net/ethernet/ti/Kconfigdrivers/phy/ti/Kconfig
Module Build
Module build for the cpsw driver is supported. To do this, use option ‘m’ for above configs, where applicable.
3.2.2.8.3.1.3. Device tree bindings¶
The DT bindings description can be found at:
3.2.2.8.3.1.4. Multi MAC mode¶
Note
This section documents Independent MAC mode of operation with CPSW3g.
Bringing Up interface
The network interface can be configured automatically depending on root file system or configured manually. Manual configuration:
ip addr add 192.168.1.1/24 dev eth0
ip link set dev eth0 up
< or >
ifconfig eth0 <ip> netmask <mask> up
Get driver information
The MCU_CPSW0 interface can be identified by using ethtool -i|--driver command.
It also provides some information about supported features.
# ethtool -i <dev>
driver: am65-cpsw-nuss
version: 0.1
firmware-version:
expansion-rom-version:
bus-info: 46000000.ethernet
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes
ethtool - Display standard information about device/link
# ethtool eth0
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)
Link detected: yes
RX checksum offload
The Driver enables RX checksum offload by default. It can be disabled/enabled by using ethtool -K command:
# ethtool -k <dev>
....
rx-checksumming: on
ethtool -K <dev> rx-checksum on|off
VLAN Config
VLAN can be added/deleted using ip or vconfig utility.
VLAN Add
ip link add link eth0 name eth0.5 type vlan id 5
< or >
vconfig add eth0 5
VLAN del
ip link del eth0.5
< or >
vconfig rem eth0 5
VLAN IP assigning
IP address can be assigned to the VLAN interface either via udhcpc
when a VLAN aware dhcp server is present or via static ip assigning
using ip or ifconfig.
Once VLAN is added, it will create a new entry in Ethernet interfaces like eth0.5, below is an example how it check the vlan interface
ip addr add 192.168.1.1/24 dev eth0.5
< or >
ifconfig eth0.5
....
eth0.5 Link encap:Ethernet HWaddr 20:CD:39:2B:C7:BE
inet addr:192.168.10.5 Bcast:192.168.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
VLAN Packet Send/Receive
To Send or receive packets with the VLAN tag, bind the socket to the proper Ethernet interface shown above and can send/receive via that socket-fd.
Multicast Add/Delete
Multicast MAC address can be added/deleted using ip maddr commands or Linux socket ioctl SIOCADDMULTI/SIOCDELMULTI.
Show muliticast address
# ip maddr show dev <dev>
2: eth0
link 01:00:5e:00:00:01
link 01:80:c2:00:00:00
link 01:80:c2:00:00:03
link 01:80:c2:00:00:0e
link 01:00:5e:00:00:fc
inet 224.0.0.252
inet 224.0.0.1
Add muliticast address
# ip maddr add 01:00:5e:00:00:05 dev eth0
# ip maddr show dev eth0
2: eth0
link 01:00:5e:00:00:01
link 01:80:c2:00:00:00
link 01:80:c2:00:00:03
link 01:80:c2:00:00:0e
link 01:00:5e:00:00:fc
link 01:00:5e:00:00:05 static
inet 224.0.0.252
inet 224.0.0.1
Delete muliticast address
# ip maddr del 01:00:5e:00:00:05 dev eth0
ethtool -P|--show-permaddr DEVNAME Show permanent hardware
address
# ethtool -P eth0
Permanent address: a0:f6:fd:a6:46:6e"
ethtool -s|--change DEVNAME Change generic options
Below commands will be redirected to the phy driver:
# ethtool -s <dev>
[ speed %d ]
[ duplex half|full ]
[ autoneg on|off ]
[ wol p|u|m|b|a|g|s|d... ]
[ sopass %x:%x:%x:%x:%x:%x ]
Note
CPSW driver do not perform any kind of WOL specific actions or configurations.
#ethtool -s eth0 duplex half speed 100
[ 3550.892112] cpsw 48484000.ethernet eth0: Link is Down
[ 3556.088704] cpsw 48484000.ethernet eth0: Link is Up - 100Mbps/Half - flow control off
Sets the driver message type flags by name or number
[ msglvl %d | msglvl type on|off ... ]
# ethtool -s eth0 msglvl drv off
# ethtool -s eth0 msglvl ifdown off
# ethtool -s eth0 msglvl ifup off
# ethtool eth0
Current message level: 0x00000031 (49)
drv ifdown ifup
ethtool -r|--negotiate DEVNAME Restart N-WAY negotiation
# ethtool -r eth0
[ 4338.167685] cpsw 48484000.ethernet eth0: Link is Down
[ 4341.288695] cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx"
ethtool -a|--show-pause DEVNAME Show pause options
# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate: off
RX: off
TX: off
ethtool -A|--pause DEVNAME Set pause options
# ethtool -A eth0 rx on tx on
cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate: off
RX: on
TX: on
ethtool -g|--show-ring DEVNAME Query RX/TX ring parameters
# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 0
RX Mini: 0
RX Jumbo: 0
TX: 0
Current hardware settings:
RX: 500
RX Mini: 0
RX Jumbo: 0
TX: 512
ethtool-l|--show-channels DEVNAME Query Channels
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 1
TX: 8
Other: 0
Combined: 0
Current hardware settings:
RX: 1
TX: 8
Other: 0
Combined: 0
ethtool -L\|--set-channels DEVNAME Set Channels.
Allows to control number of TX channels driver is allowed to work with at DMA level. The maximum number of TX channels is 8.
Supported options [ tx N ]:
# ethtool -L eth0 tx 3
ethtool --show-priv-flags/--set-priv-flags DEVNAME Show/Set private flags
Allows to control private flags supported by driver.
Flag |
|
|---|---|
p0-rx-ptype-rrobin |
Controls TX DMA channels processing mode: round-robin or priority mode. In case priority mode is enabled, the high number channel will have higher priority: TX 7 - prio 7 … TX 0 - prio 0. |
iet-frame-preemption |
Enables support for Interspersed Express Traffic (IET) IEEE 802.3br (frame preemption). |
iet-mac-verify |
Enables Interspersed Express Traffic (IET) MAC verification procedure on link up event. |
# ethtool --show-priv-flags eth0
Private flags for eth0:
p0-rx-ptype-rrobin : on
iet-frame-preemption: off
iet-mac-verify : off
# ethtool --set-priv-flags eth0 p0-rx-ptype-rrobin off
Note
The network interface have to be down for private flags configuration.
ethtool -S|--statistics DEVNAME Show adapter statistics
“ethtool -S” command displays statistic for both external Port 1 and Host port 0. Host port 0 statistics prefixed with p0_.
# ethtool -S eth0
NIC statistics:
p0_rx_good_frames: 347
p0_rx_broadcast_frames: 4
p0_rx_multicast_frames: 264
p0_rx_crc_errors: 0
p0_rx_oversized_frames: 0
p0_rx_undersized_frames: 0
p0_ale_drop: 0
p0_ale_overrun_drop: 0
p0_rx_octets: 25756
p0_tx_good_frames: 4816
p0_tx_broadcast_frames: 3629
p0_tx_multicast_frames: 1120
p0_tx_octets: 878055
p0_tx_64B_frames: 735
p0_tx_65_to_127B_frames: 1023
...
rx_good_frames: 4816
rx_broadcast_frames: 3629
rx_multicast_frames: 1120
rx_pause_frames: 0
rx_crc_errors: 0
rx_align_code_errors: 0
rx_oversized_frames: 0
rx_jabber_frames: 0
rx_undersized_frames: 0
rx_fragments: 0
ale_drop: 0
ale_overrun_drop: 0
rx_octets: 878055
tx_good_frames: 347
tx_broadcast_frames: 4
tx_multicast_frames: 264
tx_pause_frames: 0
tx_deferred_frames: 0
tx_collision_frames: 0
tx_single_coll_frames: 0
tx_mult_coll_frames: 0
tx_excessive_collisions: 0
tx_late_collisions: 0
...
ethtool -T|--show-time-stamping DEVNAME Show time stamping
capabilities.
Accessible when CPTS is enabled.
# ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 1
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
none (HWTSTAMP_FILTER_NONE)
all (HWTSTAMP_FILTER_ALL)
ethtool --show-eee DEVNAME Show EEE settings
#ethtool --show-eee eth0
EEE Settings for eth0:
EEE status: not supported
ethtool --set-eee DEVNAME Set EEE settings.
Note
Full EEE is not supported in driver, but it enables reading and writing of EEE advertising settings in Ethernet PHY. This way one can disable advertising EEE for certain speeds.
ethtool -d|--register-dump DEVNAME Do a register dump
This command dumps all CPSW MMIO regions in the below format. The TI switch-config tool can be used for CPSW NUSS register dump parsing.
MMIO region header (8 Bytes) |
|
|---|---|
module_id (u32) |
MMIO region id NUSS = 1, RGMII_STATUS = 2, MDIO = 3, CPSW = 4, CPSW_P0 = 5, CPSW_P1 = 6, CPSW_CPTS = 7, CPSW_ALE = 8, CPSW_ALE_TBL = 9 |
len (u32) |
MMIO region dump length, including header |
MMIO region registers dump (num_regs * 8 Bytes) |
|
reg_offset (u32) |
register offset from the start of MCU NAVSS MMIO space |
reg_value (u32) |
MMIO region dump length, including header |
Exception: ALE table dumped as raw array of ALE records (3 * u32 per record).
# ethtool -d eth0
Offset Values
------ ------
0x0000: 01 00 00 00 48 00 00 00 00 00 00 00 00 71 a0 6b
0x0010: 04 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00
0x0020: 0c 00 00 00 00 00 00 00 10 00 00 00 01 00 00 00
0x0030: 14 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00
0x0040: 1c 00 00 00 00 00 00 00 02 00 00 00 48 00 00 00
0x0050: 30 00 00 00 0b 00 00 00 34 00 00 00 00 00 00 00
0x0060: 38 00 00 00 00 00 00 00 3c 00 00 00 00 00 00 00
...
3.2.2.8.3.1.4.1. Interrupt pacing¶
The Interrupt pacing (IRQ coalescing) based on hrtimers for RX/TX data path separately can be enabled by following ethtool commands (min value is 20us).
The RX data path - only one queue:
# ethtool -C ethX rx-usecs N
The TX data path - any of enabed TX queue (up to 8):
- by default enables coalesing for TX0
# ethtool -C ethX tx-usecs N
- configure TX0
# ethtool -Q ethX queue_mask 1 --coalesce tx-usecs N
- configure TX1
# ethtool -Q ethX queue_mask 2 --coalesce tx-usecs N
- configure TX0 and TX1
# ethtool -Q ethX queue_mask 3 --coalesce tx-usecs N --coalesce tx-usecs N
The Interrupt pacing (IRQ coalescing) configuration can be retrieved by commands:
# ethtool -c ethX
- show configuration for TX0 and TX1:
# ethtool -Q eth0 queue_mask 3 --show-coalesce
It is also possible to use standard Linux Net core hard irqs deferral feature which can be enabled by configuring:
/sys/class/net/ethX/
gro_flush_timeout (in ns)
napi_defer_hard_irqs (number of retries)
Enabling of hard IRQ will be deferred napi_defer_hard_irqs times with gro_flush_timeout timeout.
The main difference of the hard irqs deferral feature from ethtool interrupt pacing (IRQ coalescing) is that it affects on both RX/TX data path and all TX/RX queues simultaneously.
TI AM65x switch-config tool
The TI Processor SDK includes precompiled correct version of AM64x switch-config tool.
The TI AM64x switch-config tool sources for AM64x SoC can be found at:
git@git.ti.com:switch-config/switch-config.git
branch:
origin/am65x-v1.0
Usage:
# switch-config -h
Switch configuration commands.....
switch-config -I,--ndev <dev> <command>
commands:
switch-config -d,--dump-ale :dump ALE table
switch-config -D,--dump=<0..9> :dump registers (0 - all)
switch-config -v,--version
dump values:
:1 - cpsw-nuss regs
:2 - cpsw-nuss-rgmii-status regs
:3 - cpsw-nuss-mdio regs
:4 - cpsw-nu regs
:5 - cpsw-nu-p0 regs
:6 - cpsw-nu-p1 regs
:7 - cpsw-nu-cpts regs
:8 - cpsw-nu-ale regs
:9 - cpsw-nu-ale-tbl regs
Example, ALE table dump:
# switch-config -d
K3 cpsw dump version (1) len(6328)
ALE table dump ents(64):
0 : type: vlan , vid = 0, untag_force = 0x3, reg_mcast = 0x0, unreg_mcast = 0x0, member_list = 0x3
1 : type: ucast, addr = f4:84:4c:eb:a0:00, ucast_type = persistant, port_num = 0x0, Secure
2 : type: mcast, addr = ff:ff:ff:ff:ff:ff, mcast_state = f, no super, port_mask = 0x3
3 : type: mcast, addr = 01:00:5e:00:00:01, mcast_state = f, no super, port_mask = 0x3
4 : type: mcast, addr = 01:80:c2:00:00:00, mcast_state = f, no super, port_mask = 0x3
5 : type: mcast, addr = 01:80:c2:00:00:03, mcast_state = f, no super, port_mask = 0x3
6 : type: mcast, addr = 01:80:c2:00:00:0e, mcast_state = f, no super, port_mask = 0x3
8 : type: mcast, addr = 01:00:5e:00:00:fc, mcast_state = f, no super, port_mask = 0x3
9 : type: ucast, vid = 0, addr = 9c:b6:d0:89:0d:85, ucast_type = touched , port_num = 0x1
26 : type: ucast, vid = 0, addr = c4:71:54:a9:6e:b4, ucast_type = touched , port_num = 0x1
27 : type: ucast, vid = 0, addr = 00:25:22:a9:4c:b3, ucast_type = touched , port_num = 0x1
Example, CPTS registers dump:
switch-config -D7
K3 cpsw dump version (1) len(6328)
cpsw-nu-cpts regdump: num_regs(38)
0003d000:reg(4E8A2109)
0003d004:reg(00000C21)
0003d008:reg(00000000)
0003d00c:reg(00000000)
0003d010:reg(7EA3BA9B)
0003d014:reg(00000000)
0003d018:reg(00000000)
0003d01c:reg(00000000)
0003d020:reg(00000000)
0003d024:reg(00000000)
0003d028:reg(00000001)
0003d02c:reg(00000000)
0003d030:reg(00000000)
0003d034:reg(C7298A99)
0003d038:reg(03300000)
0003d03c:reg(00000000)
0003d040:reg(0000028E)
0003d044:reg(00000000)
0003d048:reg(00000000)
3.2.2.8.3.1.4.2. Errata: i2329 MDIO interface corruption (CPSW and PRUSS)¶
3.2.2.8.3.1.4.2.1. Description¶
It is possible that the MDIO interface of all instances of CPSW and PRUSS peripherals (if present) returns corrupt read data on MDIO reads (e.g. returning stale or previous data), or sends incorrect data on MDIO writes. It is also possible that the MDIO interface becomes unavailable until the next peripheral reset (either by LPSC reset or global device reset with reset isolation disabled in case of CPSW).
Possible system level manifestations of this issue could be (1) erroneous ethernet PHY link down status (2) inability to properly configure an ethernet PHY over MDIO (3) incorrect PHY detection (e.g. wrong address) (4) read or write timeouts when attempting to configure PHY over MDIO.
The most common issue with Linux is observation of following prints in the kernel log
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
am65-cpsw-nuss 46000000.ethernet eth0: Link is Down
am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
3.2.2.8.3.1.4.2.2. Workaround for TI SDK Version > 8.4¶
MDIO protocol can be emulated by reading and writing to the appropriate bits within the MDIO_MANUAL_IF_REG register of the MDIO peripheral to directly manipulate the MDIO clock and data pins. Refer to TRM for full details of manual mode register bits and their function.
In this case the device pin multiplexing should be configured to allow the IO to be controlled by the CPSW or PRUSS peripherals (same as in normal intended operation), but the MDIO state machine must be disabled by ensuring MDIO_CONTROL_REG.ENABLE bit is 0 in the MDIO_CONTROL_REG and enable manual mode by setting MDIO_POLL_REG.MANUALMODE bit to 1.
The implementation of the above workaround is available from Processor SDK v8.5.
3.2.2.8.3.1.5. Promiscous Mode¶
By default promiscous mode is disabled. It can be enabled by using the below command.
Please note running a tool like tcpdump will itself enable promiscous mode.
ip link set eth0 promisc on
3.2.2.8.3.1.6. Set MacAddress manually¶
While the default MacAddress of the port is obtained from EEPROM, it’s possible to change the MacAddress manually from shell.
ip link set dev <eth0> address <macaddress>
3.2.2.8.3.1.7. Multi port Switch mode¶
Note
This section documents the Switch mode features available with CPSW3g
The Switch mode can be enabled by configuring devlink driver parameter “switch_mode” to 1/true
devlink dev param set platform/8000000.ethernet \
name switch_mode value true cmode runtime
Above setting can be done regardless of the state of Port’s netdev devices - UP/DOWN, but Port’s netdev devices have to be in UP state before joining the bridge. This is to avoid overwriting of bridge configuration as CPSW switch driver completely reloads its configuration when first port changes its state to UP.
When all the interfaces have joined the bridge - CPSW switch driver will enable marking packets with offload_fwd_mark flag.
All configuration is implemented via switchdev API.
3.2.2.8.3.1.7.1. Bridge setup¶
devlink dev param set platform/8000000.ethernet \
name switch_mode value true cmode runtime
ip link add name br0 type bridge
ip link set dev br0 type bridge ageing_time 1000
ip link set dev eth0 up
ip link set dev eth1 up
ip link set dev eth0 master br0
ip link set dev eth1 master br0
[*] bridge vlan add dev br0 vid 1 self
[*] bridge vlan add dev br0 vid 1 pvid untagged self
[*] if vlan_filtering=1, where default_pvid=1
Note: Steps [*] are mandatory.
3.2.2.8.3.1.7.2. Turn On/Off Spanning Tree Protocol (STP)¶
ip link set dev br0 type bridge stp_state 1/0
3.2.2.8.3.1.7.3. VLAN configuration¶
bridge vlan add dev br0 vid 1 self # <---- add VLAN as a Bridge Entry
bridge vlan add dev br0 vid 1 pvid untagged self # <---- add cpu port to VLAN 1
This step is mandatory for bridge/default_pvid.
3.2.2.8.3.1.7.4. Adding extra VLANs¶
untagged
bridge vlan add dev eth0 vid 100 pvid untagged master bridge vlan add dev sw0p2 vid 100 pvid untagged master bridge vlan add dev br0 vid 100 self # <---- add VLAN as a Bridge Entry bridge vlan add dev br0 vid 100 pvid untagged self # <---- Add cpu port to VLAN100
tagged
bridge vlan add dev eth0 vid 100 master bridge vlan add dev sw0p2 vid 100 master bridge vlan add dev br0 vid 100 self # <---- add VLAN as a Bridge Entry bridge vlan add dev br0 vid 100 pvid tagged self # <---- Add cpu port to VLAN100
3.2.2.8.3.1.7.5. Forwarding Data Bases (FDBs)¶
Forwarding entries for MAC addresses are automatically added on the appropriate switch port upon detection as default operation as an unmanaged bridge. For managed bridge operation manually add FDB entries as required.
Manually adding FDBs
bridge fdb add aa:bb:cc:dd:ee:ff dev eth0 master vlan 100
bridge fdb add aa:bb:cc:dd:ee:fe dev sw0p2 master # <---- Add on all VLANs
Note
For untagged traffic, the PVID is 1 by default. Therefore to add FDB entry for untagged traffic, the commands mentioned above have to use “vid 1” at the respective places.
3.2.2.8.3.1.7.6. Multicast Data Bases (MDBs)¶
Multicast entries are automatically added on the appropriate switch port upon detection as default operation as an unmanaged bridge. For managed bridge operation manually add MDB entries as required.
Manually adding MDBs
bridge mdb add dev br0 port eth0 grp 239.1.1.1 permanent vid 100
bridge mdb add dev br0 port eth0 grp 239.1.1.1 permanent # <---- Add on all VLANs
Note
For untagged traffic, the PVID is 1 by default. Therefore to add MDB entry for untagged traffic, the commands mentioned above have to use “vid 1” at the respective places.
3.2.2.8.3.1.7.7. Multicast flooding¶
CPU port mcast_flooding is always on
Turning flooding on/off on switch ports:
bridge link set dev eth0 mcast_flood on/off
3.2.2.8.3.1.7.8. Enabling Cut Through Forwarding (CTF)¶
Caution
The Cut Through configuration interface could be changed significantly in the future depending on Linux Kernel mainline development.
Cut Through feature allows forwarding packet from one external port to another without being stored in Port FIFOs thus reducing overall latency for packet forwarding.
Limitations:
Feature is only available in switch mode.
Cut Through forwarding is only supported between external ports and no support for Cut Through to host port currently.
Cut-Thru is not supported with 10/100 half-duplex.
Cut-Thru is not supported with any form of flow control.
If Intersperced Express Traffic (IET) is enabled, then Cut Through can only be enabled on an express priority queue and not on preemptible queues.
Note
Currently per port priority mask is set via debugfs entries. This may change in future.
Here is the commands to setup cut-through for priority 0 traffic:
ip link set dev eth0 down
ip link set dev eth1 down
devlink dev param set platform/8000000.ethernet name switch_mode value true cmode runtime
echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_rx_pri_mask
echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_tx_pri_mask
echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_rx_pri_mask
echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_tx_pri_mask
ethtool --set-priv-flags eth0 cut-thru on
ethtool --set-priv-flags eth1 cut-thru on
ip link add name br0 type bridge
ip link set dev br0 type bridge ageing_time 1000
ip link set dev eth0 up
sleep 1
ip link set dev eth1 up
sleep 1
ip link set dev eth0 master br0
ip link set dev eth1 master br0
ip link set dev br0 type bridge stp_state 1
ip link set dev br0 up
ip addr add 10.0.0.1/8 dev br0
bridge vlan add dev br0 vid 1 self
bridge vlan add dev br0 vid 1 pvid untagged self
The value being written to cut_thru_rx_pri_mask and cut_thru_tx_pri_mask represents the priorties queues for which cut through feature needs to be enabled. Can be configured only when all external ports are down.
With above settings, generate a iperf3 traffic from client on Port1 to client on Port 2 and observe the ethtool statistics to see cut through taking effect
root@evm:~# ethtool -S eth1 | grep col
tx_collision_frames: 1796093
tx_single_coll_frames: 0
tx_mult_coll_frames: 3
tx_excessive_collisions: 0
tx_late_collisions: 0
root@evm:~# ethtool -S eth0 | grep col
tx_collision_frames: 3
tx_single_coll_frames: 0
tx_mult_coll_frames: 2002396
tx_excessive_collisions: 52
tx_late_collisions: 23
tx_collision_frames: Enet_Pn_TxCut Enet Port n Cut Thru with and without delay (full-duplex)
tx_single_coll_frames: Enet_pn_TxCut_SAF Enet Port n Tx Store and Forward (full-duplex)
tx_mult_coll_frames: Enet_Pn_RxCut_NoDelay Enet Port n Rx Cut Thru with no delay (full-duplex)
tx_excessive_collisions: Enet_Pn_RxCut_Delay Enet Port n Rx Cut Thru with delay (full-duplex)
tx_late_collisions: Enet_Pn_RxCut_SAF Enet Port n Rx Store and Forward (full-duplex)
3.2.2.8.3.1.7.9. Transmit Traffic Control and Rate Limiting¶
The main difference between one port and multi port devices is that TX CPPI channels are shared between all network devices while External Ports FIFO are per port. The MQPRIO Qdisk can be used to assign different TX CPPI channels to different ports and this way improve over all TX performance.
The configured External Ports Fifos rate should must not be oversubscribed. If some Ext. port and Host port both send to the same priority then Ext. Ports Fifos rate for this priority has to be set as sum of Ext. and Host port rates plus some margin.
Example Host port ingress with separate TX CPPI channel per port, no QoS
switch mode: on | off
Port 1 assigned TX CPPI channel 0
Port 2 assigned TX CPPI channel 1
TX CPPI channels processing mode: Round Robin
ip link set dev eth0 down
ip link set dev eth1 down
ethtool -L eth0 tx 2
ethtool --set-priv-flags eth1 p0-rx-ptype-rrobin on
ip link set dev eth0 up
ip link set dev eth1 up
tc qdisc add dev eth0 handle 100: parent root mqprio num_tc 1 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@0 hw 1 mode channel
tc qdisc add dev eth1 handle 100: parent root mqprio num_tc 1 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@1 hw 1 mode channel
If TX CPPI channels processing mode is Round Robin (p0-rx-ptype-rrobin off) then Rate Limiting should for selected TX CPPI channels.
echo 800 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
echo 800 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
Example Host port ingress with separate TX CPPI channel per port and Rate Limiting
Linux Host send bulk and rate limited traffic to both ports. No traffic switching between P1 and P2.
to Host1│ CPSW3g Host bridge │ to Host2
│ / \ │
│ P1 P2 │
│ | | │
│ Host1 Host2 │
▼ ▼
switch mode: on
tc filters are used to assign pri for iperf3 traffic using port as filter value
VLAN tagged traffic vid=100
eth0: Ports 1
pri7 traffic routed to TX CPPI channel 7, rate limit 100Mbit
pri6 traffic routed to TX CPPI channel 6, rate limit 200Mbit
pri0-5 traffic routed to TX CPPI channel 0
pri7 traffic mapped to TC2, External Ports FIFO2, cir=100Mbit
pri6 traffic mapped to TC1, External Ports FIFO1, cir=200Mbit
pri0-5 traffic mapped to TC0, External Ports FIFO0
eth1: Ports 2
pri7 traffic routed to TX CPPI channel 5, rate limit 100Mbit
pri6 traffic routed to TX CPPI channel 4, rate limit 200Mbit
pri0-5 traffic routed to TX CPPI channel 0
pri7 traffic mapped to TC2, External Ports FIFO2, cir=100Mbit
pri6 traffic mapped to TC1, External Ports FIFO1, cir=200Mbit
pri0-5 traffic mapped to TC1, External Ports FIFO0
ip link set dev eth0 down
ip link set dev eth1 down
ethtool -L eth0 tx 8
ethtool --set-priv-flags eth1 p0-rx-ptype-rrobin off
ip link add name br0 type bridge
ip link set dev eth0 up
ip link set dev eth1 up
ip link set dev eth0 master br0
ip link set dev eth1 master br0
ip link set dev br0 up
#configure bridge...
#configure bridge vlan
ip link add link br0 name br0.100 type vlan id 100
ip link set br0.100 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
bridge vlan add dev eth0 vid 100 master
bridge vlan add dev eth1 vid 100 master
bridge vlan add dev br0 vid 100 self
# set IP to br0.100 192.168.100.1
echo 100 > /sys/class/net/eth0/queues/tx-7/tx_maxrate
echo 200 > /sys/class/net/eth0/queues/tx-6/tx_maxrate
echo 100 > /sys/class/net/eth0/queues/tx-5/tx_maxrate
echo 200 > /sys/class/net/eth0/queues/tx-4/tx_maxrate
tc qdisc add dev br0.100 clsact
tc filter add dev br0.100 egress protocol ip prio 1 u32 match ip dport 5001 0xffff action skbedit priority 7
tc filter add dev br0.100 egress protocol ip prio 1 u32 match ip dport 5002 0xffff action skbedit priority 6
#eth0
tc qdisc add dev eth0 parent root handle 100: mqprio num_tc 3 \
map 0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 \
queues 1@0 1@6 1@7 hw 1 mode channel \
shaper bw_rlimit min_rate 0 200mbit 100mbit
#eth1
tc qdisc add dev eth1 parent root handle 100: mqprio num_tc 3 \
map 0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 \
queues 1@0 1@4 1@5 hw 1 mode channel \
shaper bw_rlimit min_rate 0 200mbit 100mbit
iperf3 -c 192.168.100.3 -t10 -p5001 -Tpri71 & \
iperf3 -c 192.168.100.3 -t10 -p5002 -Tpri61 & \
iperf3 -c 192.168.100.3 -t10 -p5003 -Tpri01 & \
iperf3 -c 192.168.100.2 -t10 -p5001 -Tpri72 & \
iperf3 -c 192.168.100.2 -t10 -p5002 -Tpri62
Example Bridging with TX CPPI channel and External Ports FIFO shapers
Linux Host send bulk and rate limited traffic to Host 2 (Port 2) pri3 200Mbit and Host 1 sends rate limited traffic to Host 2 (Port 2) pri3 200Mbit.
CPSW3g Host bridge │ to Host2
/ \ │ 200Mbit
P1 P2 │
| | │
Host1 Host2 │
▼
──────────────►
to Host2
200Mbit
switch mode: on
tc filters are used to assign pri for iperf3 traffic using port as filter value
VLAN tagged traffic vid=100
assume pri3 is class A and pri 2 is class B (not used)
eth0: Ports 1
pri0-7 traffic routed to TX CPPI channel 0, no HW offload
eth1: Ports 2
pri3 traffic routed to TX CPPI channel 7, rate limit 200Mbit
pri2 traffic routed to TX CPPI channel 6, rate limit 100Mbit
pri0,1,4-7 traffic routed to TX CPPI channel 0
pri3 traffic mapped to TC2, External Ports FIFO2, cir=200Mbit
pri2 traffic mapped to TC1, External Ports FIFO1, cir=100Mbit
pri0,1,4-7 traffic mapped to TC0, External Ports FIFO0
CPSW3g Host bridge configuration
ip link set dev eth0 down
ip link set dev eth1 down
ethtool -L eth0 tx 8
ethtool --set-priv-flags eth1 p0-rx-ptype-rrobin off
ip link add name br0 type bridge
ip link set dev eth0 up
ip link set dev eth1 up
ip link set dev eth0 master br0
ip link set dev eth1 master br0
ip link set dev br0 up
#configure bridge...
#configure bridge vlan
ip link add link br0 name br0.100 type vlan id 100
ip link set br0.100 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
bridge vlan add dev eth0 vid 100 master
bridge vlan add dev eth1 vid 100 master
bridge vlan add dev br0 vid 100 self
# set IP to br0.100 192.168.100.1
echo 200 > /sys/class/net/eth0/queues/tx-7/tx_maxrate
echo 100 > /sys/class/net/eth0/queues/tx-6/tx_maxrate
tc qdisc add dev br0.100 clsact
tc filter add dev br0.100 egress protocol ip prio 1 u32 match ip dport 5002 0xffff action skbedit priority 3
#eth0
tc qdisc add dev eth0 handle 100: parent root mqprio num_tc 1 \
map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@0 hw 0
#eth1
tc qdisc add dev eth1 parent root handle 100: mqprio num_tc 3 \
map 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@6 1@7 hw 1 mode channel \
shaper bw_rlimit min_rate 0 100mbit 410mbit
iperf3 -c 192.168.100.2 -t10 -p5002 -Tpri22 -t30 & \
iperf3 -c 192.168.100.2 -t10 -p5003 -Tpri02 -t30 &
Host 1 configuration
ip link add link eth0 name eth0.100 type vlan id 100
ip link set eth0.100 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
# set IP to br0.100 192.168.100.3
tc qdisc add dev eth0.100 clsact
tc filter add dev eth0.100 egress protocol ip prio 1 u32 match ip dport 5001 0xffff action skbedit priority 3
#Real QoS configuration depends Host 2 functionality
iperf3 -c 192.168.100.2 -t10 -p5001 -Tpri3 -t30
# - or -
iperf3 -c 192.168.100.2 -t10 -p5001 -Tpri3 -t30 -u -b210M
Host 2 configuration
ip link add link eth0 name eth0.100 type vlan id 100
ip link set eth0.100 type vlan egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
# set IP to br0.100 192.168.100.3
iperf3 -s -p 5001&
iperf3 -s -p 5002&
iperf3 -s -p 5003&