-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Medium
-
Linux Core SDK
-
LCPD-46920
-
11.01
-
12.01.00
-
am64xx-evm
Customer reports that UDP packets are received out-of-order on AM64x EVM running TI SDK 11.01.05.03 when using HSR offload mode with a Hirschmann RSP35 RedBox as the traffic source.
Environment:
Platform: AM64x EVM (AM6442)
SDK: TI SDK 11.01.05.03 / 11.02
PRU Firmware: REL.HSR_1G_01.02.03.05
RedBox: Hirschmann RSP35
Link Speed: 1Gbps
Test Setup:
Run
PC (iperf client) --> RedBox --> AM64x EVM (iperf server)
Reproduction Steps:
Configure HSR on AM64x with offload enabled
Run iperf3 server: iperf3 -s -B 192.168.1.2
Capture traffic: tcpdump -i hsr0 -p -w /tmp/capture.pcap
Send UDP traffic from PC: iperf3 -c 192.168.1.2 -l 300 -u -t 1 -b 100M
Key Findings:
Out-of-order issue only observed at 1Gbps link speed with RedBox
- Issue not observed at 100Mbps link speed
- Issue not observed with AM64x-to-AM64x HSR ring (no RedBox)
- Issue not observed with software HSR (non-offloaded)
- Profishark captures on wire show packets arriving in correct order
- tcpdump on AM64x shows packets out-of-order
- PRU firmware debug confirms packets processed in correct order at firmware level
Root Cause Analysis (TI Investigation):
- Firmware ruled out as the source - debug code confirms packets passed to Linux driver in correct order
- Issue appears to be in Linux kernel packet processing path
- Multicore processing may cause packet reordering when packets are processed on different CPU cores
- tcpdump orders packets by kernel processing timestamp, not MAC-layer arrival timestamp
Workaround Attempts:
- maxcpus=1 boot parameter: Reduced out-of-order but caused packet loss
- RX IRQ CPU affinity to single core: Improved at lower bitrates (50Mbps OK), issues persist at higher rates
- tcpdump -j adapter_unsynced: Did not resolve the issue
Additional Data:
ethtool stats show FW_DROPPED_PKT counter incrementing
Issue correlates with packet loss at higher bitrates --> but not always, sometimes packet loss is seen but no out of order is seen
Reference:
E2E Thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1595726/am6442-am64x-evm-hsr-packet-order-error/6180300