ICSSG: HSR packet out of order

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Medium
    • Linux Core SDK
    • LCPD-46920
    • 11.01
    • 12.01.00
    • am64xx-evm

      Customer reports that UDP packets are received out-of-order on AM64x EVM running TI SDK 11.01.05.03 when using HSR offload mode with a Hirschmann RSP35 RedBox as the traffic source.

      Environment:

      Platform: AM64x EVM (AM6442)
      SDK: TI SDK 11.01.05.03 / 11.02
      PRU Firmware: REL.HSR_1G_01.02.03.05
      RedBox: Hirschmann RSP35
      Link Speed: 1Gbps
      Test Setup:

      Run
      PC (iperf client) --> RedBox --> AM64x EVM (iperf server)
      Reproduction Steps:

      Configure HSR on AM64x with offload enabled
      Run iperf3 server: iperf3 -s -B 192.168.1.2
      Capture traffic: tcpdump -i hsr0 -p -w /tmp/capture.pcap
      Send UDP traffic from PC: iperf3 -c 192.168.1.2 -l 300 -u -t 1 -b 100M
      Key Findings:

      Out-of-order issue only observed at 1Gbps link speed with RedBox

      • Issue not observed at 100Mbps link speed
      • Issue not observed with AM64x-to-AM64x HSR ring (no RedBox)
      • Issue not observed with software HSR (non-offloaded)
      • Profishark captures on wire show packets arriving in correct order
      • tcpdump on AM64x shows packets out-of-order
      • PRU firmware debug confirms packets processed in correct order at firmware level

      Root Cause Analysis (TI Investigation):

      • Firmware ruled out as the source - debug code confirms packets passed to Linux driver in correct order
      • Issue appears to be in Linux kernel packet processing path
      • Multicore processing may cause packet reordering when packets are processed on different CPU cores
      • tcpdump orders packets by kernel processing timestamp, not MAC-layer arrival timestamp

      Workaround Attempts:

      • maxcpus=1 boot parameter: Reduced out-of-order but caused packet loss
      • RX IRQ CPU affinity to single core: Improved at lower bitrates (50Mbps OK), issues persist at higher rates
      • tcpdump -j adapter_unsynced: Did not resolve the issue

      Additional Data:
      ethtool stats show FW_DROPPED_PKT counter incrementing
      Issue correlates with packet loss at higher bitrates --> but not always, sometimes packet loss is seen but no out of order is seen

      Reference:
      E2E Thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1595726/am6442-am64x-evm-hsr-packet-order-error/6180300

            Assignee:
            TI User
            Reporter:
            TI User
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:

                Connection: Intermediate to External PROD System
                EXTSYNC-6578 - ICSSG: HSR packet out of order
                SYNCHRONIZED
                • Last Sync Date: