-
Notifications
You must be signed in to change notification settings - Fork 84
Fix XDP documentation to a unified location for ICSSG and CPSW #642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
|
MeghanaMalladiTI marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,8 @@ | ||
| .. _pru_icssg_xdp: | ||
| .. _kernel_xdp: | ||
|
Check warning on line 1 in source/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/XDP.rst
|
||
|
|
||
| ############# | ||
| PRU_ICSSG XDP | ||
| ############# | ||
| ### | ||
| XDP | ||
|
Check warning on line 4 in source/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/XDP.rst
|
||
| ### | ||
|
|
||
| .. contents:: :local: | ||
| :depth: 3 | ||
|
|
@@ -11,24 +11,24 @@ | |
| Introduction | ||
| ************ | ||
|
|
||
| XDP (eXpress Data Path) provides a framework for BPF that enables high-performance programmable packet processing in the Linux kernel. It runs the BPF program at the earliest possible point in software, namely at the moment the network driver receives the packet. | ||
| eXpress Data Path (XDP) provides a framework for extended Berkeley Packet Filters (eBPF) that enables high-performance programmable packet processing in the Linux kernel. It runs the eBPF program at the earliest possible point in software, namely at the moment the network driver receives the packet. | ||
|
|
||
| XDP allows running a BPF program just before the skbs are allocated in the driver, the BPF program can look at the packet and return the following things. | ||
| XDP allows running an eBPF program just before the socket buffers (skbs) are allocated in the driver. The eBPF program can examine the packet and return one of the following actions. | ||
|
|
||
| - XDP_DROP :- The packet is dropped right away, without wasting any resources. Useful for firewall etc. | ||
| - XDP_ABORTED :- Similar to drop, an exception is generated. | ||
| - XDP_PASS :- Pass the packet to kernel stack, i.e. the skbs are allocated and it works normally. | ||
| - XDP_TX :- Send the packet back to same NIC with modification(if done by the program). | ||
| - XDP_REDIRECT :- Send the packet to another NIC or to the user space through AF_XDP Socket(discussed below). | ||
| - ``XDP_DROP`` :- The packet is dropped right away, without wasting any resources. Useful for firewall etc. | ||
| - ``XDP_ABORTED`` :- Similar to drop, an exception is generated. | ||
| - ``XDP_PASS`` :- Pass the packet to kernel stack, i.e. the skbs are allocated and it works normally. | ||
| - ``XDP_TX`` :- Send the packet back to same NIC with modification(if done by the program). | ||
| - ``XDP_REDIRECT`` :- Send the packet to another NIC or to the user space through AF_XDP Socket(discussed below). | ||
|
|
||
| .. Image:: /images/XDP-packet-processing.png | ||
| .. image:: /images/XDP-packet-processing.png | ||
|
|
||
| As explained before, the XDP_REDIRECT sends packets directly to the user space. | ||
| As explained before, the ``XDP_REDIRECT`` sends packets directly to the user space. | ||
| This works by using the AF_XDP socket type which was introduced specifically for this usecase. | ||
|
|
||
| In this process, the packet is directly sent to the user space without going through the kernel network stack. | ||
|
|
||
| .. Image:: /images/xdp-packet.png | ||
| .. image:: /images/xdp-packet.png | ||
|
|
||
| Use cases for XDP | ||
| ================= | ||
|
|
@@ -42,10 +42,10 @@ | |
| 5. **Network Analytics**: Real-time traffic analysis and monitoring | ||
| 6. **Custom Network Functions**: Specialized packet handling for unique requirements | ||
|
|
||
| How to run XDP with PRU_ICSSG | ||
| ============================= | ||
| How to run XDP on EVM | ||
| ===================== | ||
|
|
||
| The kernel configuration requires the following changes to use XDP with PRU_ICSSG: | ||
| The kernel configuration requires the following changes to use XDP: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
|
|
@@ -89,8 +89,8 @@ | |
| AF_XDP sockets operate through a shared memory mechanism: | ||
|
|
||
| 1. XDP program intercepts packets at driver level | ||
| 2. XDP_REDIRECT action sends packets to the socket | ||
| 3. Shared memory rings (RX/TX/FILL/COMPLETION) manage packet data | ||
| 2. ``XDP_REDIRECT`` action sends packets to the socket | ||
| 3. Shared memory rings (``RX``/``TX``/``FILL``/``COMPLETION``) manage packet data | ||
| 4. Userspace application directly accesses the packet data | ||
| 5. Zero or minimal copying depending on the mode used | ||
|
|
||
|
|
@@ -99,22 +99,13 @@ | |
| - **RX Ring**: Received packets ready for consumption | ||
| - **TX Ring**: Packets to be transmitted | ||
| - **FILL Ring**: Pre-allocated buffers for incoming packets | ||
| - **COMPLETION Ring**: Tracks completed TX operations | ||
| - **COMPLETION Ring**: Tracks completed ``TX`` operations | ||
|
|
||
| For more details on AF_XDP please refer to the official documentation: `AF_XDP <https://www.kernel.org/doc/html/latest/networking/af_xdp.html>`_. | ||
|
|
||
| Current Support Status in PRU_ICSSG | ||
| =================================== | ||
|
|
||
| The PRU_ICSSG Ethernet driver currently supports: | ||
|
|
||
| - Native XDP mode | ||
| - Generic XDP mode (SKB-based) | ||
| - Zero-copy mode | ||
|
|
||
| ************************** | ||
| XDP Zero-Copy in PRU_ICSSG | ||
| ************************** | ||
| ************* | ||
| XDP Zero-Copy | ||
| ************* | ||
|
|
||
| Introduction to Zero-Copy Mode | ||
| ============================== | ||
|
|
@@ -131,31 +122,88 @@ | |
| 3. Managing memory ownership through descriptor rings rather than data movement | ||
|
|
||
| This approach provides several benefits: | ||
|
|
||
| - Reduced CPU utilization | ||
| - Lower memory bandwidth consumption | ||
| - Decreased latency for packet processing | ||
| - Improved overall throughput | ||
|
|
||
| Requirements for Zero-Copy | ||
| Performance Considerations | ||
| ========================== | ||
|
|
||
| For zero-copy to function properly with PRU_ICSSG, ensure: | ||
| When implementing XDP applications, consider these performance factors: | ||
|
|
||
| 1. **Memory Alignment**: Buffers should be aligned to page boundaries for optimal performance | ||
| 2. **Batch Processing**: Process multiple packets in batches when possible | ||
| 3. **Poll Mode**: Use poll() or similar mechanisms to avoid blocking on socket operations | ||
| 4. **Core Affinity**: Bind application threads to specific CPU cores to reduce cache contention | ||
|
|
||
| ****************** | ||
| Testing XDP on EVM | ||
| ****************** | ||
|
|
||
| The `xdp-tools <https://github.com/xdp-project/xdp-tools>`__ package provides | ||
| utility tools for testing XDP and AF_XDP such as :command:`xdp-bench`, :command:`xdp-trafficgen` etc. | ||
|
|
||
| TI SDK packages the latest version of ``xdp-tools`` utilities and provides it as part of the SDK. | ||
|
|
||
| Both CPSW and ICSSG Ethernet drivers supports Native XDP, Generic XDP, and Zero-copy mode. | ||
|
|
||
| .. note:: | ||
|
|
||
| In case of testing with CPSW please note that when running XDP in Zero-copy mode, non-XDP traffic will be dropped. | ||
|
|
||
| **XDP Transmit test** — generate traffic using XDP (copy mode): | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-trafficgen udp -m ff:ff:ff:ff:ff:ff <interface> | ||
|
|
||
| **XDP Drop test** — receive and drop packets using XDP (copy mode): | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-bench xdp-bench drop <interface> | ||
|
|
||
| **XDP Pass test** — receive and pass packets through XDP allowing normal network stack processing (copy mode): | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-bench xdp-bench pass <interface> | ||
|
|
||
| **XDP TX test** — Hairpins (bounces back) received packets on the same interface (copy mode): | ||
|
|
||
| 1. **Driver Support**: Verify the PRU_ICSSG driver is loaded with zero-copy support enabled | ||
| 2. **Memory Alignment**: Buffer addresses must be properly aligned to page boundaries | ||
| 3. **UMEM Configuration**: The UMEM area must be correctly configured: | ||
| - Properly aligned memory allocation | ||
| - Sufficient number of packet buffers | ||
| - Appropriate buffer sizes | ||
| 4. **Hugepages**: Using hugepages for UMEM allocation is recommended for optimal performance | ||
| .. code-block:: console | ||
|
|
||
| xdp-bench xdp-bench tx <interface> | ||
|
|
||
| **XDP Redirect test** — Redirects received packets on the from one interface to another (copy mode): | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-bench xdp-bench redirect <interface1> <interface2> | ||
|
|
||
| **XSK Drop test** — receive and drop packets using AF_XDP socket in zero-copy mode: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-bench xsk-drop -q 0 -C zero-copy <interface> | ||
|
|
||
| **XSK Transmit test** — generate traffic using AF_XDP socket in zero-copy mode: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| xdp-trafficgen xsk-udp -m ff:ff:ff:ff:ff:ff -q 0 -C zero-copy <interface> | ||
|
|
||
| While xdpsock is not packaged into the SDK, the same functionality can be done with xsk-trafficgen and xsk-bench from the xdp-tools package. | ||
| For more details on xdpsock and how it performs XDP zero copy testing refer to `xdpsock <https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example>`_ | ||
|
|
||
| Performance Comparison | ||
| ====================== | ||
|
|
||
| Performance testing shows that zero-copy mode can provide substantial throughput improvements compared to copy mode: | ||
|
|
||
| `xdpsock <https://github.com/xdp-project/bpf-examples/tree/main/AF_XDP-example>`_ opensource tool was used for testing XDP zero copy. | ||
| AF_XDP performance while using 64 byte packets in Kpps: | ||
| AF_XDP performance while using 64 byte packets for ICSSG (in Kpps): | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
|
|
@@ -173,13 +221,20 @@ | |
| - 354 | ||
| - 855 | ||
|
|
||
| Performance Considerations | ||
| ========================== | ||
| AF_XDP performance while using 64 byte packets for CPSW (in Kpps): | ||
|
|
||
| When implementing XDP applications, consider these performance factors: | ||
| .. list-table:: | ||
| :header-rows: 1 | ||
|
|
||
| 1. **Memory Alignment**: Buffers should be aligned to page boundaries for optimal performance | ||
| 2. **Batch Processing**: Process multiple packets in batches when possible | ||
| 3. **Poll Mode**: Use poll() or similar mechanisms to avoid blocking on socket operations | ||
| 4. **Core Affinity**: Bind application threads to specific CPU cores to reduce cache contention | ||
| 5. **NUMA Awareness**: Consider NUMA topology when allocating memory for packet buffers | ||
| * - Benchmark | ||
| - XDP-SKB | ||
| - XDP-Native | ||
| - XDP-Native(ZeroCopy) | ||
| * - rxdrop | ||
| - 322 | ||
| - 491 | ||
| - 845 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't extensively test this but on some latest test with all AM6x platforms and all configurations but I found that on AM62Px tests on SDK 11.2 the actual kps rate was was much less. For documentation purposes I can approve this for now but later I might create JIRA if I indeed see kps rate is poor on latest SDK |
||
| * - txonly | ||
| - 390 | ||
| - 394 | ||
| - 723 | ||
Uh oh!
There was an error while loading. Please reload this page.