or would like to run DPDK software applications on Cisco UCS servers using
Cisco VIC adapters the following documentation is relevant.
-Version Information
--------------------
-
-The version of the ENIC PMD driver is 1.0.0.6 and will be printed by ENIC PMD
-during the initialization.
-
How to obtain ENIC PMD integrated DPDK
--------------------------------------
ENIC PMD support is integrated into the DPDK suite. dpdk-<version>.tar.gz
-should be downloaded from http://dpdk.org
+should be downloaded from http://core.dpdk.org/download/
Configuration information
- **Number of Queues**
- The maximum number of receive and transmit queues are configurable on a per
- vNIC basis through the Cisco UCS Manager (CIMC or UCSM). These values
- should be configured to be greater than or equal to the nb_rx_q and nb_tx_q
- parameters expected to used in the call to the rte_eth_dev_configure()
- function.
+ The maximum number of receive queues (RQs), work queues (WQs) and
+ completion queues (CQs) are configurable on a per vNIC basis
+ through the Cisco UCS Manager (CIMC or UCSM).
+
+ These values should be configured as follows:
+
+ - The number of WQs should be greater or equal to the value of the
+ expected nb_tx_q parameter in the call to the
+ rte_eth_dev_configure()
+
+ - The number of RQs configured in the vNIC should be greater or
+ equal to *twice* the value of the expected nb_rx_q parameter in
+ the call to rte_eth_dev_configure(). With the addition of rx
+ scatter, a pair of RQs on the vnic is needed for each receive
+ queue used by DPDK, even if rx scatter is not being used.
+ Having a vNIC with only 1 RQ is not a valid configuration, and
+ will fail with an error message.
+
+ - The number of CQs should set so that there is one CQ for each
+ WQ, and one CQ for each pair of RQs.
+
+ For example: If the application requires 3 Rx queues, and 3 Tx
+ queues, the vNIC should be configured to have at least 3 WQs, 6
+ RQs (3 pairs), and 6 CQs (3 for use by WQs + 3 for use by the 3
+ pairs of RQs).
- **Size of Queues**
a per vNIC bases via the UCS Manager and should be greater than or equal to
the nb_rx_desc and nb_tx_desc parameters expected to be used in the calls
to rte_eth_rx_queue_setup() and rte_eth_tx_queue_setup() respectively.
+ An application requesting more than the set size will be limited to that
+ size.
+
+ Unless there is a lack of resources due to creating many vNICs, it
+ is recommended that the WQ and RQ sizes be set to the maximum. This
+ gives the application the greatest amount of flexibility in its
+ queue configuration.
+
+ - *Note*: Since the introduction of rx scatter, for performance
+ reasons, this PMD uses two RQs on the vNIC per receive queue in
+ DPDK. One RQ holds descriptors for the start of a packet the
+ second RQ holds the descriptors for the rest of the fragments of
+ a packet. This means that the nb_rx_desc parameter to
+ rte_eth_rx_queue_setup() can be a greater than 4096. The exact
+ amount will depend on the size of the mbufs being used for
+ receives, and the MTU size.
+
+ For example: If the mbuf size is 2048, and the MTU is 9000, then
+ receiving a full size packet will take 5 descriptors, 1 from the
+ start of packet queue, and 4 from the second queue. Assuming
+ that the RQ size was set to the maximum of 4096, then the
+ application can specify up to 1024 + 4096 as the nb_rx_desc
+ parameter to rte_eth_rx_queue_setup().
- **Interrupts**
Only one interrupt per vNIC interface should be configured in the UCS
manager regardless of the number receive/transmit queues. The ENIC PMD
- uses this interrupt to get information about errors in the fast path.
+ uses this interrupt to get information about link status and errors
+ in the fast path.
+
+.. _enic-flow-director:
+
+Flow director support
+---------------------
+
+Advanced filtering support was added to 1300 series VIC firmware starting
+with version 2.0.13 for C-series UCS servers and version 3.1.2 for UCSM
+managed blade servers. In order to enable advanced filtering the 'Advanced
+filter' radio button should be enabled via CIMC or UCSM followed by a reboot
+of the server.
+
+With advanced filters, perfect matching of all fields of IPv4, IPv6 headers
+as well as TCP, UDP and SCTP L4 headers is available through flow director.
+Masking of these feilds for partial match is also supported.
+
+Without advanced filter support, the flow director is limited to IPv4
+perfect filtering of the 5-tuple with no masking of fields supported.
+
+Ingress VLAN Rewrite
+--------------------
+
+VIC adapters can tag, untag, or modify the VLAN headers of ingress
+packets. The ingress VLAN rewrite mode controls this behavior. By
+default, it is set to pass-through, where the NIC does not modify the
+VLAN header in any way so that the application can see the original
+header. This mode is sufficient for many applications, but may not be
+suitable for others. Such applications may change the mode by setting
+``devargs`` parameter ``ig-vlan-rewrite`` to one of the following.
+
+- ``pass``: Pass-through mode. The NIC does not modify the VLAN
+ header. This is the default mode.
+
+- ``priority``: Priority-tag default VLAN mode. If the ingress packet
+ is tagged with the default VLAN, the NIC replaces its VLAN header
+ with the priority tag (VLAN ID 0).
+
+- ``trunk``: Default trunk mode. The NIC tags untagged ingress packets
+ with the default VLAN. Tagged ingress packets are not modified. To
+ the application, every packet appears as tagged.
+
+- ``untag``: Untag default VLAN mode. If the ingress packet is tagged
+ with the default VLAN, the NIC removes or untags its VLAN header so
+ that the application sees an untagged packet. As a result, the
+ default VLAN becomes `untagged`. This mode can be useful for
+ applications such as OVS-DPDK performance benchmarks that utilize
+ only the default VLAN and want to see only untagged packets.
Limitations
-----------
In test setups where an Ethernet port of a Cisco adapter in TRUNK mode is
connected point-to-point to another adapter port or connected though a router
instead of a switch, all ingress packets will be VLAN tagged. Programs such
- as l3fwd which do not account for VLAN tags in packets will misbehave. The
- solution is to enable VLAN stripping on ingress. The follow code fragment is
- example of how to accomplish this:
+ as l3fwd may not account for VLAN tags in packets and may misbehave. One
+ solution is to enable VLAN stripping on ingress so the VLAN tag is removed
+ from the packet and put into the mbuf->vlan_tci field. Here is an example
+ of how to accomplish this:
.. code-block:: console
vlan_offload |= ETH_VLAN_STRIP_OFFLOAD;
rte_eth_dev_set_vlan_offload(port, vlan_offload);
+Another alternative is modify the adapter's ingress VLAN rewrite mode so that
+packets with the default VLAN tag are stripped by the adapter and presented to
+DPDK as untagged packets. In this case mbuf->vlan_tci and the PKT_RX_VLAN and
+PKT_RX_VLAN_STRIPPED mbuf flags would not be set. This mode is enabled with the
+``devargs`` parameter ``ig-vlan-rewrite=untag``. For example::
+
+ -w 12:00.0,ig-vlan-rewrite=untag
+
+- Limited flow director support on 1200 series and 1300 series Cisco VIC
+ adapters with old firmware. Please see :ref:`enic-flow-director`.
+
+- Flow director features are not supported on generation 1 Cisco VIC adapters
+ (M81KR and P81E)
+
How to build the suite?
-----------------------
The build instructions for the DPDK suite should be followed. By default
- VIC 1385
- VIC 1387
-- Flow director features are not supported on generation 1 Cisco VIC adapters
- (M81KR and P81E)
-
Supported Operating Systems
---------------------------
Any Linux distribution fulfilling the conditions described in Dependencies
- IP checksum offload
- Receive side VLAN stripping
- Multiple receive and transmit queues
-- Flow Director ADD, UPDATE, DELETE, STATS operation support for IPV4 5-TUPLE
- flows
+- Flow Director ADD, UPDATE, DELETE, STATS operation support IPv4 and IPv6
- Promiscuous mode
- Setting RX VLAN (supported via UCSM/CIMC only)
- VLAN filtering (supported via UCSM/CIMC only)
- Execution of application by unprivileged system users
- IPV4, IPV6 and TCP RSS hashing
+- Scattered Rx
+- MTU update
Known bugs and Unsupported features in this release
---------------------------------------------------
- non-IPV4 flow direction
- Setting of extended VLAN
- UDP RSS hashing
+- MTU update only works if Scattered Rx mode is disabled
Prerequisites
-------------
- DPDK suite should be configured based on the user's decision to use VFIO or
UIO framework
- If the vNIC device(s) to be used is bound to the kernel mode Ethernet driver
- (enic), use 'ifconfig' to bring the interface down. The dpdk_nic_bind.py tool
+ (enic), use 'ifconfig' to bring the interface down. The dpdk-devbind.py tool
can then be used to unbind the device's bus id from the enic kernel mode
driver.
- Bind the intended vNIC to vfio-pci in case the user wants ENIC PMD to use
- VFIO framework using dpdk_nic_bind.py.
+ VFIO framework using dpdk-devbind.py.
- Bind the intended vNIC to igb_uio in case the user wants ENIC PMD to use
- UIO framework using dpdk_nic_bind.py.
+ UIO framework using dpdk-devbind.py.
At this point the system should be ready to run DPDK applications. Once the
application runs to completion, the vNIC can be detached from vfio-pci or
Code Review / deb_dpdk.git / blobdiff