--- /dev/null
+.. BSD LICENSE
+ Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+DPDK Xen Based Packet-Switching Solution
+========================================
+
+Introduction
+------------
+
+DPDK provides a para-virtualization packet switching solution, based on the Xen hypervisor's Grant Table, Note 1,
+which provides simple and fast packet switching capability between guest domains and host domain based on MAC address or VLAN tag.
+
+This solution is comprised of two components;
+a Poll Mode Driver (PMD) as the front end in the guest domain and a switching back end in the host domain.
+XenStore is used to exchange configure information between the PMD front end and switching back end,
+including grant reference IDs for shared Virtio RX/TX rings,
+MAC address, device state, and so on. XenStore is an information storage space shared between domains,
+see further information on XenStore below.
+
+The front end PMD can be found in the DPDK directory lib/ librte_pmd_xenvirt and back end example in examples/vhost_xen.
+
+The PMD front end and switching back end use shared Virtio RX/TX rings as para- virtualized interface.
+The Virtio ring is created by the front end, and Grant table references for the ring are passed to host.
+The switching back end maps those grant table references and creates shared rings in a mapped address space.
+
+The following diagram describes the functionality of the DPDK Xen Packet- Switching Solution.
+
+
+.. _figure_dpdk_xen_pkt_switch:
+
+.. figure:: img/dpdk_xen_pkt_switch.*
+
+ Functionality of the DPDK Xen Packet Switching Solution.
+
+
+Note 1 The Xen hypervisor uses a mechanism called a Grant Table to share memory between domains
+(`http://wiki.xen.org/wiki/Grant Table <http://wiki.xen.org/wiki/Grant%20Table>`_).
+
+A diagram of the design is shown below, where "gva" is the Guest Virtual Address,
+which is the data pointer of the mbuf, and "hva" is the Host Virtual Address:
+
+
+.. _figure_grant_table:
+
+.. figure:: img/grant_table.*
+
+ DPDK Xen Layout
+
+
+In this design, a Virtio ring is used as a para-virtualized interface for better performance over a Xen private ring
+when packet switching to and from a VM.
+The additional performance is gained by avoiding a system call and memory map in each memory copy with a XEN private ring.
+
+Device Creation
+---------------
+
+Poll Mode Driver Front End
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Mbuf pool allocation:
+
+ To use a Xen switching solution, the DPDK application should use rte_mempool_gntalloc_create()
+ to reserve mbuf pools during initialization.
+ rte_mempool_gntalloc_create() creates a mempool with objects from memory allocated and managed via gntalloc/gntdev.
+
+ The DPDK now supports construction of mempools from allocated virtual memory through the rte_mempool_xmem_create() API.
+
+ This front end constructs mempools based on memory allocated through the xen_gntalloc driver.
+ rte_mempool_gntalloc_create() allocates Grant pages, maps them to continuous virtual address space,
+ and calls rte_mempool_xmem_create() to build mempools.
+ The Grant IDs for all Grant pages are passed to the host through XenStore.
+
+* Virtio Ring Creation:
+
+ The Virtio queue size is defined as 256 by default in the VQ_DESC_NUM macro.
+ Using the queue setup function,
+ Grant pages are allocated based on ring size and are mapped to continuous virtual address space to form the Virtio ring.
+ Normally, one ring is comprised of several pages.
+ Their Grant IDs are passed to the host through XenStore.
+
+ There is no requirement that this memory be physically continuous.
+
+* Interrupt and Kick:
+
+ There are no interrupts in DPDK Xen Switching as both front and back ends work in polling mode.
+ There is no requirement for notification.
+
+* Feature Negotiation:
+
+ Currently, feature negotiation through XenStore is not supported.
+
+* Packet Reception & Transmission:
+
+ With mempools and Virtio rings created, the front end can operate Virtio devices,
+ as it does in Virtio PMD for KVM Virtio devices with the exception that the host
+ does not require notifications or deal with interrupts.
+
+XenStore is a database that stores guest and host information in the form of (key, value) pairs.
+The following is an example of the information generated during the startup of the front end PMD in a guest VM (domain ID 1):
+
+.. code-block:: console
+
+ xenstore -ls /local/domain/1/control/dpdk
+ 0_mempool_gref="3042,3043,3044,3045"
+ 0_mempool_va="0x7fcbc6881000"
+ 0_tx_vring_gref="3049"
+ 0_rx_vring_gref="3053"
+ 0_ether_addr="4e:0b:d0:4e:aa:f1"
+ 0_vring_flag="3054"
+ ...
+
+Multiple mempools and multiple Virtios may exist in the guest domain, the first number is the index, starting from zero.
+
+The idx#_mempool_va stores the guest virtual address for mempool idx#.
+
+The idx#_ether_adder stores the MAC address of the guest Virtio device.
+
+For idx#_rx_ring_gref, idx#_tx_ring_gref, and idx#_mempool_gref, the value is a list of Grant references.
+Take idx#_mempool_gref node for example, the host maps those Grant references to a continuous virtual address space.
+The real Grant reference information is stored in this virtual address space,
+where (gref, pfn) pairs follow each other with -1 as the terminator.
+
+
+.. _figure_grant_refs:
+
+.. figure:: img/grant_refs.*
+
+ Mapping Grant references to a continuous virtual address space
+
+
+After all gref# IDs are retrieved, the host maps them to a continuous virtual address space.
+With the guest mempool virtual address, the host establishes 1:1 address mapping.
+With multiple guest mempools, the host establishes multiple address translation regions.
+
+Switching Back End
+~~~~~~~~~~~~~~~~~~
+
+The switching back end monitors changes in XenStore.
+When the back end detects that a new Virtio device has been created in a guest domain, it will:
+
+#. Retrieve Grant and configuration information from XenStore.
+
+#. Map and create a Virtio ring.
+
+#. Map mempools in the host and establish address translation between the guest address and host address.
+
+#. Select a free VMDQ pool, set its affinity with the Virtio device, and set the MAC/ VLAN filter.
+
+Packet Reception
+~~~~~~~~~~~~~~~~
+
+When packets arrive from an external network, the MAC?VLAN filter classifies packets into queues in one VMDQ pool.
+As each pool is bonded to a Virtio device in some guest domain, the switching back end will:
+
+#. Fetch an available entry from the Virtio RX ring.
+
+#. Get gva, and translate it to hva.
+
+#. Copy the contents of the packet to the memory buffer pointed to by gva.
+
+The DPDK application in the guest domain, based on the PMD front end,
+is polling the shared Virtio RX ring for available packets and receives them on arrival.
+
+Packet Transmission
+~~~~~~~~~~~~~~~~~~~
+
+When a Virtio device in one guest domain is to transmit a packet,
+it puts the virtual address of the packet's data area into the shared Virtio TX ring.
+
+The packet switching back end is continuously polling the Virtio TX ring.
+When new packets are available for transmission from a guest, it will:
+
+#. Fetch an available entry from the Virtio TX ring.
+
+#. Get gva, and translate it to hva.
+
+#. Copy the packet from hva to the host mbuf's data area.
+
+#. Compare the destination MAC address with all the MAC addresses of the Virtio devices it manages.
+ If a match exists, it directly copies the packet to the matched VIrtio RX ring.
+ Otherwise, it sends the packet out through hardware.
+
+.. note::
+
+ The packet switching back end is for demonstration purposes only.
+ The user could implement their switching logic based on this example.
+ In this example, only one physical port on the host is supported.
+ Multiple segments are not supported. The biggest mbuf supported is 4KB.
+ When the back end is restarted, all front ends must also be restarted.
+
+Running the Application
+-----------------------
+
+The following describes the steps required to run the application.
+
+Validated Environment
+~~~~~~~~~~~~~~~~~~~~~
+
+Host:
+
+ Xen-hypervisor: 4.2.2
+
+ Distribution: Fedora release 18
+
+ Kernel: 3.10.0
+
+ Xen development package (including Xen, Xen-libs, xen-devel): 4.2.3
+
+Guest:
+
+ Distribution: Fedora 16 and 18
+
+ Kernel: 3.6.11
+
+Xen Host Prerequisites
+~~~~~~~~~~~~~~~~~~~~~~
+
+Note that the following commands might not be the same on different Linux* distributions.
+
+* Install xen-devel package:
+
+ .. code-block:: console
+
+ yum install xen-devel.x86_64
+
+* Start xend if not already started:
+
+ .. code-block:: console
+
+ /etc/init.d/xend start
+
+* Mount xenfs if not already mounted:
+
+ .. code-block:: console
+
+ mount -t xenfs none /proc/xen
+
+* Enlarge the limit for xen_gntdev driver:
+
+ .. code-block:: console
+
+ modprobe -r xen_gntdev
+ modprobe xen_gntdev limit=1000000
+
+.. note::
+
+ The default limit for earlier versions of the xen_gntdev driver is 1024.
+ That is insufficient to support the mapping of multiple Virtio devices into multiple VMs,
+ so it is necessary to enlarge the limit by reloading this module.
+ The default limit of recent versions of xen_gntdev is 1048576.
+ The rough calculation of this limit is:
+
+ limit=nb_mbuf# * VM#.
+
+ In DPDK examples, nb_mbuf# is normally 8192.
+
+Building and Running the Switching Backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Edit config/common_linuxapp, and change the default configuration value for the following two items:
+
+ .. code-block:: console
+
+ CONFIG_RTE_LIBRTE_XEN_DOM0=y
+ CONFIG RTE_LIBRTE_PMD_XENVIRT=n
+
+#. Build the target:
+
+ .. code-block:: console
+
+ make install T=x86_64-native-linuxapp-gcc
+
+#. Ensure that RTE_SDK and RTE_TARGET are correctly set. Build the switching example:
+
+ .. code-block:: console
+
+ make -C examples/vhost_xen/
+
+#. Load the Xen DPDK memory management module and preallocate memory:
+
+ .. code-block:: console
+
+ insmod ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/xen_dom0/rte_dom0_mm.ko
+ echo 2048> /sys/kernel/mm/dom0-mm/memsize-mB/memsize
+
+ .. note::
+
+ On Xen Dom0, there is no hugepage support.
+ Under Xen Dom0, the DPDK uses a special memory management kernel module
+ to allocate chunks of physically continuous memory.
+ Refer to the *DPDK Getting Started Guide* for more information on memory management in the DPDK.
+ In the above command, 4 GB memory is reserved (2048 of 2 MB pages) for DPDK.
+
+#. Load uio_pci_generic and bind one Intel NIC controller to it:
+
+ .. code-block:: console
+
+ modprobe uio_pci_generic
+ python tools/dpdk_nic_bind.py -b uio_pci_generic 0000:09:00:00.0
+
+ In this case, 0000:09:00.0 is the PCI address for the NIC controller.
+
+#. Run the switching back end example:
+
+ .. code-block:: console
+
+ examples/vhost_xen/build/vhost-switch -c f -n 3 --xen-dom0 -- -p1
+
+.. note::
+
+ The -xen-dom0 option instructs the DPDK to use the Xen kernel module to allocate memory.
+
+Other Parameters:
+
+* -vm2vm
+
+ The vm2vm parameter enables/disables packet switching in software.
+ Disabling vm2vm implies that on a VM packet transmission will always go to the Ethernet port
+ and will not be switched to another VM
+
+* -Stats
+
+ The Stats parameter controls the printing of Virtio-net device statistics.
+ The parameter specifies the interval (in seconds) at which to print statistics,
+ an interval of 0 seconds will disable printing statistics.
+
+Xen PMD Frontend Prerequisites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Install xen-devel package for accessing XenStore:
+
+ .. code-block:: console
+
+ yum install xen-devel.x86_64
+
+#. Mount xenfs, if it is not already mounted:
+
+ .. code-block:: console
+
+ mount -t xenfs none /proc/xen
+
+#. Enlarge the default limit for xen_gntalloc driver:
+
+ .. code-block:: console
+
+ modprobe -r xen_gntalloc
+ modprobe xen_gntalloc limit=6000
+
+.. note::
+
+ Before the Linux kernel version 3.8-rc5, Jan 15th 2013,
+ a critical defect occurs when a guest is heavily allocating Grant pages.
+ The Grant driver allocates fewer pages than expected which causes kernel memory corruption.
+ This happens, for example, when a guest uses the v1 format of a Grant table entry and allocates
+ more than 8192 Grant pages (this number might be different on different hypervisor versions).
+ To work around this issue, set the limit for gntalloc driver to 6000.
+ (The kernel normally allocates hundreds of Grant pages with one Xen front end per virtualized device).
+ If the kernel allocates a lot of Grant pages, for example, if the user uses multiple net front devices,
+ it is best to upgrade the Grant alloc driver.
+ This defect has been fixed in kernel version 3.8-rc5 and later.
+
+Building and Running the Front End
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Edit config/common_linuxapp, and change the default configuration value:
+
+ .. code-block:: console
+
+ CONFIG_RTE_LIBRTE_XEN_DOM0=n
+ CONFIG_RTE_LIBRTE_PMD_XENVIRT=y
+
+#. Build the package:
+
+ .. code-block:: console
+
+ make install T=x86_64-native-linuxapp-gcc
+
+#. Enable hugepages. Refer to the *DPDK Getting Started Guide* for instructions on
+ how to use hugepages in the DPDK.
+
+#. Run TestPMD. Refer to *DPDK TestPMD Application User Guide* for detailed parameter usage.
+
+ .. code-block:: console
+
+ ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11"
+ testpmd>set fwd mac
+ testpmd>start
+
+ As an example to run two TestPMD instances over 2 Xen Virtio devices:
+
+ .. code-block:: console
+
+ --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" --vdev="eth_xenvirt1;mac=00:00:00:00:00:22"
+
+
+Usage Examples: Injecting a Packet Stream Using a Packet Generator
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Loopback Mode
+^^^^^^^^^^^^^
+
+Run TestPMD in a guest VM:
+
+.. code-block:: console
+
+ ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" -- -i --eth-peer=0,00:00:00:00:00:22
+ testpmd> set fwd mac
+ testpmd> start
+
+Example output of the vhost_switch would be:
+
+.. code-block:: console
+
+ DATA:(0) MAC_ADDRESS 00:00:00:00:00:11 and VLAN_TAG 1000 registered.
+
+The above message indicates that device 0 has been registered with MAC address 00:00:00:00:00:11 and VLAN tag 1000.
+Any packets received on the NIC with these values is placed on the device's receive queue.
+
+Configure a packet stream in the packet generator, set the destination MAC address to 00:00:00:00:00:11, and VLAN to 1000,
+the guest Virtio receives these packets and sends them out with destination MAC address 00:00:00:00:00:22.
+
+Inter-VM Mode
+^^^^^^^^^^^^^
+
+Run TestPMD in guest VM1:
+
+.. code-block:: console
+
+ ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" -- -i --eth-peer=0,00:00:00:00:00:22 -- -i
+
+Run TestPMD in guest VM2:
+
+.. code-block:: console
+
+ ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:22" -- -i --eth-peer=0,00:00:00:00:00:33
+
+Configure a packet stream in the packet generator, and set the destination MAC address to 00:00:00:00:00:11 and VLAN to 1000.
+The packets received in Virtio in guest VM1 will be forwarded to Virtio in guest VM2 and
+then sent out through hardware with destination MAC address 00:00:00:00:00:33.
+
+The packet flow is:
+
+packet generator->Virtio in guest VM1->switching backend->Virtio in guest VM2->switching backend->wire
Code Review / deb_dpdk.git / blobdiff