docs/report/vpp_functional_tests/overview.rst

   1 Overview\r
   2 ========\r
   3 \r
   4 Tested Virtual Topologies\r
   5 -------------------------\r
   6 \r
   7 CSIT VPP functional tests are executed on virtualized topologies created using\r
   8 Virtual Internet Routing Lab (VIRL) simulation platform contributed by Cisco.\r
   9 VIRL runs on physical baremetal servers hosted by LF FD.io project.  Majority\r
  10 of the tests are executed in the three node logical test topology - Traffic\r
  11 Generator (TG) node and two Systems Under Test (SUT) nodes connected in a\r
  12 loop. Some tests use two node logical test topology - TG node and SUT1 node.\r
  13 Both logical test topologies are shown in the figures below.\r
  14 \r
  15 ::\r
  16 \r
  17     +------------------------+           +------------------------+\r
  18     |                        |           |                        |\r
  19     |  +------------------+  |           |  +------------------+  |\r
  20     |  |                  <----------------->                  |  |\r
  21     |  |                  |  |           |  |                  |  |\r
  22     |  |       DUT1       <----------------->       DUT2       |  |\r
  23     |  +--^--^------------+  |           |  +------------^--^--+  |\r
  24     |     |  |               |           |               |  |     |\r
  25     |     |  |         SUT1  |           |  SUT2         |  |     |\r
  26     +------------------------+           +------------------------+\r
  27           |  |                                           |  |\r
  28           |  |                                           |  |\r
  29           |  |               +-----------+               |  |\r
  30           |  +--------------->           <---------------+  |\r
  31           |                  |    TG     |                  |\r
  32           +------------------>           <------------------+\r
  33                              +-----------+\r
  34 \r
  35                        +------------------------+\r
  36                        |                        |\r
  37                        |  +------------------+  |\r
  38           +--------------->                  <--------------+\r
  39           |            |  |                  |  |           |\r
  40           |  |------------>       DUT1       <-----------+  |\r
  41           |  |         |  +--^--^------------+  |        |  |\r
  42           |  |         |                        |        |  |\r
  43           |  |         |                  SUT1  |        |  |\r
  44           |  |         +------------------------+        |  |\r
  45           |  |                                           |  |\r
  46           |  |                                           |  |\r
  47           |  |               +-----------+               |  |\r
  48           |  +--------------->           <---------------+  |\r
  49           |                  |    TG     |                  |\r
  50           +------------------>           <------------------+\r
  51                              +-----------+\r
  52 \r
  53 SUT1 and SUT2 are two VMs (Ubuntu or Centos, depending on the test suite), TG\r
  54 is a Traffic Generator (TG, another Ubuntu VM). SUTs run VPP SW application in\r
  55 Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy SW\r
  56 application as a packet Traffic Generator. Logical connectivity between SUTs\r
  57 and to TG is provided using virtual NICs using VMs' virtio driver.\r
  58 \r
  59 Virtual testbeds are created on-demand whenever a verification job is started\r
  60 (e.g. triggered by the gerrit patch submission) and destroyed upon completion\r
  61 of all functional tests. Each node is a Virtual Machine and each connection\r
  62 that is drawn on the diagram is available for use in any test case. During the\r
  63 test execution, all nodes are reachable thru the Management network connected\r
  64 to every node via dedicated virtual NICs and virtual links (not shown above\r
  65 for clarity).\r
  66 \r
  67 For the test cases that require DUT (VPP) to communicate with VM over the\r
  68 vhost-user interfaces, a nested VM is created on SUT1 and/or SUT2 for the\r
  69 duration of these particular test cases only. DUT (VPP) test topology with VM\r
  70 is shown in the figure below including the applicable packet flow thru the VM\r
  71 (marked in the figure with ``***``).\r
  72 \r
  73 ::\r
  74 \r
  75     +------------------------+           +------------------------+\r
  76     |      +----------+      |           |      +----------+      |\r
  77     |      |    VM    |      |           |      |    VM    |      |\r
  78     |      |  ******  |      |           |      |  ******  |      |\r
  79     |      +--^----^--+      |           |      +--^----^--+      |\r
  80     |        *|    |*        |           |        *|    |*        |\r
  81     |  +------v----v------+  |           |  +------v----v------+  |\r
  82     |  |      *    *      |**|***********|**|      *    *      |  |\r
  83     |  |  *****    *******<----------------->*******    *****  |  |\r
  84     |  |  *    DUT1       |  |           |  |       DUT2    *  |  |\r
  85     |  +--^---------------+  |           |  +---------------^--+  |\r
  86     |    *|                  |           |                  |*    |\r
  87     |    *|            SUT1  |           |  SUT2            |*    |\r
  88     +------------------------+           +------------------^-----+\r
  89          *|                                                 |*\r
  90          *|                                                 |*\r
  91          *|                  +-----------+                  |*\r
  92          *|                  |           |                  |*\r
  93          *+------------------>    TG     <------------------+*\r
  94          ******************* |           |********************\r
  95                              +-----------+\r
  96 \r
  97 Functional Tests Coverage\r
  98 -------------------------\r
  99 \r
 100 Following VPP functional test areas are covered in the CSIT |release| with\r
 101 results listed in this report:\r
 102 \r
 103 - **DHCP - Client and Proxy** - Dynamic Host Control Protocol Client and Proxy\r
 104   for IPv4, IPv6.\r
 105 - **GRE Overlay Tunnels** - Generic Routing Encapsulation for IPv4.\r
 106 - **L2BD Ethernet Switching** - L2 Bridge-Domain switched-forwarding for\r
 107   untagged Ethernet, dot1q and dot1ad tagged.\r
 108 - **L2XC Ethernet Switching** - L2 Cross-Connect switched-forwarding for\r
 109   untagged Ethernet, dot1q and dot1ad tagged.\r
 110 - **LISP Overlay Tunnels** - Locator/ID Separation Protocol overlay tunnels and\r
 111   locator/id mapping control.\r
 112 - **Softwire Tunnels** - IPv4-in-IPv6 softwire tunnels.\r
 113 - **Cop Address Security** - address white-list and black-list filtering for\r
 114   IPv4, IPv6.\r
 115 - **IPSec - Tunnels and Transport** - IPSec tunnel and transport modes.\r
 116 - **IPv6 Routed-Forwarding** - IPv6 routed-forwarding, NS/ND, RA, ICMPv6.\r
 117 - **uRPF Source Security** - unicast Reverse Path Forwarding security.\r
 118 - **Tap Interface** - baseline Linux tap interface tests.\r
 119 - **Telemetry - IPFIX and SPAN** - IPFIX netflow statistics and SPAN port\r
 120   mirroring.\r
 121 - **VRF Routed-Forwarding** - multi-context IPVPN routed-forwarding for IPv4,\r
 122   IPv6.\r
 123 - **iACL Security** - ingress Access Control List security for IPv4, IPv6, MAC.\r
 124 - **IPv4 Routed-Forwarding** - IPv4 routed-forwarding, RPF, ARP, Proxy ARP,\r
 125   ICMPv4.\r
 126 - **QoS Policer Metering** - ingress packet rate measuring and marking for IPv4,\r
 127   IPv6.\r
 128 - **VLAN Tag Translation** - L2 VLAN tag translation 2to2, 2to1, 1to2, 1to1.\r
 129 - **VXLAN Overlay Tunnels** - VXLAN tunneling for L2-over-IP, for IPv4, IPv6.\r
 130 \r
 131 Functional Tests Naming\r
 132 -----------------------\r
 133 \r
 134 CSIT |release| introduced a common structured naming convention for all\r
 135 performance and functional tests. This change was driven by substantially\r
 136 growing number and type of CSIT test cases. Firstly, the original practice did\r
 137 not always follow any strict naming convention. Secondly test names did not\r
 138 always clearly capture tested packet encapsulations, and the actual type or\r
 139 content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
 140 their locality were not captured either. These were but few reasons that drove\r
 141 the decision to change and define a new more complete and stricter test naming\r
 142 convention, and to apply this to all existing and new test cases.\r
 143 \r
 144 The new naming should be intuitive for majority of the tests. The complete\r
 145 description of CSIT test naming convention is provided on `CSIT test naming\r
 146 page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
 147 \r
 148 Here few illustrative examples of the new naming usage for functional test\r
 149 suites:\r
 150 \r
 151 #. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
 152 \r
 153     - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
 154       routed forwarding, functional tests.\r
 155 \r
 156 #. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
 157    P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
 158 \r
 159     - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
 160       Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
 161       interfaces and one VM, functional tests.\r