docs/report/honeycomb_performance_tests/overview.rst

   1 Overview
   2 ========
   3
   4 Tested Physical Topologies
   5 --------------------------
   6
   7 CSIT VPP performance tests are executed on physical baremetal servers hosted by
   8 LF FD.io project. Testbed physical topology is shown in the figure below.::
   9
  10     +------------------------+           +------------------------+
  11     |                        |           |                        |
  12     |  +------------------+  |           |  +------------------+  |
  13     |  |                  |  |           |  |                  |  |
  14     |  |                  <----------------->                  |  |
  15     |  |       DUT1       |  |           |  |       DUT2       |  |
  16     |  +--^---------------+  |           |  +---------------^--+  |
  17     |     |                  |           |                  |     |
  18     |     |            SUT1  |           |  SUT2            |     |
  19     +------------------------+           +------------------^-----+
  20           |                                                 |
  21           |                                                 |
  22           |                  +-----------+                  |
  23           |                  |           |                  |
  24           +------------------>    TG     <------------------+
  25                              |           |
  26                              +-----------+
  27
  28 SUT1 runs VPP SW application in Linux user-mode as a
  29 Device Under Test (DUT), and a python script to generate traffic. SUT2 and TG
  30 are unused.
  31 sical connectivity between SUTs and to TG is provided using
  32 different NIC model. Currently installed NIC models include:
  33
  34 Performance tests involve sending Netconf requests over localhost to the
  35 Honeycomb listener port, and measuring response time.
  36
  37 Note that reported performance results are specific to the SUTs tested.
  38 Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with
  39 other CPUs are likely to yield different results.
  40
  41 For detailed LF FD.io test bed specification and physical topology please refer
  42 to `LF FDio CSIT testbed wiki page
  43 <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
  44
  45 Performance Tests Coverage
  46 --------------------------
  47
  48 As of right now, there is only a single Honeycomb performance test. Measuring
  49 response time for a simple read operation, performed synchronously and using
  50 single (not batch) requests.
  51
  52 Currently the tests do not trigger automatically, but can be run on-demand from
  53 the hc2vpp project.
  54
  55 Performance Tests Naming
  56 ------------------------
  57
  58 CSIT |release| follows a common structured naming convention for all
  59 performance and system functional tests, introduced in CSIT |release-1|.
  60
  61 The naming should be intuitive for majority of the tests. Complete
  62 description of CSIT test naming convention is provided on `CSIT test naming wiki
  63 <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
  64
  65 Here few illustrative examples of the new naming usage for performance test
  66 suites:
  67
  68 #. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
  69
  70    - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
  71      PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
  72    - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
  73      Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
  74      with MAC learning, NDR throughput discovery.
  75    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
  76      on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
  77      switching with MAC learning, NDR throughput discovery.
  78    - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
  79      x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
  80    - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
  81      Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
  82      discovery.
  83
  84 #. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
  85    P2V2P, NIC2VMchain2NIC, P2V2V2P**
  86
  87    - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
  88      PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
  89      VirtPortConfig-VMconfig-TestType*
  90    - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
  91      of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
  92      switching to/from two vhost interfaces and one VM, NDR throughput
  93      discovery.
  94    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
  95      ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
  96      switching to/from two vhost interfaces and one VM, NDR throughput
  97      discovery.
  98    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
  99      ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
 100      switching to/from four vhost interfaces and two VMs, NDR throughput
 101      discovery.
 102
 103 Methodology: Multi-Core
 104 -----------------------
 105
 106 **Multi-core Test** - CSIT |release| multi-core tests are executed in the
 107 following thread and core configurations:
 108
 109 #. 1t - 1 Honeycomb Netconf thread on 1 CPU physical core.
 110 #. 8t - 8 Honeycomb Netconf thread on 8 CPU physical core.
 111 #. 16t - 16 Honeycomb Netconf thread on 16 CPU physical core.
 112
 113 Traffic generator also uses multiple threads/cores, to simulate multiple
 114 Netconf clients accessing the Honeycomb server.
 115
 116 Methodology: Performance measurement
 117 ------------------------------------
 118
 119 The following values are measured and reported in tests:
 120
 121 - Average request rate. Averaged over the entire test duration, over all client
 122   threads. Negative replies (if any) are not counted and are reported separately.