CSIT-744 Update report content for proper parsing
[csit.git] / docs / report / honeycomb_performance_tests / overview.rst
1 Overview
2 ========
3
4 Tested Physical Topologies
5 --------------------------
6
7 CSIT VPP performance tests are executed on physical baremetal servers hosted by
8 LF FD.io project. Testbed physical topology is shown in the figure below.::
9
10     +------------------------+           +------------------------+
11     |                        |           |                        |
12     |  +------------------+  |           |  +------------------+  |
13     |  |                  |  |           |  |                  |  |
14     |  |                  <----------------->                  |  |
15     |  |       DUT1       |  |           |  |       DUT2       |  |
16     |  +--^---------------+  |           |  +---------------^--+  |
17     |     |                  |           |                  |     |
18     |     |            SUT1  |           |  SUT2            |     |
19     +------------------------+           +------------------^-----+
20           |                                                 |
21           |                                                 |
22           |                  +-----------+                  |
23           |                  |           |                  |
24           +------------------>    TG     <------------------+
25                              |           |
26                              +-----------+
27
28 SUT1 runs VPP SW application in Linux user-mode as a
29 Device Under Test (DUT), and a python script to generate traffic. SUT2 and TG
30 are unused.
31 sical connectivity between SUTs and to TG is provided using
32 different NIC model. Currently installed NIC models include:
33
34 Performance tests involve sending Netconf requests over localhost to the
35 Honeycomb listener port, and measuring response time.
36
37 Note that reported performance results are specific to the SUTs tested.
38 Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with
39 other CPUs are likely to yield different results.
40
41 For detailed LF FD.io test bed specification and physical topology please refer
42 to `LF FDio CSIT testbed wiki page
43 <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
44
45 Performance Tests Coverage
46 --------------------------
47
48 As of right now, there is only a single Honeycomb performance test. Measuring
49 response time for a simple read operation, performed synchronously and using
50 single (not batch) requests.
51
52 Currently the tests do not trigger automatically, but can be run on-demand from
53 the hc2vpp project.
54
55 Performance Tests Naming
56 ------------------------
57
58 CSIT |release| follows a common structured naming convention for all
59 performance and system functional tests, introduced in CSIT |release-1|.
60
61 The naming should be intuitive for majority of the tests. Complete
62 description of CSIT test naming convention is provided on `CSIT test naming wiki
63 <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
64
65 Here few illustrative examples of the new naming usage for performance test
66 suites:
67
68 #. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
69
70    - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
71      PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
72    - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
73      Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
74      with MAC learning, NDR throughput discovery.
75    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
76      on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
77      switching with MAC learning, NDR throughput discovery.
78    - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
79      x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
80    - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
81      Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
82      discovery.
83
84 #. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
85    P2V2P, NIC2VMchain2NIC, P2V2V2P**
86
87    - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
88      PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
89      VirtPortConfig-VMconfig-TestType*
90    - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
91      of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
92      switching to/from two vhost interfaces and one VM, NDR throughput
93      discovery.
94    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
95      ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
96      switching to/from two vhost interfaces and one VM, NDR throughput
97      discovery.
98    - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
99      ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
100      switching to/from four vhost interfaces and two VMs, NDR throughput
101      discovery.
102
103 Methodology: Multi-Core
104 -----------------------
105
106 **Multi-core Test** - CSIT |release| multi-core tests are executed in the
107 following thread and core configurations:
108
109 #. 1t - 1 Honeycomb Netconf thread on 1 CPU physical core.
110 #. 8t - 8 Honeycomb Netconf thread on 8 CPU physical core.
111 #. 16t - 16 Honeycomb Netconf thread on 16 CPU physical core.
112
113 Traffic generator also uses multiple threads/cores, to simulate multiple
114 Netconf clients accessing the Honeycomb server.
115
116 Methodology: Performance measurement
117 ------------------------------------
118
119 The following values are measured and reported in tests:
120
121 - Average request rate. Averaged over the entire test duration, over all client
122   threads. Negative replies (if any) are not counted and are reported separately.

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.