docs/report/vpp_performance_tests/csit_release_notes.rst

   1 CSIT Release Notes\r
   2 ==================\r
   3 \r
   4 Changes in CSIT |release|\r
   5 -------------------------\r
   6 \r
   7 #. Added VPP performance tests\r
   8 \r
   9    - **Linux Container VPP memif tests**\r
  10 \r
  11      - Tests with VPP in L2 Bridge-Domain configuration connecting over\r
  12        memif virtual interfaces to VPPs running in LXCs;\r
  13 \r
  14    - **Docker Container VPP memif tests**\r
  15 \r
  16      - Tests with VPP in L2 Cross-Connect configuration connecting over\r
  17        memif virtual interfaces VPPs running in Docker containers;\r
  18 \r
  19    - **Container Topologies Orchestrated by K8s with VPP memif tests**\r
  20 \r
  21      - Tests with VPP in L2 Cross-Connect and Bridge-Domain configurations\r
  22        connecting over memif virtual interfaces VPPs running in Docker\r
  23        containers, with service chain topologies orchestrated by Kubernetes;\r
  24 \r
  25    - **Stateful Security Groups**\r
  26 \r
  27      - m-thread m-core VPP stateful and stateless security-groups tests;\r
  28 \r
  29    - **MAC-IP binding**\r
  30 \r
  31      - MACIP input access-lists, single-thread single-core and m-thread\r
  32        m-core tests;\r
  33 \r
  34      - Statistical analysis of repeatibility of results;\r
  35 \r
  36 Performance Improvements\r
  37 ------------------------\r
  38 \r
  39 Substantial improvements in measured packet throughput have been observed in a\r
  40 number of CSIT |release| tests listed below, with relative increase of\r
  41 double-digit percentage points. Relative improvements for this release are\r
  42 calculated against the test results listed in CSIT |release-1| report. The\r
  43 comparison is calculated between the mean values based on collected and\r
  44 archived test results' samples for involved VPP releases. Standard deviation\r
  45 has been also listed for CSIT |release|. Performance numbers since release\r
  46 VPP-16.09 are provided for reference.\r
  47 \r
  48 NDR Throughput\r
  49 ~~~~~~~~~~~~~~\r
  50 \r
  51 Non-Drop Rate Throughput discovery tests:\r
  52 \r
  53 .. only:: html\r
  54 \r
  55    .. csv-table::\r
  56       :align: center\r
  57       :file: performance_improvements/performance_improvements_ndr_top.csv\r
  58 \r
  59 .. only:: latex\r
  60 \r
  61    .. raw:: latex\r
  62 \r
  63       \makeatletter\r
  64       \csvset{\r
  65         perfimprovements column width/.style={after head=\csv@pretable\begin{longtable}{ m{1.5cm} m{4cm} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1}}\csv@tablehead},\r
  66       }\r
  67       \makeatother\r
  68 \r
  69       {\tiny\r
  70       \csvautobooklongtable[separator=comma,\r
  71         respect all,\r
  72         no check column count,\r
  73         perfimprovements column width=1cm,\r
  74         late after line={\\\hline},\r
  75         late after last line={\end{longtable}}\r
  76         ]{../_tmp/src/vpp_performance_tests/performance_improvements/performance_improvements_ndr_top.csv}\r
  77       }\r
  78 \r
  79 \r
  80 PDR Throughput\r
  81 ~~~~~~~~~~~~~~\r
  82 \r
  83 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
  84 \r
  85 .. only:: html\r
  86 \r
  87    .. csv-table::\r
  88       :align: center\r
  89       :file: performance_improvements/performance_improvements_pdr_top.csv\r
  90 \r
  91 .. only:: latex\r
  92 \r
  93    .. raw:: latex\r
  94 \r
  95       \makeatletter\r
  96       \csvset{\r
  97         perfimprovements column width/.style={after head=\csv@pretable\begin{longtable}{ m{1.5cm} m{4cm} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1}}\csv@tablehead},\r
  98       }\r
  99       \makeatother\r
 100 \r
 101       {\tiny\r
 102       \csvautobooklongtable[separator=comma,\r
 103         respect all,\r
 104         no check column count,\r
 105         perfimprovements column width=1cm,\r
 106         late after line={\\\hline},\r
 107         late after last line={\end{longtable}}\r
 108         ]{../_tmp/src/vpp_performance_tests/performance_improvements/performance_improvements_pdr_top.csv}\r
 109       }\r
 110 \r
 111 \r
 112 Measured improvements are in line with VPP code optimizations listed in\r
 113 `VPP-17.10 release notes\r
 114 <https://docs.fd.io/vpp/17.10/release_notes_1710.html>`_.\r
 115 \r
 116 Other Performance Changes\r
 117 -------------------------\r
 118 \r
 119 Other changes in measured packet throughput, with either minor relative increase\r
 120 or decrease, have been observed in a number of CSIT |release| tests listed\r
 121 below. Relative changes are calculated against the test results listed in CSIT\r
 122 |release-1| report.\r
 123 \r
 124 NDR Throughput\r
 125 ~~~~~~~~~~~~~~\r
 126 \r
 127 Non-Drop Rate Throughput discovery tests:\r
 128 \r
 129 .. only:: html\r
 130 \r
 131    .. csv-table::\r
 132       :align: center\r
 133       :file: performance_improvements/performance_improvements_ndr_low.csv\r
 134 \r
 135 .. only:: latex\r
 136 \r
 137    .. raw:: latex\r
 138 \r
 139       \makeatletter\r
 140       \csvset{\r
 141         perfimprovements column width/.style={after head=\csv@pretable\begin{longtable}{ m{1.5cm} m{4cm} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1}}\csv@tablehead},\r
 142       }\r
 143       \makeatother\r
 144 \r
 145       {\tiny\r
 146       \csvautobooklongtable[separator=comma,\r
 147         respect all,\r
 148         no check column count,\r
 149         perfimprovements column width=1cm,\r
 150         late after line={\\\hline},\r
 151         late after last line={\end{longtable}}\r
 152         ]{../_tmp/src/vpp_performance_tests/performance_improvements/performance_improvements_ndr_low.csv}\r
 153       }\r
 154 \r
 155 \r
 156 PDR Throughput\r
 157 ~~~~~~~~~~~~~~\r
 158 \r
 159 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
 160 \r
 161 .. only:: html\r
 162 \r
 163    .. csv-table::\r
 164       :align: center\r
 165       :file: performance_improvements/performance_improvements_pdr_low.csv\r
 166 \r
 167 .. only:: latex\r
 168 \r
 169    .. raw:: latex\r
 170 \r
 171       \makeatletter\r
 172       \csvset{\r
 173         perfimprovements column width/.style={after head=\csv@pretable\begin{longtable}{ m{1.5cm} m{4cm} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1} m{#1}}\csv@tablehead},\r
 174       }\r
 175       \makeatother\r
 176 \r
 177       {\tiny\r
 178       \csvautobooklongtable[separator=comma,\r
 179         respect all,\r
 180         no check column count,\r
 181         perfimprovements column width=1cm,\r
 182         late after line={\\\hline},\r
 183         late after last line={\end{longtable}}\r
 184         ]{../_tmp/src/vpp_performance_tests/performance_improvements/performance_improvements_pdr_low.csv}\r
 185       }\r
 186 \r
 187 \r
 188 Known Issues\r
 189 ------------\r
 190 \r
 191 Here is the list of known issues in CSIT |release| for VPP performance tests:\r
 192 \r
 193 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 194 | # | Issue                                           | Jira ID    | Description                                                     |\r
 195 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 196 | 1 | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |\r
 197 |   |                                                 |            |                                                                 |\r
 198 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 199 | 2 | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 200 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 201 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 202 | 3 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
 203 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
 204 |   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |\r
 205 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 206 | 4 | Lower than expected NDR throughput with         | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
 207 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r
 208 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 209 \r