docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. **VPP performance tests**
   8
   9    - *MRR tests* - New Maximum Receive Rate tests measure the packet
  10      forwarding rate under the maximum load offered by traffic
  11      generator over a set trial duration, regardless of packet loss.
  12      MRR tests are used for continuous performance trending and for
  13      comparison between releases.
  14
  15    - *Service Chaining with SRv6* - New SRv6 (Segment Routing IPv6) proxy
  16      tests measure performance of SRv6 Endpoint fronting SR-unaware
  17      appliance via masquerading (End.AM), dynamic proxy (End.AD) or
  18      static proxy (End.AS) SR functions.
  19
  20 #. **Presentation and Analytics Layer**
  21
  22    - *Performance trending* - Added continuous performance trending and
  23      analysis. New Performance Trending and Performance Analysis jobs
  24      executed regular throughput tests, with results being subsequently
  25      analysed and trend and anomalies summarized and presented in VPP
  26      Performance Dashboard and trendline graphs.
  27
  28 #. **Test Framework Optimizations**
  29
  30    - *Performance tests efficiency* - Qemu build/install optimizations,
  31      warmup phase handling, vpp restart handling. Resulted in improved
  32      stability and reduced total execution time by 30% for single pkt
  33      size e.g. 64B/78B.
  34
  35    - *General code housekeeping* - ongoing RF keywords optimizations,
  36      removal of redundant RF keywords.
  37
  38 Performance Changes
  39 -------------------
  40
  41 Relative performance changes in measured packet throughput in |csit-release|
  42 are calculated against the results from |csit-release-1|
  43 report. Listed mean and standard deviation values are computed based on
  44 a series of the same tests executed against respective VPP releases to
  45 verify test results repeatability, with percentage change calculated for
  46 mean values. Note that the standard deviation is quite high for a small
  47 number of packet throughput tests, what indicates poor test results
  48 repeatability and makes the relative change of mean throughput value not
  49 fully representative for these tests. The root causes behind poor
  50 results repeatability vary between the test cases.
  51
  52 NDR Changes
  53 ~~~~~~~~~~~
  54
  55 NDR throughput changes between releases are available in a
  56 CSV and pretty ASCII formats:
  57
  58   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_,
  59   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_,
  60   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_,
  61   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_.
  62
  63 .. note::
  64
  65     Test results have been generated by
  66     `FD.io test executor vpp performance job 3n-hsw`_,
  67     with Robot Framework result
  68     files csit-vpp-perf-|srelease|-\*.zip
  69     `archived here <../_static/archive/>`_.
  70
  71 PDR Changes
  72 ~~~~~~~~~~~
  73
  74 PDR throughput changes between releases are available in a
  75 CSV and pretty ASCII formats:
  76
  77   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_,
  78   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_,
  79   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_,
  80   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_.
  81
  82 .. note::
  83
  84     Test results have been generated by
  85     `FD.io test executor vpp performance job 3n-hsw`_,
  86     with Robot Framework result
  87     files csit-vpp-perf-|srelease|-\*.zip
  88     `archived here <../_static/archive/>`_.
  89
  90 MRR Changes
  91 ~~~~~~~~~~~
  92
  93 MRR small packet throughput changes between releases are available in a
  94 CSV and pretty ASCII formats:
  95
  96   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_,
  97   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_,
  98   - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_,
  99   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_,
 100   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_,
 101   - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_.
 102
 103 .. note::
 104
 105     Test results have been generated by
 106     `FD.io test executor vpp mrr jobs <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master/>`_
 107     with Robot Framework result files csit-vpp-perf-mrr-daily-master__*__output.xml.gz
 108     `archived here <https://docs.fd.io/csit/master/trending/_static/archive/>`_.
 109
 110 Comparison Across Testbeds
 111 --------------------------
 112
 113 Relative performance changes in measured packet throughputon 3-Node Skx testbed
 114 are calculated against the results measured on 3-Node Hsw testbed.
 115
 116 NDR Changes
 117 ~~~~~~~~~~~
 118
 119 NDR changes between testbeds are available in a
 120 CSV and pretty ASCII formats:
 121
 122   - `csv format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
 123   - `pretty ASCII format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.
 124
 125 .. note::
 126
 127     Test results have been generated by
 128     `FD.io test executor vpp performance job 3n-hsw`_ and
 129     `FD.io test executor vpp performance job 3n-skx`_
 130     with Robot Framework result
 131     files csit-vpp-perf-|srelease|-\*.zip
 132     `archived here <../_static/archive/>`_.
 133
 134 PDR Changes
 135 ~~~~~~~~~~~
 136
 137 PDR throughput changes between testbeds are available in a
 138 CSV and pretty ASCII formats:
 139
 140   - `csv format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
 141   - `pretty ASCII format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.
 142
 143 .. note::
 144
 145     Test results have been generated by
 146     `FD.io test executor vpp performance job 3n-hsw`_ and
 147     `FD.io test executor vpp performance job 3n-skx`_
 148     with Robot Framework result
 149     files csit-vpp-perf-|srelease|-\*.zip
 150     `archived here <../_static/archive/>`_.
 151
 152 Throughput Trending
 153 -------------------
 154
 155 In addition to reporting throughput changes between VPP releases, CSIT
 156 provides continuous performance trending for VPP master branch:
 157
 158 #. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_
 159    - per VPP test case throughput trend, trend compliance and summary of
 160    detected anomalies.
 161
 162 #. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_
 163    - throughput test metrics, trend calculations and anomaly
 164    classification (progression, regression, outlier).
 165
 166 #. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_
 167    - per VPP build MRR throughput measurements against the trendline
 168    with anomaly highlights, with associated CSIT test jobs.
 169
 170 Known Issues
 171 ------------
 172
 173 List of known issues in |csit-release| for VPP performance tests:
 174
 175 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 176 | # | Issue                                           | Jira ID    | Description                                                     |
 177 +===+=================================================+============+=================================================================+
 178 | 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570   | DPDK reporting rx-errors, indicating L1 issue. Suspected issue  |
 179 |   | on x520.                                        |            | with HW combination of X710-X520 in LF testbeds. Not observed   |
 180 |   |                                                 |            | outside of LF testbeds.                                         |
 181 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 182 | 2 | Lower than expected NDR throughput of DPDK      | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |
 183 |   | testpmd and VPP L2 path NDR throughput with     |            | PDR throughput on XL710 and X710 NICs.                          |
 184 |   | xl710 and x710 NICs, compared to x520 NICs.     |            |                                                                 |
 185 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 186 | 3 | Tagged Ethernet dot1q and dot1ad L2 path        | CSIT-1066  | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: |
 187 |   | throughput regression.                          |            | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd          |
 188 |   |                                                 |            | performance tests.                                              |
 189 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 190 | 4 | IPSec (software, no QAT HW) throughput          | CSIT-1064  | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR    |
 191 |   | regression.                                     |            | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected.    |
 192 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 193 | 5 | High failure rate of creating working container | CSIT-1065  | About 20% of orchestrated container topology tests failing data |
 194 |   | topologies with K8s/Ligato orchestration.       |            | plane verification indicating configuration issue. Suspected    |
 195 |   |                                                 |            | issue with Ligato vpp-agent.                                    |
 196 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+