docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. **VPP performance tests**
   8
   9    - *MRR tests* - New Maximum Receive Rate tests measure the packet
  10      forwarding rate under the maximum load offered by traffic
  11      generator over a set trial duration, regardless of packet loss.
  12      MRR tests are used for continuous performance trending and for
  13      comparison between releases.
  14
  15    - *Service Chaining with SRv6* - New SRv6 (Segment Routing IPv6) proxy
  16      tests measure performance of SRv6 Endpoint fronting SR-unaware
  17      appliance via masquerading (End.AM), dynamic proxy (End.AD) or
  18      static proxy (End.AS) SR functions.
  19
  20 #. **Presentation and Analytics Layer**
  21
  22    - *Performance trending* - Added continuous performance trending and
  23      analysis. New Performance Trending and Performance Analysis jobs
  24      executed regular throughput tests, with results being subsequently
  25      analysed and trend and anomalies summarized and presented in VPP
  26      Performance Dashboard and trendline graphs.
  27
  28 #. **Test Framework Optimizations**
  29
  30    - *Performance tests efficiency* - Qemu build/install optimizations,
  31      warmup phase handling, vpp restart handling. Resulted in improved
  32      stability and reduced total execution time by 30% for single pkt
  33      size e.g. 64B/78B.
  34
  35    - *General code housekeeping* - ongoing RF keywords optimizations,
  36      removal of redundant RF keywords.
  37
  38 Performance Changes
  39 -------------------
  40
  41 Relative performance changes in measured packet throughput in CSIT
  42 |release| are calculated against the results from CSIT |release-1|
  43 report. Listed mean and standard deviation values are computed based on
  44 a series of the same tests executed against respective VPP releases to
  45 verify test results repeatibility, with percentage change calculated for
  46 mean values. Note that the standard deviation is quite high for a small
  47 number of packet throughput tests, what indicates poor test results
  48 repeatability and makes the relative change of mean throughput value not
  49 fully representative for these tests. The root causes behind poor
  50 results repeatibility vary between the test cases.
  51
  52 NDR Changes
  53 ~~~~~~~~~~~
  54
  55 NDR small packet throughput changes between releases are available in a
  56 CSV and pretty ASCII formats:
  57
  58   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_,
  59   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_,
  60   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_,
  61   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_.
  62
  63 .. note::
  64
  65     Test results have been generated by
  66     `FD.io test executor vpp performance jobs`_ with Robot Framework result
  67     files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_.
  68
  69 PDR Changes
  70 ~~~~~~~~~~~
  71
  72 NDR small packet throughput changes between releases are available in a
  73 CSV and pretty ASCII formats:
  74
  75   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_,
  76   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_,
  77   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_,
  78   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_.
  79
  80 .. note::
  81
  82     Test results have been generated by
  83     `FD.io test executor vpp performance jobs`_ with Robot Framework result
  84     files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_.
  85
  86 MRR Changes
  87 ~~~~~~~~~~~
  88
  89 MRR small packet throughput changes between releases are available in a
  90 CSV and pretty ASCII formats:
  91
  92   - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_,
  93   - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_,
  94   - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_,
  95   - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_,
  96   - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_,
  97   - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_.
  98
  99 .. note::
 100
 101     Test results have been generated by
 102     `FD.io test executor vpp mrr jobs <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master/>`_
 103     with Robot Framework result files csit-vpp-perf-mrr-daily-master__*__output.xml.gz
 104     `archived here <https://docs.fd.io/csit/master/trending/_static/archive/>`_.
 105
 106 Comparison Across Testbeds
 107 --------------------------
 108
 109 .. warning::
 110
 111     TODO: Add:
 112
 113     Table 1.
 114     Test Case     3-Node Hsw    3-Node Skx    Skx vs. Hsw Delta [%]
 115
 116     Table 2.
 117     Test Case     3-Node Skx    2-Node Skx    2-Node vs. 3-Node Delta [%]
 118
 119 Throughput Trending
 120 -------------------
 121
 122 In addition to reporting throughput changes between VPP releases, CSIT
 123 provides continuous performance trending for VPP master branch:
 124
 125 #. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_
 126    - per VPP test case throughput trend, trend compliance and summary of
 127    detected anomalies.
 128
 129 #. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_
 130    - throughput test metrics, trend calculations and anomaly
 131    classification (progression, regression, outlier).
 132
 133 #. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_
 134    - per VPP build MRR throughput measurements against the trendline
 135    with anomaly highlights, with associated CSIT test jobs.
 136
 137 Known Issues
 138 ------------
 139
 140 List of known issues in |csit-release| for VPP performance tests:
 141
 142 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 143 | # | Issue                                           | Jira ID    | Description                                                     |
 144 +===+=================================================+============+=================================================================+
 145 | 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570   | DPDK reporting rx-errors, indicating L1 issue. Suspected issue  |
 146 |   | on x520.                                        |            | with HW combination of X710-X520 in LF testbeds. Not observed   |
 147 |   |                                                 |            | outside of LF testbeds.                                         |
 148 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 149 | 2 | Lower than expected NDR throughput of DPDK      | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |
 150 |   | testpmd and VPP L2 path NDR throughput with     |            | PDR throughput on XL710 and X710 NICs.                          |
 151 |   | xl710 and x710 NICs, compared to x520 NICs.     |            |                                                                 |
 152 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 153 | 3 | Tagged Ethernet dot1q and dot1ad L2 path        | CSIT-1066  | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: |
 154 |   | throughput regression.                          |            | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd          |
 155 |   |                                                 |            | performance tests.                                              |
 156 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 157 | 4 | IPSec (software, no QAT HW) throughput          | CSIT-1064  | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR    |
 158 |   | regression.                                     |            | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected.    |
 159 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 160 | 5 | High failure rate of creating working container | CSIT-1065  | About 20% of orchestrated container topology tests failing data |
 161 |   | topologies with K8s/Ligato orchestration.       |            | plane verification indicating configuration issue. Suspected    |
 162 |   |                                                 |            | issue with Ligato vpp-agent.                                    |
 163 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+