4 This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous
5 **S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT
6 |release| system testing of |vpp-release|.
8 This is the full html version, there is also a reduced
9 `PDF version of this report`_.
11 The report describes CSIT functional and performance tests and their
12 continuous execution delivered in CSIT |release|. A high-level overview is
13 provided for each CSIT test environment running in :abbr:`LF (Linux Foundation)`
14 FD.io Continuous Performance Labs. This is followed by summary of all executed
15 tests against the |vpp-release| and associated FD.io projects and sub-systems
16 (Honeycomb, DPDK, NSH_SFC), CSIT |release| release notes, result highlights and
17 known issues discovered in CSIT. More detailed description of each environment,
18 pointers to CSIT test code documentation and detailed test resuls with links to
19 the source data files are also provided.
21 CSIT |release| report contains following main sections and sub-sections:
23 #. **Introduction** - general introduction to CSIT project; *Overview* -
24 this section; *CSIT Test Naming* - CSIT general naming convention for test
25 suites and test cases, important to recognize the high-level test content
26 and interpret reported results; *General Notes* - general notes related to
29 #. **VPP Performance Tests** - VPP performance tests executed in physical
30 FD.io testbeds; *Overview* - tested topologies, test coverage and naming
31 specifics, methodology for multi-core, packet throughput and latency, and
32 KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added
33 tests, performance changes, environment or methodology changes, known CSIT
34 issues; *Packet Throughput Graphs* and *Packet Latency
35 Graphs* - plotted NDR, PDR throughput and latency results from multiple
36 test job executions; *Throughput Speedup Multi-Core* - plotted core
37 configuration speedup comparision; *Test Environment* - environment
38 description; *VPP HTTP Server Performance Results* - plotted HTTP Server
39 performance; *Documentation* - CSIT source code documentation for VPP
42 #. **DPDK Performance Tests** - DPDK performance tests executed in
43 physical FD.io testbeds; *Overview* - tested topologies, test coverage;
44 *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues;
45 *Packet Throughput Graphs* and *Packet Latency Graphs*
46 - plotted NDR, PDR throughput and latency results from multiple test job
47 executions; *Test Environment* - environment description; *Documentation* -
48 CSIT source code documentation for DPDK performance tests.
50 #. **VPP Functional Tests** - VPP functional tests executed in virtual
51 FD.io testbeds; *Overview* - tested virtual topologies, test coverage and
52 naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added
53 tests, environment or methodology changes, known CSIT issues, tests to be
54 added; *Test Environment* - environment description ; *Documentation* -
55 source code documentation for VPP functional tests.
57 #. **Honeycomb Functional Tests** - Honeycomb functional tests executed in
58 virtual FD.io testbeds; *Overview* - tested virtual topologies, test
59 coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
60 |release|, added tests, environment or methodology changes, known CSIT issues;
61 *Test Environment* - environment description ;
62 *Documentation* - source code documentation for Honeycomb functional tests.
64 #. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in
65 virtual FD.io testbeds; *Overview* - tested virtual topologies, test
66 coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
67 |release|, added tests, environment or methodology changes, known CSIT issues;
68 *Test Environment* - environment description ;
69 *Documentation* - source code documentation for NSH_SFC functional tests.
71 #. **Detailed Test Results** - auto-generated results from CSIT jobs
72 executions using CSIT Robot Framework output files as source data; *VPP
73 Performance Results*, *DPDK Performance Results*, *VPP Functional
74 Results*, *Honeycomb Functional Results*, *VPPtest Functional Results*.
76 #. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs
77 executions using CSIT Robot Framework output files as source data; *VPP
78 Performance Test Configs*, *VPP Functional Test Configs*.
80 #. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs
81 executions using CSIT Robot Framework output files as source data; *VPP
82 Performance Operational Data*.
84 #. **CSIT Framework Documentation** - description of the overall CSIT
85 framework design hierarchy, CSIT test naming convention, followed by
86 description of Presentation and Analytics Layer (PAL) introduced in