4 This is FD.io CSIT (Fast Data i/o Continuous System Integration and
5 Testing) project report for |csit-release| system performance and
6 functional testing of |vpp-release|.
8 There is also a downloadable `PDF version of this report`_.
10 This report describes CSIT performance and functional tests delivered in
11 |csit-release|. A high-level overview is provided for each CSIT test
12 environment running in :abbr:`LF (Linux Foundation)` FD.io Continuous
13 Performance Labs. This is followed by summary of all executed tests
14 against the |vpp-release| and associated FD.io projects and sub-systems
15 (Honeycomb, DPDK, NSH_SFC, DMM), |csit-release| release notes, result
16 highlights and known issues discovered in CSIT. More detailed
17 description of each environment, pointers to CSIT test code
18 documentation and detailed test resuls with links to the source data
19 files are also provided.
21 |csit-release| report contains following main sections and sub-sections:
23 #. **Introduction**: general introduction to CSIT project.
25 - **Introduction**: this section.
26 - **Test Scenarios Overview**: a brief overview of test scenarios
27 covered in this report.
28 - **Physical Testbeds**: description of physical testbeds.
29 - **Performance Test Methodology**: benchmarking methodologies.
31 #. **VPP Performance**: VPP performance tests executed in physical
34 - **Overview**: tested logical topologies, test coverage and naming
36 - **Release Notes**: changes in |csit-release|, added tests,
37 performance changes, environment or methodology changes, known
39 - **Packet Throughput**: NDR, PDR throughput graphs based on results
40 from repeated same test job executions to verify repeatibility of
42 - **Packet Latency**: Latency graphs based on results from test job
44 - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
45 graphs based on results from test job executions.
46 - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance
48 - **Test Environment**: performance test environment configuration.
49 - **Documentation**: documentation of K8s Pod/Container orchestration
50 in CSIT and pointers to CSIT source code documentation for VPP
53 #. **DPDK Performance**: DPDK performance tests executed in physical
56 - **Overview**: tested logical topologies, test coverage.
57 - **Release Notes**: changes in |csit-release|, known issues.
58 - **Packet Throughput**: NDR, PDR throughput graphs based on results
59 from repeated same test job executions to verify repeatibility of
61 - **Packet Latency**: Latency graphs based on results from test job
63 - **Test Environment**: performance test environment configuration.
64 - **Documentation**: pointers to CSIT source code documentation for
65 DPDK performance tests.
67 #. **VPP Functional**: VPP functional tests executed in virtual FD.io
70 - **Overview**: tested virtual topologies, test coverage and naming
72 - **Release Notes** - changes in |csit-release|, added tests,
73 environment or methodology changes, known issues.
74 - **Test Environment**: functional test environment configuration.
75 - **Documentation**: pointers to CSIT source code documentation for
78 #. **Honeycomb Functional**: Honeycomb functional tests executed in
79 virtual FD.io testbeds.
81 - **Overview**: tested virtual topologies, test coverage and naming
83 - **Release Notes** - changes in |csit-release|, known issues.
84 - **Test Environment**: functional test environment configuration.
85 - **Documentation**: pointers to CSIT source code documentation for
86 Honeycomb functional tests.
88 #. **NSH_SFC Functional**: NSH_SFC functional tests executed in
89 virtual FD.io testbeds.
91 - **Overview**: tested virtual topologies, test coverage and naming
93 - **Release Notes** - changes in |csit-release|, known issues.
94 - **Test Environment**: functional test environment configuration.
95 - **Documentation**: pointers to CSIT source code documentation for
96 NSH_SFC functional tests.
98 #. **DMM Functional**: DMM functional tests executed in
99 virtual FD.io testbeds.
101 - **Overview**: tested virtual topologies, test coverage and naming
103 - **Release Notes** - changes in |csit-release|, known issues.
104 - **Test Environment**: functional test environment configuration.
105 - **Documentation**: pointers to CSIT source code documentation for
106 DMM functional tests.
108 #. **Detailed Results**: detailed result tables auto-generated from CSIT
109 test job executions using RF (Robot Framework) output files as
112 - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency.
113 - **VPP Performance MRR**: VPP MRR throughput.
114 - **VPP K8s Container Memif**: VPP K8s Container/Pod topologies
116 - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput
118 - **VPP Functional**: detailed VPP functional results.
119 - **Honeycomb Functional**: detailed HoneyComb functional results.
120 - **NSH_SFC Functional**: detailed nsh-plugin functional results.
121 - **DMM Functional**: detailed DMM functional results.
123 #. **Test Configuration**: VPP DUT configuration data based on VPP API
124 Test (VAT) Commands History auto-generated from CSIT test job
125 executions using RF output files as sources.
127 - **VPP Performance NDR/PDR**: configuration data.
128 - **VPP Performance MRR**: configuration data.
129 - **VPP K8s Container Memif**: configuration data.
130 - **VPP Functional**: configuration data.
132 #. **Test Operational Data**: VPP DUT operational data auto-generated
133 from CSIT test job executions using RFoutput files as sources.
135 - **VPP Performance NDR/PDR**: VPP `show run` outputs under test
138 #. **CSIT Framework Documentation**: description of the overall FD.io
141 - **Design**: framework modular design hierarchy.
142 - **Test naming**: test naming convention.
143 - **Presentation and Analytics Layer**: description of PAL CSIT
145 - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and
146 test case grouping and selection.