4 FD.io |csit-release| report contains system performance and functional
5 testing data of |vpp-release|. `PDF version of this report`_ is
6 available for download.
8 |csit-release| report is structured as follows:
10 #. INTRODUCTION: General introduction to FD.io |csit-release|.
12 - **Introduction**: This section.
13 - **Test Scenarios Overview**: A brief overview of test scenarios
14 covered in this report.
15 - **Physical Testbeds**: Description of physical testbeds.
16 - **Test Methodology**: Performance benchmarking and functional test
19 #. VPP PERFORMANCE: VPP performance tests executed in physical
22 - **Overview**: Tested logical topologies, test coverage and naming
24 - **Release Notes**: Changes in |csit-release|, added tests,
25 environment or methodology changes, known issues.
26 - **Packet Throughput**: NDR, PDR throughput graphs based on results
27 from repeated same test job executions to verify repeatibility of
29 - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
30 graphs based on results from test job executions.
31 - **Packet Latency**: Latency graphs based on results from test job
33 - **Soak Tests**: Long duration soak tests are executed using PLRsearch
35 - **NFV Service Density**: Network Function Virtualization (NFV) service
36 density tests focus on measuring total per server throughput at varied NFV
37 service "packing" densities with vswitch providing host dataplane.
38 - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance
40 - **Comparisons**: Performance comparisons between VPP releases and
41 between different testbed types.
42 - **Throughput Trending**: References to continuous VPP performance
44 - **Test Environment**: Performance test environment configuration.
45 - **Documentation**: Documentation of K8s Pod/Container orchestration
46 in CSIT and pointers to CSIT source code documentation for VPP
49 #. DPDK PERFORMANCE: DPDK performance tests executed in physical
52 - **Overview**: Tested logical topologies, test coverage.
53 - **Release Notes**: Changes in |csit-release|, known issues.
54 - **Packet Throughput**: NDR, PDR throughput graphs based on results
55 from repeated same test job executions to verify repeatibility of
57 - **Packet Latency**: Latency graphs based on results from test job
59 - **Comparisons**: Performance comparisons between DPDK releases and
60 between different testbed types.
61 - **Throughput Trending**: References to regular DPDK performance
63 - **Test Environment**: Performance test environment configuration.
64 - **Documentation**: Pointers to CSIT source code documentation for
65 DPDK performance tests.
67 #. VPP DEVICE: VPP functional tests executed in physical FD.io
68 testbeds using containers.
70 - **Overview**: Tested virtual topologies, test coverage and naming
72 - **Release Notes**: Changes in |csit-release|, added tests,
73 environment or methodology changes, known issues.
74 - **Integration Tests**: Functional test environment configuration.
75 - **Documentation**: Pointers to CSIT source code documentation for
78 #. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io
81 - **Overview**: Tested virtual topologies, test coverage and naming
83 - **Release Notes**: Changes in |csit-release|, added tests,
84 environment or methodology changes, known issues.
85 - **Test Environment**: Functional test environment configuration.
86 - **Documentation**: Pointers to CSIT source code documentation for
89 #. HONEYCOMB FUNCTIONAL: Honeycomb functional tests executed in
90 virtual FD.io testbeds.
92 - **Overview**: Tested virtual topologies, test coverage and naming
94 - **Release Notes**: Changes in |csit-release|, known issues.
95 - **Test Environment**: Functional test environment configuration.
96 - **Documentation**: Pointers to CSIT source code documentation for
97 Honeycomb functional tests.
99 #. NSH_SFC FUNCTIONAL: NSH_SFC functional tests executed in
100 virtual FD.io testbeds.
102 - **Overview**: Tested virtual topologies, test coverage and naming
104 - **Release Notes**: Changes in |csit-release|, known issues.
105 - **Test Environment**: Functional test environment configuration.
106 - **Documentation**: Pointers to CSIT source code documentation for
107 NSH_SFC functional tests.
109 #. DMM FUNCTIONAL: DMM functional tests executed in
110 virtual FD.io testbeds.
112 - **Overview**: Tested virtual topologies, test coverage and naming
114 - **Release Notes**: Changes in |csit-release|, known issues.
115 - **Test Environment**: Functional test environment configuration.
116 - **Documentation**: Pointers to CSIT source code documentation for
117 DMM functional tests.
119 #. DETAILED RESULTS: Detailed result tables auto-generated from CSIT
120 test job executions using RF (Robot Framework) output files as
123 - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency.
124 - **VPP Performance MRR**: VPP MRR throughput.
125 - **VPP K8s Container Memif**: VPP K8s Container/Pod topologies
127 - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput
129 - **VPP Functional**: Detailed VPP functional results.
130 - **Honeycomb Functional**: Detailed HoneyComb functional results.
131 - **NSH_SFC Functional**: Detailed nsh-plugin functional results.
132 - **DMM Functional**: Detailed DMM functional results.
134 #. TEST CONFIGURATION: VPP DUT configuration data based on VPP API
135 Test (VAT) Commands History auto-generated from CSIT test job
136 executions using RF output files as sources.
138 - **VPP Performance NDR/PDR**: Configuration data.
139 - **VPP Performance MRR**: Configuration data.
140 - **VPP K8s Container Memif**: Configuration data.
141 - **VPP Functional**: Configuration data.
143 #. TEST OPERATIONAL DATA: VPP DUT operational data auto-generated
144 from CSIT test job executions using RFoutput files as sources.
146 - **VPP Performance NDR/PDR**: VPP `show run` outputs under test
149 #. CSIT FRAMEWORK DOCUMENTATION: Description of the overall FD.io
152 - **Design**: Framework modular design hierarchy.
153 - **Test naming**: Test naming convention.
154 - **Presentation and Analytics Layer**: Description of PAL CSIT
156 - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and
157 test case grouping and selection.