4 FD.io |csit-release| report contains system performance and functional
5 testing data of |vpp-release|. `PDF version of this report`_ is
6 available for download.
8 |csit-release| report is structured as follows:
10 #. INTRODUCTION: General introduction to FD.io |csit-release|.
12 - **Introduction**: This section.
13 - **Test Scenarios Overview**: A brief overview of test scenarios
14 covered in this report.
15 - **Physical Testbeds**: Description of physical testbeds.
16 - **Performance Test Methodology**: Benchmarking test methodologies.
18 #. VPP PERFORMANCE: VPP performance tests executed in physical
21 - **Overview**: Tested logical topologies, test coverage and naming
23 - **Release Notes**: Changes in |csit-release|, added tests,
24 environment or methodology changes, known issues.
25 - **Packet Throughput**: NDR, PDR throughput graphs based on results
26 from repeated same test job executions to verify repeatibility of
28 - **Packet Latency**: Latency graphs based on results from test job
30 - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
31 graphs based on results from test job executions.
32 - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance
34 - **Comparisons**: Performance comparisons between VPP releases and
35 between different testbed types.
36 - **Throughput Trending**: References to continuous VPP performance
38 - **Test Environment**: Performance test environment configuration.
39 - **Documentation**: Documentation of K8s Pod/Container orchestration
40 in CSIT and pointers to CSIT source code documentation for VPP
43 #. DPDK PERFORMANCE: DPDK performance tests executed in physical
46 - **Overview**: Tested logical topologies, test coverage.
47 - **Release Notes**: Changes in |csit-release|, known issues.
48 - **Packet Throughput**: NDR, PDR throughput graphs based on results
49 from repeated same test job executions to verify repeatibility of
51 - **Packet Latency**: Latency graphs based on results from test job
53 - **Comparisons**: Performance comparisons between DPDK releases and
54 between different testbed types.
55 - **Throughput Trending**: References to regular DPDK performance
57 - **Test Environment**: Performance test environment configuration.
58 - **Documentation**: Pointers to CSIT source code documentation for
59 DPDK performance tests.
61 #. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io
64 - **Overview**: Tested virtual topologies, test coverage and naming
66 - **Release Notes**: Changes in |csit-release|, added tests,
67 environment or methodology changes, known issues.
68 - **Test Environment**: Functional test environment configuration.
69 - **Documentation**: Pointers to CSIT source code documentation for
72 #. HONEYCOMBE FUNCTIONAL: Honeycomb functional tests executed in
73 virtual FD.io testbeds.
75 - **Overview**: Tested virtual topologies, test coverage and naming
77 - **Release Notes**: Changes in |csit-release|, known issues.
78 - **Test Environment**: Functional test environment configuration.
79 - **Documentation**: Pointers to CSIT source code documentation for
80 Honeycomb functional tests.
82 #. NSH_SFC FUNCTIONAL: NSH_SFC functional tests executed in
83 virtual FD.io testbeds.
85 - **Overview**: Tested virtual topologies, test coverage and naming
87 - **Release Notes**: Changes in |csit-release|, known issues.
88 - **Test Environment**: Functional test environment configuration.
89 - **Documentation**: Pointers to CSIT source code documentation for
90 NSH_SFC functional tests.
92 #. DMM FUNCTIONAL: DMM functional tests executed in
93 virtual FD.io testbeds.
95 - **Overview**: Tested virtual topologies, test coverage and naming
97 - **Release Notes**: Changes in |csit-release|, known issues.
98 - **Test Environment**: Functional test environment configuration.
99 - **Documentation**: Pointers to CSIT source code documentation for
100 DMM functional tests.
102 #. DETAILED RESULTS: Detailed result tables auto-generated from CSIT
103 test job executions using RF (Robot Framework) output files as
106 - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency.
107 - **VPP Performance MRR**: VPP MRR throughput.
108 - **VPP K8s Container Memif**: VPP K8s Container/Pod topologies
110 - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput
112 - **VPP Functional**: Detailed VPP functional results.
113 - **Honeycomb Functional**: Detailed HoneyComb functional results.
114 - **NSH_SFC Functional**: Detailed nsh-plugin functional results.
115 - **DMM Functional**: Detailed DMM functional results.
117 #. TEST CONFIGURATION: VPP DUT configuration data based on VPP API
118 Test (VAT) Commands History auto-generated from CSIT test job
119 executions using RF output files as sources.
121 - **VPP Performance NDR/PDR**: Configuration data.
122 - **VPP Performance MRR**: Configuration data.
123 - **VPP K8s Container Memif**: Configuration data.
124 - **VPP Functional**: Configuration data.
126 #. TEST OPERATIONAL DATA: VPP DUT operational data auto-generated
127 from CSIT test job executions using RFoutput files as sources.
129 - **VPP Performance NDR/PDR**: VPP `show run` outputs under test
132 #. CSIT FRAMEWORK DOCUMENTATION: Description of the overall FD.io
135 - **Design**: Framework modular design hierarchy.
136 - **Test naming**: Test naming convention.
137 - **Presentation and Analytics Layer**: Description of PAL CSIT
139 - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and
140 test case grouping and selection.