1 # CSIT - Continuous System Integration Testing
3 1. [Architecture](#architecture)
4 1. [Directory Structure](#directory-structure)
6 1. [Keywords](#keywords)
7 1. [Other Resources](#other-resources)
8 1. [Quickstart](#quick-start)
10 1. [Physical Testbed](#physical-testbed)
12 1. [Trending](#trending)
13 1. [Code Documentation](#code-documentation)
14 1. [Coding Guidelines](#coding-guidelines)
18 FD.io CSIT system design needs to meet continuously expanding requirements of
19 FD.io projects including VPP, related sub-systems (e.g. plugin applications,
20 DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as
21 growing number of compute platforms running those applications. With CSIT
22 project scope and charter including both FD.io continuous testing AND
23 performance trending/comparisons, those evolving requirements further amplify
24 the need for CSIT framework modularity, flexibility and usability.
26 CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level
27 of the hierarchy, presentation level at the top level and a number of functional
28 layers in-between. The current CSIT system design including CSIT framework is
29 depicted in the figure below.
31 ![csit design](docs/report/csit_framework_documentation/csit_design_picture.svg "CSIT architecture")
33 A brief bottom-up description is provided here:
36 - SUTs - Systems Under Test;
37 - DUTs - Devices Under Test;
38 - TGs - Traffic Generators;
39 1. Level-1 libraries - Robot and Python
40 - Lowest level CSIT libraries abstracting underlying test environment, SUT,
42 - Used commonly across multiple L2 KWs;
43 - Performance and functional tests:
44 - L1 KWs (KeyWords) are implemented as RF libraries and Python
46 - Performance TG L1 KWs:
47 - All L1 KWs are implemented as Python libraries:
48 - Support for TRex only today;
49 - Performance data plane traffic profiles:
50 - TG-specific stream profiles provide full control of:
51 - Packet definition – layers, MACs, IPs, ports, combinations thereof
52 e.g. IPs and UDP ports;
53 - Stream definitions - different streams can run together, delayed,
55 - Stream profiles are independent of CSIT framework and can be used
56 in any T-rex setup, can be sent anywhere to repeat tests with
57 exactly the same setup;
58 - Easily extensible – one can create a new stream profile that meets
60 - Same stream profile can be used for different tests with the same
62 - Functional data plane traffic scripts:
63 - Scapy specific traffic scripts;
64 1. Level-2 libraries - Robot resource files
65 - Higher level CSIT libraries abstracting required functions for executing
67 - L2 KWs are classified into the following functional categories:
68 - Configuration, test, verification, state report;
69 - Suite setup, suite teardown;
70 - Test setup, test teardown;
72 - Test suites with test cases;
73 - Performance tests using physical testbed environment:
77 - VPP Container K8s orchestrated topologies;
79 - Documentation generator;
81 - Testbed environment setup ansible playbooks;
82 - Operational debugging scripts;
84 ## Directory Structure
92 │ ├── dpdk_scripts # DPDK helper scripts
93 │ └── perf # DPDK performance tests
95 │ ├── func # TLDK functional VIRL tests
96 │ ├── tldk_scripts # TLDK helper scripts
97 │ └── tldk_testconfig # TLDK test configuration
99 ├── device # VPP device tests
100 ├── func # VPP functional VIRL tests
101 └── perf # VPP performance tests
112 │ ├── entry # Main bootstrap entry directory
113 │ ├── function # Bootstrap function library
114 │ └── shell # Various functions
115 ├── python # Python L1 KWs
116 └── robot # Robot Framework L2 KWs
123 ├── docs # Main documentaion
124 ├── PyPI # PyPI packages provided by CSIT
128 │ ├── templates # Templates (vpp_api_test, kubernetes, ...)
129 │ ├── test_data # Robot Test configuration
131 │ │ ├── disk-image-builder # Utilities for building (DCR, VM) images
132 │ │ ├── doc_gen # Code documentation generator
133 │ │ ├── papi # PAPI driver
134 │ │ ├── presentation # Report generator
135 │ │ ├── scripts # Various tools
136 │ │ ├── testbed-setup # Physical testbed setup scripts
137 │ │ ├── topology # Helper scripts for topology manipulation
138 │ │ ├── trex # TRex driver
139 │ │ ├── vagrant # VPP device vagrant environment
140 │ │ └── wrk # WRK driver
141 │ ├── topology_schemas
142 │ ├── traffic_profiles # Performance tests traffic profiles
145 │ └── traffic_scripts # Functional tests traffic profiles
148 └── topologies # Linux Foundation topology files
157 [Vagrant environment preparation](docs/testing_in_vagrant.rst) documentaion is
158 describing local VPP Device functional testing.
162 [Physical testbed preparation](resources/tools/testbed-setup/README.rst)
163 documentation is describing PXE and Ansible setup process. All the software
164 requirements for running Performance Teste are part of Ansible playbooks.
168 [CSIT Report](https://docs.fd.io/csit/master/report/).
172 [CSIT Trending](https://docs.fd.io/csit/master/trending/).
174 ## Code Documentation
176 [CSIT Code Documentation](https://docs.fd.io/csit/master/doc/).
180 If you are interested in contributing, please see the
181 [coding guidelines](docs/test_code_guidelines.rst).