docs: added csit tests job specs.
[csit.git] / README.md
1 # CSIT - Continuous System Integration Testing
2
3 1. [Architecture](#architecture)
4 1. [Directory Structure](#directory-structure)
5    1. [Tests](#tests)
6    1. [Keywords](#keywords)
7    1. [Other Resources](#other-resources)
8 1. [Quickstart](#quick-start)
9    1. [Vagrant](#vagrant)
10    1. [Physical Testbed](#physical-testbed)
11 1. [Report](#report)
12 1. [Trending](#trending)
13 1. [Code Documentation](#code-documentation)
14 1. [Coding Guidelines](#coding-guidelines)
15
16 ## Architecture
17
18 FD.io CSIT system design needs to meet continuously expanding requirements of
19 FD.io projects including VPP, related sub-systems (e.g. plugin applications,
20 DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as
21 growing number of compute platforms running those applications. With CSIT
22 project scope and charter including both FD.io continuous testing AND
23 performance trending/comparisons, those evolving requirements further amplify
24 the need for CSIT framework modularity, flexibility and usability.
25
26 CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level
27 of the hierarchy, presentation level at the top level and a number of functional
28 layers in-between. The current CSIT system design including CSIT framework is
29 depicted in the figure below.
30
31 ![csit design](docs/report/csit_framework_documentation/csit_design_picture.svg "CSIT architecture")
32
33 A brief bottom-up description is provided here:
34
35 1. SUTs, DUTs, TGs
36    - SUTs - Systems Under Test;
37    - DUTs - Devices Under Test;
38    - TGs - Traffic Generators;
39 1. Level-1 libraries - Robot and Python
40    - Lowest level CSIT libraries abstracting underlying test environment, SUT,
41      DUT and TG specifics;
42    - Used commonly across multiple L2 KWs;
43    - Performance and functional tests:
44      - L1 KWs (KeyWords) are implemented as RF libraries and Python
45        libraries;
46    - Performance TG L1 KWs:
47      - All L1 KWs are implemented as Python libraries:
48        - Support for TRex only today;
49    - Performance data plane traffic profiles:
50      - TG-specific stream profiles provide full control of:
51        - Packet definition – layers, MACs, IPs, ports, combinations thereof
52          e.g. IPs and UDP ports;
53        - Stream definitions - different streams can run together, delayed,
54          one after each other;
55        - Stream profiles are independent of CSIT framework and can be used
56          in any T-rex setup, can be sent anywhere to repeat tests with
57          exactly the same setup;
58        - Easily extensible – one can create a new stream profile that meets
59          tests requirements;
60        - Same stream profile can be used for different tests with the same
61          traffic needs;
62    - Functional data plane traffic scripts:
63      - Scapy specific traffic scripts;
64 1. Level-2 libraries - Robot resource files
65    - Higher level CSIT libraries abstracting required functions for executing
66      tests;
67    - L2 KWs are classified into the following functional categories:
68      - Configuration, test, verification, state report;
69      - Suite setup, suite teardown;
70      - Test setup, test teardown;
71 1. Tests - Robot
72    - Test suites with test cases;
73    - Functional tests using VIRL environment:
74      - VPP;
75      - Honeycomb;
76      - NSH_SFC;
77      - DMM;
78      - TLDK;
79    - Performance tests using physical testbed environment:
80      - VPP;
81      - DPDK-Testpmd;
82      - DPDK-L3Fwd;
83      - Honeycomb;
84      - VPP Container K8s orchestrated topologies;
85    - Tools:
86      - Documentation generator;
87      - Report generator;
88      - Testbed environment setup ansible playbooks;
89      - Operational debugging scripts;
90
91 ## Directory Structure
92
93 ### Tests
94
95 ```
96 .
97 └── tests
98     ├── dmm
99     │   └── func                    # DMM functional VIRL tests
100     ├── dpdk
101     │   ├── dpdk_scripts            # DPDK helper scripts
102     │   └── perf                    # DPDK performance tests
103     ├── honeycomb
104     │   ├── func                    # Honeycomb functional VIRL tests
105     │   └── perf                    # Honeycomb functional performance tests
106     ├── kubernetes
107     │   └── perf                    # VPP K8S orchestration performance tests
108     ├── nsh_sfc
109     │   ├── func                    # NSH_SFC functional tests
110     │   └── sfc_scripts             # NSH_SFC helper scripts
111     ├── tldk
112     │   ├── func                    # TLDK functional VIRL tests
113     │   ├── tldk_scripts            # TLDK helper scripts
114     │   └── tldk_testconfig         # TLDK test configuration
115     └── vpp
116         ├── device                  # VPP device tests
117         ├── func                    # VPP functional VIRL tests
118         └── perf                    # VPP performance tests
119 ```
120
121 ### Keywords
122
123 ```
124 .
125 resources
126 └── libraries
127     ├── bash
128     │   ├── config
129     │   ├── entry                   # Main bootstrap entry directory
130     │   ├── function                # Bootstrap function library
131     │   ├── qemu_patches            # Custom QEMU patches (see KVM methodology)
132     │   └── shell                   # Various functions
133     ├── python                      # Python L1 KWs
134     └── robot                       # Robot Framework L2 KWs
135 ```
136
137 ### Other Resources
138
139 ```
140 .
141 ├── docs                            # Main documentaion
142 ├── PyPI                            # PyPI packages provided by CSIT
143 │   ├── jumpavg
144 │   └── MLRsearch
145 ├── resources
146 │   ├── templates                   # Templates (vpp_api_test, kubernetes, ...)
147 │   ├── test_data                   # Robot Test configuration
148 │   ├── tools
149 │   │   ├── disk-image-builder      # Utilities for building (DCR, VM) images
150 │   │   ├── doc_gen                 # Code documentation generator
151 │   │   ├── papi                    # PAPI driver
152 │   │   ├── presentation            # Report generator
153 │   │   ├── scripts                 # Various tools
154 │   │   ├── testbed-setup           # Physical testbed setup scripts
155 │   │   ├── topology                # Helper scripts for topology manipulation
156 │   │   ├── trex                    # TRex driver
157 │   │   ├── vagrant                 # VPP device vagrant environment
158 │   │   ├── virl                    # VIRL helper scripts
159 │   │   └── wrk                     # WRK driver
160 │   ├── topology_schemas
161 │   ├── traffic_profiles            # Performance tests traffic profiles
162 │   │   ├── trex
163 │   │   └── wrk
164 │   └── traffic_scripts             # Functional tests traffic profiles
165 │       ├── dhcp
166 │       ├── honeycomb
167 │       └── lisp
168 └── topologies                      # Linux Foundation topology files
169     ├── available
170     └── enabled
171 ```
172
173 ## Quickstart
174
175 ### Vagrant
176
177 [Vagrant environment preparation](docs/testing_in_vagrant.rst) documentaion is
178 describing local VPP Device functional testing.
179
180 ### Physical Testbed
181
182 [Physical testbed preparation](resources/tools/testbed-setup/README.rst)
183 documentation is describing PXE and Ansible setup process. All the software
184 requirements for running Performance Teste are part of Ansible playbooks.
185
186 ## Report
187
188 [CSIT Report](https://docs.fd.io/csit/master/report/).
189
190 ## Trending
191
192 [CSIT Trending](https://docs.fd.io/csit/master/trending/).
193
194 ## Code Documentation
195
196 [CSIT Code Documentation](https://docs.fd.io/csit/master/doc/).
197
198 ## Coding Guidelines
199
200 If you are interested in contributing, please see the
201 [coding guidelines](docs/test_code_guidelines.rst).

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.