API: Add collection for 21706/17
[csit.git] / README.md
1 # CSIT - Continuous System Integration Testing
2
3 1. [Architecture](#architecture)
4 1. [Directory Structure](#directory-structure)
5    1. [Tests](#tests)
6    1. [Keywords](#keywords)
7    1. [Other Resources](#other-resources)
8 1. [Quickstart](#quick-start)
9    1. [Vagrant](#vagrant)
10    1. [Physical Testbed](#physical-testbed)
11 1. [Report](#report)
12 1. [Trending](#trending)
13 1. [Code Documentation](#code-documentation)
14 1. [Coding Guidelines](#coding-guidelines)
15
16 ## Architecture
17
18 FD.io CSIT system design needs to meet continuously expanding requirements of
19 FD.io projects including VPP, related sub-systems (e.g. plugin applications,
20 DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as
21 growing number of compute platforms running those applications. With CSIT
22 project scope and charter including both FD.io continuous testing AND
23 performance trending/comparisons, those evolving requirements further amplify
24 the need for CSIT framework modularity, flexibility and usability.
25
26 CSIT follows a hierarchical system design with SUTs and DUTs at the bottom level
27 of the hierarchy, presentation level at the top level and a number of functional
28 layers in-between. The current CSIT system design including CSIT framework is
29 depicted in the figure below.
30
31 ![csit design](docs/report/csit_framework_documentation/csit_design_picture.svg "CSIT architecture")
32
33 A brief bottom-up description is provided here:
34
35 1. SUTs, DUTs, TGs
36    - SUTs - Systems Under Test;
37    - DUTs - Devices Under Test;
38    - TGs - Traffic Generators;
39 1. Level-1 libraries - Robot and Python
40    - Lowest level CSIT libraries abstracting underlying test environment, SUT,
41      DUT and TG specifics;
42    - Used commonly across multiple L2 KWs;
43    - Performance and functional tests:
44      - L1 KWs (KeyWords) are implemented as RF libraries and Python
45        libraries;
46    - Performance TG L1 KWs:
47      - All L1 KWs are implemented as Python libraries:
48        - Support for TRex only today;
49    - Performance data plane traffic profiles:
50      - TG-specific stream profiles provide full control of:
51        - Packet definition – layers, MACs, IPs, ports, combinations thereof
52          e.g. IPs and UDP ports;
53        - Stream definitions - different streams can run together, delayed,
54          one after each other;
55        - Stream profiles are independent of CSIT framework and can be used
56          in any T-rex setup, can be sent anywhere to repeat tests with
57          exactly the same setup;
58        - Easily extensible – one can create a new stream profile that meets
59          tests requirements;
60        - Same stream profile can be used for different tests with the same
61          traffic needs;
62    - Functional data plane traffic scripts:
63      - Scapy specific traffic scripts;
64 1. Level-2 libraries - Robot resource files
65    - Higher level CSIT libraries abstracting required functions for executing
66      tests;
67    - L2 KWs are classified into the following functional categories:
68      - Configuration, test, verification, state report;
69      - Suite setup, suite teardown;
70      - Test setup, test teardown;
71 1. Tests - Robot
72    - Test suites with test cases;
73    - Functional tests using VIRL environment:
74      - VPP;
75      - DMM;
76      - TLDK;
77    - Performance tests using physical testbed environment:
78      - VPP;
79      - DPDK-Testpmd;
80      - DPDK-L3Fwd;
81      - VPP Container K8s orchestrated topologies;
82    - Tools:
83      - Documentation generator;
84      - Report generator;
85      - Testbed environment setup ansible playbooks;
86      - Operational debugging scripts;
87
88 ## Directory Structure
89
90 ### Tests
91
92 ```
93 .
94 └── tests
95     ├── dmm
96     │   └── func                    # DMM functional VIRL tests
97     ├── dpdk
98     │   ├── dpdk_scripts            # DPDK helper scripts
99     │   └── perf                    # DPDK performance tests
100     ├── kubernetes
101     │   └── perf                    # VPP K8S orchestration performance tests
102     ├── tldk
103     │   ├── func                    # TLDK functional VIRL tests
104     │   ├── tldk_scripts            # TLDK helper scripts
105     │   └── tldk_testconfig         # TLDK test configuration
106     └── vpp
107         ├── device                  # VPP device tests
108         ├── func                    # VPP functional VIRL tests
109         └── perf                    # VPP performance tests
110 ```
111
112 ### Keywords
113
114 ```
115 .
116 resources
117 └── libraries
118     ├── bash
119     │   ├── config
120     │   ├── entry                   # Main bootstrap entry directory
121     │   ├── function                # Bootstrap function library
122     │   ├── qemu_patches            # Custom QEMU patches (see KVM methodology)
123     │   └── shell                   # Various functions
124     ├── python                      # Python L1 KWs
125     └── robot                       # Robot Framework L2 KWs
126 ```
127
128 ### Other Resources
129
130 ```
131 .
132 ├── docs                            # Main documentaion
133 ├── PyPI                            # PyPI packages provided by CSIT
134 │   ├── jumpavg
135 │   └── MLRsearch
136 ├── resources
137 │   ├── templates                   # Templates (vpp_api_test, kubernetes, ...)
138 │   ├── test_data                   # Robot Test configuration
139 │   ├── tools
140 │   │   ├── disk-image-builder      # Utilities for building (DCR, VM) images
141 │   │   ├── doc_gen                 # Code documentation generator
142 │   │   ├── papi                    # PAPI driver
143 │   │   ├── presentation            # Report generator
144 │   │   ├── scripts                 # Various tools
145 │   │   ├── testbed-setup           # Physical testbed setup scripts
146 │   │   ├── topology                # Helper scripts for topology manipulation
147 │   │   ├── trex                    # TRex driver
148 │   │   ├── vagrant                 # VPP device vagrant environment
149 │   │   ├── virl                    # VIRL helper scripts
150 │   │   └── wrk                     # WRK driver
151 │   ├── topology_schemas
152 │   ├── traffic_profiles            # Performance tests traffic profiles
153 │   │   ├── trex
154 │   │   └── wrk
155 │   └── traffic_scripts             # Functional tests traffic profiles
156 │       ├── dhcp
157 │       └── lisp
158 └── topologies                      # Linux Foundation topology files
159     ├── available
160     └── enabled
161 ```
162
163 ## Quickstart
164
165 ### Vagrant
166
167 [Vagrant environment preparation](docs/testing_in_vagrant.rst) documentaion is
168 describing local VPP Device functional testing.
169
170 ### Physical Testbed
171
172 [Physical testbed preparation](resources/tools/testbed-setup/README.rst)
173 documentation is describing PXE and Ansible setup process. All the software
174 requirements for running Performance Teste are part of Ansible playbooks.
175
176 ## Report
177
178 [CSIT Report](https://docs.fd.io/csit/master/report/).
179
180 ## Trending
181
182 [CSIT Trending](https://docs.fd.io/csit/master/trending/).
183
184 ## Code Documentation
185
186 [CSIT Code Documentation](https://docs.fd.io/csit/master/doc/).
187
188 ## Coding Guidelines
189
190 If you are interested in contributing, please see the
191 [coding guidelines](docs/test_code_guidelines.rst).

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.