report 1807: fixed index.html for introduction section.
[csit.git] / docs / report / honeycomb_functional_tests / overview.rst
1 Overview
2 ========
3
4 Virtual Topologies
5 ------------------
6
7 CSIT HoneyComb functional tests are executed in VM-based virtual topologies
8 created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)`
9 simulation platform contributed by Cisco. VIRL runs on physical
10 baremetal servers hosted by LF FD.io project.
11
12 All tests are executed in two-node virtual test topology shown in the
13 figure below.
14
15 .. only:: latex
16
17     .. raw:: latex
18
19         \begin{figure}[H]
20             \centering
21                 \graphicspath{{../_tmp/src/vpp_functional_tests/}}
22                 \includegraphics[width=0.90\textwidth]{virtual-2n-nic2nic}
23                 \label{fig:virtual-2n-nic2nic}
24         \end{figure}
25
26 .. only:: html
27
28     .. figure:: ../vpp_functional_tests/virtual-2n-nic2nic.svg
29         :alt: virtual-2n-nic2nic
30         :align: center
31
32 SUT (System Under Test) is a VM running Ubuntu Linux (or Centos,
33 depending on the test suite), TG (Traffic Generator) is another VM
34 running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP  in
35 Linux user-mode as a combined DUT (Device Under Test). TG runs Scapy
36 application as a packet Traffic Generator. Virtual connectivity between
37 SUT and TG is provided using virtual NICs using VMs' virtio drivers.
38
39 Functional Tests Coverage
40 -------------------------
41
42 |csit-release| includes following HoneyComb functionality tested in
43 virtual VM environment:
44
45 +-----------------------+----------------------------------------------+
46 | Functionality         |  Description                                 |
47 +=======================+==============================================+
48 | ACL                   | CRD for low-level classifiers: table and     |
49 |                       | session management, interface assignment.    |
50 |                       |                                              |
51 |                       | - Configure up to 2 classify tables.         |
52 |                       | - Configure up to 2 classify sessions on one |
53 |                       |   table.                                     |
54 |                       | - Assign classify session to a physical      |
55 |                       |   interface.                                 |
56 |                       | - Remove tables, sessions, interface         |
57 |                       |   assignments.                               |
58 |                       | - Test case count: 9.                        |
59 +-----------------------+----------------------------------------------+
60 | ACL-PLUGIN            | CRD for high-level classifier.               |
61 |                       |                                              |
62 |                       | - MAC + IP address classification.           |
63 |                       | - IPv4, IPv6 address classification.         |
64 |                       | - TCP, UDP, ICMP, ICMPv6 protocol and        |
65 |                       |   next-header classification.                |
66 |                       | - port number classification.                |
67 |                       | - ICMP, ICMPv6 code and type classification. |
68 |                       | - Test case count: 15.                       |
69 +-----------------------+----------------------------------------------+
70 | Basic interface       | CRUD for interface state.                    |
71 | management            |                                              |
72 |                       | - ipv4/ipv6 address, ipv4 neighbor, MTU      |
73 |                       |   value.                                     |
74 |                       | - Test case count: 14.                       |
75 +-----------------------+----------------------------------------------+
76 | Border Gateway        | CRUD and functional tests for BGP.           |
77 | Protocol              |                                              |
78 |                       | - Configure peers and routes                 |
79 |                       | - Check interactions with another BGP peer.  |
80 |                       | - Test case count: 13.                       |
81 +-----------------------+----------------------------------------------+
82 | DHCP Relay            | CRD for DHCP relay feature.                  |
83 |                       |                                              |
84 |                       | - Configure DHCP Relays.                     |
85 |                       | - IPv4 and IPv6 variants.                    |
86 |                       | - Test case count: 4.                        |
87 +-----------------------+----------------------------------------------+
88 | Honeycomb             | Configuration persistence.                   |
89 | Infractructure        |                                              |
90 |                       | - Netconf notifications for interface        |
91 |                       |   events.                                    |
92 |                       | - Netconf negative tests aimed at specific   |
93 |                       |   issues.                                    |
94 |                       | - Netconf/Restconf northbound over IPv6.     |
95 |                       | - Test case count: 12.                       |
96 +-----------------------+----------------------------------------------+
97 | L2BD                  | CRUD for L2 Bridge-Domain, interface         |
98 |                       | assignment.                                  |
99 |                       |                                              |
100 |                       | - Create up to two bridge domains with all   |
101 |                       |   implemented functions turned on:           |
102 |                       |   flooding, unknown-unicast flooding,        |
103 |                       |   forwarding, learning, arp-termination.     |
104 |                       | - Assign up to two physical interfaces to a  |
105 |                       |   single bridge domain.                      |
106 |                       | - Remove interface assignments, remove       |
107 |                       |   bridge domains.                            |
108 |                       | - Test case count: 5.                        |
109 +-----------------------+----------------------------------------------+
110 | L2FIB                 | CRD for L2-FIB entries.                      |
111 |                       |                                              |
112 |                       | - Create 4 FIB entries:                      |
113 |                       |   one of each for filter/forward,            |
114 |                       |   static/dynamic combinations.               |
115 |                       | - Remove FIB entries.                        |
116 |                       | - Test case count: 7.                        |
117 +-----------------------+----------------------------------------------+
118 | LISP                  | CRD for Lisp: mapping, locator set,          |
119 |                       | adjacency, mapresolver.                      |
120 |                       |                                              |
121 |                       | - Toggle Lisp feature status.                |
122 |                       | - Configure and delete Lisp mapping as local |
123 |                       |   and remote.                                |
124 |                       | - Configure and delete Lisp adjacency        |
125 |                       |   mapping.                                   |
126 |                       | - Configure and delete Lisp map resolver,    |
127 |                       |   proxy ITR.                                 |
128 |                       | - Test case count: 18.                       |
129 +-----------------------+----------------------------------------------+
130 | LISP GPE              | CRUD for LISP GPE mappings.                  |
131 |                       |                                              |
132 |                       | - Toggle Lisp GPE feature status.            |
133 |                       | - Configure Lisp GPE mappings.               |
134 |                       | - Traffic test verifying encapsulation.      |
135 |                       | - Test case count: 12.                       |
136 +-----------------------+----------------------------------------------+
137 | NAT                   | CRD for NAT entries, interface assignment.   |
138 |                       |                                              |
139 |                       | - Configure and delete up to two NAT         |
140 |                       |   entries.                                   |
141 |                       | - Assign NAT entries to a physical           |
142 |                       |   interface.                                 |
143 |                       | - Test case count: 6.                        |
144 +-----------------------+----------------------------------------------+
145 | NSH_SFC               | CRD for NSH maps and entries, using NSH_SFC  |
146 |                       | plugin.                                      |
147 |                       |                                              |
148 |                       | - Configure up to 2 NSH entries.             |
149 |                       | - Configure up to 2 NSH maps.                |
150 |                       | - Modify and delete NSH maps and entries.    |
151 |                       | - Test case count: 8.                        |
152 +-----------------------+----------------------------------------------+
153 | PBB                   | CRD for provider backbone bridge             |
154 |                       | sub-interface.                               |
155 |                       |                                              |
156 |                       | - Configure, modify and remove a PBB         |
157 |                       |   sub-interface over a physical interface.   |
158 |                       | - Test case count: 8.                        |
159 +-----------------------+----------------------------------------------+
160 | Policer               | CRD for traffic policing feature.            |
161 |                       |                                              |
162 |                       | - Configure Policing rules.                  |
163 |                       | - Assign to interface.                       |
164 |                       | - Test case count: 6.                        |
165 +-----------------------+----------------------------------------------+
166 | Port mirroring        | CRD for SPAN port mirroring, interface       |
167 |                       | assignment.                                  |
168 |                       |                                              |
169 |                       | - Configure SPAN port mirroring on a         |
170 |                       |   physical interface, mirroring.             |
171 |                       | - up to 2 interfaces.                        |
172 |                       | - Remove SPAN configuration from interfaces. |
173 |                       | - Test case count: 14.                       |
174 +-----------------------+----------------------------------------------+
175 | ProxyARP              | CRD for proxyARP feature.                    |
176 |                       |                                              |
177 |                       | - Configure proxyARP.                        |
178 |                       | - Assign to interface.                       |
179 |                       | - Test case count: 3.                        |
180 +-----------------------+----------------------------------------------+
181 | ProxyND6              | CRD for Neighbor Discovery Proxy.            |
182 |                       |                                              |
183 |                       | - Configure ProxyND6 feature on interface.   |
184 |                       | - Test case count: 4.                        |
185 +-----------------------+----------------------------------------------+
186 | Routing               | CRD for routing.                             |
187 |                       |                                              |
188 |                       | - Configure single-hop route.                |
189 |                       | - Configure multi-hop routes.                |
190 |                       | - Configure blackhole route.                 |
191 |                       | - IPv4 and IPv6 variants.                    |
192 |                       | - Test case count: 6.                        |
193 +-----------------------+----------------------------------------------+
194 | SLAAC                 | CRD for Stateless Address AutoConfiguration. |
195 |                       |                                              |
196 |                       | - Configure SLAAC feature on interfaces.     |
197 |                       | - Test case count: 7.                        |
198 +-----------------------+----------------------------------------------+
199 | Vhost-user            | CRUD for Vhost-user interfaces.              |
200 |                       |                                              |
201 |                       | - Create, modify and delete Vhost-user       |
202 |                       |   interface, as client and server.           |
203 |                       | - Test case count: 8.                        |
204 +-----------------------+----------------------------------------------+
205 | VLAN                  | CRUD for VLAN sub-interface management.      |
206 |                       |                                              |
207 |                       | - Create VLAN sub-interface over a physical  |
208 |                       |   interface.                                 |
209 |                       | - Toggle interface state separately for      |
210 |                       |   super-interface and sub-interface.         |
211 |                       | - Configure IP address and bridge domain     |
212 |                       |   assignment on sub-interface.               |
213 |                       | - Configure VLAN tag rewrite on              |
214 |                       |   sub-interface.                             |
215 |                       | - Test case count: 24.                       |
216 +-----------------------+----------------------------------------------+
217 | VxLAN                 | CRD for VxLAN tunnels.                       |
218 |                       |                                              |
219 |                       | - Create VxLAN interface.                    |
220 |                       | - Disable VxLAN interface.                   |
221 |                       | - Re-create a disabled VxLAN interface.      |
222 |                       | - Test case count: 6.                        |
223 +-----------------------+----------------------------------------------+
224 | VxLAN-GPE             | CRD for VxLAN GPE tunnels.                   |
225 |                       |                                              |
226 |                       | - Create VxLAN GPE interface.                |
227 |                       | - Disable VxLAN interface.                   |
228 |                       | - Re-create a disabled VxLAN interface.      |
229 |                       | - Test case count: 7.                        |
230 +-----------------------+----------------------------------------------+
231 | TAP                   | CRUD for Tap interface management.           |
232 |                       |                                              |
233 |                       | - Create, modify and delete TAP interface.   |
234 |                       | - Test case count: 3.                        |
235 +-----------------------+----------------------------------------------+
236
237 Total 219 Honeycomb functional tests in the |csit-release|.
238
239 Operational data in Honeycomb should mirror configuration data at all
240 times. Because of this, test cases follow this general pattern:
241
242 #. read operational data of the feature using restconf.
243 #. read status of the feature using VPP API dump.
244 #. modify configuration of the feature using restconf.
245 #. verify changes to operational data using restconf.
246 #. verify changes using VPP API dump, OR
247 #. send a packet to VPP node and observe behaviour to verify configuration.
248
249 Test cases involving network interfaces utilize the first two interfaces
250 on the DUT node.
251
252 Functional Tests Naming
253 -----------------------
254
255 |csit-release| follows a common structured naming convention for all
256 performance and system functional tests, introduced in CSIT-17.01.
257
258 The naming should be intuitive for majority of the tests. Complete
259 description of CSIT test naming convention is provided on
260 :ref:`csit_test_naming`.

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.