-Overview\r
-========\r
-\r
-Tested Virtual Topologies\r
--------------------------\r
-\r
-CSIT Honeycomb functional tests are executed on virtualized topologies created\r
-using Virtual Internet Routing Lab (VIRL) simulation platform contributed by\r
-Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io project. All\r
-tests are executed in two node logical test topology - Traffic Generator (TG)\r
-node and Systems Under Test (SUT1) node connected in a loop. Logical test\r
-topology is shown in the figure below.\r
-\r
-::\r
-\r
- +------------------------+\r
- | |\r
- | +------------------+ |\r
- +---------------> <--------------+\r
- | | | | | |\r
- | |------------> DUT1 <-----------+ |\r
- | | | +------------------+ | | |\r
- | | | | | |\r
- | | | SUT1 | | |\r
- | | +------------------------+ | |\r
- | | | |\r
- | | | |\r
- | | +-----------+ | |\r
- | +---------------> <---------------+ |\r
- | | TG | |\r
- +------------------> <------------------+\r
- +-----------+\r
-\r
-SUT1 is a VM (Ubuntu or Centos, depending on the test suite), TG is a Traffic\r
-Generator (TG, another Ubuntu VM). SUTs run Honeycomb and VPP SW applications\r
-in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy\r
-SW application as a packet Traffic Generator. Logical connectivity between\r
-SUTs and to TG is provided using virtual NICs using VMs' virtio driver.\r
-\r
-Virtual testbeds are created on-demand whenever a verification job is started\r
-(e.g. triggered by the gerrit patch submission) and destroyed upon completion\r
-of all functional tests. Each node is a Virtual Machine and each connection\r
-that is drawn on the diagram is available for use in any test case. During the\r
-test execution, all nodes are reachable thru the Management network connected\r
-to every node via dedicated virtual NICs and virtual links (not shown above\r
-for clarity).\r
-\r
-Functional Tests Coverage\r
--------------------------\r
-\r
-The following Honeycomb functional test areas are included in the CSIT |release|\r
-with results listed in this report:\r
-\r
-- **Basic interface management** - CRUD for interface state,\r
- - ipv4/ipv6 address, ipv4 neighbor, MTU value.\r
- - Test case count: 7\r
-- **L2BD** - CRUD for L2 Bridge-Domain, interface assignment.\r
- - Create up to two bridge domains with all implemented functions turned on.\r
- - (flooding, unknown-unicast flooding, forwarding, learning, arp-termination)\r
- - Assign up to two physical interfaces to a single bridge domain.\r
- - Remove interface assignments, remove bridge domains.\r
- - Test case count: 5\r
-- **L2FIB** - CRD for L2-FIB entries.\r
- - Create 4 FIB entries\r
- - (one of each for filter/forward, static/dynamic combinations).\r
- - Remove FIB entries.\r
- - Test case count: 7\r
-- **VxLAN** - CRD for VxLAN tunnels.\r
- - Create VxLAN interface.\r
- - Disable VxLAN interface.\r
- - Re-create a disabled VxLAN interface.\r
- - Test case count: 6\r
-- **VxLAN-GPE** - CRD for VxLAN GPE tunnels.\r
- - Create VxLAN GPE interface.\r
- - Disable VxLAN interface.\r
- - Re-create a disabled VxLAN interface.\r
- - Test case count: 7\r
-- **Vhost-user** - CRUD for Vhost-user interfaces.\r
- - Create, modify and delete Vhost-user interface, as client and server.\r
- - Test case count: 8\r
-- **TAP** - CRUD for Tap interface management.\r
- - Create, modify and delete TAP interface.\r
- - Test case count: 3\r
-- **VLAN** - CRUD for VLAN sub-interface management.\r
- - Create VLAN sub-interface over a physical interface.\r
- - Toggle interface state separately for super-interface and sub-interface.\r
- - Configure IP address and bridge domain assignment on sub-interface.\r
- - Configure VLAN tag rewrite on sub-interface.\r
- - Test case count: 17\r
-- **ACL** - CRD for low-level classifiers: table and session management,\r
- - interface assignment.\r
- - Configure up to 2 classify tables.\r
- - Configure up to 2 classify sessions on one table.\r
- - Assign classify session to a physical interface.\r
- - Remove tables, sessions, interface assignments.\r
- - Test case count: 9\r
-- **PBB** - CRD for provider backbone bridge sub-interface.\r
- - Configure, modify and remove a PBB sub-interface over a physical interface.\r
- - Test case count: 9\r
-- **NSH_SFC** - CRD for NSH maps and entries, using NSH_SFC plugin.\r
- - Configure up to 2 NSH entries.\r
- - Configure up to 2 NSH maps.\r
- - Modify and delete NSH maps and entries.\r
- - Test case count: 8\r
-- **LISP** - CRD for Lisp: mapping, locator set, adjacency, map resolver.\r
- - Toggle Lisp feature status.\r
- - Configure and delete Lisp mapping as local and remote.\r
- - Configure and delete Lisp adjacency mapping\r
- - Configure and delete Lisp map resolver, proxy ITR.\r
- - Test case count: 11\r
-- **NAT** - CRD for NAT entries, interface assignment.\r
- - Configure and delete up to two NAT entries.\r
- - Assign NAT entries to a physical interface.\r
- - Test case count: 6\r
-- **Port mirroring** - CRD for SPAN port mirroring, interface assignment.\r
- - Configure SPAN port mirroring on a physical interface, mirroring\r
- - up to 2 interfaces.\r
- - Remove SPAN configuration from interfaces.\r
- - Test case count: 3\r
-- **ACL-PLUGIN** - CRD for high-level classifier\r
- - MAC + IP address classification.\r
- - IPv4, IPv6 address classification.\r
- - TCP, UDP, ICMP, ICMPv6 protocol/next-header classification.\r
- - port number classification.\r
- - ICMP, ICMPv6 code and type classification.\r
- - Test case count: 15\r
-- **ProxyARP** - CRD for proxyARP feature.\r
- - Configure proxyARP.\r
- - Assign to interface.\r
- - Test case count: 3\r
-- **ProxyND6** - CRD for Neighbor Discovery Proxy.\r
- - Configure ProxyND6 feature on interface.\r
- - Test case count: 4\r
-- **DHCP Relay** - CRD for DHCP relay feature.\r
- - Configure DHCP Relays.\r
- - IPv4 and IPv6 variants.\r
- - Test case count: 4\r
-- **SLAAC** - CRD for Stateless Address AutoConfiguration.\r
- - Configure SLAAC feature on interfaces.\r
- - Test case count: 7\r
-- **Routing** - CRD for routing.\r
- - Configure single-hop route.\r
- - Configure multi-hop routes.\r
- - Configure blackhole route.\r
- - IPv4 and IPv6 variants.\r
- - Test case count: 6\r
-- **Honeycomb Infractructure** - configuration persistence,\r
- - Netconf notifications for interface events,\r
- - Netconf negative tests aimed at specific issues\r
-\r
-Total 111 Honeycomb tests in the CSIT |release|.\r
-\r
-Operational data in Honeycomb should mirror configuration data at all times.\r
-Because of this, test cases follow this general pattern:\r
-\r
-#. read operational data of the feature using restconf.\r
-#. read status of the feature using VPP API dump.\r
-#. modify configuration of the feature using restconf.\r
-#. verify changes to operational data using restconf.\r
-#. verify changes using VPP API dump, OR\r
-#. send a packet to VPP node and observe behaviour to verify configuration\r
-\r
-Test cases involving network interfaces utilize the first two interfaces on\r
-the DUT node.\r
-\r
-Functional Tests Naming\r
------------------------\r
-\r
-CSIT |release| introduced a common structured naming convention for all\r
-performance and functional tests. This change was driven by substantially\r
-growing number and type of CSIT test cases. Firstly, the original practice did\r
-not always follow any strict naming convention. Secondly test names did not\r
-always clearly capture tested packet encapsulations, and the actual type or\r
-content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
-their locality were not captured either. These were but few reasons that drove\r
-the decision to change and define a new more complete and stricter test naming\r
-convention, and to apply this to all existing and new test cases.\r
-\r
-The new naming should be intuitive for majority of the tests. The complete\r
-description of CSIT test naming convention is provided on `CSIT test naming\r
-page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
-\r
-Here few illustrative examples of the new naming usage for functional test\r
-suites:\r
-\r
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
-\r
- - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
- routed forwarding, functional tests.\r
-\r
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
- P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
-\r
- - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
- Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
- interfaces and one VM, functional tests.\r
+Overview
+========
+
+Virtual Topologies
+------------------
+
+CSIT HoneyComb functional tests are executed in VM-based virtual topologies
+created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)`
+simulation platform contributed by Cisco. VIRL runs on physical
+baremetal servers hosted by LF FD.io project.
+
+All tests are executed in two-node virtual test topology shown in the
+figure below.
+
+.. only:: latex
+
+ .. raw:: latex
+
+ \begin{figure}[H]
+ \centering
+ \graphicspath{{../_tmp/src/vpp_functional_tests/}}
+ \includegraphics[width=0.90\textwidth]{virtual-2n-nic2nic}
+ \label{fig:virtual-2n-nic2nic}
+ \end{figure}
+
+.. only:: html
+
+ .. figure:: ../vpp_functional_tests/virtual-2n-nic2nic.svg
+ :alt: virtual-2n-nic2nic
+ :align: center
+
+SUT (System Under Test) is a VM running Ubuntu Linux (or Centos,
+depending on the test suite), TG (Traffic Generator) is another VM
+running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP in
+Linux user-mode as a combined DUT (Device Under Test). TG runs Scapy
+application as a packet Traffic Generator. Virtual connectivity between
+SUT and TG is provided using virtual NICs using VMs' virtio drivers.
+
+Functional Tests Coverage
+-------------------------
+
+|csit-release| includes following HoneyComb functionality tested in
+virtual VM environment:
+
++-----------------------+----------------------------------------------+
+| Functionality | Description |
++=======================+==============================================+
+| ACL | - CRD for low-level classifiers: table and |
+| | session management, interface assignment. |
+| | - Configure up to 2 classify tables. |
+| | - Configure up to 2 classify sessions on one |
+| | table. |
+| | - Assign classify session to a physical |
+| | interface. |
+| | - Remove tables, sessions, interface |
+| | assignments. |
+| | - Test case count: 9. |
++-----------------------+----------------------------------------------+
+| ACL-PLUGIN | - CRD for high-level classifier. |
+| | - MAC + IP address classification. |
+| | - IPv4, IPv6 address classification. |
+| | - TCP, UDP, ICMP, ICMPv6 protocol and |
+| | next-header classification. |
+| | - port number classification. |
+| | - ICMP, ICMPv6 code and type classification. |
+| | - Test case count: 15. |
++-----------------------+----------------------------------------------+
+| Basic interface | - CRUD for interface state. |
+| management | - ipv4/ipv6 address, ipv4 neighbor, MTU |
+| | value. |
+| | - Test case count: 14. |
++-----------------------+----------------------------------------------+
+| Border Gateway | - CRUD and functional tests for BGP. |
+| Protocol | - Configure peers and routes |
+| | - Check interactions with another BGP peer. |
+| | - Test case count: 13. |
++-----------------------+----------------------------------------------+
+| DHCP Relay | - CRD for DHCP relay feature. |
+| | - Configure DHCP Relays. |
+| | - IPv4 and IPv6 variants. |
+| | - Test case count: 4. |
++-----------------------+----------------------------------------------+
+| Honeycomb | - Configuration persistence. |
+| Infractructure | - Netconf notifications for interface |
+| | events. |
+| | - Netconf negative tests aimed at specific |
+| | issues. |
+| | - Netconf/Restconf northbound over IPv6. |
+| | - Test case count: 14. |
++-----------------------+----------------------------------------------+
+| L2BD | - CRUD for L2 Bridge-Domain, interface |
+| | assignment. |
+| | - Create up to two bridge domains with all |
+| | implemented functions turned on: |
+| | flooding, unknown-unicast flooding, |
+| | forwarding, learning, arp-termination. |
+| | - Assign up to two physical interfaces to a |
+| | single bridge domain. |
+| | - Remove interface assignments, remove |
+| | bridge domains. |
+| | - Test case count: 5. |
++-----------------------+----------------------------------------------+
+| L2FIB | - CRD for L2-FIB entries. |
+| | - Create 4 FIB entries: |
+| | one of each for filter/forward, |
+| | static/dynamic combinations. |
+| | - Remove FIB entries. |
+| | - Test case count: 7. |
++-----------------------+----------------------------------------------+
+| LISP | - CRD for Lisp: mapping, locator set, |
+| | adjacency, mapresolver. |
+| | - Toggle Lisp feature status. |
+| | - Configure and delete Lisp mapping as local |
+| | and remote. |
+| | - Configure and delete Lisp adjacency |
+| | mapping. |
+| | - Configure and delete Lisp map resolver, |
+| | proxy ITR. |
+| | - Test case count: 18. |
++-----------------------+----------------------------------------------+
+| LISP GPE | - CRUD for LISP GPE mappings. |
+| | - Toggle Lisp GPE feature status. |
+| | - Configure Lisp GPE mappings. |
+| | - Traffic test verifying encapsulation. |
+| | - Test case count: 12. |
++-----------------------+----------------------------------------------+
+| NAT | - CRD for NAT entries, interface assignment. |
+| | - Configure and delete up to two NAT |
+| | entries. |
+| | - Assign NAT entries to a physical |
+| | interface. |
+| | - Test case count: 6. |
++-----------------------+----------------------------------------------+
+| NSH_SFC (excluded) | - CRD for NSH maps and entries, using |
+| | NSH_SFC plugin. |
+| | - Configure up to 2 NSH entries. |
+| | - Configure up to 2 NSH maps. |
+| | - Modify and delete NSH maps and entries. |
+| | - Test case count: 8. |
++-----------------------+----------------------------------------------+
+| PBB | - CRD for provider backbone bridge |
+| | sub-interface. |
+| | - Configure, modify and remove a PBB |
+| | sub-interface over a physical interface. |
+| | - Test case count: 8. |
++-----------------------+----------------------------------------------+
+| Policer | - CRD for traffic policing feature. |
+| | - Configure Policing rules. |
+| | - Assign to interface. |
+| | - Test case count: 6. |
++-----------------------+----------------------------------------------+
+| Port mirroring | - CRD for SPAN port mirroring, interface |
+| | assignment. |
+| | - Configure SPAN port mirroring on a |
+| | physical interface, mirroring. |
+| | - up to 2 interfaces. |
+| | - Remove SPAN configuration from interfaces. |
+| | - Test case count: 14. |
++-----------------------+----------------------------------------------+
+| ProxyARP | - CRD for proxyARP feature. |
+| | - Configure proxyARP. |
+| | - Assign to interface. |
+| | - Test case count: 3. |
++-----------------------+----------------------------------------------+
+| ProxyND6 | - CRD for Neighbor Discovery Proxy. |
+| | - Configure ProxyND6 feature on interface. |
+| | - Test case count: 4. |
++-----------------------+----------------------------------------------+
+| Routing | - CRD for routing. |
+| | - Configure single-hop route. |
+| | - Configure multi-hop routes. |
+| | - Configure blackhole route. |
+| | - IPv4 and IPv6 variants. |
+| | - Test case count: 6. |
++-----------------------+----------------------------------------------+
+| SLAAC | - CRD for Stateless Address |
+| | AutoConfiguration. |
+| | - Configure SLAAC feature on interfaces. |
+| | - Test case count: 7. |
++-----------------------+----------------------------------------------+
+| Vhost-user | - CRUD for Vhost-user interfaces. |
+| | - Create, modify and delete Vhost-user |
+| | interface, as client and server. |
+| | - Test case count: 8. |
++-----------------------+----------------------------------------------+
+| VLAN | - CRUD for VLAN sub-interface management. |
+| | - Create VLAN sub-interface over a physical |
+| | interface. |
+| | - Toggle interface state separately for |
+| | super-interface and sub-interface. |
+| | - Configure IP address and bridge domain |
+| | assignment on sub-interface. |
+| | - Configure VLAN tag rewrite on |
+| | sub-interface. |
+| | - Test case count: 24. |
++-----------------------+----------------------------------------------+
+| VxLAN | - CRD for VxLAN tunnels. |
+| | - Create VxLAN interface. |
+| | - Disable VxLAN interface. |
+| | - Re-create a disabled VxLAN interface. |
+| | - Test case count: 6. |
++-----------------------+----------------------------------------------+
+| VxLAN-GPE | - CRD for VxLAN GPE tunnels. |
+| | - Create VxLAN GPE interface. |
+| | - Disable VxLAN interface. |
+| | - Re-create a disabled VxLAN interface. |
+| | - Test case count: 7. |
++-----------------------+----------------------------------------------+
+| TAP | - CRUD for Tap interface management. |
+| | - Create, modify and delete TAP interface. |
+| | - Test case count: 3. |
++-----------------------+----------------------------------------------+
+
+Total 213 Honeycomb functional tests in the |csit-release|.
+
+Operational data in Honeycomb should mirror configuration data at all
+times. Because of this, test cases follow this general pattern:
+
+#. read operational data of the feature using restconf.
+#. read status of the feature using VPP API dump.
+#. modify configuration of the feature using restconf.
+#. verify changes to operational data using restconf.
+#. verify changes using VPP API dump, OR
+#. send a packet to VPP node and observe behaviour to verify configuration.
+
+Test cases involving network interfaces utilize the first two interfaces
+on the DUT node.
+
+Functional Tests Naming
+-----------------------
+
+|csit-release| follows a common structured naming convention for all
+performance and system functional tests, introduced in CSIT-17.01.
+
+The naming should be intuitive for majority of the tests. Complete
+description of CSIT test naming convention is provided on
+:ref:`csit_test_naming`.
Code Review / csit.git / blobdiff