--- /dev/null
+Overview\r
+========\r
+\r
+Tested Virtual Topologies\r
+-------------------------\r
+\r
+CSIT Honeycomb functional tests are executed on virtualized topologies created\r
+using Virtual Internet Routing Lab (VIRL) simulation platform contributed by\r
+Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io project. All\r
+tests are executed in two node logical test topology - Traffic Generator (TG)\r
+node and Systems Under Test (SUT1) node connected in a loop. Logical test\r
+topology is shown in the figure below.\r
+\r
+::\r
+\r
+ +------------------------+\r
+ | |\r
+ | +------------------+ |\r
+ +---------------> <--------------+\r
+ | | | | | |\r
+ | |------------> DUT1 <-----------+ |\r
+ | | | +------------------+ | | |\r
+ | | | | | |\r
+ | | | SUT1 | | |\r
+ | | +------------------------+ | |\r
+ | | | |\r
+ | | | |\r
+ | | +-----------+ | |\r
+ | +---------------> <---------------+ |\r
+ | | TG | |\r
+ +------------------> <------------------+\r
+ +-----------+\r
+\r
+SUT1 is a VM (Ubuntu or Centos, depending on the test suite), TG is a Traffic\r
+Generator (TG, another Ubuntu VM). SUTs run Honeycomb and VPP SW applications\r
+in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy\r
+SW application as a packet Traffic Generator. Logical connectivity between\r
+SUTs and to TG is provided using virtual NICs using VMs' virtio driver.\r
+\r
+Virtual testbeds are created on-demand whenever a verification job is started\r
+(e.g. triggered by the gerrit patch submission) and destroyed upon completion\r
+of all functional tests. Each node is a Virtual Machine and each connection\r
+that is drawn on the diagram is available for use in any test case. During the\r
+test execution, all nodes are reachable thru the Management network connected\r
+to every node via dedicated virtual NICs and virtual links (not shown above\r
+for clarity).\r
+\r
+Functional Tests Coverage\r
+-------------------------\r
+\r
+The following Honeycomb functional test areas are included in the CSIT |release|\r
+with results listed in this report:\r
+\r
+- **Basic interface management** - CRUD for interface state,\r
+ - ipv4/ipv6 address, ipv4 neighbor, MTU value.\r
+ - Test case count: 7\r
+- **L2BD** - CRUD for L2 Bridge-Domain, interface assignment.\r
+ - Create up to two bridge domains with all implemented functions turned on.\r
+ - (flooding, unknown-unicast flooding, forwarding, learning, arp-termination)\r
+ - Assign up to two physical interfaces to a single bridge domain.\r
+ - Remove interface assignments, remove bridge domains.\r
+ - Test case count: 5\r
+- **L2FIB** - CRD for L2-FIB entries.\r
+ - Create 4 FIB entries\r
+ - (one of each for filter/forward, static/dynamic combinations).\r
+ - Remove FIB entries.\r
+ - Test case count: 7\r
+- **VxLAN** - CRD for VxLAN tunnels.\r
+ - Create VxLAN interface.\r
+ - Disable VxLAN interface.\r
+ - Re-create a disabled VxLAN interface.\r
+ - Test case count: 6\r
+- **VxLAN-GPE** - CRD for VxLAN GPE tunnels.\r
+ - Create VxLAN GPE interface.\r
+ - Disable VxLAN interface.\r
+ - Re-create a disabled VxLAN interface.\r
+ - Test case count: 7\r
+- **Vhost-user** - CRUD for Vhost-user interfaces.\r
+ - Create, modify and delete Vhost-user interface, as client and server.\r
+ - Test case count: 8\r
+- **TAP** - CRUD for Tap interface management.\r
+ - Create, modify and delete TAP interface.\r
+ - Test case count: 3\r
+- **VLAN** - CRUD for VLAN sub-interface management.\r
+ - Create VLAN sub-interface over a physical interface.\r
+ - Toggle interface state separately for super-interface and sub-interface.\r
+ - Configure IP address and bridge domain assignment on sub-interface.\r
+ - Configure VLAN tag rewrite on sub-interface.\r
+ - Test case count: 17\r
+- **ACL** - CRD for low-level classifiers: table and session management,\r
+ - interface assignment.\r
+ - Configure up to 2 classify tables.\r
+ - Configure up to 2 classify sessions on one table.\r
+ - Assign classify session to a physical interface.\r
+ - Remove tables, sessions, interface assignments.\r
+ - Test case count: 9\r
+- **PBB** - CRD for provider backbone bridge sub-interface.\r
+ - Configure, modify and remove a PBB sub-interface over a physical interface.\r
+ - Test case count: 9\r
+- **NSH_SFC** - CRD for NSH maps and entries, using NSH_SFC plugin.\r
+ - Configure up to 2 NSH entries.\r
+ - Configure up to 2 NSH maps.\r
+ - Modify and delete NSH maps and entries.\r
+ - Test case count: 8\r
+- **LISP** - CRD for Lisp: mapping, locator set, adjacency, map resolver.\r
+ - Toggle Lisp feature status.\r
+ - Configure and delete Lisp mapping as local and remote.\r
+ - Configure and delete Lisp adjacency mapping\r
+ - Configure and delete Lisp map resolver, proxy ITR.\r
+ - Test case count: 11\r
+- **NAT** - CRD for NAT entries, interface assignment.\r
+ - Configure and delete up to two NAT entries.\r
+ - Assign NAT entries to a physical interface.\r
+ - Test case count: 6\r
+- **Port mirroring** - CRD for SPAN port mirroring, interface assignment.\r
+ - Configure SPAN port mirroring on a physical interface, mirroring\r
+ - up to 2 interfaces.\r
+ - Remove SPAN configuration from interfaces.\r
+ - Test case count: 3\r
+- **Honeycomb Infractructure** - configuration persistence,\r
+ - Netconf notifications for interface events,\r
+ - Netconf negative tests aimed at specific issues\r
+\r
+Total 111 Honeycomb tests in the CSIT |release|.\r
+\r
+Operational data in Honeycomb should mirror configuration data at all times.\r
+Because of this, test cases follow this general pattern:\r
+\r
+#. read operational data of the feature using restconf.\r
+#. read status of the feature using VPP API dump.\r
+#. modify configuration of the feature using restconf.\r
+#. verify changes to operational data using restconf.\r
+#. verify changes using VPP API dump.\r
+\r
+Test cases involving network interfaces utilize the first two interfaces on\r
+the DUT node.\r
+\r
+Functional Tests Naming\r
+-----------------------\r
+\r
+CSIT |release| introduced a common structured naming convention for all\r
+performance and functional tests. This change was driven by substantially\r
+growing number and type of CSIT test cases. Firstly, the original practice did\r
+not always follow any strict naming convention. Secondly test names did not\r
+always clearly capture tested packet encapsulations, and the actual type or\r
+content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
+their locality were not captured either. These were but few reasons that drove\r
+the decision to change and define a new more complete and stricter test naming\r
+convention, and to apply this to all existing and new test cases.\r
+\r
+The new naming should be intuitive for majority of the tests. The complete\r
+description of CSIT test naming convention is provided on `CSIT test naming\r
+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
+\r
+Here few illustrative examples of the new naming usage for functional test\r
+suites:\r
+\r
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
+\r
+ - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
+ routed forwarding, functional tests.\r
+\r
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
+\r
+ - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
+ interfaces and one VM, functional tests.\r
Code Review / csit.git / blobdiff