From c2ce046f71f7de1f247be68513e3d4e10c8c2a04 Mon Sep 17 00:00:00 2001 From: Maciek Konstantynowicz Date: Tue, 14 Aug 2018 17:30:45 +0100 Subject: [PATCH] 1807 report: updated virtual topologies for all functional tests. Change-Id: I3053c7a9733ca7c03df993698472ad8d031dba96 Signed-off-by: Maciek Konstantynowicz --- docs/report/dmm_functional_tests/overview.rst | 80 +- docs/report/dpdk_performance_tests/overview.rst | 2 - .../report/honeycomb_functional_tests/overview.rst | 394 ++++--- docs/report/nsh_sfc_functional_tests/overview.rst | 130 ++- docs/report/vpp_functional_tests/overview.rst | 211 ++-- .../vpp_functional_tests/virtual-2n-nic2nic.svg | 315 ++++++ .../vpp_functional_tests/virtual-3n-nic2nic.svg | 452 ++++++++ .../vpp_functional_tests/virtual-3n-vm-vhost.svg | 1110 ++++++++++++++++++++ docs/report/vpp_performance_tests/overview.rst | 6 +- 9 files changed, 2313 insertions(+), 387 deletions(-) create mode 100644 docs/report/vpp_functional_tests/virtual-2n-nic2nic.svg create mode 100644 docs/report/vpp_functional_tests/virtual-3n-nic2nic.svg create mode 100644 docs/report/vpp_functional_tests/virtual-3n-vm-vhost.svg diff --git a/docs/report/dmm_functional_tests/overview.rst b/docs/report/dmm_functional_tests/overview.rst index d30398a282..aa9e853817 100644 --- a/docs/report/dmm_functional_tests/overview.rst +++ b/docs/report/dmm_functional_tests/overview.rst @@ -4,20 +4,13 @@ Overview Virtual Topologies ------------------ -CSIT DMM functional tests are executed on virtualized topologies created using -:abbr:`VIRL (Virtual Internet Routing Lab)` simulation platform contributed by -Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io project. -Based on the packet path through server SUTs, one logical topology type -is used for DMM DUT data plane testing: +CSIT DMM functional tests are executed in VM-based virtual topologies +created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)` +simulation platform contributed by Cisco. VIRL runs on physical +baremetal servers hosted by LF FD.io project. -#. NIC-to-NIC switching topologies. - -NIC-to-NIC Switching -~~~~~~~~~~~~~~~~~~~~ - -The simplest logical topology for software data plane application like -DMM is NIC-to-NIC switching. Tested topologies for the 3-Node -testbed is shown in the figure below. +All tests are executed in three-node virtual test topology shown in the +figure below. .. only:: latex @@ -25,42 +18,37 @@ testbed is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-3n-nic2nic} - \label{fig:logical-3n-nic2nic} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-3n-nic2nic} + \label{fig:virtual-3n-nic2nic} \end{figure} .. only:: html - .. figure:: ../vpp_performance_tests/logical-3n-nic2nic.svg - :alt: logical-3n-nic2nic + .. figure:: ../vpp_functional_tests/virtual-3n-nic2nic.svg + :alt: virtual-3n-nic2nic :align: center -SUT1 and SUT2 are two VMs (Ubuntu), TG -is a Traffic Generator (TG, another Ubuntu VM). SUTs run sample server-client -application using the DMM libs in Linux user-mode as a Device Under Test (DUT) -within the VM. Currently TG node is not being used in DMM-CSIT. Logical -connectivity between SUTs is provided using virtual NICs using VMs' virtio -driver. - -Virtual testbeds are created on-demand whenever a verification job is started -(e.g. triggered by the gerrit patch submission) and destroyed upon completion -of all functional tests. Each node is a Virtual Machine and each connection -that is drawn on the diagram is available for use in any test case. During the -test execution, all nodes are reachable through the Management network connected -to every node via dedicated virtual NICs and virtual links (not shown above -for clarity). - -DMM Functional Tests Coverage ------------------------------ - -Following DMM functional test areas are covered in the |csit-release| with -results listed in this report: - -- **DMM basic testcase** - DMM has only one test case right now. - The testcase demonstrates single server[DUT1] and single client[DUT2] scenario - using DMM framework and kernel tcp/ip stack. - - - Test case count: 1 - -Total 1 DMM functional test in the |csit-release|. +SUT1 and SUT2 are two VMs (running Ubuntu), TG is a Traffic Generator VM +(running Ubuntu). SUTs run +sample server-client application using the DMM libraries in Linux user- +mode as a Device Under Test (DUT) within the VM. Currently TG node is +not being used in DMM-CSIT. Network connectivity between SUTs and to TG +is provided using virtual NICs and VMs' virtio drivers. + +Functional Tests Coverage +------------------------- + +|csit-release| includes following DMM functionality tested in virtual VM +environment: + ++-----------------------+----------------------------------------------+ +| Functionality | Description | ++=======================+==============================================+ +| DMM basic operation | DMM has only one test case right now. The | +| | testcase demonstrates single server[DUT1] | +| | and single client[DUT2] scenario using DMM | +| | framework and kernel tcp/ip stack. | +| | | +| | Test case count: 1. | ++-----------------------+----------------------------------------------+ diff --git a/docs/report/dpdk_performance_tests/overview.rst b/docs/report/dpdk_performance_tests/overview.rst index 41881b8c8e..499f0068e4 100644 --- a/docs/report/dpdk_performance_tests/overview.rst +++ b/docs/report/dpdk_performance_tests/overview.rst @@ -4,8 +4,6 @@ Overview For description of physical testbeds used for DPDK performance tests please refer to :ref:`tested_physical_topologies`. -.. _tested_logical_topologies: - Logical Topologies ------------------ diff --git a/docs/report/honeycomb_functional_tests/overview.rst b/docs/report/honeycomb_functional_tests/overview.rst index 3ff8a6e2e9..b57d2b4094 100644 --- a/docs/report/honeycomb_functional_tests/overview.rst +++ b/docs/report/honeycomb_functional_tests/overview.rst @@ -1,15 +1,16 @@ Overview ======== -Tested Virtual Topologies -------------------------- +Virtual Topologies +------------------ + +CSIT HoneyComb functional tests are executed in VM-based virtual topologies +created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)` +simulation platform contributed by Cisco. VIRL runs on physical +baremetal servers hosted by LF FD.io project. -CSIT Honeycomb functional tests are executed on virtualized topologies created -using :abbr:`VIRL (Virtual Internet Routing Lab)` simulation platform -contributed by Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io -project. All tests are executed in two node logical test topology - Traffic -Generator (TG) node and Systems Under Test (SUT1) node connected in a loop. -Logical test topology is shown in the figure below. +All tests are executed in two-node virtual test topology shown in the +figure below. .. only:: latex @@ -17,153 +18,226 @@ Logical test topology is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-2n-nic2nic} - \label{fig:logical-2n-nic2nic} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-2n-nic2nic} + \label{fig:virtual-2n-nic2nic} \end{figure} .. only:: html - .. figure:: logical-2n-nic2nic.svg - :alt: logical-2n-nic2nic + .. figure:: ../vpp_functional_tests/virtual-2n-nic2nic.svg + :alt: virtual-2n-nic2nic :align: center -SUT1 is a VM (Ubuntu or Centos, depending on the test suite), TG is a Traffic -Generator (TG, another Ubuntu VM). SUTs run Honeycomb and VPP SW applications -in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy -SW application as a packet Traffic Generator. Logical connectivity between -SUTs and to TG is provided using virtual NICs using VMs' virtio driver. - -Virtual testbeds are created on-demand whenever a verification job is started -(e.g. triggered by the gerrit patch submission) and destroyed upon completion -of all functional tests. Each node is a Virtual Machine and each connection -that is drawn on the diagram is available for use in any test case. During the -test execution, all nodes are reachable thru the Management network connected -to every node via dedicated virtual NICs and virtual links (not shown above -for clarity). +SUT (System Under Test) is a VM running Ubuntu Linux (or Centos, +depending on the test suite), TG (Traffic Generator) is another VM +running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP in +Linux user-mode as a combined DUT (Device Under Test). TG runs Scapy +application as a packet Traffic Generator. Virtual connectivity between +SUT and TG is provided using virtual NICs using VMs' virtio drivers. Functional Tests Coverage ------------------------- -The following Honeycomb functional test areas are included in the |csit-release| -with results listed in this report: - -- **Basic interface management** - CRUD for interface state, - - ipv4/ipv6 address, ipv4 neighbor, MTU value. - - Test case count: 14 -- **L2BD** - CRUD for L2 Bridge-Domain, interface assignment. - - Create up to two bridge domains with all implemented functions turned on. - - (flooding, unknown-unicast flooding, forwarding, learning, arp-termination) - - Assign up to two physical interfaces to a single bridge domain. - - Remove interface assignments, remove bridge domains. - - Test case count: 5 -- **L2FIB** - CRD for L2-FIB entries. - - Create 4 FIB entries - - (one of each for filter/forward, static/dynamic combinations). - - Remove FIB entries. - - Test case count: 7 -- **VxLAN** - CRD for VxLAN tunnels. - - Create VxLAN interface. - - Disable VxLAN interface. - - Re-create a disabled VxLAN interface. - - Test case count: 6 -- **VxLAN-GPE** - CRD for VxLAN GPE tunnels. - - Create VxLAN GPE interface. - - Disable VxLAN interface. - - Re-create a disabled VxLAN interface. - - Test case count: 7 -- **Vhost-user** - CRUD for Vhost-user interfaces. - - Create, modify and delete Vhost-user interface, as client and server. - - Test case count: 8 -- **TAP** - CRUD for Tap interface management. - - Create, modify and delete TAP interface. - - Test case count: 3 -- **VLAN** - CRUD for VLAN sub-interface management. - - Create VLAN sub-interface over a physical interface. - - Toggle interface state separately for super-interface and sub-interface. - - Configure IP address and bridge domain assignment on sub-interface. - - Configure VLAN tag rewrite on sub-interface. - - Test case count: 24 -- **ACL** - CRD for low-level classifiers: table and session management, - - interface assignment. - - Configure up to 2 classify tables. - - Configure up to 2 classify sessions on one table. - - Assign classify session to a physical interface. - - Remove tables, sessions, interface assignments. - - Test case count: 9 -- **PBB** - CRD for provider backbone bridge sub-interface. - - Configure, modify and remove a PBB sub-interface over a physical interface. - - Test case count: 8 -- **NSH_SFC** - CRD for NSH maps and entries, using NSH_SFC plugin. - - Configure up to 2 NSH entries. - - Configure up to 2 NSH maps. - - Modify and delete NSH maps and entries. - - Test case count: 8 -- **LISP** - CRD for Lisp: mapping, locator set, adjacency, map resolver. - - Toggle Lisp feature status. - - Configure and delete Lisp mapping as local and remote. - - Configure and delete Lisp adjacency mapping - - Configure and delete Lisp map resolver, proxy ITR. - - Test case count: 18 -- **LISP GPE** - CRUD for LISP GPE mappings. - - Toggle Lisp GPE feature status. - - Configure Lisp GPE mappings. - - Traffic test verifying encapsulation. - - Test case count: 12 -- **NAT** - CRD for NAT entries, interface assignment. - - Configure and delete up to two NAT entries. - - Assign NAT entries to a physical interface. - - Test case count: 6 -- **Port mirroring** - CRD for SPAN port mirroring, interface assignment. - - Configure SPAN port mirroring on a physical interface, mirroring - - up to 2 interfaces. - - Remove SPAN configuration from interfaces. - - Test case count: 14 -- **ACL-PLUGIN** - CRD for high-level classifier - - MAC + IP address classification. - - IPv4, IPv6 address classification. - - TCP, UDP, ICMP, ICMPv6 protocol/next-header classification. - - port number classification. - - ICMP, ICMPv6 code and type classification. - - Test case count: 15 -- **ProxyARP** - CRD for proxyARP feature. - - Configure proxyARP. - - Assign to interface. - - Test case count: 3 -- **ProxyND6** - CRD for Neighbor Discovery Proxy. - - Configure ProxyND6 feature on interface. - - Test case count: 4 -- **DHCP Relay** - CRD for DHCP relay feature. - - Configure DHCP Relays. - - IPv4 and IPv6 variants. - - Test case count: 4 -- **SLAAC** - CRD for Stateless Address AutoConfiguration. - - Configure SLAAC feature on interfaces. - - Test case count: 7 -- **Routing** - CRD for routing. - - Configure single-hop route. - - Configure multi-hop routes. - - Configure blackhole route. - - IPv4 and IPv6 variants. - - Test case count: 6 -- **Policer** - CRD for traffic policing feature. - - Configure Policing rules. - - Assign to interface. - - Test case count: 6 -- **Border Gateway Protocol** - CRUD and functional tests for BGP. - - Configure peers and routes - - Check interactions with another BGP peer. - - Test case count: 13 -- **Honeycomb Infractructure** - configuration persistence, - - Netconf notifications for interface events, - - Netconf negative tests aimed at specific issues - - Netconf/Restconf northbound over IPv6 - - Test case count: 12 +|csit-release| includes following HoneyComb functionality tested in +virtual VM environment: + ++-----------------------+----------------------------------------------+ +| Functionality | Description | ++=======================+==============================================+ +| ACL | CRD for low-level classifiers: table and | +| | session management, interface assignment. | +| | | +| | - Configure up to 2 classify tables. | +| | - Configure up to 2 classify sessions on one | +| | table. | +| | - Assign classify session to a physical | +| | interface. | +| | - Remove tables, sessions, interface | +| | assignments. | +| | - Test case count: 9. | ++-----------------------+----------------------------------------------+ +| ACL-PLUGIN | CRD for high-level classifier. | +| | | +| | - MAC + IP address classification. | +| | - IPv4, IPv6 address classification. | +| | - TCP, UDP, ICMP, ICMPv6 protocol and | +| | next-header classification. | +| | - port number classification. | +| | - ICMP, ICMPv6 code and type classification. | +| | - Test case count: 15. | ++-----------------------+----------------------------------------------+ +| Basic interface | CRUD for interface state. | +| management | | +| | - ipv4/ipv6 address, ipv4 neighbor, MTU | +| | value. | +| | - Test case count: 14. | ++-----------------------+----------------------------------------------+ +| Border Gateway | CRUD and functional tests for BGP. | +| Protocol | | +| | - Configure peers and routes | +| | - Check interactions with another BGP peer. | +| | - Test case count: 13. | ++-----------------------+----------------------------------------------+ +| DHCP Relay | CRD for DHCP relay feature. | +| | | +| | - Configure DHCP Relays. | +| | - IPv4 and IPv6 variants. | +| | - Test case count: 4. | ++-----------------------+----------------------------------------------+ +| Honeycomb | Configuration persistence. | +| Infractructure | | +| | - Netconf notifications for interface | +| | events. | +| | - Netconf negative tests aimed at specific | +| | issues. | +| | - Netconf/Restconf northbound over IPv6. | +| | - Test case count: 12. | ++-----------------------+----------------------------------------------+ +| L2BD | CRUD for L2 Bridge-Domain, interface | +| | assignment. | +| | | +| | - Create up to two bridge domains with all | +| | implemented functions turned on: | +| | flooding, unknown-unicast flooding, | +| | forwarding, learning, arp-termination. | +| | - Assign up to two physical interfaces to a | +| | single bridge domain. | +| | - Remove interface assignments, remove | +| | bridge domains. | +| | - Test case count: 5. | ++-----------------------+----------------------------------------------+ +| L2FIB | CRD for L2-FIB entries. | +| | | +| | - Create 4 FIB entries: | +| | one of each for filter/forward, | +| | static/dynamic combinations. | +| | - Remove FIB entries. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| LISP | CRD for Lisp: mapping, locator set, | +| | adjacency, mapresolver. | +| | | +| | - Toggle Lisp feature status. | +| | - Configure and delete Lisp mapping as local | +| | and remote. | +| | - Configure and delete Lisp adjacency | +| | mapping. | +| | - Configure and delete Lisp map resolver, | +| | proxy ITR. | +| | - Test case count: 18. | ++-----------------------+----------------------------------------------+ +| LISP GPE | CRUD for LISP GPE mappings. | +| | | +| | - Toggle Lisp GPE feature status. | +| | - Configure Lisp GPE mappings. | +| | - Traffic test verifying encapsulation. | +| | - Test case count: 12. | ++-----------------------+----------------------------------------------+ +| NAT | CRD for NAT entries, interface assignment. | +| | | +| | - Configure and delete up to two NAT | +| | entries. | +| | - Assign NAT entries to a physical | +| | interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| NSH_SFC | CRD for NSH maps and entries, using NSH_SFC | +| | plugin. | +| | | +| | - Configure up to 2 NSH entries. | +| | - Configure up to 2 NSH maps. | +| | - Modify and delete NSH maps and entries. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| PBB | CRD for provider backbone bridge | +| | sub-interface. | +| | | +| | - Configure, modify and remove a PBB | +| | sub-interface over a physical interface. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| Policer | CRD for traffic policing feature. | +| | | +| | - Configure Policing rules. | +| | - Assign to interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| Port mirroring | CRD for SPAN port mirroring, interface | +| | assignment. | +| | | +| | - Configure SPAN port mirroring on a | +| | physical interface, mirroring. | +| | - up to 2 interfaces. | +| | - Remove SPAN configuration from interfaces. | +| | - Test case count: 14. | ++-----------------------+----------------------------------------------+ +| ProxyARP | CRD for proxyARP feature. | +| | | +| | - Configure proxyARP. | +| | - Assign to interface. | +| | - Test case count: 3. | ++-----------------------+----------------------------------------------+ +| ProxyND6 | CRD for Neighbor Discovery Proxy. | +| | | +| | - Configure ProxyND6 feature on interface. | +| | - Test case count: 4. | ++-----------------------+----------------------------------------------+ +| Routing | CRD for routing. | +| | | +| | - Configure single-hop route. | +| | - Configure multi-hop routes. | +| | - Configure blackhole route. | +| | - IPv4 and IPv6 variants. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| SLAAC | CRD for Stateless Address AutoConfiguration. | +| | | +| | - Configure SLAAC feature on interfaces. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| Vhost-user | CRUD for Vhost-user interfaces. | +| | | +| | - Create, modify and delete Vhost-user | +| | interface, as client and server. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| VLAN | CRUD for VLAN sub-interface management. | +| | | +| | - Create VLAN sub-interface over a physical | +| | interface. | +| | - Toggle interface state separately for | +| | super-interface and sub-interface. | +| | - Configure IP address and bridge domain | +| | assignment on sub-interface. | +| | - Configure VLAN tag rewrite on | +| | sub-interface. | +| | - Test case count: 24. | ++-----------------------+----------------------------------------------+ +| VxLAN | CRD for VxLAN tunnels. | +| | | +| | - Create VxLAN interface. | +| | - Disable VxLAN interface. | +| | - Re-create a disabled VxLAN interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| VxLAN-GPE | CRD for VxLAN GPE tunnels. | +| | | +| | - Create VxLAN GPE interface. | +| | - Disable VxLAN interface. | +| | - Re-create a disabled VxLAN interface. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| TAP | CRUD for Tap interface management. | +| | | +| | - Create, modify and delete TAP interface. | +| | - Test case count: 3. | ++-----------------------+----------------------------------------------+ Total 219 Honeycomb functional tests in the |csit-release|. -Operational data in Honeycomb should mirror configuration data at all times. -Because of this, test cases follow this general pattern: +Operational data in Honeycomb should mirror configuration data at all +times. Because of this, test cases follow this general pattern: #. read operational data of the feature using restconf. #. read status of the feature using VPP API dump. @@ -172,37 +246,15 @@ Because of this, test cases follow this general pattern: #. verify changes using VPP API dump, OR #. send a packet to VPP node and observe behaviour to verify configuration. -Test cases involving network interfaces utilize the first two interfaces on -the DUT node. +Test cases involving network interfaces utilize the first two interfaces +on the DUT node. Functional Tests Naming ----------------------- -CSIT-17.01 introduced a common structured naming convention for all -performance and functional tests. This change was driven by substantially -growing number and type of CSIT test cases. Firstly, the original practice did -not always follow any strict naming convention. Secondly test names did not -always clearly capture tested packet encapsulations, and the actual type or -content of the tests. Thirdly HW configurations in terms of NICs, ports and -their locality were not captured either. These were but few reasons that drove -the decision to change and define a new more complete and stricter test naming -convention, and to apply this to all existing and new test cases. - -The new naming should be intuitive for majority of the tests. The complete -description of CSIT test naming convention is provided on `CSIT test naming -page `_. - -Here few illustrative examples of the new naming usage for functional test -suites: - -#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - - - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline - routed forwarding, functional tests. - -#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, - P2V2P, NIC2VMchain2NIC, P2V2V2P** +|csit-release| follows a common structured naming convention for all +performance and system functional tests, introduced in CSIT-17.01. - - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of - Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost - interfaces and one VM, functional tests. +The naming should be intuitive for majority of the tests. Complete +description of CSIT test naming convention is provided on +:ref:`csit_test_naming`. diff --git a/docs/report/nsh_sfc_functional_tests/overview.rst b/docs/report/nsh_sfc_functional_tests/overview.rst index c9156fb22d..6d36f377ce 100644 --- a/docs/report/nsh_sfc_functional_tests/overview.rst +++ b/docs/report/nsh_sfc_functional_tests/overview.rst @@ -4,20 +4,13 @@ Overview Virtual Topologies ------------------ -CSIT NSH_SFC functional tests are executed on virtualized topologies created -using :abbr:`VIRL (Virtual Internet Routing Lab)` simulation platform -contributed by Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io -project. Based on the packet path thru server SUTs, one logical -topology type is used for VPP DUT data plane testing: +CSIT NSH_SFC functional tests are executed in VM-based virtual topologies +created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)` +simulation platform contributed by Cisco. VIRL runs on physical +baremetal servers hosted by LF FD.io project. -#. NIC-to-NIC switching topologies. - -NIC-to-NIC Switching -~~~~~~~~~~~~~~~~~~~~ - -The simplest logical topology for software data plane application like -VPP is NIC-to-NIC switching. Tested topology for the 3-Node -testbeds is shown in the figure below. +All tests are executed in three-node virtual test topology shown in the +figure below. .. only:: latex @@ -25,62 +18,67 @@ testbeds is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-3n-nic2nic} - \label{fig:logical-3n-nic2nic} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-3n-nic2nic} + \label{fig:virtual-3n-nic2nic} \end{figure} .. only:: html - .. figure:: ../vpp_performance_tests/logical-3n-nic2nic.svg - :alt: logical-3n-nic2nic + .. figure:: ../vpp_functional_tests/virtual-3n-nic2nic.svg + :alt: virtual-3n-nic2nic :align: center -SUT1 and SUT2 are two VMs (Ubuntu or Centos, depending on the test suite), TG -is a Traffic Generator (TG, another Ubuntu VM). SUTs run NSH_SFC SW application -in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy SW -application as a packet Traffic Generator. Logical connectivity between SUTs -and to TG is provided using virtual NICs using VMs' virtio driver. - -Virtual testbeds are created on-demand whenever a verification job is started -(e.g. triggered by the gerrit patch submission) and destroyed upon completion -of all functional tests. Each node is a Virtual Machine and each connection -that is drawn on the diagram is available for use in any test case. During the -test execution, all nodes are reachable thru the Management network connected -to every node via dedicated virtual NICs and virtual links (not shown above -for clarity). - -NSH_SFC Functional Tests Coverage ---------------------------------- - -Following NSH_SFC functional test areas are covered in the |csit-release| with -results listed in this report: - -- **NSH SFC Classifier** - TG sends some TCP packets to test NSH SFC - Classifier functional. DUT1 will receive these packets from one NIC and - loopback the VXLAN-GPE-NSH encapsulated packets to the TG from other NIC. - - - Test case count: 7 - -- **NSH SFC Proxy Inbound** - TG sends some VXLAN-GPE-NSH encapsulated packets - to test NSH SFC Proxy Inbound functional. DUT1 will receive these packets from - one NIC and loopback the VXLAN encapsulated packets to the TG from other NIC. - - - Test case count: 6 - -- **NSH SFC Proxy Outbound** - TG sends some VXLAN encapsulated packets to test - NSH SFC Proxy Outbound functional. DUT1 will receive these packets from one - NIC and loopback the VXLAN-GPE-NSH encapsulated packets to the TG from other - NIC. - - - Test case count: 6 - -- **NSH SFC Service Function Forward** - TG sends some VXLAN-GPE-NSH - encapsulated packets to test NSH SFC Service Function Forward functional. DUT1 - will receive these packets from one NIC and swap the VXLAN-GPE-NSH header, - after that DUT1 loopback the VXLAN-GPE-NSH encapsulated packtes to the TG from - other NIC. - - - Test case count: 6 - -Total 25 NSH SFC functional tests in the |csit-release|. +SUT1 and SUT2 are two VMs (running Ubuntu or Centos, depending on the +test suite), TG is a Traffic Generator (running Ubuntu VM). SUTs run VPP +with nsh-plugin in Linux user-mode as a Device Under Test (DUT) within +the VM. TG runs Scapy SW application as a packet Traffic Generator. +Network connectivity between SUTs and to TG is provided using virtual +NICs and VMs' virtio drivers. + +Functional Tests Coverage +------------------------- + +|csit-release| includes following NSH_SFC functionality tested in +virtual VM environment: + ++-----------------------+----------------------------------------------+ +| Functionality | Description | ++=======================+==============================================+ +| NSH SFC Classifier | TG sends some TCP packets to test NSH SFC | +| | Classifier functional. DUT1 will receive | +| | these packets from one NIC and loopback the | +| | VXLAN-GPE-NSH encapsulated packets to the TG | +| | from other NIC. | +| | | +| | Test case count: 7. | ++-----------------------+----------------------------------------------+ +| NSH SFC Proxy Inbound | TG sends some VXLAN-GPE-NSH encapsulated | +| | packets to test NSH SFC Proxy Inbound | +| | functional. DUT1 will receive these packets | +| | from one NIC and loopback the VXLAN | +| | encapsulated packets to the TG from other | +| | NIC. | +| | | +| | Test case count: 6. | ++-----------------------+----------------------------------------------+ +| NSH SFC Proxy | TG sends some VXLAN encapsulated packets to | +| Outbound | test NSH SFC Proxy Outbound functional. DUT1 | +| | will receive these packets from one NIC and | +| | loopback the VXLAN-GPE-NSH encapsulated | +| | packets to the TG from other NIC. | +| | | +| | Test case count: 6. | ++-----------------------+----------------------------------------------+ +| NSH SFC Service | TG sends some VXLAN-GPE-NSH | +| Function Forward | encapsulated packets to test NSH SFC Service | +| | Function Forward functional. DUT1 will | +| | receive these packets from one NIC and swap | +| | the VXLAN-GPE-NSH header, after that DUT1 | +| | loopback the VXLAN-GPE-NSH encapsulated | +| | packets to the TG from other NIC. | +| | | +| | Test case count: 6. | ++-----------------------+----------------------------------------------+ + +Total 25 NSH_SFC functional tests in the |csit-release|. diff --git a/docs/report/vpp_functional_tests/overview.rst b/docs/report/vpp_functional_tests/overview.rst index 035696dc77..8ce516cf3d 100644 --- a/docs/report/vpp_functional_tests/overview.rst +++ b/docs/report/vpp_functional_tests/overview.rst @@ -4,39 +4,23 @@ Overview Virtual Topologies ------------------ -CSIT VPP functional tests are executed on virtualized topologies created using -:abbr:`VIRL (Virtual Internet Routing Lab)` simulation platform contributed by -Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io project. -Based on the packet path thru server SUTs, two distinct logical topology types -are used for VPP DUT data plane testing: +CSIT VPP functional tests are executed in VM-based virtual topologies +created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)` +simulation platform contributed by Cisco. VIRL runs on physical +baremetal servers hosted by LF FD.io project. -#. NIC-to-NIC switching topologies. -#. VM service switching topologies. +Based on the packet path thru SUT VMs, two distinct logical topology +types are used for VPP DUT data plane testing: -NIC-to-NIC Switching -~~~~~~~~~~~~~~~~~~~~ +#. vNIC-to-vNIC switching topologies. +#. Nested-VM service switching topologies. -The simplest logical topology for software data plane application like -VPP is NIC-to-NIC switching. Tested topologies for 2-Node and 3-Node -testbeds are shown in figures below. - -.. only:: latex - - .. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-2n-nic2nic} - \label{fig:logical-2n-nic2nic} - \end{figure} - -.. only:: html - - .. figure:: ../vpp_performance_tests/logical-2n-nic2nic.svg - :alt: logical-2n-nic2nic - :align: center +vNIC-to-vNIC Switching +~~~~~~~~~~~~~~~~~~~~~~ +The simplest virtual topology for software data plane application like +VPP is vNIC-to-vNIC switching. Tested virtual topologies for 2-Node and +3-Node testbeds are shown in figures below. .. only:: latex @@ -44,38 +28,17 @@ testbeds are shown in figures below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-3n-nic2nic} - \label{fig:logical-3n-nic2nic} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-2n-nic2nic} + \label{fig:virtual-2n-nic2nic} \end{figure} .. only:: html - .. figure:: ../vpp_performance_tests/logical-3n-nic2nic.svg - :alt: logical-3n-nic2nic + .. figure:: ../vpp_functional_tests/virtual-2n-nic2nic.svg + :alt: virtual-2n-nic2nic :align: center -SUT1 and SUT2 are two VMs (Ubuntu or Centos, depending on the test suite), TG -is a Traffic Generator (TG, another Ubuntu VM). SUTs run VPP SW application in -Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy SW -application as a packet Traffic Generator. Logical connectivity between SUTs -and to TG is provided using virtual NICs using VMs' virtio driver. - -Virtual testbeds are created on-demand whenever a verification job is started -(e.g. triggered by the gerrit patch submission) and destroyed upon completion -of all functional tests. Each node is a Virtual Machine and each connection -that is drawn on the diagram is available for use in any test case. During the -test execution, all nodes are reachable thru the Management network connected -to every node via dedicated virtual NICs and virtual links (not shown above -for clarity). - -VM Service Switching -~~~~~~~~~~~~~~~~~~~~ - -VM service switching topology test cases require VPP DUT to communicate -with Virtual Machines (VMs) over vhost-user virtual interfaces. A nested VM is -created on SUT1 and/or SUT2 for the duration of these particular test cases -only. DUT (VPP) test topology with VM is shown in the figure below. .. only:: latex @@ -83,17 +46,40 @@ only. DUT (VPP) test topology with VM is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-2n-vm-vhost} - \label{fig:logical-2n-vm-vhost} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-3n-nic2nic} + \label{fig:virtual-3n-nic2nic} \end{figure} .. only:: html - .. figure:: ../vpp_performance_tests/logical-2n-vm-vhost.svg - :alt: logical-2n-vm-vhost + .. figure:: ../vpp_functional_tests/virtual-3n-nic2nic.svg + :alt: virtual-3n-nic2nic :align: center +SUT1 and SUT2 are two VMs (running Ubuntu or Centos, depending on the test +suite), TG is a Traffic Generator (running Ubuntu VM). SUTs run VPP +SW application in Linux user-mode as a Device Under Test (DUT) within +the VM. TG runs Scapy SW application as a packet Traffic Generator. +Network connectivity between SUTs and to TG is provided using virtual +NICs and VMs' virtio drivers. + +Virtual testbeds are created on-demand whenever a verification job is +started (e.g. triggered by the gerrit patch submission) and destroyed +upon completion of all functional tests. Each node is a Virtual Machine +and each connection that is drawn on the diagram is available for use in +any test case. During the test execution, all nodes are reachable thru +the Management network connected to every node via dedicated virtual +NICs and virtual links (not shown above for clarity). + +Nested-VM Service Switching +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Nested-VM (Virtual Machine) service switching topology test cases +require VPP DUT to communicate with nested-VM(s) over vhost-user virtual +interfaces. Nested-VM(s) is(are) created on SUT1 and/or SUT2 for the +duration of these particular test cases only. Virtual test topology with +nested-VM(s) is shown in the figure below. .. only:: latex @@ -101,56 +87,83 @@ only. DUT (VPP) test topology with VM is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-3n-vm-vhost} - \label{fig:logical-3n-vm-vhost} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-3n-vm-vhost} + \label{fig:virtual-3n-vm-vhost} \end{figure} .. only:: html - .. figure:: ../vpp_performance_tests/logical-3n-vm-vhost.svg - :alt: logical-3n-vm-vhost + .. figure:: ../vpp_functional_tests/virtual-3n-vm-vhost.svg + :alt: virtual-3n-vm-vhost :align: center Functional Tests Coverage ------------------------- -Following VPP functional test areas are covered in the |csit-release| with -results listed in this report: - -- **DHCP - Client and Proxy** - Dynamic Host Control Protocol Client and Proxy - for IPv4, IPv6. -- **GRE Overlay Tunnels** - Generic Routing Encapsulation for IPv4. -- **L2BD Ethernet Switching** - L2 Bridge-Domain switched-forwarding for - untagged Ethernet, dot1q and dot1ad tagged. -- **L2XC Ethernet Switching** - L2 Cross-Connect switched-forwarding for - untagged Ethernet, dot1q and dot1ad tagged. -- **LISP Overlay Tunnels** - Locator/ID Separation Protocol overlay tunnels and - locator/id mapping control. -- **Softwire Tunnels** - IPv4-in-IPv6 softwire tunnels. -- **Cop Address Security** - address white-list and black-list filtering for - IPv4, IPv6. -- **IPSec - Tunnels and Transport** - IPSec tunnel and transport modes. -- **IPv6 Routed-Forwarding** - IPv6 routed-forwarding, NS/ND, RA, ICMPv6. -- **uRPF Source Security** - unicast Reverse Path Forwarding security. -- **Tap Interface** - baseline Linux tap interface tests. -- **Telemetry - IPFIX and SPAN** - IPFIX netflow statistics and SPAN port - mirroring. -- **VRF Routed-Forwarding** - multi-context IPVPN routed-forwarding for IPv4, - IPv6. -- **iACL Security** - ingress Access Control List security for IPv4, IPv6, MAC. -- **IPv4 Routed-Forwarding** - IPv4 routed-forwarding, RPF, ARP, Proxy ARP, - ICMPv4. -- **QoS Policer Metering** - ingress packet rate measuring and marking for IPv4, - IPv6. -- **VLAN Tag Translation** - L2 VLAN tag translation 2to2, 2to1, 1to2, 1to1. -- **VXLAN Overlay Tunnels** - VXLAN tunneling for L2-over-IP, for IPv4, IPv6. +|csit-release| includes following VPP functionality tested in virtual VM +environment: + ++-----------------------+----------------------------------------------+ +| Functionality | Description | ++=======================+==============================================+ +| ACL | Ingress Access Control List security for L2 | +| | Bridge-Domain MAC switching, IPv4 routing, | +| | IPv6 routing. | ++-----------------------+----------------------------------------------+ +| COP | COP address white-list and black-list | +| | filtering for IPv4 and IPv6 routing. | ++-----------------------+----------------------------------------------+ +| DHCP | Dynamic Host Control Protocol Client and | +| | Proxy for IPv4 and IPv6 routing. | ++-----------------------+----------------------------------------------+ +| GRE | Generic Routing Encapsulation Overlay | +| | Tunnels for IPv4. | ++-----------------------+----------------------------------------------+ +| IPSec | IPSec tunnel and transport modes. | ++-----------------------+----------------------------------------------+ +| IPv4 | IPv4 routing, RPF, ARP, Proxy ARP, ICMPv4. | ++-----------------------+----------------------------------------------+ +| IPv6 | IPv6 routing, NS/ND, RA, ICMPv6. | ++-----------------------+----------------------------------------------+ +| L2BD | L2 Bridge-Domain switching for untagged | +| | Ethernet, dot1q and dot1ad tagged. | ++-----------------------+----------------------------------------------+ +| L2XC | L2 Cross-Connect switching for untagged | +| | Ethernet, dot1q and dot1ad tagged. | ++-----------------------+----------------------------------------------+ +| LISP | Locator/ID Separation Protocol overlay | +| | tunnels and locator/id mapping control. | ++-----------------------+----------------------------------------------+ +| QoS Policer Metering | Ingress packet rate metering and marking for | +| | IPv4, IPv6. | ++-----------------------+----------------------------------------------+ +| Softwire Tunnels | IPv4-in-IPv6 softwire tunnels. | ++-----------------------+----------------------------------------------+ +| Tap Interface | Baseline Linux tap interface tests. | ++-----------------------+----------------------------------------------+ +| IPFIX and SPAN | Telemetry IPFIX netflow statistics and SPAN | +| | port mirroring. | ++-----------------------+----------------------------------------------+ +| uRPF Source Security | Unicast Reverse Path Forwarding security for | +| | IPv4 and IPv6 routing. | ++-----------------------+----------------------------------------------+ +| VLAN Tag Translation | L2 VLAN tag translation 2to2, 2to1, 1to2, | +| | 1to1. | ++-----------------------+----------------------------------------------+ +| VRF Routing | Multi-context VRF IPVPN routing for IPv4 and | +| | IPv6. | ++-----------------------+----------------------------------------------+ +| VXLAN | VXLAN overlay tunneling for L2-over-IPv4 and | +| | -over-IPv6. | ++-----------------------+----------------------------------------------+ Functional Tests Naming ----------------------- -|csit-release| follows a common structured naming convention for all performance -and system functional tests, introduced in CSIT-17.01. +|csit-release| follows a common structured naming convention for all +performance and system functional tests, introduced in CSIT-17.01. -The naming should be intuitive for majority of the tests. Complete description -of CSIT test naming convention is provided on :ref:`csit_test_naming`. +The naming should be intuitive for majority of the tests. Complete +description of CSIT test naming convention is provided on +:ref:`csit_test_naming`. diff --git a/docs/report/vpp_functional_tests/virtual-2n-nic2nic.svg b/docs/report/vpp_functional_tests/virtual-2n-nic2nic.svg new file mode 100644 index 0000000000..3bc5d0d410 --- /dev/null +++ b/docs/report/vpp_functional_tests/virtual-2n-nic2nic.svg @@ -0,0 +1,315 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SUT VM + + + + + + + + + + + + + + DUT + + + + + + + + TG VM + + + + + + + + vNIC + + + + + + + + + + + + + + + + + + + + + + + + + + + LinuxKernel + + + + + + + + + + + + + + Linux-Guest + + + + + + + + User-Space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Virtual 2-Node Topology: vNIC-to-vNIC Switching + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ForwardingContext + + + + + + + + \ No newline at end of file diff --git a/docs/report/vpp_functional_tests/virtual-3n-nic2nic.svg b/docs/report/vpp_functional_tests/virtual-3n-nic2nic.svg new file mode 100644 index 0000000000..14942eae92 --- /dev/null +++ b/docs/report/vpp_functional_tests/virtual-3n-nic2nic.svg @@ -0,0 +1,452 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SUT1 VM + + + + + + + + + + + + + + DUT1 + + + + + + + + TG VM + + + + + + + + vNIC + + + + + + + + + + + + + + + + + + + + + + + + + + + LinuxKernel + + + + + + + + + + + + + + Linux-Guest + + + + + + + + User-Space + + + + + + + + + + + + + + + SUT2 VM + + + + + + + + + + + + + + DUT2 + + + + + + + + vNIC + + + + + + + + + + + + + + + + + + + + + + + + + + + LinuxKernel + + + + + + + + + + + + + + Linux-Guest + + + + + + + + User-Space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Virtual 3-Node Topology: vNIC-to-vNIC Switching + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ForwardingContext + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ForwardingContext + + + + + + + + \ No newline at end of file diff --git a/docs/report/vpp_functional_tests/virtual-3n-vm-vhost.svg b/docs/report/vpp_functional_tests/virtual-3n-vm-vhost.svg new file mode 100644 index 0000000000..93c1c6e954 --- /dev/null +++ b/docs/report/vpp_functional_tests/virtual-3n-vm-vhost.svg @@ -0,0 +1,1110 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Virtual 3-Node Topology: nested-VM Service Switching + + + + + + + + SUT1 VM + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + nested-VM[n] + + + + + + + + + + + + + + + + nested-VM[1] + + + + + + + + + + + + + + + + nested-VM[2] + + + + + + + + DUT1 + + + + + + + + TG VM + + + + + + + + + + + + + LinuxKernel + + + + + + + + Linux-Guest + + + + + + + + User-Space + + + + + + + + VNF[1] + + + + + + … + + + + + + + + VNF[2] + + + + + + + + VNF[n] + + + + + + + + vNIC + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + … + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + FwdCxt[0] + + + + + + + + + + FwdCxt[1] + + + + + + + + + + FwdCxt[2] + + + + + + + + + + FwdCxt[n] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SUT2 VM + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + nested-VM[n] + + + + + + + + + + + + + + + + nested-VM[1] + + + + + + + + + + + + + + + + nested-VM[2] + + + + + + + + DUT2 + + + + + + + + + + + + + LinuxKernel + + + + + + + + Linux-Guest + + + + + + + + User-Space + + + + + + + + VNF[1] + + + + + + … + + + + + + + + VNF[2] + + + + + + + + VNF[n] + + + + + + + + vNIC + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + … + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + FwdCxt[0] + + + + + + + + + + FwdCxt[1] + + + + + + + + + + FwdCxt[2] + + + + + + + + + + FwdCxt[n] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst index 5220dc562a..8ddcaec798 100644 --- a/docs/report/vpp_performance_tests/overview.rst +++ b/docs/report/vpp_performance_tests/overview.rst @@ -265,11 +265,11 @@ topologies and configurations: regardless of packet loss. Maximum load for specified Ethernet frame size is set to the bi-directional link rate. -|csit-release| includes following VPP data plane features performance -tested across a range of NIC drivers and NIC models: +|csit-release| includes following VPP data plane functionality +performance tested across a range of NIC drivers and NIC models: +-----------------------+----------------------------------------------+ -| Features | Description | +| Functionality | Description | +=======================+==============================================+ | ACL | L2 Bridge-Domain switching and | | | IPv4and IPv6 routing with iACL and oACL IP | -- 2.16.6