X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Ftag_documentation.rst;h=e860d5a8eee5ccc0b7f7dd555f4cc3dfc25cfeb9;hp=c6064c61a30bff5443e319b29a93d40e18a54588;hb=cf2a4f97f261a8ab27578232e9f419968e8a1fb1;hpb=32188ccdce50accbb80b405c5813543a4131e744 diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst index c6064c61a3..e860d5a8ee 100644 --- a/docs/tag_documentation.rst +++ b/docs/tag_documentation.rst @@ -1,13 +1,23 @@ -CSIT TAGs Descriptions -====================== +CSIT Tags +========= All CSIT test cases are labelled with Robot Framework tags used to allow for easy test case type identification, test case grouping and selection for -execution. Following sections list currently used CSIT TAGs and their -documentation. +execution. Following sections list currently used CSIT tags and their +descriptions. -Topology TAGs -------------- +Testbed Topology Tags +--------------------- + +.. topic:: 2_NODE_DOUBLE_LINK_TOPO + + 2 nodes connected in a circular topology with two links interconnecting + the devices. + +.. topic:: 2_NODE_SINGLE_LINK_TOPO + + 2 nodes connected in a circular topology with at least one link + interconnecting devices. .. topic:: 3_NODE_DOUBLE_LINK_TOPO @@ -19,7 +29,7 @@ Topology TAGs 3 nodes connected in a circular topology with at least one link interconnecting devices. -Objective TAGs +Objective Tags -------------- .. topic:: SKIP_PATCH @@ -31,7 +41,7 @@ Objective TAGs Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch). -Environment TAGs +Environment Tags ---------------- .. topic:: HW_ENV @@ -46,7 +56,7 @@ Environment TAGs DUTs with VPP and capable of running Virtual Machine. -NIC model tags +NIC Model Tags -------------- .. topic:: NIC_Intel-X520-DA2 @@ -61,6 +71,10 @@ NIC model tags Intel X710 NIC. +.. topic:: NIC_Intel-XXV710 + + Intel XXV710 NIC. + .. topic:: NIC_Cisco-VIC-1227 VIC-1227 by Cisco. @@ -69,7 +83,7 @@ NIC model tags VIC-1385 by Cisco. -Scaling TAGs +Scaling Tags ------------ .. topic:: FIB_20K @@ -84,9 +98,61 @@ Scaling TAGs 2x1,000,000 entries in single fib table -.. topic:: TNL_1000 +.. topic:: L2BD_1 + + Test with 1 L2 bridge domain. + +.. topic:: L2BD_10 + + Test with 10 L2 bridge domains. + +.. topic:: L2BD_100 + + Test with 100 L2 bridge domains. + +.. topic:: L2BD_1K + + Test with 1000 L2 bridge domains. + +.. topic:: VLAN_1 + + Test with 1 VLAN sub-interface. + +.. topic:: VLAN_10 + + Test with 10 VLAN sub-interfaces. + +.. topic:: VLAN_100 + + Test with 100 VLAN sub-interfaces. + +.. topic:: VLAN_1K + + Test with 1000 VLAN sub-interfaces. + +.. topic:: VXLAN_1 + + Test with 1 VXLAN tunnel. + +.. topic:: VXLAN_10 - IPSec in tunnel mode - 1000 tunnels. + Test with 10 VXLAN tunnels. + +.. topic:: VXLAN_100 + + Test with 100 VXLAN tunnels. + +.. topic:: VXLAN_1K + + Test with 1000 VXLAN tunnels. + +.. topic:: TNL_{t} + + IPSec in tunnel mode - {t} tunnels. + +.. topic:: SRC_USER_1 + + Traffic flow with 1 unique IP (users) in one direction. .. topic:: SRC_USER_10 @@ -123,8 +189,62 @@ Scaling TAGs Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in one direction. -Tags marking functional vs. performance of tests ------------------------------------------------- +.. topic:: HOSTS_1024 + + Stateless or stateful traffic stream with 1024 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_4096 + + Stateless or stateful traffic stream with 4096 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_16384 + + Stateless or stateful traffic stream with 16384 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_65536 + + Stateless or stateful traffic stream with 65536 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_262144 + + Stateless or stateful traffic stream with 262144 client source IP4 addresses + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: GENEVE4_1TUN + + Test with 1 GENEVE IPv4 tunnel. + +.. topic:: GENEVE4_4TUN + + Test with 4 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_16TUN + + Test with 16 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_64TUN + + Test with 64 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_256TUN + + Test with 256 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_1024TUN + + Test with 1024 GENEVE IPv4 tunnels. + +Test Category Tags +------------------ .. topic:: FUNCTEST @@ -134,39 +254,53 @@ Tags marking functional vs. performance of tests All performance test cases. -Performance testing tags ------------------------- +Performance Type Tags +--------------------- + +.. topic:: NDRPDR -.. topic:: PDRDISC + Single test finding both No Drop Rate and Partial Drop Rate simultaneously. + The search is done by optimized algorithm which performs + multiple trial runs at different durations and transmit rates. + The results come from the final trials, which have duration of 30 seconds. - Partial Drop Rate evaluation of single run result, with non-zero packet - loss tolerance (LT) expressed in percentage of packets transmitted. +.. topic:: MRR -.. topic:: NDRDISC + Performance tests where TG sends the traffic at maximum rate (line rate) + and reports total sent/received packets over trial duration. + The result is an average of 10 trials of 1 second duration. - Non Drop Rate evaluation of results. Loss acceptance of dropped packets is - set to zero lost packets. +.. topic:: SOAK -.. topic:: NDRCHK + Performance tests using PLRsearch to find the critical load. - Performance tests where TG verifies DUTs' throughput at ref-NDR (reference - Non Drop Rate) with zero packet loss tolerance. +.. topic:: RECONF -.. topic:: NDRPDRDISC + Performance tests aimed to measure lost packets (time) when performing + reconfiguration while full throughput offered load is applied. - Find performance of DUT based on :rfc:`2544` with linear / binary / combined - search. (Previous LONG tests.) +Ethernet Frame Size Tags +------------------------ -Ethernet frame size tags for performance tests ----------------------------------------------- +These are describing the traffic offered by Traffic Generator, +"primary" traffic in case of asymmetric load. +For traffic between DUTs, or for "secondary" traffic, see ${overhead} value. .. topic:: 64B - 64B frames used for test. + 64B frames used for test. Generic ethernet or IPv4. .. topic:: 78B - 78B frames used for test. + 78B frames used for test. Ipv6. + +.. topic:: 114B + + 114B frames used for test. IPv4+vxlan. + +.. topic:: 118B + + 118B frames used for test. Dot1q+IPv4+vxlan. .. topic:: IMIX @@ -192,17 +326,18 @@ Ethernet frame size tags for performance tests 9000B frames used for test. -Test type tags +Test Type Tags -------------- .. topic:: BASE Baseline test cases, no encapsulation, no feature(s) configured in tests. + No scaling whatsoever, beyond minimum needed for RSS. .. topic:: IP4BASE IPv4 baseline test cases, no encapsulation, no feature(s) configured in - tests. + tests. Minimal number of routes. Other quantities may be scaled. .. topic:: IP6BASE @@ -219,9 +354,15 @@ Test type tags L2BD baseline test cases, no encapsulation, no feature(s) configured in tests. +.. topic:: L2PATCH + + L2PATCH baseline test cases, no encapsulation, no feature(s) configured in + tests. + .. topic:: SCALE - Scale test cases. + Scale test cases. Other tags specify which quantities are scaled. + Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE). .. topic:: ENCAP @@ -231,11 +372,103 @@ Test type tags At least one feature is configured in test cases. Use also feature tag(s). -.. topic:: TLDK +.. topic:: UDP + + Tests which use any kind of UDP traffic (STL or ASTF profile). + +.. topic:: TCP + + Tests which use any kind of TCP traffic (STL or ASTF profile). + +.. + TODO: Should we define tags STL and ASTF? + +.. topic:: UDP_UDIR + + Tests which use unidirectional UDP traffic (STL profile only). + +.. topic:: UDP_BIDIR + + Tests which use bidirectional UDP traffic (STL profile only). + +.. topic:: UDP_CPS + + Tests which measure connections per second on minimal UDP pseudoconnections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: TCP_CPS + + Tests which measure connections per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: UDP_PPS + + Tests which measure packets per second on lightweight UDP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: TCP_PPS + + Tests which measure packets per second on lightweight TCP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: HTTP + + Tests which use traffic formed of valid HTTP requests (and responses). + +.. + TODO: Add HTTP tag to the current hoststack tests. + TODO: Document other tags already used by hoststack tests. + +.. topic:: NF_DENSITY + + Performance tests that measure throughput of multiple VNF and CNF + service topologies at different service densities. + +NF Service Density Tags +----------------------- + +.. topic:: CHAIN + + NF service density tests with VNF or CNF service chain topology(ies). + +.. topic:: PIPE + + NF service density tests with CNF service pipeline topology(ies). + +.. topic:: NF_L3FWDIP4 + + NF service density tests with DPDK l3fwd IPv4 routing as NF workload. + +.. topic:: NF_VPPIP4 + + NF service density tests with VPP IPv4 routing as NF workload. + +.. topic:: {r}R{c}C + + Service density matrix locator {r}R{c}C, {r}Row denoting number of + service instances, {c}Column denoting number of NFs per service + instance. {r}=(1,2,4,6,8,10), {c}=(1,2,4,6,8,10). + +.. topic:: {n}VM{t}T + + Service density {n}VM{t}T, {n}Number of NF Qemu VMs, {t}Number of threads + per NF. + +.. topic:: {n}DCRt}T + + Service density {n}DCR{t}T, {n}Number of NF Docker containers, {t}Number of + threads per NF. + +.. topic:: {n}_ADDED_CHAINS - Functional test cases for TLDK. + {n}Number of chains (or pipelines) added (and/or removed) + during RECONF test. -Forwarding mode tags +Forwarding Mode Tags -------------------- .. topic:: L2BDMACSTAT @@ -258,7 +491,19 @@ Forwarding mode tags VPP IPv6 routed forwarding. -Underlay tags +.. topic:: LOADBALANCER_MAGLEV + + VPP Load balancer maglev mode. + +.. topic:: LOADBALANCER_L3DSR + + VPP Load balancer l3dsr mode. + +.. topic:: LOADBALANCER_NAT4 + + VPP Load balancer nat4 mode. + +Underlay Tags ------------- .. topic:: IP4UNRLAY @@ -273,7 +518,7 @@ Underlay tags MPLS underlay. -Overlay tags +Overlay Tags ------------ .. topic:: L2OVRLAY @@ -288,7 +533,7 @@ Overlay tags IPv6 overlay (IPv6 payload). -Tagging tags +Tagging Tags ------------ .. topic:: DOT1Q @@ -299,7 +544,7 @@ Tagging tags All test cases with dot1ad. -Encapsulation tags +Encapsulation Tags ------------------ .. topic:: ETH @@ -314,6 +559,26 @@ Encapsulation tags All test cases with LISP-GPE. +.. topic:: LISP_IP4o4 + + All test cases with LISP_IP4o4. + +.. topic:: LISPGPE_IP4o4 + + All test cases with LISPGPE_IP4o4. + +.. topic:: LISPGPE_IP6o4 + + All test cases with LISPGPE_IP6o4. + +.. topic:: LISPGPE_IP4o6 + + All test cases with LISPGPE_IP4o6. + +.. topic:: LISPGPE_IP6o6 + + All test cases with LISPGPE_IP6o6. + .. topic:: VXLAN All test cases with Vxlan. @@ -330,24 +595,58 @@ Encapsulation tags All test cases with IPSEC. -Interface tags +.. topic:: SRv6 + + All test cases with Segment routing over IPv6 dataplane. + +.. topic:: SRv6_1SID + + All SRv6 test cases with single SID. + +.. topic:: SRv6_2SID_DECAP + + All SRv6 test cases with two SIDs and with decapsulation. + +.. topic:: SRv6_2SID_NODECAP + + All SRv6 test cases with two SIDs and without decapsulation. + +.. topic:: GENEVE + + All test cases with GENEVE. + +.. topic:: GENEVE_L3MODE + + All test cases with GENEVE tunnel in L3 mode. + +Interface Tags -------------- .. topic:: PHY All test cases which use physical interface(s). +.. topic:: GSO + + All test cases which uses Generic Segmentation Offload. + .. topic:: VHOST All test cases which uses VHOST. -.. topic:: VHOST_256 +.. topic:: VHOST_1024 - All test cases which uses VHOST with qemu queue size set to 256. + All test cases which uses VHOST DPDK driver with qemu queue size set + to 1024. -.. topic:: VHOST_1024 +.. topic:: VIRTIO + + All test cases which uses VIRTIO native VPP driver. + +.. topic:: VIRTIO_1024 - All test cases which uses VHOST with qemu queue size set to 1024. + All test cases which uses VIRTIO native VPP driver with qemu queue size set + to 1024. .. topic:: CFS_OPT @@ -369,16 +668,82 @@ Interface tags All test cases which uses Memif. -Feature tags +.. topic:: SINGLE_MEMIF + + All test cases which uses only single Memif connection per DUT. One DUT + instance is running in container having one physical interface exposed to + container. + +.. topic:: LBOND + + All test cases which uses link bonding (BondEthernet interface). + +.. topic:: LBOND_DPDK + + All test cases which uses DPDK link bonding. + +.. topic:: LBOND_VPP + + All test cases which uses VPP link bonding. + +.. topic:: LBOND_MODE_XOR + + All test cases which uses link bonding with mode XOR. + +.. topic:: LBOND_MODE_LACP + + All test cases which uses link bonding with mode LACP. + +.. topic:: LBOND_LB_L34 + + All test cases which uses link bonding with load-balance mode l34. + +.. topic:: LBOND_1L + + All test cases which uses one link for link bonding. + +.. topic:: LBOND_2L + + All test cases which uses two links for link bonding. + +.. topic:: DRV_AVF + + All test cases which uses Intel Adaptive Virtual Function (AVF) device + plugin for VPP. This plugins provides native device support for Intel AVF. + AVF is driver specification for current and future Intel Virtual Function + devices. In essence, today this driver can be used only with Intel + XL710 / X710 / XXV710 adapters. + +.. topic:: DRV_VFIO_PCI + + All test cases which uses vfio-pci device driver. It supports variety of NIC + adapters. + +.. topic:: DRV_RDMA_CORE + + All test cases which uses rdma-core device driver. It supports Mellanox + NIC adapters. + +.. topic:: RXQ_SIZE_{n} + + All test cases which RXQ size (RX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +.. topic:: TXQ_SIZE_{n} + + All test cases which TXQ size (TX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +Feature Tags ------------ .. topic:: IACLDST iACL destination. -.. topic:: COPWHLIST +.. topic:: ADLALWLIST - COP whitelist. + ADL allowlist. .. topic:: NAT44 @@ -420,7 +785,23 @@ Feature tags ACL plugin configured and tested with 50 not-hitting ACEs. -Encryption tags +.. topic:: SRv6_PROXY + + SRv6 endpoint to SR-unaware appliance via proxy. + +.. topic:: SRv6_PROXY_STAT + + SRv6 endpoint to SR-unaware appliance via static proxy. + +.. topic:: SRv6_PROXY_DYN + + SRv6 endpoint to SR-unaware appliance via dynamic proxy. + +.. topic:: SRv6_PROXY_MASQ + + SRv6 endpoint to SR-unaware appliance via masquerading proxy. + +Encryption Tags --------------- .. topic:: IPSECSW @@ -439,7 +820,39 @@ Encryption tags IPSec in tunnel mode. -Client-workload tags +.. topic:: IPSECINT + + IPSec in interface mode. + +.. topic:: AES + + IPSec using AES algorithms. + +.. topic:: AES_128_CBC + + IPSec using AES 128 CBC algorithms. + +.. topic:: AES_128_GCM + + IPSec using AES 128 GCM algorithms. + +.. topic:: AES_256_GCM + + IPSec using AES 256 GCM algorithms. + +.. topic:: HMAC + + IPSec using HMAC integrity algorithms. + +.. topic:: HMAC_SHA_256 + + IPSec using HMAC SHA 256 integrity algorithms. + +.. topic:: HMAC_SHA_512 + + IPSec using HMAC SHA 512 integrity algorithms. + +Client-Workload Tags -------------------- .. topic:: VM @@ -450,6 +863,10 @@ Client-workload tags All test cases which use Linux container and LXC utils. +.. topic:: DRC + + All test cases which use at least one Docker container. + .. topic:: DOCKER All test cases which use Docker as container manager. @@ -458,23 +875,9 @@ Client-workload tags All test cases with specific APP use. -Container orchestration tags +Container Orchestration Tags ---------------------------- -.. topic:: K8S - - All test cases which use Kubernetes for orchestration. - -.. topic:: SFC_CONTROLLER - - All test cases which use ligato/sfc_controller for driving configuration - of vpp inside container. - -.. topic:: VPP_AGENT - - All test cases which use Golang implementation of a control/management plane - for VPP - .. topic:: 1VSWITCH VPP running in Docker container acting as VSWITCH. @@ -487,15 +890,21 @@ Container orchestration tags 2 VPP running in 2 Docker containers acting as VNF work load. -Multi-threading tags +.. topic:: 4VNF + + 4 VPP running in 4 Docker containers acting as VNF work load. + +Multi-Threading Tags -------------------- .. topic:: STHREAD - All test cases using single poll mode thread. + *Dynamic tag*. + All test cases using single poll mode thread. .. topic:: MTHREAD + *Dynamic tag*. All test cases using more then one poll mode driver thread. .. topic:: 1NUMA @@ -506,55 +915,58 @@ Multi-threading tags All test cases with packet processing on two sockets. -.. topic:: SMT +.. topic:: 1C + + 1 worker thread pinned to 1 dedicated physical core; or if HyperThreading is + enabled, 2 worker threads each pinned to a separate logical core within 1 + dedicated physical core. Main thread pinned to core 1. - All test cases with symmetric Multi-Threading (HyperThreading) enabled. +.. topic:: 2C -.. topic:: NOSMT + 2 worker threads pinned to 2 dedicated physical cores; or if HyperThreading + is enabled, 4 worker threads each pinned to a separate logical core within 2 + dedicated physical cores. Main thread pinned to core 1. - All test cases with symmetric Multi-Threading (HyperThreading) disabled. +.. topic:: 4C + + 4 worker threads pinned to 4 dedicated physical cores; or if HyperThreading + is enabled, 8 worker threads each pinned to a separate logical core within 4 + dedicated physical cores. Main thread pinned to core 1. .. topic:: 1T1C + *Dynamic tag*. 1 worker thread pinned to 1 dedicated physical core. 1 receive queue per - interface. Main thread pinned to core 0. + interface. Main thread pinned to core 1. .. topic:: 2T2C + *Dynamic tag*. 2 worker threads pinned to 2 dedicated physical cores. 1 receive queue per - interface. Main thread pinned to core 0. + interface. Main thread pinned to core 1. .. topic:: 4T4C + *Dynamic tag*. 4 worker threads pinned to 4 dedicated physical cores. 2 receive queues per - interface. Main thread pinned to core 0. - -.. topic:: 6T6C - - 6 worker threads pinned to 6 dedicated physical cores. 3 receive queues per - interface. Main thread pinned to core 0. - -.. topic:: 8T8C - - 8 worker threads pinned to 8 dedicated physical cores. 4 receive queues per - interface. Main thread pinned to core 0. - -Honeycomb tags --------------- - -.. topic:: HC_FUNC - - Honeycomb functional test cases. + interface. Main thread pinned to core 1. -.. topic:: HC_NSH +.. topic:: 2T1C - Honeycomb NSH test cases. + *Dynamic tag*. + 2 worker threads each pinned to a separate logical core within 1 dedicated + physical core. 1 receive queue per interface. Main thread pinned to core 1. -.. topic:: HC_PERSIST +.. topic:: 4T2C - Honeycomb persistence test cases. + *Dynamic tag*. + 4 worker threads each pinned to a separate logical core within 2 dedicated + physical cores. 2 receive queues per interface. Main thread pinned to core + 1. -.. topic:: HC_REST_ONLY +.. topic:: 8T4C - (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode - using ODL client for Restfconf -> Netconf translation. + *Dynamic tag*. + 8 worker threads each pinned to a separate logical core within 4 dedicated + physical cores. 4 receive queues per interface. Main thread pinned to core + 1.