X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Ftag_documentation.rst;h=b2130c1f28de28c252936778a94eeeb0be254de7;hp=6247fc5dffddc198ebea1e1efda724dbd0e8ed3d;hb=31fb6cc78ec42a05522f9849210e4f84313e1f88;hpb=18cd1f996429c380c9eef0d3a52ec95b9e55c996 diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst index 6247fc5dff..b2130c1f28 100644 --- a/docs/tag_documentation.rst +++ b/docs/tag_documentation.rst @@ -1,13 +1,13 @@ -CSIT TAGs +CSIT Tags ========= All CSIT test cases are labelled with Robot Framework tags used to allow for easy test case type identification, test case grouping and selection for -execution. Following sections list currently used CSIT TAGs and their -documentation. +execution. Following sections list currently used CSIT tags and their +descriptions. -Topology TAGs -------------- +Testbed Topology Tags +--------------------- .. topic:: 2_NODE_DOUBLE_LINK_TOPO @@ -29,7 +29,7 @@ Topology TAGs 3 nodes connected in a circular topology with at least one link interconnecting devices. -Objective TAGs +Objective Tags -------------- .. topic:: SKIP_PATCH @@ -41,7 +41,7 @@ Objective TAGs Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch). -Environment TAGs +Environment Tags ---------------- .. topic:: HW_ENV @@ -56,7 +56,7 @@ Environment TAGs DUTs with VPP and capable of running Virtual Machine. -NIC model tags +NIC Model Tags -------------- .. topic:: NIC_Intel-X520-DA2 @@ -83,7 +83,11 @@ NIC model tags VIC-1385 by Cisco. -Scaling TAGs +.. topic:: NIC_Amazon-Nitro-50G + + Amazon EC2 ENA NIC. + +Scaling Tags ------------ .. topic:: FIB_20K @@ -146,29 +150,14 @@ Scaling TAGs Test with 1000 VXLAN tunnels. -.. topic:: TNL_1000 - - IPSec in tunnel mode - 1000 tunnels. - -.. topic:: SRC_USER_10 - - Traffic flow with 10 unique IPs (users) in one direction. - -.. topic:: SRC_USER_100 - - Traffic flow with 100 unique IPs (users) in one direction. - -.. topic:: SRC_USER_1000 - - Traffic flow with 1000 unique IPs (users) in one direction. +.. topic:: TNL_{t} -.. topic:: SRC_USER_2000 + IPSec in tunnel mode - {t} tunnels. - Traffic flow with 2000 unique IPs (users) in one direction. +.. topic:: SRC_USER_{u} -.. topic:: SRC_USER_4000 - - Traffic flow with 4000 unique IPs (users) in one direction. + Traffic flow with {u} unique IPs (users) in one direction. + {u}=(1,10,100,1000,2000,4000). .. topic:: 100_FLOWS @@ -185,19 +174,37 @@ Scaling TAGs Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in one direction. -Tags marking functional vs. performance of tests ------------------------------------------------- +.. topic:: HOSTS_{h} + + Stateless or stateful traffic stream with {h} client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + {h}=(1024,4096,16384,65536,262144). + +.. topic:: GENEVE4_{t}TUN + + Test with {t} GENEVE IPv4 tunnel. {t}=(1,4,16,64,256,1024) + +Test Category Tags +------------------ -.. topic:: FUNCTEST +.. topic:: DEVICETEST - All functional test cases. + All vpp_device functional test cases. .. topic:: PERFTEST All performance test cases. -Performance testing tags ------------------------- +VPP Device Type Tags +-------------------- + +.. topic:: SCAPY + + All test cases that uses Scapy for packet generation and validation. + +Performance Type Tags +--------------------- .. topic:: NDRPDR @@ -212,56 +219,42 @@ Performance testing tags and reports total sent/received packets over trial duration. The result is an average of 10 trials of 1 second duration. -Ethernet frame size tags for performance tests ----------------------------------------------- +.. topic:: SOAK + + Performance tests using PLRsearch to find the critical load. -.. topic:: 64B +.. topic:: RECONF - 64B frames used for test. + Performance tests aimed to measure lost packets (time) when performing + reconfiguration while full throughput offered load is applied. -.. topic:: 78B +Ethernet Frame Size Tags +------------------------ - 78B frames used for test. +These are describing the traffic offered by Traffic Generator, +"primary" traffic in case of asymmetric load. +For traffic between DUTs, or for "secondary" traffic, see ${overhead} value. -.. topic:: 114B +.. topic:: {b}B - 114B frames used for test. + {b} Bytes frames used for test. .. topic:: IMIX IMIX frame sequence (28x 64B, 16x 570B, 4x 1518B) used for test. -.. topic:: 1460B - - 1460B frames used for test. - -.. topic:: 1480B - - 1480B frames used for test. - -.. topic:: 1514B - - 1514B frames used for test. - -.. topic:: 1518B - - 1518B frames used for test. - -.. topic:: 9000B - - 9000B frames used for test. - -Test type tags +Test Type Tags -------------- .. topic:: BASE Baseline test cases, no encapsulation, no feature(s) configured in tests. + No scaling whatsoever, beyond minimum needed for RSS. .. topic:: IP4BASE IPv4 baseline test cases, no encapsulation, no feature(s) configured in - tests. + tests. Minimal number of routes. Other quantities may be scaled. .. topic:: IP6BASE @@ -285,7 +278,8 @@ Test type tags .. topic:: SCALE - Scale test cases. + Scale test cases. Other tags specify which quantities are scaled. + Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE). .. topic:: ENCAP @@ -295,31 +289,117 @@ Test type tags At least one feature is configured in test cases. Use also feature tag(s). -.. topic:: TLDK +.. topic:: UDP + + Tests which use any kind of UDP traffic (STL or ASTF profile). - Functional test cases for TLDK. +.. topic:: TCP -.. topic:: DMM + Tests which use any kind of TCP traffic (STL or ASTF profile). - Functional test cases for DMM. +.. topic:: TREX -.. topic:: TCP + Tests which test trex traffic without any software DUTs in the traffic path. + +.. + TODO: Should we define tags STL and ASTF? + +.. topic:: UDP_UDIR + + Tests which use unidirectional UDP traffic (STL profile only). - Tests which use TCP. +.. topic:: UDP_BIDIR + + Tests which use bidirectional UDP traffic (STL profile only). + +.. topic:: UDP_CPS + + Tests which measure connections per second on minimal UDP pseudoconnections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_CPS - Performance tests which measure connections per second using http requests. + Tests which measure connections per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_RPS - Performance tests which measure requests per second using http requests. + Tests which measure requests per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: UDP_PPS + + Tests which measure packets per second on lightweight UDP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: TCP_PPS + + Tests which measure packets per second on lightweight TCP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: HTTP - Tests which use HTTP. + Tests which use traffic formed of valid HTTP requests (and responses). + +.. + TODO: Add HTTP tag to the current hoststack tests. + TODO: Document other tags already used by hoststack tests. + +.. topic:: LDP_NGINX + + LDP NGINX is un-modified NGINX with VPP via LD_PRELOAD. + +.. topic:: NF_DENSITY + + Performance tests that measure throughput of multiple VNF and CNF + service topologies at different service densities. + +NF Service Density Tags +----------------------- + +.. topic:: CHAIN + + NF service density tests with VNF or CNF service chain topology(ies). + +.. topic:: PIPE + + NF service density tests with CNF service pipeline topology(ies). + +.. topic:: NF_L3FWDIP4 + + NF service density tests with DPDK l3fwd IPv4 routing as NF workload. + +.. topic:: NF_VPPIP4 -Forwarding mode tags + NF service density tests with VPP IPv4 routing as NF workload. + +.. topic:: {r}R{c}C + + Service density matrix locator {r}R{c}C, {r}Row denoting number of + service instances, {c}Column denoting number of NFs per service + instance. {r}=(1,2,4,6,8,10), {c}=(1,2,4,6,8,10). + +.. topic:: {n}VM{t}T + + Service density {n}VM{t}T, {n}Number of NF Qemu VMs, {t}Number of threads + per NF. + +.. topic:: {n}DCRt}T + + Service density {n}DCR{t}T, {n}Number of NF Docker containers, {t}Number of + threads per NF. + +.. topic:: {n}_ADDED_CHAINS + + {n}Number of chains (or pipelines) added (and/or removed) + during RECONF test. + +Forwarding Mode Tags -------------------- .. topic:: L2BDMACSTAT @@ -342,7 +422,24 @@ Forwarding mode tags VPP IPv6 routed forwarding. -Underlay tags +.. topic:: LOADBALANCER_MAGLEV + + VPP Load balancer maglev mode. + +.. topic:: LOADBALANCER_L3DSR + + VPP Load balancer l3dsr mode. + +.. topic:: LOADBALANCER_NAT4 + + VPP Load balancer nat4 mode. + +.. topic:: N2N + + Mode, where NICs from the same physical server are directly + connected with a cable. + +Underlay Tags ------------- .. topic:: IP4UNRLAY @@ -357,7 +454,7 @@ Underlay tags MPLS underlay. -Overlay tags +Overlay Tags ------------ .. topic:: L2OVRLAY @@ -372,7 +469,7 @@ Overlay tags IPv6 overlay (IPv6 payload). -Tagging tags +Tagging Tags ------------ .. topic:: DOT1Q @@ -383,7 +480,7 @@ Tagging tags All test cases with dot1ad. -Encapsulation tags +Encapsulation Tags ------------------ .. topic:: ETH @@ -398,6 +495,26 @@ Encapsulation tags All test cases with LISP-GPE. +.. topic:: LISP_IP4o4 + + All test cases with LISP_IP4o4. + +.. topic:: LISPGPE_IP4o4 + + All test cases with LISPGPE_IP4o4. + +.. topic:: LISPGPE_IP6o4 + + All test cases with LISPGPE_IP6o4. + +.. topic:: LISPGPE_IP4o6 + + All test cases with LISPGPE_IP4o6. + +.. topic:: LISPGPE_IP6o6 + + All test cases with LISPGPE_IP6o6. + .. topic:: VXLAN All test cases with Vxlan. @@ -410,6 +527,10 @@ Encapsulation tags All test cases with GRE. +.. topic:: GTPU + + All test cases with GTPU. + .. topic:: IPSEC All test cases with IPSEC. @@ -418,24 +539,54 @@ Encapsulation tags All test cases with Segment routing over IPv6 dataplane. -Interface tags +.. topic:: SRv6_1SID + + All SRv6 test cases with single SID. + +.. topic:: SRv6_2SID_DECAP + + All SRv6 test cases with two SIDs and with decapsulation. + +.. topic:: SRv6_2SID_NODECAP + + All SRv6 test cases with two SIDs and without decapsulation. + +.. topic:: GENEVE + + All test cases with GENEVE. + +.. topic:: GENEVE_L3MODE + + All test cases with GENEVE tunnel in L3 mode. + +Interface Tags -------------- .. topic:: PHY All test cases which use physical interface(s). +.. topic:: GSO + + All test cases which uses Generic Segmentation Offload. + .. topic:: VHOST All test cases which uses VHOST. -.. topic:: VHOST_256 +.. topic:: VHOST_1024 - All test cases which uses VHOST with qemu queue size set to 256. + All test cases which uses VHOST DPDK driver with qemu queue size set + to 1024. -.. topic:: VHOST_1024 +.. topic:: VIRTIO + + All test cases which uses VIRTIO native VPP driver. - All test cases which uses VHOST with qemu queue size set to 1024. +.. topic:: VIRTIO_1024 + + All test cases which uses VIRTIO native VPP driver with qemu queue size set + to 1024. .. topic:: CFS_OPT @@ -487,32 +638,40 @@ Interface tags All test cases which uses link bonding with load-balance mode l34. -.. topic:: LBOND_1L +.. topic:: LBOND_{n}L + + All test cases which use {n} link(s) for link bonding. + +.. topic:: DRV_{d} + + All test cases which NIC Driver for DUT is set to {d}. Default is VFIO_PCI. + {d}=(AVF, RDMA_CORE, VFIO_PCI, AF_XDP). + +.. topic:: TG_DRV_{d} - All test cases which uses one link for link bonding. + All test cases which NIC Driver for TG is set to {d}. Default is IGB_UIO. + {d}=(RDMA_CORE, IGB_UIO). -.. topic:: LBOND_2L +.. topic:: RXQ_SIZE_{n} - All test cases which uses two links for link bonding. + All test cases which RXQ size (RX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. -.. topic:: DRV_AVF +.. topic:: TXQ_SIZE_{n} - All test cases which uses Intel Adaptive Virtual Function (AVF) device - plugin for VPP. This plugins provides native device support for Intel AVF. - AVF is driver specification for current and future Intel Virtual Function - devices. In essence, today this driver can be used only with Intel - XL710 / X710 / XXV710 adapters. + All test cases which TXQ size (TX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. -Feature tags +Feature Tags ------------ .. topic:: IACLDST iACL destination. -.. topic:: COPWHLIST +.. topic:: ADLALWLIST - COP whitelist. + ADL allowlist. .. topic:: NAT44 @@ -570,7 +729,7 @@ Feature tags SRv6 endpoint to SR-unaware appliance via masquerading proxy. -Encryption tags +Encryption Tags --------------- .. topic:: IPSECSW @@ -589,151 +748,105 @@ Encryption tags IPSec in tunnel mode. -Client-workload tags --------------------- +.. topic:: IPSECINT -.. topic:: VM + IPSec in interface mode. - All test cases which use at least one virtual machine. +.. topic:: AES -.. topic:: LXC + IPSec using AES algorithms. - All test cases which use Linux container and LXC utils. +.. topic:: AES_128_CBC -.. topic:: DOCKER + IPSec using AES 128 CBC algorithms. - All test cases which use Docker as container manager. +.. topic:: AES_128_GCM -.. topic:: APP + IPSec using AES 128 GCM algorithms. - All test cases with specific APP use. +.. topic:: AES_256_GCM -Container orchestration tags ----------------------------- + IPSec using AES 256 GCM algorithms. -.. topic:: K8S +.. topic:: HMAC - All test cases which use Kubernetes for orchestration. + IPSec using HMAC integrity algorithms. -.. topic:: SFC_CONTROLLER +.. topic:: HMAC_SHA_256 - All test cases which use ligato/sfc_controller for driving configuration - of vpp inside container. + IPSec using HMAC SHA 256 integrity algorithms. -.. topic:: VPP_AGENT +.. topic:: HMAC_SHA_512 - All test cases which use Golang implementation of a control/management plane - for VPP + IPSec using HMAC SHA 512 integrity algorithms. -.. topic:: 1VSWITCH +.. topic:: SCHEDULER - VPP running in Docker container acting as VSWITCH. + IPSec using crypto sw scheduler engine. -.. topic:: 1VNF - - 1 VPP running in Docker container acting as VNF work load. - -.. topic:: 2VNF - - 2 VPP running in 2 Docker containers acting as VNF work load. - -.. topic:: 4VNF - - 4 VPP running in 4 Docker containers acting as VNF work load. - -Multi-threading tags +Client-Workload Tags -------------------- -.. topic:: STHREAD +.. topic:: VM - *Dynamic tag*. - All test cases using single poll mode thread. + All test cases which use at least one virtual machine. -.. topic:: MTHREAD +.. topic:: LXC - *Dynamic tag*. - All test cases using more then one poll mode driver thread. + All test cases which use Linux container and LXC utils. -.. topic:: 1NUMA +.. topic:: DRC - All test cases with packet processing on single socket. + All test cases which use at least one Docker container. -.. topic:: 2NUMA +.. topic:: DOCKER - All test cases with packet processing on two sockets. + All test cases which use Docker as container manager. -.. topic:: 1C +.. topic:: APP - 1 worker thread pinned to 1 dedicated physical core; or if HyperThreading is - enabled, 2 worker threads each pinned to a separate logical core within 1 - dedicated physical core. Main thread pinned to core 1. + All test cases with specific APP use. -.. topic:: 2C +Container Orchestration Tags +---------------------------- - 2 worker threads pinned to 2 dedicated physical cores; or if HyperThreading - is enabled, 4 worker threads each pinned to a separate logical core within 2 - dedicated physical cores. Main thread pinned to core 1. +.. topic:: {n}VSWITCH -.. topic:: 4C + {n} VPP running in {n} Docker container(s) acting as a VSWITCH. + {n}=(1). - 4 worker threads pinned to 4 dedicated physical cores; or if HyperThreading - is enabled, 8 worker threads each pinned to a separate logical core within 4 - dedicated physical cores. Main thread pinned to core 1. +.. topic:: {n}VNF -.. topic:: 1T1C + {n} VPP running in {n} Docker container(s) acting as a VNF work load. + {n}=(1). - *Dynamic tag*. - 1 worker thread pinned to 1 dedicated physical core. 1 receive queue per - interface. Main thread pinned to core 1. +Multi-Threading Tags +-------------------- -.. topic:: 2T2C +.. topic:: STHREAD *Dynamic tag*. - 2 worker threads pinned to 2 dedicated physical cores. 1 receive queue per - interface. Main thread pinned to core 1. + All test cases using single poll mode thread. -.. topic:: 4T4C +.. topic:: MTHREAD *Dynamic tag*. - 4 worker threads pinned to 4 dedicated physical cores. 2 receive queues per - interface. Main thread pinned to core 1. + All test cases using more then one poll mode driver thread. -.. topic:: 2T1C +.. topic:: {n}NUMA - *Dynamic tag*. - 2 worker threads each pinned to a separate logical core within 1 dedicated - physical core. 1 receive queue per interface. Main thread pinned to core 1. + All test cases with packet processing on {n} socket(s). {n}=(1,2). -.. topic:: 4T2C +.. topic:: {c}C - *Dynamic tag*. - 4 worker threads each pinned to a separate logical core within 2 dedicated - physical cores. 2 receive queues per interface. Main thread pinned to core - 1. + {c} worker thread pinned to {c} dedicated physical core; or if + HyperThreading is enabled, {c}*2 worker threads each pinned to a separate + logical core within 1 dedicated physical core. Main thread pinned to core 1. + {t}=(1,2,4). -.. topic:: 8T4C +.. topic:: {t}T{c}C *Dynamic tag*. - 8 worker threads each pinned to a separate logical core within 4 dedicated - physical cores. 4 receive queues per interface. Main thread pinned to core - 1. - -Honeycomb tags --------------- - -.. topic:: HC_FUNC - - Honeycomb functional test cases. - -.. topic:: HC_NSH - - Honeycomb NSH test cases. - -.. topic:: HC_PERSIST - - Honeycomb persistence test cases. - -.. topic:: HC_REST_ONLY - - (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode - using ODL client for Restfconf -> Netconf translation. + {t} worker threads pinned to {c} dedicated physical cores. Main thread + pinned to core 1. By default CSIT is configuring same amount of receive + queues per interface as worker threads. {t}=(1,2,4,8), {t}=(1,2,4).