X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Ftag_documentation.rst;h=fa015759a196b4cf223d45823793fb73821a1b59;hp=6247fc5dffddc198ebea1e1efda724dbd0e8ed3d;hb=9377c956a86e42727039d9dab8879c10c9399f4c;hpb=18cd1f996429c380c9eef0d3a52ec95b9e55c996 diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst index 6247fc5dff..fa015759a1 100644 --- a/docs/tag_documentation.rst +++ b/docs/tag_documentation.rst @@ -1,13 +1,13 @@ -CSIT TAGs +CSIT Tags ========= All CSIT test cases are labelled with Robot Framework tags used to allow for easy test case type identification, test case grouping and selection for -execution. Following sections list currently used CSIT TAGs and their -documentation. +execution. Following sections list currently used CSIT tags and their +descriptions. -Topology TAGs -------------- +Testbed Topology Tags +--------------------- .. topic:: 2_NODE_DOUBLE_LINK_TOPO @@ -29,7 +29,7 @@ Topology TAGs 3 nodes connected in a circular topology with at least one link interconnecting devices. -Objective TAGs +Objective Tags -------------- .. topic:: SKIP_PATCH @@ -41,7 +41,7 @@ Objective TAGs Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch). -Environment TAGs +Environment Tags ---------------- .. topic:: HW_ENV @@ -56,7 +56,7 @@ Environment TAGs DUTs with VPP and capable of running Virtual Machine. -NIC model tags +NIC Model Tags -------------- .. topic:: NIC_Intel-X520-DA2 @@ -83,7 +83,11 @@ NIC model tags VIC-1385 by Cisco. -Scaling TAGs +.. topic:: NIC_Amazon-Nitro-50G + + Amazon EC2 ENA NIC. + +Scaling Tags ------------ .. topic:: FIB_20K @@ -146,9 +150,13 @@ Scaling TAGs Test with 1000 VXLAN tunnels. -.. topic:: TNL_1000 +.. topic:: TNL_{t} + + IPSec in tunnel mode - {t} tunnels. + +.. topic:: SRC_USER_1 - IPSec in tunnel mode - 1000 tunnels. + Traffic flow with 1 unique IP (users) in one direction. .. topic:: SRC_USER_10 @@ -185,8 +193,62 @@ Scaling TAGs Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in one direction. -Tags marking functional vs. performance of tests ------------------------------------------------- +.. topic:: HOSTS_1024 + + Stateless or stateful traffic stream with 1024 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_4096 + + Stateless or stateful traffic stream with 4096 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_16384 + + Stateless or stateful traffic stream with 16384 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_65536 + + Stateless or stateful traffic stream with 65536 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_262144 + + Stateless or stateful traffic stream with 262144 client source IP4 addresses + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: GENEVE4_1TUN + + Test with 1 GENEVE IPv4 tunnel. + +.. topic:: GENEVE4_4TUN + + Test with 4 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_16TUN + + Test with 16 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_64TUN + + Test with 64 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_256TUN + + Test with 256 GENEVE IPv4 tunnels. + +.. topic:: GENEVE4_1024TUN + + Test with 1024 GENEVE IPv4 tunnels. + +Test Category Tags +------------------ .. topic:: FUNCTEST @@ -196,8 +258,8 @@ Tags marking functional vs. performance of tests All performance test cases. -Performance testing tags ------------------------- +Performance Type Tags +--------------------- .. topic:: NDRPDR @@ -212,20 +274,37 @@ Performance testing tags and reports total sent/received packets over trial duration. The result is an average of 10 trials of 1 second duration. -Ethernet frame size tags for performance tests ----------------------------------------------- +.. topic:: SOAK + + Performance tests using PLRsearch to find the critical load. + +.. topic:: RECONF + + Performance tests aimed to measure lost packets (time) when performing + reconfiguration while full throughput offered load is applied. + +Ethernet Frame Size Tags +------------------------ + +These are describing the traffic offered by Traffic Generator, +"primary" traffic in case of asymmetric load. +For traffic between DUTs, or for "secondary" traffic, see ${overhead} value. .. topic:: 64B - 64B frames used for test. + 64B frames used for test. Generic ethernet or IPv4. .. topic:: 78B - 78B frames used for test. + 78B frames used for test. Ipv6. .. topic:: 114B - 114B frames used for test. + 114B frames used for test. IPv4+vxlan. + +.. topic:: 118B + + 118B frames used for test. Dot1q+IPv4+vxlan. .. topic:: IMIX @@ -251,17 +330,18 @@ Ethernet frame size tags for performance tests 9000B frames used for test. -Test type tags +Test Type Tags -------------- .. topic:: BASE Baseline test cases, no encapsulation, no feature(s) configured in tests. + No scaling whatsoever, beyond minimum needed for RSS. .. topic:: IP4BASE IPv4 baseline test cases, no encapsulation, no feature(s) configured in - tests. + tests. Minimal number of routes. Other quantities may be scaled. .. topic:: IP6BASE @@ -285,7 +365,8 @@ Test type tags .. topic:: SCALE - Scale test cases. + Scale test cases. Other tags specify which quantities are scaled. + Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE). .. topic:: ENCAP @@ -295,31 +376,113 @@ Test type tags At least one feature is configured in test cases. Use also feature tag(s). -.. topic:: TLDK +.. topic:: UDP - Functional test cases for TLDK. + Tests which use any kind of UDP traffic (STL or ASTF profile). -.. topic:: DMM +.. topic:: TCP - Functional test cases for DMM. + Tests which use any kind of TCP traffic (STL or ASTF profile). -.. topic:: TCP +.. + TODO: Should we define tags STL and ASTF? + +.. topic:: UDP_UDIR + + Tests which use unidirectional UDP traffic (STL profile only). - Tests which use TCP. +.. topic:: UDP_BIDIR + + Tests which use bidirectional UDP traffic (STL profile only). + +.. topic:: UDP_CPS + + Tests which measure connections per second on minimal UDP pseudoconnections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_CPS - Performance tests which measure connections per second using http requests. + Tests which measure connections per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_RPS - Performance tests which measure requests per second using http requests. + Tests which measure requests per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: UDP_PPS + + Tests which measure packets per second on lightweight UDP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: TCP_PPS + + Tests which measure packets per second on lightweight TCP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: HTTP - Tests which use HTTP. + Tests which use traffic formed of valid HTTP requests (and responses). + +.. + TODO: Add HTTP tag to the current hoststack tests. + TODO: Document other tags already used by hoststack tests. + +.. topic:: LDP_NGINX + + LDP NGINX is un-modified NGINX with VPP via LD_PRELOAD. + +.. topic:: NF_DENSITY + + Performance tests that measure throughput of multiple VNF and CNF + service topologies at different service densities. + +NF Service Density Tags +----------------------- + +.. topic:: CHAIN -Forwarding mode tags + NF service density tests with VNF or CNF service chain topology(ies). + +.. topic:: PIPE + + NF service density tests with CNF service pipeline topology(ies). + +.. topic:: NF_L3FWDIP4 + + NF service density tests with DPDK l3fwd IPv4 routing as NF workload. + +.. topic:: NF_VPPIP4 + + NF service density tests with VPP IPv4 routing as NF workload. + +.. topic:: {r}R{c}C + + Service density matrix locator {r}R{c}C, {r}Row denoting number of + service instances, {c}Column denoting number of NFs per service + instance. {r}=(1,2,4,6,8,10), {c}=(1,2,4,6,8,10). + +.. topic:: {n}VM{t}T + + Service density {n}VM{t}T, {n}Number of NF Qemu VMs, {t}Number of threads + per NF. + +.. topic:: {n}DCRt}T + + Service density {n}DCR{t}T, {n}Number of NF Docker containers, {t}Number of + threads per NF. + +.. topic:: {n}_ADDED_CHAINS + + {n}Number of chains (or pipelines) added (and/or removed) + during RECONF test. + +Forwarding Mode Tags -------------------- .. topic:: L2BDMACSTAT @@ -342,7 +505,19 @@ Forwarding mode tags VPP IPv6 routed forwarding. -Underlay tags +.. topic:: LOADBALANCER_MAGLEV + + VPP Load balancer maglev mode. + +.. topic:: LOADBALANCER_L3DSR + + VPP Load balancer l3dsr mode. + +.. topic:: LOADBALANCER_NAT4 + + VPP Load balancer nat4 mode. + +Underlay Tags ------------- .. topic:: IP4UNRLAY @@ -357,7 +532,7 @@ Underlay tags MPLS underlay. -Overlay tags +Overlay Tags ------------ .. topic:: L2OVRLAY @@ -372,7 +547,7 @@ Overlay tags IPv6 overlay (IPv6 payload). -Tagging tags +Tagging Tags ------------ .. topic:: DOT1Q @@ -383,7 +558,7 @@ Tagging tags All test cases with dot1ad. -Encapsulation tags +Encapsulation Tags ------------------ .. topic:: ETH @@ -398,6 +573,26 @@ Encapsulation tags All test cases with LISP-GPE. +.. topic:: LISP_IP4o4 + + All test cases with LISP_IP4o4. + +.. topic:: LISPGPE_IP4o4 + + All test cases with LISPGPE_IP4o4. + +.. topic:: LISPGPE_IP6o4 + + All test cases with LISPGPE_IP6o4. + +.. topic:: LISPGPE_IP4o6 + + All test cases with LISPGPE_IP4o6. + +.. topic:: LISPGPE_IP6o6 + + All test cases with LISPGPE_IP6o6. + .. topic:: VXLAN All test cases with Vxlan. @@ -418,24 +613,54 @@ Encapsulation tags All test cases with Segment routing over IPv6 dataplane. -Interface tags +.. topic:: SRv6_1SID + + All SRv6 test cases with single SID. + +.. topic:: SRv6_2SID_DECAP + + All SRv6 test cases with two SIDs and with decapsulation. + +.. topic:: SRv6_2SID_NODECAP + + All SRv6 test cases with two SIDs and without decapsulation. + +.. topic:: GENEVE + + All test cases with GENEVE. + +.. topic:: GENEVE_L3MODE + + All test cases with GENEVE tunnel in L3 mode. + +Interface Tags -------------- .. topic:: PHY All test cases which use physical interface(s). +.. topic:: GSO + + All test cases which uses Generic Segmentation Offload. + .. topic:: VHOST All test cases which uses VHOST. -.. topic:: VHOST_256 +.. topic:: VHOST_1024 + + All test cases which uses VHOST DPDK driver with qemu queue size set + to 1024. - All test cases which uses VHOST with qemu queue size set to 256. +.. topic:: VIRTIO -.. topic:: VHOST_1024 + All test cases which uses VIRTIO native VPP driver. - All test cases which uses VHOST with qemu queue size set to 1024. +.. topic:: VIRTIO_1024 + + All test cases which uses VIRTIO native VPP driver with qemu queue size set + to 1024. .. topic:: CFS_OPT @@ -503,16 +728,36 @@ Interface tags devices. In essence, today this driver can be used only with Intel XL710 / X710 / XXV710 adapters. -Feature tags +.. topic:: DRV_VFIO_PCI + + All test cases which uses vfio-pci device driver. It supports variety of NIC + adapters. + +.. topic:: DRV_RDMA_CORE + + All test cases which uses rdma-core device driver. It supports Mellanox + NIC adapters. + +.. topic:: RXQ_SIZE_{n} + + All test cases which RXQ size (RX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +.. topic:: TXQ_SIZE_{n} + + All test cases which TXQ size (TX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +Feature Tags ------------ .. topic:: IACLDST iACL destination. -.. topic:: COPWHLIST +.. topic:: ADLALWLIST - COP whitelist. + ADL allowlist. .. topic:: NAT44 @@ -570,7 +815,7 @@ Feature tags SRv6 endpoint to SR-unaware appliance via masquerading proxy. -Encryption tags +Encryption Tags --------------- .. topic:: IPSECSW @@ -589,7 +834,43 @@ Encryption tags IPSec in tunnel mode. -Client-workload tags +.. topic:: IPSECINT + + IPSec in interface mode. + +.. topic:: AES + + IPSec using AES algorithms. + +.. topic:: AES_128_CBC + + IPSec using AES 128 CBC algorithms. + +.. topic:: AES_128_GCM + + IPSec using AES 128 GCM algorithms. + +.. topic:: AES_256_GCM + + IPSec using AES 256 GCM algorithms. + +.. topic:: HMAC + + IPSec using HMAC integrity algorithms. + +.. topic:: HMAC_SHA_256 + + IPSec using HMAC SHA 256 integrity algorithms. + +.. topic:: HMAC_SHA_512 + + IPSec using HMAC SHA 512 integrity algorithms. + +.. topic:: SCHEDULER + + IPSec using crypto sw scheduler engine. + +Client-Workload Tags -------------------- .. topic:: VM @@ -600,6 +881,10 @@ Client-workload tags All test cases which use Linux container and LXC utils. +.. topic:: DRC + + All test cases which use at least one Docker container. + .. topic:: DOCKER All test cases which use Docker as container manager. @@ -608,23 +893,9 @@ Client-workload tags All test cases with specific APP use. -Container orchestration tags +Container Orchestration Tags ---------------------------- -.. topic:: K8S - - All test cases which use Kubernetes for orchestration. - -.. topic:: SFC_CONTROLLER - - All test cases which use ligato/sfc_controller for driving configuration - of vpp inside container. - -.. topic:: VPP_AGENT - - All test cases which use Golang implementation of a control/management plane - for VPP - .. topic:: 1VSWITCH VPP running in Docker container acting as VSWITCH. @@ -641,7 +912,7 @@ Container orchestration tags 4 VPP running in 4 Docker containers acting as VNF work load. -Multi-threading tags +Multi-Threading Tags -------------------- .. topic:: STHREAD @@ -717,23 +988,3 @@ Multi-threading tags 8 worker threads each pinned to a separate logical core within 4 dedicated physical cores. 4 receive queues per interface. Main thread pinned to core 1. - -Honeycomb tags --------------- - -.. topic:: HC_FUNC - - Honeycomb functional test cases. - -.. topic:: HC_NSH - - Honeycomb NSH test cases. - -.. topic:: HC_PERSIST - - Honeycomb persistence test cases. - -.. topic:: HC_REST_ONLY - - (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode - using ODL client for Restfconf -> Netconf translation.