X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Ftag_documentation.rst;h=fa015759a196b4cf223d45823793fb73821a1b59;hp=7cb71f115e8cb5be08e45d699ee33586f1e9bbc9;hb=9377c956a86e42727039d9dab8879c10c9399f4c;hpb=88e6c1f979f530397a0743e4300d42faf9b538f5 diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst index 7cb71f115e..fa015759a1 100644 --- a/docs/tag_documentation.rst +++ b/docs/tag_documentation.rst @@ -1,13 +1,13 @@ -CSIT TAGs +CSIT Tags ========= All CSIT test cases are labelled with Robot Framework tags used to allow for easy test case type identification, test case grouping and selection for -execution. Following sections list currently used CSIT TAGs and their -documentation. +execution. Following sections list currently used CSIT tags and their +descriptions. -Topology TAGs -------------- +Testbed Topology Tags +--------------------- .. topic:: 2_NODE_DOUBLE_LINK_TOPO @@ -29,7 +29,7 @@ Topology TAGs 3 nodes connected in a circular topology with at least one link interconnecting devices. -Objective TAGs +Objective Tags -------------- .. topic:: SKIP_PATCH @@ -41,7 +41,7 @@ Objective TAGs Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch). -Environment TAGs +Environment Tags ---------------- .. topic:: HW_ENV @@ -56,7 +56,7 @@ Environment TAGs DUTs with VPP and capable of running Virtual Machine. -NIC model tags +NIC Model Tags -------------- .. topic:: NIC_Intel-X520-DA2 @@ -83,7 +83,11 @@ NIC model tags VIC-1385 by Cisco. -Scaling TAGs +.. topic:: NIC_Amazon-Nitro-50G + + Amazon EC2 ENA NIC. + +Scaling Tags ------------ .. topic:: FIB_20K @@ -146,9 +150,13 @@ Scaling TAGs Test with 1000 VXLAN tunnels. -.. topic:: TNL_1000 +.. topic:: TNL_{t} + + IPSec in tunnel mode - {t} tunnels. + +.. topic:: SRC_USER_1 - IPSec in tunnel mode - 1000 tunnels. + Traffic flow with 1 unique IP (users) in one direction. .. topic:: SRC_USER_10 @@ -185,24 +193,62 @@ Scaling TAGs Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in one direction. -.. topic:: 1C1N +.. topic:: HOSTS_1024 + + Stateless or stateful traffic stream with 1024 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_4096 + + Stateless or stateful traffic stream with 4096 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_16384 + + Stateless or stateful traffic stream with 16384 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_65536 + + Stateless or stateful traffic stream with 65536 client source IP4 addresses, + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: HOSTS_262144 + + Stateless or stateful traffic stream with 262144 client source IP4 addresses + usually with 63 flow differing in source port number. Could be UDP or TCP. + If NAT is used, the clients are inside. Outside IP range can differ. + +.. topic:: GENEVE4_1TUN + + Test with 1 GENEVE IPv4 tunnel. + +.. topic:: GENEVE4_4TUN + + Test with 4 GENEVE IPv4 tunnels. - 1 service chain with 1 network function per chain. +.. topic:: GENEVE4_16TUN -.. topic:: 2C1N + Test with 16 GENEVE IPv4 tunnels. - 2 service chains with 1 network function per chain. +.. topic:: GENEVE4_64TUN -.. topic:: 1C2N + Test with 64 GENEVE IPv4 tunnels. - 1 service chain with 2 network functions per chain. +.. topic:: GENEVE4_256TUN -.. topic:: 2C2N + Test with 256 GENEVE IPv4 tunnels. - 2 service chains with 2 network functions per chain. +.. topic:: GENEVE4_1024TUN -Tags marking functional vs. performance of tests ------------------------------------------------- + Test with 1024 GENEVE IPv4 tunnels. + +Test Category Tags +------------------ .. topic:: FUNCTEST @@ -212,8 +258,8 @@ Tags marking functional vs. performance of tests All performance test cases. -Performance testing tags ------------------------- +Performance Type Tags +--------------------- .. topic:: NDRPDR @@ -228,20 +274,37 @@ Performance testing tags and reports total sent/received packets over trial duration. The result is an average of 10 trials of 1 second duration. -Ethernet frame size tags for performance tests ----------------------------------------------- +.. topic:: SOAK + + Performance tests using PLRsearch to find the critical load. + +.. topic:: RECONF + + Performance tests aimed to measure lost packets (time) when performing + reconfiguration while full throughput offered load is applied. + +Ethernet Frame Size Tags +------------------------ + +These are describing the traffic offered by Traffic Generator, +"primary" traffic in case of asymmetric load. +For traffic between DUTs, or for "secondary" traffic, see ${overhead} value. .. topic:: 64B - 64B frames used for test. + 64B frames used for test. Generic ethernet or IPv4. .. topic:: 78B - 78B frames used for test. + 78B frames used for test. Ipv6. .. topic:: 114B - 114B frames used for test. + 114B frames used for test. IPv4+vxlan. + +.. topic:: 118B + + 118B frames used for test. Dot1q+IPv4+vxlan. .. topic:: IMIX @@ -267,17 +330,18 @@ Ethernet frame size tags for performance tests 9000B frames used for test. -Test type tags +Test Type Tags -------------- .. topic:: BASE Baseline test cases, no encapsulation, no feature(s) configured in tests. + No scaling whatsoever, beyond minimum needed for RSS. .. topic:: IP4BASE IPv4 baseline test cases, no encapsulation, no feature(s) configured in - tests. + tests. Minimal number of routes. Other quantities may be scaled. .. topic:: IP6BASE @@ -301,7 +365,8 @@ Test type tags .. topic:: SCALE - Scale test cases. + Scale test cases. Other tags specify which quantities are scaled. + Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE). .. topic:: ENCAP @@ -311,36 +376,113 @@ Test type tags At least one feature is configured in test cases. Use also feature tag(s). -.. topic:: TLDK +.. topic:: UDP + + Tests which use any kind of UDP traffic (STL or ASTF profile). - Functional test cases for TLDK. +.. topic:: TCP -.. topic:: DMM + Tests which use any kind of TCP traffic (STL or ASTF profile). - Functional test cases for DMM. +.. + TODO: Should we define tags STL and ASTF? -.. topic:: TCP +.. topic:: UDP_UDIR + + Tests which use unidirectional UDP traffic (STL profile only). + +.. topic:: UDP_BIDIR - Tests which use TCP. + Tests which use bidirectional UDP traffic (STL profile only). + +.. topic:: UDP_CPS + + Tests which measure connections per second on minimal UDP pseudoconnections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_CPS - Performance tests which measure connections per second using http requests. + Tests which measure connections per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: TCP_RPS - Performance tests which measure requests per second using http requests. + Tests which measure requests per second on empty TCP connections. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: UDP_PPS + + Tests which measure packets per second on lightweight UDP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. + +.. topic:: TCP_PPS + + Tests which measure packets per second on lightweight TCP transactions. + This implies ASTF traffic profile is used. + This tag selects specific output processing in PAL. .. topic:: HTTP - Tests which use HTTP. + Tests which use traffic formed of valid HTTP requests (and responses). + +.. + TODO: Add HTTP tag to the current hoststack tests. + TODO: Document other tags already used by hoststack tests. + +.. topic:: LDP_NGINX + + LDP NGINX is un-modified NGINX with VPP via LD_PRELOAD. + +.. topic:: NF_DENSITY + + Performance tests that measure throughput of multiple VNF and CNF + service topologies at different service densities. + +NF Service Density Tags +----------------------- + +.. topic:: CHAIN + + NF service density tests with VNF or CNF service chain topology(ies). + +.. topic:: PIPE + + NF service density tests with CNF service pipeline topology(ies). + +.. topic:: NF_L3FWDIP4 -.. topic:: SERVICE_DENSITY + NF service density tests with DPDK l3fwd IPv4 routing as NF workload. - Performance tests which measures performance of service density by creating - topology of multiple sevice chains of network functions. +.. topic:: NF_VPPIP4 -Forwarding mode tags + NF service density tests with VPP IPv4 routing as NF workload. + +.. topic:: {r}R{c}C + + Service density matrix locator {r}R{c}C, {r}Row denoting number of + service instances, {c}Column denoting number of NFs per service + instance. {r}=(1,2,4,6,8,10), {c}=(1,2,4,6,8,10). + +.. topic:: {n}VM{t}T + + Service density {n}VM{t}T, {n}Number of NF Qemu VMs, {t}Number of threads + per NF. + +.. topic:: {n}DCRt}T + + Service density {n}DCR{t}T, {n}Number of NF Docker containers, {t}Number of + threads per NF. + +.. topic:: {n}_ADDED_CHAINS + + {n}Number of chains (or pipelines) added (and/or removed) + during RECONF test. + +Forwarding Mode Tags -------------------- .. topic:: L2BDMACSTAT @@ -363,7 +505,19 @@ Forwarding mode tags VPP IPv6 routed forwarding. -Underlay tags +.. topic:: LOADBALANCER_MAGLEV + + VPP Load balancer maglev mode. + +.. topic:: LOADBALANCER_L3DSR + + VPP Load balancer l3dsr mode. + +.. topic:: LOADBALANCER_NAT4 + + VPP Load balancer nat4 mode. + +Underlay Tags ------------- .. topic:: IP4UNRLAY @@ -378,7 +532,7 @@ Underlay tags MPLS underlay. -Overlay tags +Overlay Tags ------------ .. topic:: L2OVRLAY @@ -393,7 +547,7 @@ Overlay tags IPv6 overlay (IPv6 payload). -Tagging tags +Tagging Tags ------------ .. topic:: DOT1Q @@ -404,7 +558,7 @@ Tagging tags All test cases with dot1ad. -Encapsulation tags +Encapsulation Tags ------------------ .. topic:: ETH @@ -419,6 +573,26 @@ Encapsulation tags All test cases with LISP-GPE. +.. topic:: LISP_IP4o4 + + All test cases with LISP_IP4o4. + +.. topic:: LISPGPE_IP4o4 + + All test cases with LISPGPE_IP4o4. + +.. topic:: LISPGPE_IP6o4 + + All test cases with LISPGPE_IP6o4. + +.. topic:: LISPGPE_IP4o6 + + All test cases with LISPGPE_IP4o6. + +.. topic:: LISPGPE_IP6o6 + + All test cases with LISPGPE_IP6o6. + .. topic:: VXLAN All test cases with Vxlan. @@ -439,24 +613,54 @@ Encapsulation tags All test cases with Segment routing over IPv6 dataplane. -Interface tags +.. topic:: SRv6_1SID + + All SRv6 test cases with single SID. + +.. topic:: SRv6_2SID_DECAP + + All SRv6 test cases with two SIDs and with decapsulation. + +.. topic:: SRv6_2SID_NODECAP + + All SRv6 test cases with two SIDs and without decapsulation. + +.. topic:: GENEVE + + All test cases with GENEVE. + +.. topic:: GENEVE_L3MODE + + All test cases with GENEVE tunnel in L3 mode. + +Interface Tags -------------- .. topic:: PHY All test cases which use physical interface(s). +.. topic:: GSO + + All test cases which uses Generic Segmentation Offload. + .. topic:: VHOST All test cases which uses VHOST. -.. topic:: VHOST_256 +.. topic:: VHOST_1024 - All test cases which uses VHOST with qemu queue size set to 256. + All test cases which uses VHOST DPDK driver with qemu queue size set + to 1024. -.. topic:: VHOST_1024 +.. topic:: VIRTIO - All test cases which uses VHOST with qemu queue size set to 1024. + All test cases which uses VIRTIO native VPP driver. + +.. topic:: VIRTIO_1024 + + All test cases which uses VIRTIO native VPP driver with qemu queue size set + to 1024. .. topic:: CFS_OPT @@ -524,16 +728,36 @@ Interface tags devices. In essence, today this driver can be used only with Intel XL710 / X710 / XXV710 adapters. -Feature tags +.. topic:: DRV_VFIO_PCI + + All test cases which uses vfio-pci device driver. It supports variety of NIC + adapters. + +.. topic:: DRV_RDMA_CORE + + All test cases which uses rdma-core device driver. It supports Mellanox + NIC adapters. + +.. topic:: RXQ_SIZE_{n} + + All test cases which RXQ size (RX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +.. topic:: TXQ_SIZE_{n} + + All test cases which TXQ size (TX descriptors) are set to {n}. Default is 0, + which means VPP (API) default. + +Feature Tags ------------ .. topic:: IACLDST iACL destination. -.. topic:: COPWHLIST +.. topic:: ADLALWLIST - COP whitelist. + ADL allowlist. .. topic:: NAT44 @@ -591,7 +815,7 @@ Feature tags SRv6 endpoint to SR-unaware appliance via masquerading proxy. -Encryption tags +Encryption Tags --------------- .. topic:: IPSECSW @@ -610,7 +834,43 @@ Encryption tags IPSec in tunnel mode. -Client-workload tags +.. topic:: IPSECINT + + IPSec in interface mode. + +.. topic:: AES + + IPSec using AES algorithms. + +.. topic:: AES_128_CBC + + IPSec using AES 128 CBC algorithms. + +.. topic:: AES_128_GCM + + IPSec using AES 128 GCM algorithms. + +.. topic:: AES_256_GCM + + IPSec using AES 256 GCM algorithms. + +.. topic:: HMAC + + IPSec using HMAC integrity algorithms. + +.. topic:: HMAC_SHA_256 + + IPSec using HMAC SHA 256 integrity algorithms. + +.. topic:: HMAC_SHA_512 + + IPSec using HMAC SHA 512 integrity algorithms. + +.. topic:: SCHEDULER + + IPSec using crypto sw scheduler engine. + +Client-Workload Tags -------------------- .. topic:: VM @@ -621,6 +881,10 @@ Client-workload tags All test cases which use Linux container and LXC utils. +.. topic:: DRC + + All test cases which use at least one Docker container. + .. topic:: DOCKER All test cases which use Docker as container manager. @@ -629,23 +893,9 @@ Client-workload tags All test cases with specific APP use. -Container orchestration tags +Container Orchestration Tags ---------------------------- -.. topic:: K8S - - All test cases which use Kubernetes for orchestration. - -.. topic:: SFC_CONTROLLER - - All test cases which use ligato/sfc_controller for driving configuration - of vpp inside container. - -.. topic:: VPP_AGENT - - All test cases which use Golang implementation of a control/management plane - for VPP - .. topic:: 1VSWITCH VPP running in Docker container acting as VSWITCH. @@ -662,7 +912,7 @@ Container orchestration tags 4 VPP running in 4 Docker containers acting as VNF work load. -Multi-threading tags +Multi-Threading Tags -------------------- .. topic:: STHREAD @@ -738,23 +988,3 @@ Multi-threading tags 8 worker threads each pinned to a separate logical core within 4 dedicated physical cores. 4 receive queues per interface. Main thread pinned to core 1. - -Honeycomb tags --------------- - -.. topic:: HC_FUNC - - Honeycomb functional test cases. - -.. topic:: HC_NSH - - Honeycomb NSH test cases. - -.. topic:: HC_PERSIST - - Honeycomb persistence test cases. - -.. topic:: HC_REST_ONLY - - (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode - using ODL client for Restfconf -> Netconf translation.