-CSIT TAGs Descriptions
-======================
+CSIT Tags
+=========
All CSIT test cases are labelled with Robot Framework tags used to allow for
easy test case type identification, test case grouping and selection for
-execution. Following sections list currently used CSIT TAGs and their
-documentation.
+execution. Following sections list currently used CSIT tags and their
+descriptions.
-Topology TAGs
--------------
+Testbed Topology Tags
+---------------------
+
+.. topic:: 2_NODE_DOUBLE_LINK_TOPO
+
+ 2 nodes connected in a circular topology with two links interconnecting
+ the devices.
+
+.. topic:: 2_NODE_SINGLE_LINK_TOPO
+
+ 2 nodes connected in a circular topology with at least one link
+ interconnecting devices.
.. topic:: 3_NODE_DOUBLE_LINK_TOPO
3 nodes connected in a circular topology with at least one link
interconnecting devices.
-Objective TAGs
+Objective Tags
--------------
.. topic:: SKIP_PATCH
Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch).
-Environment TAGs
+Environment Tags
----------------
.. topic:: HW_ENV
DUTs with VPP and capable of running Virtual Machine.
-NIC model tags
+NIC Model Tags
--------------
.. topic:: NIC_Intel-X520-DA2
Intel X710 NIC.
+.. topic:: NIC_Intel-XXV710
+
+ Intel XXV710 NIC.
+
.. topic:: NIC_Cisco-VIC-1227
VIC-1227 by Cisco.
VIC-1385 by Cisco.
-Scaling TAGs
+.. topic:: NIC_Amazon-Nitro-50G
+
+ Amazon EC2 ENA NIC.
+
+Scaling Tags
------------
.. topic:: FIB_20K
2x1,000,000 entries in single fib table
-.. topic:: TNL_1000
+.. topic:: L2BD_1
+
+ Test with 1 L2 bridge domain.
+
+.. topic:: L2BD_10
+
+ Test with 10 L2 bridge domains.
+
+.. topic:: L2BD_100
+
+ Test with 100 L2 bridge domains.
+
+.. topic:: L2BD_1K
+
+ Test with 1000 L2 bridge domains.
+
+.. topic:: VLAN_1
+
+ Test with 1 VLAN sub-interface.
+
+.. topic:: VLAN_10
+
+ Test with 10 VLAN sub-interfaces.
+
+.. topic:: VLAN_100
+
+ Test with 100 VLAN sub-interfaces.
+
+.. topic:: VLAN_1K
- IPSec in tunnel mode - 1000 tunnels.
+ Test with 1000 VLAN sub-interfaces.
-.. topic:: SRC_USER_10
+.. topic:: VXLAN_1
- Traffic flow with 10 unique IPs (users) in one direction.
+ Test with 1 VXLAN tunnel.
-.. topic:: SRC_USER_100
+.. topic:: VXLAN_10
- Traffic flow with 100 unique IPs (users) in one direction.
+ Test with 10 VXLAN tunnels.
-.. topic:: SRC_USER_1000
+.. topic:: VXLAN_100
- Traffic flow with 1000 unique IPs (users) in one direction.
+ Test with 100 VXLAN tunnels.
-.. topic:: SRC_USER_2000
+.. topic:: VXLAN_1K
- Traffic flow with 2000 unique IPs (users) in one direction.
+ Test with 1000 VXLAN tunnels.
-.. topic:: SRC_USER_4000
+.. topic:: TNL_{t}
- Traffic flow with 4000 unique IPs (users) in one direction.
+ IPSec in tunnel mode - {t} tunnels.
+
+.. topic:: SRC_USER_{u}
+
+ Traffic flow with {u} unique IPs (users) in one direction.
+ {u}=(1,10,100,1000,2000,4000).
.. topic:: 100_FLOWS
Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in
one direction.
-Tags marking functional vs. performance of tests
-------------------------------------------------
+.. topic:: HOSTS_{h}
-.. topic:: FUNCTEST
+ Stateless or stateful traffic stream with {h} client source IP4 addresses,
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+ {h}=(1024,4096,16384,65536,262144).
- All functional test cases.
-
-.. topic:: PERFTEST
+.. topic:: GENEVE4_{t}TUN
- All performance test cases.
-
-Performance testing tags
-------------------------
+ Test with {t} GENEVE IPv4 tunnel. {t}=(1,4,16,64,256,1024)
-.. topic:: PDRDISC
+Test Category Tags
+------------------
- Partial Drop Rate evaluation of single run result, with non-zero packet
- loss tolerance (LT) expressed in percentage of packets transmitted.
+.. topic:: DEVICETEST
-.. topic:: NDRDISC
+ All vpp_device functional test cases.
- Non Drop Rate evaluation of results. Loss acceptance of dropped packets is
- set to zero lost packets.
+.. topic:: PERFTEST
-.. topic:: NDRCHK
+ All performance test cases.
- Performance tests where TG verifies DUTs' throughput at ref-NDR (reference
- Non Drop Rate) with zero packet loss tolerance.
+VPP Device Type Tags
+--------------------
-.. topic:: NDRPDRDISC
+.. topic:: SCAPY
- Find performance of DUT based on :rfc:`2544` with linear / binary / combined
- search. (Previous LONG tests.)
+ All test cases that uses Scapy for packet generation and validation.
-Ethernet frame size tags for performance tests
-----------------------------------------------
+Performance Type Tags
+---------------------
-.. topic:: 64B
+.. topic:: NDRPDR
- 64B frames used for test.
+ Single test finding both No Drop Rate and Partial Drop Rate simultaneously.
+ The search is done by optimized algorithm which performs
+ multiple trial runs at different durations and transmit rates.
+ The results come from the final trials, which have duration of 30 seconds.
-.. topic:: 78B
+.. topic:: MRR
- 78B frames used for test.
+ Performance tests where TG sends the traffic at maximum rate (line rate)
+ and reports total sent/received packets over trial duration.
+ The result is an average of 10 trials of 1 second duration.
-.. topic:: IMIX
+.. topic:: SOAK
- IMIX frame sequence (28x 64B, 16x 570B, 4x 1518B) used for test.
+ Performance tests using PLRsearch to find the critical load.
-.. topic:: 1460B
+.. topic:: RECONF
- 1460B frames used for test.
+ Performance tests aimed to measure lost packets (time) when performing
+ reconfiguration while full throughput offered load is applied.
-.. topic:: 1480B
-
- 1480B frames used for test.
-
-.. topic:: 1514B
+Ethernet Frame Size Tags
+------------------------
- 1514B frames used for test.
+These are describing the traffic offered by Traffic Generator,
+"primary" traffic in case of asymmetric load.
+For traffic between DUTs, or for "secondary" traffic, see ${overhead} value.
-.. topic:: 1518B
+.. topic:: {b}B
- 1518B frames used for test.
+ {b} Bytes frames used for test.
-.. topic:: 9000B
+.. topic:: IMIX
- 9000B frames used for test.
+ IMIX frame sequence (28x 64B, 16x 570B, 4x 1518B) used for test.
-Test type tags
+Test Type Tags
--------------
.. topic:: BASE
Baseline test cases, no encapsulation, no feature(s) configured in tests.
+ No scaling whatsoever, beyond minimum needed for RSS.
.. topic:: IP4BASE
IPv4 baseline test cases, no encapsulation, no feature(s) configured in
- tests.
+ tests. Minimal number of routes. Other quantities may be scaled.
.. topic:: IP6BASE
L2BD baseline test cases, no encapsulation, no feature(s) configured in
tests.
+.. topic:: L2PATCH
+
+ L2PATCH baseline test cases, no encapsulation, no feature(s) configured in
+ tests.
+
.. topic:: SCALE
- Scale test cases.
+ Scale test cases. Other tags specify which quantities are scaled.
+ Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE).
.. topic:: ENCAP
At least one feature is configured in test cases. Use also feature tag(s).
-.. topic:: TLDK
+.. topic:: UDP
+
+ Tests which use any kind of UDP traffic (STL or ASTF profile).
+
+.. topic:: TCP
+
+ Tests which use any kind of TCP traffic (STL or ASTF profile).
- Functional test cases for TLDK.
+.. topic:: TREX
-Forwarding mode tags
+ Tests which test trex traffic without any software DUTs in the traffic path.
+
+..
+ TODO: Should we define tags STL and ASTF?
+
+.. topic:: UDP_UDIR
+
+ Tests which use unidirectional UDP traffic (STL profile only).
+
+.. topic:: UDP_BIDIR
+
+ Tests which use bidirectional UDP traffic (STL profile only).
+
+.. topic:: UDP_CPS
+
+ Tests which measure connections per second on minimal UDP pseudoconnections.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: TCP_CPS
+
+ Tests which measure connections per second on empty TCP connections.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: TCP_RPS
+
+ Tests which measure requests per second on empty TCP connections.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: UDP_PPS
+
+ Tests which measure packets per second on lightweight UDP transactions.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: TCP_PPS
+
+ Tests which measure packets per second on lightweight TCP transactions.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: HTTP
+
+ Tests which use traffic formed of valid HTTP requests (and responses).
+
+..
+ TODO: Add HTTP tag to the current hoststack tests.
+ TODO: Document other tags already used by hoststack tests.
+
+.. topic:: LDP_NGINX
+
+ LDP NGINX is un-modified NGINX with VPP via LD_PRELOAD.
+
+.. topic:: NF_DENSITY
+
+ Performance tests that measure throughput of multiple VNF and CNF
+ service topologies at different service densities.
+
+NF Service Density Tags
+-----------------------
+
+.. topic:: CHAIN
+
+ NF service density tests with VNF or CNF service chain topology(ies).
+
+.. topic:: PIPE
+
+ NF service density tests with CNF service pipeline topology(ies).
+
+.. topic:: NF_L3FWDIP4
+
+ NF service density tests with DPDK l3fwd IPv4 routing as NF workload.
+
+.. topic:: NF_VPPIP4
+
+ NF service density tests with VPP IPv4 routing as NF workload.
+
+.. topic:: {r}R{c}C
+
+ Service density matrix locator {r}R{c}C, {r}Row denoting number of
+ service instances, {c}Column denoting number of NFs per service
+ instance. {r}=(1,2,4,6,8,10), {c}=(1,2,4,6,8,10).
+
+.. topic:: {n}VM{t}T
+
+ Service density {n}VM{t}T, {n}Number of NF Qemu VMs, {t}Number of threads
+ per NF.
+
+.. topic:: {n}DCRt}T
+
+ Service density {n}DCR{t}T, {n}Number of NF Docker containers, {t}Number of
+ threads per NF.
+
+.. topic:: {n}_ADDED_CHAINS
+
+ {n}Number of chains (or pipelines) added (and/or removed)
+ during RECONF test.
+
+Forwarding Mode Tags
--------------------
.. topic:: L2BDMACSTAT
VPP IPv6 routed forwarding.
-Underlay tags
+.. topic:: LOADBALANCER_MAGLEV
+
+ VPP Load balancer maglev mode.
+
+.. topic:: LOADBALANCER_L3DSR
+
+ VPP Load balancer l3dsr mode.
+
+.. topic:: LOADBALANCER_NAT4
+
+ VPP Load balancer nat4 mode.
+
+.. topic:: N2N
+
+ Mode, where NICs from the same physical server are directly
+ connected with a cable.
+
+Underlay Tags
-------------
.. topic:: IP4UNRLAY
MPLS underlay.
-Overlay tags
+Overlay Tags
------------
.. topic:: L2OVRLAY
IPv6 overlay (IPv6 payload).
-Tagging tags
+Tagging Tags
------------
.. topic:: DOT1Q
All test cases with dot1ad.
-Encapsulation tags
+Encapsulation Tags
------------------
.. topic:: ETH
All test cases with LISP-GPE.
+.. topic:: LISP_IP4o4
+
+ All test cases with LISP_IP4o4.
+
+.. topic:: LISPGPE_IP4o4
+
+ All test cases with LISPGPE_IP4o4.
+
+.. topic:: LISPGPE_IP6o4
+
+ All test cases with LISPGPE_IP6o4.
+
+.. topic:: LISPGPE_IP4o6
+
+ All test cases with LISPGPE_IP4o6.
+
+.. topic:: LISPGPE_IP6o6
+
+ All test cases with LISPGPE_IP6o6.
+
.. topic:: VXLAN
All test cases with Vxlan.
All test cases with GRE.
+.. topic:: GTPU
+
+ All test cases with GTPU.
+
.. topic:: IPSEC
All test cases with IPSEC.
-Interface tags
+.. topic:: SRv6
+
+ All test cases with Segment routing over IPv6 dataplane.
+
+.. topic:: SRv6_1SID
+
+ All SRv6 test cases with single SID.
+
+.. topic:: SRv6_2SID_DECAP
+
+ All SRv6 test cases with two SIDs and with decapsulation.
+
+.. topic:: SRv6_2SID_NODECAP
+
+ All SRv6 test cases with two SIDs and without decapsulation.
+
+.. topic:: GENEVE
+
+ All test cases with GENEVE.
+
+.. topic:: GENEVE_L3MODE
+
+ All test cases with GENEVE tunnel in L3 mode.
+
+.. topic:: FLOW
+
+ All test cases with FLOW.
+
+.. topic:: NTUPLE
+
+ All test cases with NTUPLE.
+
+.. topic:: L2TPV3
+
+ All test cases with L2TPV3.
+
+Interface Tags
--------------
.. topic:: PHY
All test cases which use physical interface(s).
+.. topic:: GSO
+
+ All test cases which uses Generic Segmentation Offload.
+
.. topic:: VHOST
All test cases which uses VHOST.
-.. topic:: VHOST_256
+.. topic:: VHOST_1024
- All test cases which uses VHOST with qemu queue size set to 256.
+ All test cases which uses VHOST DPDK driver with qemu queue size set
+ to 1024.
-.. topic:: VHOST_1024
+.. topic:: VIRTIO
- All test cases which uses VHOST with qemu queue size set to 1024.
+ All test cases which uses VIRTIO native VPP driver.
+
+.. topic:: VIRTIO_1024
+
+ All test cases which uses VIRTIO native VPP driver with qemu queue size set
+ to 1024.
.. topic:: CFS_OPT
All test cases which uses Memif.
-Feature tags
+.. topic:: SINGLE_MEMIF
+
+ All test cases which uses only single Memif connection per DUT. One DUT
+ instance is running in container having one physical interface exposed to
+ container.
+
+.. topic:: LBOND
+
+ All test cases which uses link bonding (BondEthernet interface).
+
+.. topic:: LBOND_DPDK
+
+ All test cases which uses DPDK link bonding.
+
+.. topic:: LBOND_VPP
+
+ All test cases which uses VPP link bonding.
+
+.. topic:: LBOND_MODE_XOR
+
+ All test cases which uses link bonding with mode XOR.
+
+.. topic:: LBOND_MODE_LACP
+
+ All test cases which uses link bonding with mode LACP.
+
+.. topic:: LBOND_LB_L34
+
+ All test cases which uses link bonding with load-balance mode l34.
+
+.. topic:: LBOND_{n}L
+
+ All test cases which use {n} link(s) for link bonding.
+
+.. topic:: DRV_{d}
+
+ All test cases which NIC Driver for DUT is set to {d}. Default is VFIO_PCI.
+ {d}=(AVF, RDMA_CORE, VFIO_PCI, AF_XDP).
+
+.. topic:: TG_DRV_{d}
+
+ All test cases which NIC Driver for TG is set to {d}. Default is IGB_UIO.
+ {d}=(RDMA_CORE, IGB_UIO).
+
+.. topic:: RXQ_SIZE_{n}
+
+ All test cases which RXQ size (RX descriptors) are set to {n}. Default is 0,
+ which means VPP (API) default.
+
+.. topic:: TXQ_SIZE_{n}
+
+ All test cases which TXQ size (TX descriptors) are set to {n}. Default is 0,
+ which means VPP (API) default.
+
+Feature Tags
------------
.. topic:: IACLDST
iACL destination.
-.. topic:: COPWHLIST
+.. topic:: ADLALWLIST
- COP whitelist.
+ ADL allowlist.
.. topic:: NAT44
ACL plugin configured and tested with 50 not-hitting ACEs.
-Encryption tags
+.. topic:: SRv6_PROXY
+
+ SRv6 endpoint to SR-unaware appliance via proxy.
+
+.. topic:: SRv6_PROXY_STAT
+
+ SRv6 endpoint to SR-unaware appliance via static proxy.
+
+.. topic:: SRv6_PROXY_DYN
+
+ SRv6 endpoint to SR-unaware appliance via dynamic proxy.
+
+.. topic:: SRv6_PROXY_MASQ
+
+ SRv6 endpoint to SR-unaware appliance via masquerading proxy.
+
+Encryption Tags
---------------
.. topic:: IPSECSW
IPSec in tunnel mode.
-Client-workload tags
---------------------
-
-.. topic:: VM
-
- All test cases which use at least one virtual machine.
-
-.. topic:: LXC
-
- All test cases which use Linux container and LXC utils.
+.. topic:: IPSECINT
-.. topic:: DOCKER
+ IPSec in interface mode.
- All test cases which use Docker as container manager.
+.. topic:: AES
-.. topic:: APP
+ IPSec using AES algorithms.
- All test cases with specific APP use.
+.. topic:: AES_128_CBC
-Container orchestration tags
-----------------------------
+ IPSec using AES 128 CBC algorithms.
-.. topic:: K8S
+.. topic:: AES_128_GCM
- All test cases which use Kubernetes for orchestration.
+ IPSec using AES 128 GCM algorithms.
-.. topic:: SFC_CONTROLLER
+.. topic:: AES_256_GCM
- All test cases which use ligato/sfc_controller for driving configuration
- of vpp inside container.
+ IPSec using AES 256 GCM algorithms.
-.. topic:: VPP_AGENT
+.. topic:: HMAC
- All test cases which use Golang implementation of a control/management plane
- for VPP
+ IPSec using HMAC integrity algorithms.
-.. topic:: 1VSWITCH
+.. topic:: HMAC_SHA_256
- VPP running in Docker container acting as VSWITCH.
+ IPSec using HMAC SHA 256 integrity algorithms.
-.. topic:: 1VNF
+.. topic:: HMAC_SHA_512
- 1 VPP running in Docker container acting as VNF work load.
+ IPSec using HMAC SHA 512 integrity algorithms.
-.. topic:: 2VNF
+.. topic:: SCHEDULER
- 2 VPP running in 2 Docker containers acting as VNF work load.
+ IPSec using crypto sw scheduler engine.
-Multi-threading tags
+Client-Workload Tags
--------------------
-.. topic:: STHREAD
-
- All test cases using single poll mode thread.
-
-.. topic:: MTHREAD
-
- All test cases using more then one poll mode driver thread.
+.. topic:: VM
-.. topic:: 1NUMA
+ All test cases which use at least one virtual machine.
- All test cases with packet processing on single socket.
+.. topic:: LXC
-.. topic:: 2NUMA
+ All test cases which use Linux container and LXC utils.
- All test cases with packet processing on two sockets.
+.. topic:: DRC
-.. topic:: SMT
+ All test cases which use at least one Docker container.
- All test cases with symmetric Multi-Threading (HyperThreading) enabled.
+.. topic:: DOCKER
-.. topic:: NOSMT
+ All test cases which use Docker as container manager.
- All test cases with symmetric Multi-Threading (HyperThreading) disabled.
+.. topic:: APP
-.. topic:: 1T1C
+ All test cases with specific APP use.
- 1 worker thread pinned to 1 dedicated physical core. 1 receive queue per
- interface. Main thread pinned to core 0.
+Container Orchestration Tags
+----------------------------
-.. topic:: 2T2C
+.. topic:: {n}VSWITCH
- 2 worker threads pinned to 2 dedicated physical cores. 1 receive queue per
- interface. Main thread pinned to core 0.
+ {n} VPP running in {n} Docker container(s) acting as a VSWITCH.
+ {n}=(1).
-.. topic:: 4T4C
+.. topic:: {n}VNF
- 4 worker threads pinned to 4 dedicated physical cores. 2 receive queues per
- interface. Main thread pinned to core 0.
+ {n} VPP running in {n} Docker container(s) acting as a VNF work load.
+ {n}=(1).
-.. topic:: 6T6C
+Multi-Threading Tags
+--------------------
- 6 worker threads pinned to 6 dedicated physical cores. 3 receive queues per
- interface. Main thread pinned to core 0.
+.. topic:: STHREAD
-.. topic:: 8T8C
+ *Dynamic tag*.
+ All test cases using single poll mode thread.
- 8 worker threads pinned to 8 dedicated physical cores. 4 receive queues per
- interface. Main thread pinned to core 0.
+.. topic:: MTHREAD
-Honeycomb tags
---------------
+ *Dynamic tag*.
+ All test cases using more then one poll mode driver thread.
-.. topic:: HC_FUNC
+.. topic:: {n}NUMA
- Honeycomb functional test cases.
+ All test cases with packet processing on {n} socket(s). {n}=(1,2).
-.. topic:: HC_PERSIST
+.. topic:: {c}C
- Honeycomb persistence test cases.
+ {c} worker thread pinned to {c} dedicated physical core; or if
+ HyperThreading is enabled, {c}*2 worker threads each pinned to a separate
+ logical core within 1 dedicated physical core. Main thread pinned to core 1.
+ {t}=(1,2,4).
-.. topic:: HC_REST_ONLY
+.. topic:: {t}T{c}C
- (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode
- using ODL client for Restfconf -> Netconf translation.
+ *Dynamic tag*.
+ {t} worker threads pinned to {c} dedicated physical cores. Main thread
+ pinned to core 1. By default CSIT is configuring same amount of receive
+ queues per interface as worker threads. {t}=(1,2,4,8), {t}=(1,2,4).
Code Review / csit.git / blobdiff