X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Ftag_documentation.rst;h=e5dd38844c53535587385be9627c1a09c37ef004;hp=2531c0242f8392103656764676a2231bd1734aa1;hb=0cd7a2d0d4e701ca61421c5de9b4b0460fdfa28a;hpb=b1a54dd8bbf32a025b998ef9d87a4c2d908c28dc diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst index 2531c0242f..e5dd38844c 100644 --- a/docs/tag_documentation.rst +++ b/docs/tag_documentation.rst @@ -1,128 +1,519 @@ -# Copyright (c) 2016 Cisco and/or its affiliates. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at: -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -Documentation for tags used to select and identify test cases. - -List of TAGs and their descriptions -=================================== +CSIT TAGs Descriptions +====================== + +All CSIT test cases are labelled with Robot Framework tags used to allow for +easy test case type identification, test case grouping and selection for +execution. Following sections list currently used CSIT TAGs and their +documentation. Topology TAGs ------------- -3_NODE_DOUBLE_LINK_TOPO +.. topic:: 3_NODE_DOUBLE_LINK_TOPO + 3 nodes connected in a circular topology with two links interconnecting the devices. -3_NODE_SINGLE_LINK_TOPO +.. topic:: 3_NODE_SINGLE_LINK_TOPO + 3 nodes connected in a circular topology with at least one link interconnecting devices. Objective TAGs -------------- -SKIP_PATCH +.. topic:: SKIP_PATCH + Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch) and csit-vpp-verify jobs (i.e. CSIT patch). +.. topic:: SKIP_VPP_PATCH + + Test case(s) marked to not run in case of vpp-csit-verify (i.e. VPP patch). + Environment TAGs ---------------- -HW_ENV +.. topic:: HW_ENV + DUTs and TGs are running on bare metal. -VM_ENV +.. topic:: VM_ENV + DUTs and TGs are running in virtual environment. -VPP_VM_ENV +.. topic:: VPP_VM_ENV + DUTs with VPP and capable of running Virtual Machine. -DUT Setup TAGs +NIC model tags -------------- -1_THREAD_NOHTT_RSS_1 - 1 worker thread pinned to dedicated core without use of Hyper-threading - technology with 1 thread per interface. Main thread pinned to core 0. +.. topic:: NIC_Intel-X520-DA2 -2_THREAD_NOHTT_RSS_1 - 2 worker threads each pinned to dedicated core without use of Hyper-threading - technology with 1 thread per interface. Main thread pinned to core 0. + Intel X520-DA2 NIC. -4_THREAD_NOHTT_RSS_2 - 4 worker threads each pinned to dedicated core without use of Hyper-threading - technology with 2 threads per interface. Main thread pinned to core 0. +.. topic:: NIC_Intel-XL710 -6_THREAD_NOHTT_RSS_3 - 6 worker threads each pinned to dedicated core without use of Hyper-threading - technology with 3 threads per interface. Main thread pinned to core 0. + Intel XL710 NIC. -8_THREAD_HTT_RSS_4 - 8 worker threads each pinned to dedicated core without use of Hyper-threading - technology with 4 threads per interface. Main thread pinned to core 0. +.. topic:: NIC_Intel-X710 -SINGLE_THREAD - All single threaded test cases. + Intel X710 NIC. -MULTI_THREAD - All test cases with more then one thread. +.. topic:: NIC_Cisco-VIC-1227 -Performance testing TAGs ------------------------- + VIC-1227 by Cisco. + +.. topic:: NIC_Cisco-VIC-1385 + + VIC-1385 by Cisco. + +Scaling TAGs +------------ + +.. topic:: FIB_20K + + 2x10,000 entries in single fib table + +.. topic:: FIB_200K + + 2x100,000 entries in single fib table + +.. topic:: FIB_2M + + 2x1,000,000 entries in single fib table + +.. topic:: TNL_1000 + + IPSec in tunnel mode - 1000 tunnels. + +.. topic:: SRC_USER_10 + + Traffic flow with 10 unique IPs (users) in one direction. + +.. topic:: SRC_USER_100 + + Traffic flow with 100 unique IPs (users) in one direction. + +.. topic:: SRC_USER_1000 + + Traffic flow with 1000 unique IPs (users) in one direction. + +.. topic:: SRC_USER_2000 + + Traffic flow with 2000 unique IPs (users) in one direction. + +.. topic:: SRC_USER_4000 + + Traffic flow with 4000 unique IPs (users) in one direction. + +.. topic:: 100_FLOWS + + Traffic stream with 100 unique flows (10 IPs/users x 10 UDP ports) in one + direction. + +.. topic:: 10k_FLOWS + + Traffic stream with 10 000 unique flows (10 IPs/users x 1000 UDP ports) in + one direction. + +.. topic:: 100k_FLOWS + + Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in + one direction. + +Tags marking functional vs. performance of tests +------------------------------------------------ + +.. topic:: FUNCTEST + + All functional test cases. + +.. topic:: PERFTEST -PERFTEST All performance test cases. -PERFTEST_VHOST - All performance test cases that include testing of VHOST with VM. +Performance testing tags +------------------------ -PERFTEST_SHORT - Performance of DUT should pass specific value. Each test case run is - executed for 10 seconds. +.. topic:: PDRDISC -PERFTEST_LONG - Find performance of DUT based on RFC2544 with linear/binary/combined - search. Each test case run is executed for 60 seconds. + Partial Drop Rate evaluation of single run result, with non-zero packet + loss tolerance (LT) expressed in percentage of packets transmitted. -PDR - Partial Drop Rate evaluation of single run result. Loss acceptance of - dropped packets from number of sent packet is set as variable in frames or - percentage. +.. topic:: NDRDISC -NDR Non Drop Rate evaluation of results. Loss acceptance of dropped packets is set to zero lost packets. -Scaling TAGs +.. topic:: NDRCHK + + Performance tests where TG verifies DUTs' throughput at ref-NDR (reference + Non Drop Rate) with zero packet loss tolerance. + +.. topic:: NDRPDRDISC + + Find performance of DUT based on RFC2544 with linear / binary / combined + search. (Previous LONG tests.) + +Ethernet frame size tags for performance tests +---------------------------------------------- + +.. topic:: 64B + + 64B frames used for test. + +.. topic:: 78B + + 78B frames used for test. + +.. topic:: IMIX + + IMIX frame sequence (28x 64B, 16x 570B, 4x 1518B) used for test. + +.. topic:: 1460B + + 1460B frames used for test. + +.. topic:: 1480B + + 1480B frames used for test. + +.. topic:: 1514B + + 1514B frames used for test. + +.. topic:: 1518B + + 1518B frames used for test. + +.. topic:: 9000B + + 9000B frames used for test. + +Test type tags +-------------- + +.. topic:: BASE + + Baseline test cases, no encapsulation, no feature(s) configured in tests. + +.. topic:: IP4BASE + + IPv4 baseline test cases, no encapsulation, no feature(s) configured in + tests. + +.. topic:: IP6BASE + + IPv6 baseline test cases, no encapsulation, no feature(s) configured in + tests. + +.. topic:: L2XCBASE + + L2XC baseline test cases, no encapsulation, no feature(s) configured in + tests. + +.. topic:: L2BDBASE + + L2BD baseline test cases, no encapsulation, no feature(s) configured in + tests. + +.. topic:: SCALE + + Scale test cases. + +.. topic:: ENCAP + + Test cases where encapsulation is used. Use also encapsulation tag(s). + +.. topic:: FEATURE + + At least one feature is configured in test cases. Use also feature tag(s). + +.. topic:: TLDK + + Functional test cases for TLDK. + +Forwarding mode tags +-------------------- + +.. topic:: L2BDMACSTAT + + VPP L2 bridge-domain, L2 MAC static. + +.. topic:: L2BDMACLRN + + VPP L2 bridge-domain, L2 MAC learning. + +.. topic:: L2XCFWD + + VPP L2 point-to-point cross-connect. + +.. topic:: IP4FWD + + VPP IPv4 routed forwarding. + +.. topic:: IP6FWD + + VPP IPv6 routed forwarding. + +Underlay tags +------------- + +.. topic:: IP4UNRLAY + + IPv4 underlay. + +.. topic:: IP6UNRLAY + + IPv6 underlay. + +.. topic:: MPLSUNRLAY + + MPLS underlay. + +Overlay tags ------------ -FIB_20K - 2x10,000 entries in single fib table +.. topic:: L2OVRLAY -FIB_200K - 2x100,000 entries in single fib table + L2 overlay. -FIB_2M - 2x1,000,000 entries in single fib table +.. topic:: IP4OVRLAY + + IPv4 overlay (IPv4 payload). + +.. topic:: IP6OVRLAY + + IPv6 overlay (IPv6 payload). + +Tagging tags +------------ + +.. topic:: DOT1Q + + All test cases with dot1q. + +.. topic:: DOT1AD + + All test cases with dot1ad. + +Encapsulation tags +------------------ + +.. topic:: ETH + + All test cases with base Ethernet (no encapsulation). + +.. topic:: LISP + + All test cases with LISP. + +.. topic:: LISPGPE + + All test cases with LISP-GPE. + +.. topic:: VXLAN + + All test cases with Vxlan. + +.. topic:: VXLANGPE + + All test cases with VXLAN-GPE. + +.. topic:: GRE + + All test cases with GRE. + +.. topic:: IPSEC + + All test cases with IPSEC. + +Interface tags +-------------- + +.. topic:: PHY + + All test cases which use physical interface(s). + +.. topic:: VHOST + + All test cases which uses VHOST. + +.. topic:: VHOST_256 + + All test cases which uses VHOST with qemu queue size set to 256. + +.. topic:: VHOST_1024 + + All test cases which uses VHOST with qemu queue size set to 1024. + +.. topic:: CFS_OPT + + All test cases which uses VM with optimised scheduler policy. + +.. topic:: TUNTAP + + All test cases which uses TUN and TAP. + +.. topic:: AFPKT + + All test cases which uses AFPKT. + +.. topic:: NETMAP + + All test cases which uses Netmap. + +.. topic:: MEMIF + + All test cases which uses Memif. + +Feature tags +------------ + +.. topic:: IACLDST + + iACL destination. + +.. topic:: COPWHLIST + + COP whitelist. + +.. topic:: SNAT + + SNAT configured and tested. + +.. topic:: ACL + + ACL plugin configured and tested. + +.. topic:: IACL + + ACL plugin configured and tested on input path. + +.. topic:: OACL + + ACL plugin configured and tested on output path. + +.. topic:: ACL_STATELESS + + ACL plugin configured and tested in stateless mode (permit action). + +.. topic:: ACL_STATEFUL + + ACL plugin configured and tested in stateful mode (permit+reflect action). + +.. topic:: ACL1 + + ACL plugin configured and tested with 1 not-hitting ACE. + +.. topic:: ACL10 + + ACL plugin configured and tested with 10 not-hitting ACEs. + +.. topic:: ACL50 + + ACL plugin configured and tested with 50 not-hitting ACEs. + +Encryption tags +--------------- + +.. topic:: IPSECSW + + Crypto in software. + +.. topic:: IPSECHW + + Crypto in hardware. + +.. topic:: IPSECTRAN + + IPSec in transport mode. + +.. topic:: IPSECTUN + + IPSec in tunnel mode. + +Client-workload tags +-------------------- + +.. topic:: VM + + All test cases which use at least one virtual machine. + +.. topic:: LXC + + All test cases which use Linux container. + +.. topic:: APP + + All test cases with specific APP use. + +Multi-threading tags +-------------------- + +.. topic:: STHREAD + + All test cases using single poll mode thread. + +.. topic:: MTHREAD + + All test cases using more then one poll mode driver thread. + +.. topic:: 1NUMA + + All test cases with packet processing on single socket. + +.. topic:: 2NUMA + + All test cases with packet processing on two sockets. + +.. topic:: SMT + + All test cases with symmetric Multi-Threading (HyperThreading) enabled. + +.. topic:: NOSMT + + All test cases with symmetric Multi-Threading (HyperThreading) disabled. + +.. topic:: 1T1C + + 1 worker thread pinned to 1 dedicated physical core. 1 receive queue per + interface. Main thread pinned to core 0. + +.. topic:: 2T2C + + 2 worker threads pinned to 2 dedicated physical cores. 1 receive queue per + interface. Main thread pinned to core 0. + +.. topic:: 4T4C + + 4 worker threads pinned to 4 dedicated physical cores. 2 receive queues per + interface. Main thread pinned to core 0. + +.. topic:: 6T6C + + 6 worker threads pinned to 6 dedicated physical cores. 3 receive queues per + interface. Main thread pinned to core 0. + +.. topic:: 8T8C + + 8 worker threads pinned to 8 dedicated physical cores. 4 receive queues per + interface. Main thread pinned to core 0. + +Honeycomb tags +-------------- + +.. topic:: HC_FUNC + + Honeycomb functional test cases. -LISP TAGs ---------- -LISP_IPv4 - Test Lisp in IPv4 topology +.. topic:: HC_PERSIST -LISP_IPv6 - Test Lisp in IPv6 topology + Honeycomb persistence test cases. -LISP_IPv4oIPv6 - Test IPv4 over IPv6 topology +.. topic:: HC_REST_ONLY -LISP_IPv6oIPv4 - Test IPv6 over IPv4 topology + (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode + using ODL client for Restfconf -> Netconf translation.