From 5d31c77092fd784e21cd50d604be30924ec9a7c5 Mon Sep 17 00:00:00 2001 From: Tibor Frank Date: Fri, 20 Oct 2017 13:35:53 +0200 Subject: [PATCH] Report: CSIT Framework Documentation Change-Id: I2eba1183a5acdd1fb34efab16fc5c89e38dedb14 Signed-off-by: Tibor Frank --- .../csit_design.rst | 388 +++--- .../csit_design_picture.svg | 1298 +++++++++--------- .../csit_tag_description.rst | 0 .../csit_test_naming.rst | 236 ++-- docs/report/csit_framework_documentation/index.rst | 9 + .../pal_func_diagram.svg | 1413 ++++++++++++++++++++ .../csit_framework_documentation/pal_layers.svg | 441 ++++++ .../csit_framework_documentation/pal_lld.rst | 1 + docs/report/index.rst | 1 + docs/report/introduction/index.rst | 3 - resources/tools/presentation/doc/pal_lld.rst | 41 +- 11 files changed, 2860 insertions(+), 971 deletions(-) rename docs/report/{introduction => csit_framework_documentation}/csit_design.rst (93%) rename docs/report/{introduction => csit_framework_documentation}/csit_design_picture.svg (99%) rename docs/report/{introduction => csit_framework_documentation}/csit_tag_description.rst (100%) rename docs/report/{introduction => csit_framework_documentation}/csit_test_naming.rst (97%) create mode 100644 docs/report/csit_framework_documentation/index.rst create mode 100644 docs/report/csit_framework_documentation/pal_func_diagram.svg create mode 100644 docs/report/csit_framework_documentation/pal_layers.svg create mode 100644 docs/report/csit_framework_documentation/pal_lld.rst diff --git a/docs/report/introduction/csit_design.rst b/docs/report/csit_framework_documentation/csit_design.rst similarity index 93% rename from docs/report/introduction/csit_design.rst rename to docs/report/csit_framework_documentation/csit_design.rst index 7798081a59..d8ea2c87d8 100644 --- a/docs/report/introduction/csit_design.rst +++ b/docs/report/csit_framework_documentation/csit_design.rst @@ -1,192 +1,196 @@ -.. _csit-design: - -CSIT Design -=========== - -FD.io CSIT system design needs to meet continuously expanding -requirements of FD.io projects including VPP, related sub-systems (e.g. -plugin applications, DPDK drivers) and FD.io applications (e.g. DPDK -applications), as well as growing number of compute platforms running -those applications. With CSIT project scope and charter including both -FD.io continuous testing AND performance trending/comparisons, those -evolving requirements further amplify the need for CSIT framework -modularity, flexibility and usability. - -Design Hierarchy ----------------- - -CSIT follows a hierarchical system design with SUTs and DUTs at the -bottom level of the hierarchy, presentation level at the top level and a -number of functional layers in-between. The current CSIT system design -including CSIT framework is depicted in the figure below. - -.. raw:: latex - - \begin{figure}[H] - \centering - \includesvg[width=0.90\textwidth]{csit_design_picture} - \label{fig:csit_design_picture} - \end{figure} - -.. figure:: csit_design_picture.svg - :alt: FD.io CSIT system design - :align: center - - *FD.io CSIT system design* - -A brief bottom-up description is provided here: - -#. SUTs, DUTs, TGs - - - SUTs - Systems Under Test; - - DUTs - Devices Under Test; - - TGs - Traffic Generators; - -#. Level-1 libraries - Robot and Python - - - Lowest level CSIT libraries abstracting underlying test environment, SUT, - DUT and TG specifics; - - Used commonly across multiple L2 KWs; - - Performance and functional tests: - - - L1 KWs (KeyWords) are implemented as RF libraries and Python - libraries; - - - Performance TG L1 KWs: - - - All L1 KWs are implemented as Python libraries: - - - Support for TRex only today; - - CSIT IXIA drivers in progress; - - - Performance data plane traffic profiles: - - - TG-specific stream profiles provide full control of: - - - Packet definition – layers, MACs, IPs, ports, combinations thereof - e.g. IPs and UDP ports; - - Stream definitions - different streams can run together, delayed, - one after each other; - - Stream profiles are independent of CSIT framework and can be used - in any T-rex setup, can be sent anywhere to repeat tests with - exactly the same setup; - - Easily extensible – one can create a new stream profile that meets - tests requirements; - - Same stream profile can be used for different tests with the same - traffic needs; - - - Functional data plane traffic scripts: - - - Scapy specific traffic scripts; - -#. Level-2 libraries - Robot resource files: - - - Higher level CSIT libraries abstracting required functions for executing - tests; - - L2 KWs are classified into the following functional categories: - - - Configuration, test, verification, state report; - - Suite setup, suite teardown; - - Test setup, test teardown; - -#. Tests - Robot: - - - Test suites with test cases; - - Functional tests using VIRL environment: - - - VPP; - - Honeycomb; - - NSH_SFC; - - - Performance tests using physical testbed environment: - - - VPP; - - DPDK-Testpmd; - - DPDK-L3Fwd; - - - Tools: - - - Documentation generator; - - Report generator; - - Testbed environment setup ansible playbooks; - - Operational debugging scripts; - -Test Lifecycle Abstraction --------------------------- - -A well coded test must follow a disciplined abstraction of the test -lifecycles that includes setup, configuration, test and verification. In -addition to improve test execution efficiency, the commmon aspects of -test setup and configuration shared across multiple test cases should be -done only once. Translating these high-level guidelines into the Robot -Framework one arrives to definition of a well coded RF tests for FD.io -CSIT. Anatomy of Good Tests for CSIT: - -#. Suite Setup - Suite startup Configuration common to all Test Cases in suite: - uses Configuration KWs, Verification KWs, StateReport KWs; -#. Test Setup - Test startup Configuration common to multiple Test Cases: uses - Configuration KWs, StateReport KWs; -#. Test Case - uses L2 KWs with RF Gherkin style: - - - prefixed with {Given} - Verification of Test setup, reading state: uses - Configuration KWs, Verification KWs, StateReport KWs; - - prefixed with {When} - Test execution: Configuration KWs, Test KWs; - - prefixed with {Then} - Verification of Test execution, reading state: uses - Verification KWs, StateReport KWs; - -#. Test Teardown - post Test teardown with Configuration cleanup and - Verification common to multiple Test Cases - uses: Configuration KWs, - Verification KWs, StateReport KWs; -#. Suite Teardown - Suite post-test Configuration cleanup: uses Configuration - KWs, Verification KWs, StateReport KWs; - -RF Keywords Functional Classification -------------------------------------- - -CSIT RF KWs are classified into the functional categories matching the test -lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped -into the following functional categories: - -#. Configuration; -#. Test; -#. Verification; -#. StateReport; -#. SuiteSetup; -#. TestSetup; -#. SuiteTeardown; -#. TestTeardown; - -RF Keywords Naming Guidelines ------------------------------ - -Readability counts: "..code is read much more often than it is written." -Hence following a good and consistent grammar practice is important when -writing :abbr:`RF (Robot Framework)` KeyWords and Tests. All CSIT test cases -are coded using Gherkin style and include only L2 KWs references. L2 KWs are -coded using simple style and include L2 KWs, L1 KWs, and L1 python references. -To improve readability, the proposal is to use the same grammar for both -:abbr:`RF (Robot Framework)` KW styles, and to formalize the grammar of English -sentences used for naming the :abbr:`RF (Robot Framework)` KWs. :abbr:`RF (Robot -Framework)` KWs names are short sentences expressing functional description of -the command. They must follow English sentence grammar in one of the following -forms: - -#. **Imperative** - verb-object(s): *"Do something"*, verb in base form. -#. **Declarative** - subject–verb–object(s): *"Subject does something"*, verb in - a third-person singular present tense form. -#. **Affirmative** - modal_verb-verb-object(s): *"Subject should be something"*, - *"Object should exist"*, verb in base form. -#. **Negative** - modal_verb-Not-verb-object(s): *"Subject should not be - something"*, *"Object should not exist"*, verb in base form. - -Passive form MUST NOT be used. However a usage of past participle as an -adjective is okay. See usage examples provided in the Coding guidelines -section below. Following sections list applicability of the above -grammar forms to different :abbr:`RF (Robot Framework)` KW categories. Usage -examples are provided, both good and bad. - -Coding guidelines ------------------ - -Coding guidelines can be found on `Design optimizations wiki page -`_. +.. _csit-design: + +CSIT Design +=========== + +FD.io CSIT system design needs to meet continuously expanding +requirements of FD.io projects including VPP, related sub-systems (e.g. +plugin applications, DPDK drivers) and FD.io applications (e.g. DPDK +applications), as well as growing number of compute platforms running +those applications. With CSIT project scope and charter including both +FD.io continuous testing AND performance trending/comparisons, those +evolving requirements further amplify the need for CSIT framework +modularity, flexibility and usability. + +Design Hierarchy +---------------- + +CSIT follows a hierarchical system design with SUTs and DUTs at the +bottom level of the hierarchy, presentation level at the top level and a +number of functional layers in-between. The current CSIT system design +including CSIT framework is depicted in the figure below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \includesvg[width=0.90\textwidth]{../_tmp/src/csit_framework_documentation/csit_design_picture} + \label{fig:csit_design_picture} + \end{figure} + +.. only:: html + + .. figure:: csit_design_picture.svg + :alt: FD.io CSIT system design + :align: center + + *FD.io CSIT system design* + +A brief bottom-up description is provided here: + +#. SUTs, DUTs, TGs + + - SUTs - Systems Under Test; + - DUTs - Devices Under Test; + - TGs - Traffic Generators; + +#. Level-1 libraries - Robot and Python + + - Lowest level CSIT libraries abstracting underlying test environment, SUT, + DUT and TG specifics; + - Used commonly across multiple L2 KWs; + - Performance and functional tests: + + - L1 KWs (KeyWords) are implemented as RF libraries and Python + libraries; + + - Performance TG L1 KWs: + + - All L1 KWs are implemented as Python libraries: + + - Support for TRex only today; + - CSIT IXIA drivers in progress; + + - Performance data plane traffic profiles: + + - TG-specific stream profiles provide full control of: + + - Packet definition – layers, MACs, IPs, ports, combinations thereof + e.g. IPs and UDP ports; + - Stream definitions - different streams can run together, delayed, + one after each other; + - Stream profiles are independent of CSIT framework and can be used + in any T-rex setup, can be sent anywhere to repeat tests with + exactly the same setup; + - Easily extensible – one can create a new stream profile that meets + tests requirements; + - Same stream profile can be used for different tests with the same + traffic needs; + + - Functional data plane traffic scripts: + + - Scapy specific traffic scripts; + +#. Level-2 libraries - Robot resource files: + + - Higher level CSIT libraries abstracting required functions for executing + tests; + - L2 KWs are classified into the following functional categories: + + - Configuration, test, verification, state report; + - Suite setup, suite teardown; + - Test setup, test teardown; + +#. Tests - Robot: + + - Test suites with test cases; + - Functional tests using VIRL environment: + + - VPP; + - Honeycomb; + - NSH_SFC; + + - Performance tests using physical testbed environment: + + - VPP; + - DPDK-Testpmd; + - DPDK-L3Fwd; + + - Tools: + + - Documentation generator; + - Report generator; + - Testbed environment setup ansible playbooks; + - Operational debugging scripts; + +Test Lifecycle Abstraction +-------------------------- + +A well coded test must follow a disciplined abstraction of the test +lifecycles that includes setup, configuration, test and verification. In +addition to improve test execution efficiency, the commmon aspects of +test setup and configuration shared across multiple test cases should be +done only once. Translating these high-level guidelines into the Robot +Framework one arrives to definition of a well coded RF tests for FD.io +CSIT. Anatomy of Good Tests for CSIT: + +#. Suite Setup - Suite startup Configuration common to all Test Cases in suite: + uses Configuration KWs, Verification KWs, StateReport KWs; +#. Test Setup - Test startup Configuration common to multiple Test Cases: uses + Configuration KWs, StateReport KWs; +#. Test Case - uses L2 KWs with RF Gherkin style: + + - prefixed with {Given} - Verification of Test setup, reading state: uses + Configuration KWs, Verification KWs, StateReport KWs; + - prefixed with {When} - Test execution: Configuration KWs, Test KWs; + - prefixed with {Then} - Verification of Test execution, reading state: uses + Verification KWs, StateReport KWs; + +#. Test Teardown - post Test teardown with Configuration cleanup and + Verification common to multiple Test Cases - uses: Configuration KWs, + Verification KWs, StateReport KWs; +#. Suite Teardown - Suite post-test Configuration cleanup: uses Configuration + KWs, Verification KWs, StateReport KWs; + +RF Keywords Functional Classification +------------------------------------- + +CSIT RF KWs are classified into the functional categories matching the test +lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped +into the following functional categories: + +#. Configuration; +#. Test; +#. Verification; +#. StateReport; +#. SuiteSetup; +#. TestSetup; +#. SuiteTeardown; +#. TestTeardown; + +RF Keywords Naming Guidelines +----------------------------- + +Readability counts: "..code is read much more often than it is written." +Hence following a good and consistent grammar practice is important when +writing :abbr:`RF (Robot Framework)` KeyWords and Tests. All CSIT test cases +are coded using Gherkin style and include only L2 KWs references. L2 KWs are +coded using simple style and include L2 KWs, L1 KWs, and L1 python references. +To improve readability, the proposal is to use the same grammar for both +:abbr:`RF (Robot Framework)` KW styles, and to formalize the grammar of English +sentences used for naming the :abbr:`RF (Robot Framework)` KWs. :abbr:`RF (Robot +Framework)` KWs names are short sentences expressing functional description of +the command. They must follow English sentence grammar in one of the following +forms: + +#. **Imperative** - verb-object(s): *"Do something"*, verb in base form. +#. **Declarative** - subject–verb–object(s): *"Subject does something"*, verb in + a third-person singular present tense form. +#. **Affirmative** - modal_verb-verb-object(s): *"Subject should be something"*, + *"Object should exist"*, verb in base form. +#. **Negative** - modal_verb-Not-verb-object(s): *"Subject should not be + something"*, *"Object should not exist"*, verb in base form. + +Passive form MUST NOT be used. However a usage of past participle as an +adjective is okay. See usage examples provided in the Coding guidelines +section below. Following sections list applicability of the above +grammar forms to different :abbr:`RF (Robot Framework)` KW categories. Usage +examples are provided, both good and bad. + +Coding guidelines +----------------- + +Coding guidelines can be found on `Design optimizations wiki page +`_. diff --git a/docs/report/introduction/csit_design_picture.svg b/docs/report/csit_framework_documentation/csit_design_picture.svg similarity index 99% rename from docs/report/introduction/csit_design_picture.svg rename to docs/report/csit_framework_documentation/csit_design_picture.svg index e48dc96664..154509a94c 100644 --- a/docs/report/introduction/csit_design_picture.svg +++ b/docs/report/csit_framework_documentation/csit_design_picture.svg @@ -1,649 +1,649 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/report/introduction/csit_tag_description.rst b/docs/report/csit_framework_documentation/csit_tag_description.rst similarity index 100% rename from docs/report/introduction/csit_tag_description.rst rename to docs/report/csit_framework_documentation/csit_tag_description.rst diff --git a/docs/report/introduction/csit_test_naming.rst b/docs/report/csit_framework_documentation/csit_test_naming.rst similarity index 97% rename from docs/report/introduction/csit_test_naming.rst rename to docs/report/csit_framework_documentation/csit_test_naming.rst index c88ec493a3..b108235cc3 100644 --- a/docs/report/introduction/csit_test_naming.rst +++ b/docs/report/csit_framework_documentation/csit_test_naming.rst @@ -1,118 +1,118 @@ -CSIT Test Naming -================ - -Background ----------- - -CSIT |release| follows a common structured naming convention for all -performance and system functional tests, introduced in CSIT |release-1|. - -The naming should be intuitive for majority of the tests. Complete -description of CSIT test naming convention is provided on -`CSIT test naming wiki page `_. -Below few illustrative examples of the naming usage for test suites across CSIT -performance, functional and Honeycomb management test areas. - -Naming Convention ------------------ - -The CSIT approach is to use tree naming convention and to encode following -testing information into test suite and test case names: - -#. packet network port configuration - - * port type, physical or virtual; - * number of ports; - * NIC model, if applicable; - * port-NIC locality, if applicable; - -#. packet encapsulations; - -#. VPP packet processing - - * packet forwarding mode; - * packet processing function(s); - -#. packet forwarding path - - * if present, network functions (processes, containers, VMs) and their - topology within the computer; - -#. main measured variable, type of test. - -Proposed convention is to encode ports and NICs on the left (underlay), -followed by outer-most frame header, then other stacked headers up to the -header processed by vSwitch-VPP, then VPP forwarding function, then encap on -vhost interface, number of vhost interfaces, number of VMs. If chained VMs -present, they get added on the right. Test topology is expected to be -symmetric, in other words packets enter and leave SUT through ports specified -on the left of the test name. Here some examples to illustrate the convention -followed by the complete legend, and tables mapping the new test filenames to -old ones. - -Naming Examples ---------------- - -CSIT test suite naming examples (filename.robot) for common tested VPP -topologies: - -1. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - - * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* - * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel - x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with - MAC learning, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on - Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching - with MAC learning, NDR throughput discovery. - * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520 - NIC, IPv4 baseline routed forwarding, NDR throughput discovery. - * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel - x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery. - * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on - Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control - Lists baseline matching on destination, NDR throughput discovery. - * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco - vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. - * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline - routed forwarding, functional tests. - -2. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, - P2V2P, NIC2VMchain2NIC, P2V2V2P** - - * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- - VirtPortConfig-VMconfig-TestType* - * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports - of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching - to/from two vhost interfaces and one VM, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from two vhost interfaces and one VM, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from four vhost interfaces and two VMs, NDR throughput - discovery. - * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of - Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost - interfaces and one VM, functional tests. - -3. **API CRUD tests - Create (Write), Read (Retrieve), Update (Modify), Delete - (Destroy) operations for configuration and operational data** - - * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...- - ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType* - * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls, - functional tests. - * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with - Honeycomb API and VAT API calls, functional tests. - * *mgmt-oper-int-apihcnc-func* => reading status and operational data of - interface with Honeycomb NetConf API calls, functional tests. - * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with - Honeycomb NetConf API calls, functional tests. - * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and - sub-interface events with Honeycomb NetConf Notifications, functional tests. - -For complete description of CSIT test naming convention please refer to `CSIT -test naming wiki page `_. +CSIT Test Naming +================ + +Background +---------- + +CSIT |release| follows a common structured naming convention for all +performance and system functional tests, introduced in CSIT |release-1|. + +The naming should be intuitive for majority of the tests. Complete +description of CSIT test naming convention is provided on +`CSIT test naming wiki page `_. +Below few illustrative examples of the naming usage for test suites across CSIT +performance, functional and Honeycomb management test areas. + +Naming Convention +----------------- + +The CSIT approach is to use tree naming convention and to encode following +testing information into test suite and test case names: + +#. packet network port configuration + + * port type, physical or virtual; + * number of ports; + * NIC model, if applicable; + * port-NIC locality, if applicable; + +#. packet encapsulations; + +#. VPP packet processing + + * packet forwarding mode; + * packet processing function(s); + +#. packet forwarding path + + * if present, network functions (processes, containers, VMs) and their + topology within the computer; + +#. main measured variable, type of test. + +Proposed convention is to encode ports and NICs on the left (underlay), +followed by outer-most frame header, then other stacked headers up to the +header processed by vSwitch-VPP, then VPP forwarding function, then encap on +vhost interface, number of vhost interfaces, number of VMs. If chained VMs +present, they get added on the right. Test topology is expected to be +symmetric, in other words packets enter and leave SUT through ports specified +on the left of the test name. Here some examples to illustrate the convention +followed by the complete legend, and tables mapping the new test filenames to +old ones. + +Naming Examples +--------------- + +CSIT test suite naming examples (filename.robot) for common tested VPP +topologies: + +1. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** + + * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* + * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with + MAC learning, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching + with MAC learning, NDR throughput discovery. + * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520 + NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery. + * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control + Lists baseline matching on destination, NDR throughput discovery. + * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco + vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline + routed forwarding, functional tests. + +2. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, + P2V2P, NIC2VMchain2NIC, P2V2V2P** + + * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- + VirtPortConfig-VMconfig-TestType* + * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports + of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching + to/from two vhost interfaces and one VM, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from four vhost interfaces and two VMs, NDR throughput + discovery. + * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of + Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost + interfaces and one VM, functional tests. + +3. **API CRUD tests - Create (Write), Read (Retrieve), Update (Modify), Delete + (Destroy) operations for configuration and operational data** + + * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...- + ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType* + * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls, + functional tests. + * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with + Honeycomb API and VAT API calls, functional tests. + * *mgmt-oper-int-apihcnc-func* => reading status and operational data of + interface with Honeycomb NetConf API calls, functional tests. + * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with + Honeycomb NetConf API calls, functional tests. + * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and + sub-interface events with Honeycomb NetConf Notifications, functional tests. + +For complete description of CSIT test naming convention please refer to `CSIT +test naming wiki page `_. diff --git a/docs/report/csit_framework_documentation/index.rst b/docs/report/csit_framework_documentation/index.rst new file mode 100644 index 0000000000..0b3228dc51 --- /dev/null +++ b/docs/report/csit_framework_documentation/index.rst @@ -0,0 +1,9 @@ +CSIT Framework Documentation +============================ + +.. toctree:: + + csit_design + csit_test_naming + pal_lld + csit_tag_description diff --git a/docs/report/csit_framework_documentation/pal_func_diagram.svg b/docs/report/csit_framework_documentation/pal_func_diagram.svg new file mode 100644 index 0000000000..14f59605f9 --- /dev/null +++ b/docs/report/csit_framework_documentation/pal_func_diagram.svg @@ -0,0 +1,1413 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Specification.YAML + + + + + + + + Data to process.xml + + + + + + + + Static content.rst + + + + + + + + + + read_specification + + + + + + + + + + read_data + + + + + + + + Specification + + + + + + + + Input data + + + + + + + + + + filter_data + + + + + + + + + + filter_data + + + + + + + + + + generate_files + + + + + + + + Tables + + + + + + + + Plots + + + + + + + + Files + + + + + + + + + + generate_report + + + + + + + + Report + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + generate_tables + + + + + + + + + + generate_plots + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + sL1 - Data + + + + + + sL2 - Data processing + + + + + + sL3 - Data presentation + + + + + + sL4 - Report generation + + + + + + + \ No newline at end of file diff --git a/docs/report/csit_framework_documentation/pal_layers.svg b/docs/report/csit_framework_documentation/pal_layers.svg new file mode 100644 index 0000000000..dfb05d3106 --- /dev/null +++ b/docs/report/csit_framework_documentation/pal_layers.svg @@ -0,0 +1,441 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + .YAMLSpecification (CSIT gerrit) + + + + + + Data + + + + + + + + .RSTStatic content (CSIT gerrit) + + + + + + + + .ZIP (.XML)Data to process (Jenkins) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + pandasData model in JSONSpecification, Input data (Pandas.Series) + + + + + + Data processing + + + + + + Data presentation + + + + + + + + Plotsplot.ly → .html + + + + + + + + Files.RST + + + + + + + + TablesPandas → .csv + + + + + + Report generation + + + + + + + + Sphinx.html / .pdf (then stored in nexus) + + + + + + Jenkins plots + + + + + + + + Jenkins plotplugin.html + + + + + + sL1 + + + + + + sL2 + + + + + + sL3 + + + + + + sL4 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Read files + + + + + + Python calls + + + + + + Python calls + + + + + + Python calls + + + + + + + \ No newline at end of file diff --git a/docs/report/csit_framework_documentation/pal_lld.rst b/docs/report/csit_framework_documentation/pal_lld.rst new file mode 100644 index 0000000000..ddd4de0994 --- /dev/null +++ b/docs/report/csit_framework_documentation/pal_lld.rst @@ -0,0 +1 @@ +.. include:: ../../../../../../resources/tools/presentation/doc/pal_lld.rst diff --git a/docs/report/index.rst b/docs/report/index.rst index 8b4bb7955c..b36c0e50cd 100644 --- a/docs/report/index.rst +++ b/docs/report/index.rst @@ -15,3 +15,4 @@ CSIT 17.07 detailed_test_results/index test_configuration/index test_operational_data/index + csit_framework_documentation/index \ No newline at end of file diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index eabbf7b7ae..a5c7f5acff 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -5,6 +5,3 @@ Introduction overview general_notes - csit_design - csit_test_naming - csit_tag_description diff --git a/resources/tools/presentation/doc/pal_lld.rst b/resources/tools/presentation/doc/pal_lld.rst index 3a2137366d..9158b889b9 100644 --- a/resources/tools/presentation/doc/pal_lld.rst +++ b/resources/tools/presentation/doc/pal_lld.rst @@ -1,6 +1,5 @@ -=================================================== -Presentation and Analytics Layer - Low Level Design -=================================================== +Presentation and Analytics Layer +================================ Overview -------- @@ -38,9 +37,21 @@ sub-layers, bottom up: - formats: html, pdf - versions: minimal, full (TODO: define the names and scope of versions) -.. figure:: pal_layers.svg - :alt: PAL Layers - :align: center +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \includesvg[width=0.90\textwidth]{../_tmp/src/csit_framework_documentation/pal_layers} + \label{fig:pal_layers} + \end{figure} + +.. only:: html + + .. figure:: pal_layers.svg + :alt: PAL Layers + :align: center Data ---- @@ -1212,9 +1223,21 @@ List of modules, classes, methods and functions PAL functional diagram `````````````````````` -.. figure:: pal_func_diagram.svg - :alt: PAL functional diagram - :align: center +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \includesvg[width=0.90\textwidth]{../_tmp/src/csit_framework_documentation/pal_func_diagram} + \label{fig:pal_func_diagram} + \end{figure} + +.. only:: html + + .. figure:: pal_func_diagram.svg + :alt: PAL functional diagram + :align: center How to add an element -- 2.16.6