X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fintroduction%2Foverview.rst;h=4aac352d0c1c46c33454caa87f7ccb93aaed8f14;hp=dc86f8600ed5bc67fad89d763e94fd2b1cfe5c8b;hb=4457640600a2b875da17e97dba2dcab8a411eb95;hpb=35e5a327a66c05a3356d214dcfdece59ca725998 diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index dc86f8600e..4aac352d0c 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -1,49 +1,82 @@ -Overview -======== - -This is the **F**\ast **D**\ata **I**/**O** Project (FD.io) **C**\ontinuous -**S**\ystem **I**\ntegration and **T**\esting (CSIT) project report for CSIT -|release| system testing of VPP-17.01 release. Initial sections of the report -describe FD.io functional and performance tests and their continuous execution -achieved in CSIT |release|. This is followed by high-level overview of FD.io -test environments running in Linux Foundation (LF) FD.io Continuous -Performance Labs), CSIT |release| release notes and result highlights. -Subsequent sections list the functional and performance detailed test results -generated by LF FD.io Jenkins system executing CSIT jobs running tests -against the VPP-17.01 release and associated sub-systems (HoneyComb, DPDK). - -The exact structure of the report is as follows: - -#. ``Introduction`` - Includes: ``Overview`` - this section; - ``CSIT Test Naming`` - describing CSIT naming convention for test suites and - test cases, important to understand to correctly interpret the reported - results; ``General Remarks`` - additional notes related to this report. -#. ``VPP Performance Tests HW`` - Summary of VPP performance tests executed in - physical Hardware testbed. Includes: ``Overview`` - containing topology - description and list of executed tests; ``Performance Tests Naming`` - lists - CSIT naming specifics for performance tests; ``CSIT Release Notes`` - with - changes in CSIT |release| including major test additions, performance - improvements, changes in testing environment and methodology and known - issues; ``Performance Measurements Methodology`` - details testing - methodology followed for multi-thread multi-core, packet throughput and - latency, and KVM VM vhost tests; ``Report Addendum Targets`` - lists content - to be added to this report as part of CSIT |release|. -#. ``Testpmd Performance Tests HW`` - ... -#. ``VPP Functional Tests VIRL`` - ... -#. ``VPP Functional Tests VPPtest`` - ... -#. ``HoneyComb Functional Tests VIRL`` - .. -#. ``VPP Performance Results HW`` - ... -#. ``Testpmd Performance Results HW`` - ... -#. ``VPP Functional Results VIRL`` - ... -#. ``VPP Functional Results VPPtest`` - ... -#. ``HoneyComb Functional Results VIRL`` - ... - -FD.io CSIT system is developed using two main coding platforms: Robot -Framework (RF) and Python. CSIT |release| source code for the executed test -suites is available in CSIT branch |release| in the directory -"./tests/". A local copy of CSIT source code can be -obtained by cloning CSIT git repository - "git clone -https://gerrit.fd.io/r/csit". The CSIT testing virtual environment can be run -on a local computer workstation (laptop, server) using Vagrant by following -the instructions in `CSIT tutorials -`_. +Overview +======== + +This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous +**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT +|release| system testing of |vpp-release|. + +The report describes CSIT functional and performance tests and their +continuous execution delivered in CSIT |release|. A high-level overview is +provided for each CSIT test environment running in Linux Foundation (LF) FD.io +Continuous Performance Labs. This is followed by summary of all executed tests +against the |vpp-release| and associated FD.io projects and sub-systems +(HoneyComb, DPDK, NSH_SFC), CSIT |release| release notes, result highlights and +known issues discovered in CSIT. More detailed description of each environment, +pointers to CSIT test code documentation and detailed test resuls with links to +the source data files are also provided. + +CSIT |release| report contains following main sections and sub-sections: + +#. **Introduction** - general introduction to CSIT project; *Overview* - + this section; *CSIT Test Naming* - CSIT general naming convention for test + suites and test cases, important to recognize the high-level test content + and interpret reported results; *General Notes* - general notes related to + this report. + +#. **VPP Performance Tests** - VPP performance tests executed in physical + FD.io testbeds; *Overview* - tested topologies, test coverage and naming + specifics, methodology for multi-core, packet throughput and latency, and + KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added + tests, performance changes, environment or methodology changes, known CSIT + issues; *Packet Throughput Graphs* and *Packet Latency + Graphs* - plotted NDR, PDR throughput and latency results from multiple + test job executions; *Test Environment* - environment description; + *Documentation* - CSIT source code documentation for VPP performance tests. + +#. **DPDK Performance Tests** - DPDK performance tests executed in + physical FD.io testbeds; *Overview* - tested topologies, test coverage; + *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues; + *Packet Throughput Graphs* and *Packet Latency Graphs* + - plotted NDR, PDR throughput and latency results from multiple test job + executions; *Test Environment* - environment description; *Documentation* - + CSIT source code documentation for DPDK performance tests. + +#. **VPP Functional Tests** - VPP functional tests executed in virtual + FD.io testbeds; *Overview* - tested virtual topologies, test coverage and + naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added + tests, environment or methodology changes, known CSIT issues, tests to be + added; *Test Environment* - environment description ; *Documentation* - + source code documentation for VPP functional tests. + +#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for Honeycomb functional tests. + +#. **VPP Unit Tests** - refers to VPP functional unit tests executed as + part of vpp make test verify option within the FD.io VPP project; listed in + this report to give a more complete view about executed VPP functional tests; + *Overview* - short overview of unit test framework and executed tests; + *Documentation* - source code documentation of VPP unit tests. + +#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for NSH_SFC functional tests. + +#. **Detailed Test Results** - auto-generated results from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Results*, *DPDK Performance Results*, *VPP Functional + Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*. + +#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Test Configs*, *VPP Functional Test Configs*. + +#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Operational Data*.