From: Tibor Frank Date: Thu, 19 Apr 2018 06:41:24 +0000 (+0200) Subject: CSIT-1052: Remove HC Performance from 1804 report X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=commitdiff_plain;h=98f7c4743d36b03cbe9952ff7e3e60e39f851e2d CSIT-1052: Remove HC Performance from 1804 report Change-Id: I924028fd6d56fff1859bc1fd27d7a5273d54435f Signed-off-by: Tibor Frank --- diff --git a/docs/report/honeycomb_performance_tests/csit_release_notes.rst b/docs/report/honeycomb_performance_tests/csit_release_notes.rst deleted file mode 100644 index d79907d4ca..0000000000 --- a/docs/report/honeycomb_performance_tests/csit_release_notes.rst +++ /dev/null @@ -1,20 +0,0 @@ -CSIT Release Notes -================== - -Changes in CSIT |release| -------------------------- - -#. No changes since previous release. - -Known Issues ------------- - -Here is the list of known issues in CSIT |release| for Honeycomb performance -tests in VIRL: - -+---+--------------------------------------------+------------+----------------------------------------------------------------------------+ -| # | Issue | Jira ID | Description | -+---+--------------------------------------------+------------+----------------------------------------------------------------------------+ -| 1 | Intermittent failures in Honeycomb startup | HC2VPP-199 | During test setup Honeycomb sometimes fails to start. Does not appear to | -| | | | affect measured results, only fails test execution ocassionally. | -+---+--------------------------------------------+------------+----------------------------------------------------------------------------+ diff --git a/docs/report/honeycomb_performance_tests/documentation.rst b/docs/report/honeycomb_performance_tests/documentation.rst deleted file mode 100644 index 6b15bde6ee..0000000000 --- a/docs/report/honeycomb_performance_tests/documentation.rst +++ /dev/null @@ -1,5 +0,0 @@ -Documentation -============= - -`CSIT Honeycomb Performance Tests Documentation`_ contains detailed -functional description and input parameters for each test case. diff --git a/docs/report/honeycomb_performance_tests/index.rst b/docs/report/honeycomb_performance_tests/index.rst deleted file mode 100644 index 3177494395..0000000000 --- a/docs/report/honeycomb_performance_tests/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -Honeycomb Performance Tests -=========================== - -.. toctree:: - - overview - csit_release_notes - test_environment - documentation - test_result_data - diff --git a/docs/report/honeycomb_performance_tests/overview.rst b/docs/report/honeycomb_performance_tests/overview.rst deleted file mode 100644 index ee9788ed26..0000000000 --- a/docs/report/honeycomb_performance_tests/overview.rst +++ /dev/null @@ -1,122 +0,0 @@ -Overview -======== - -Tested Physical Topologies --------------------------- - -CSIT VPP performance tests are executed on physical baremetal servers hosted by -LF FD.io project. Testbed physical topology is shown in the figure below.:: - - +------------------------+ +------------------------+ - | | | | - | +------------------+ | | +------------------+ | - | | | | | | | | - | | <-----------------> | | - | | DUT1 | | | | DUT2 | | - | +--^---------------+ | | +---------------^--+ | - | | | | | | - | | SUT1 | | SUT2 | | - +------------------------+ +------------------^-----+ - | | - | | - | +-----------+ | - | | | | - +------------------> TG <------------------+ - | | - +-----------+ - -SUT1 runs VPP SW application in Linux user-mode as a -Device Under Test (DUT), and a python script to generate traffic. SUT2 and TG -are unused. -sical connectivity between SUTs and to TG is provided using -different NIC model. Currently installed NIC models include: - -Performance tests involve sending Netconf requests over localhost to the -Honeycomb listener port, and measuring response time. - -Note that reported performance results are specific to the SUTs tested. -Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with -other CPUs are likely to yield different results. - -For detailed LF FD.io test bed specification and physical topology please refer -to `LF FDio CSIT testbed wiki page -`_. - -Performance Tests Coverage --------------------------- - -As of right now, there is only a single Honeycomb performance test. Measuring -response time for a simple read operation, performed synchronously and using -single (not batch) requests. - -Currently the tests do not trigger automatically, but can be run on-demand from -the hc2vpp project. - -Performance Tests Naming ------------------------- - -CSIT |release| follows a common structured naming convention for all -performance and system functional tests, introduced in CSIT |release-1|. - -The naming should be intuitive for majority of the tests. Complete -description of CSIT test naming convention is provided on `CSIT test naming wiki -`_. - -Here few illustrative examples of the new naming usage for performance test -suites: - -#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - - - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* - - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on - Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching - with MAC learning, NDR throughput discovery. - - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE - on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline - switching with MAC learning, NDR throughput discovery. - - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel - x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. - - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on - Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput - discovery. - -#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, - P2V2P, NIC2VMchain2NIC, P2V2V2P** - - - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- - VirtPortConfig-VMconfig-TestType* - - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports - of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain - switching to/from two vhost interfaces and one VM, NDR throughput - discovery. - - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from two vhost interfaces and one VM, NDR throughput - discovery. - - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from four vhost interfaces and two VMs, NDR throughput - discovery. - -Methodology: Multi-Core ------------------------ - -**Multi-core Test** - CSIT |release| multi-core tests are executed in the -following thread and core configurations: - -#. 1t - 1 Honeycomb Netconf thread on 1 CPU physical core. -#. 8t - 8 Honeycomb Netconf thread on 8 CPU physical core. -#. 16t - 16 Honeycomb Netconf thread on 16 CPU physical core. - -Traffic generator also uses multiple threads/cores, to simulate multiple -Netconf clients accessing the Honeycomb server. - -Methodology: Performance measurement ------------------------------------- - -The following values are measured and reported in tests: - -- Average request rate. Averaged over the entire test duration, over all client - threads. Negative replies (if any) are not counted and are reported separately. diff --git a/docs/report/honeycomb_performance_tests/test_environment.rst b/docs/report/honeycomb_performance_tests/test_environment.rst deleted file mode 100644 index db187b2c1b..0000000000 --- a/docs/report/honeycomb_performance_tests/test_environment.rst +++ /dev/null @@ -1,22 +0,0 @@ -Test Environment -================ - -To execute performance tests, there are three identical testbeds, each testbed -consists of two DUTs and one TG. - -Server HW Configuration ------------------------ - -See `Performance HW Configuration <../vpp_performance_tests/test_environment.html>`_ - -Additionally, configuration for the Honeycomb client: - - -**Honeycomb Startup Command** - -Use the server mode JIT compiler, increase the default memory size, -metaspace size, and enable NUMA optimizations for the JVM. - -:: - - $ java -server -Xms128m -Xmx512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m -XX:+UseNUMA -XX:+UseParallelGC diff --git a/docs/report/honeycomb_performance_tests/test_result_data.rst b/docs/report/honeycomb_performance_tests/test_result_data.rst deleted file mode 100644 index 9e791d6e88..0000000000 --- a/docs/report/honeycomb_performance_tests/test_result_data.rst +++ /dev/null @@ -1,19 +0,0 @@ -Test Result Data -================ - -This section includes summary of Netconf read operation performance. -Performance is reported for Honeycomb running in multiple configurations of -netconf thread(s) and their physical CPU core(s) placement, and for different -read operation targets. - -.. note:: - - Test results have been generated by - `FD.io test executor honeycomb performance jobs`_ with Robot Framework - result files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_. - -Honeycomb + Netconf -=================== - -No new data has been generated in the course of this release. Results from -the last successful run are available in `17.07 release report `_. \ No newline at end of file diff --git a/docs/report/index.rst b/docs/report/index.rst index bbcd5cbab8..158259f1af 100644 --- a/docs/report/index.rst +++ b/docs/report/index.rst @@ -1,4 +1,4 @@ -CSIT 18.01 +CSIT 18.04 ========== .. toctree:: @@ -7,7 +7,6 @@ CSIT 18.01 introduction/index vpp_performance_tests/index dpdk_performance_tests/index - honeycomb_performance_tests/index vpp_functional_tests/index honeycomb_functional_tests/index vpp_unit_tests/index diff --git a/docs/report/introduction/general_notes.rst b/docs/report/introduction/general_notes.rst index 994a53e6d0..b51ed159c5 100644 --- a/docs/report/introduction/general_notes.rst +++ b/docs/report/introduction/general_notes.rst @@ -49,11 +49,6 @@ is listed separately, as follows: Testpmd/L3FWD application runs in host user-mode. TRex is used as a traffic generator. -#. **Honeycomb Performance** - Honeycomb performance tests are executed in - physical FD.io testbeds, focusing on the performance of Honeycomb management - and programming functionality of VPP. Tests cover a range of CRUD operations - executed against VPP. - #. **VPP Functional** - VPP functional tests are executed in virtual FD.io testbeds focusing on VPP packet processing functionality, including network data plane and in -line control plane. Tests cover vNIC-to-vNIC diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index 22bc29f071..536d5d3cf1 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -44,13 +44,6 @@ CSIT |release| report contains following main sections and sub-sections: executions; *Test Environment* - environment description; *Documentation* - CSIT source code documentation for DPDK performance tests. -#. **Honeycomb Performance Tests** - Honeycomb performance tests executed in - physical FD.io testbeds; *Overview* - tested topologies, test coverage - and naming specifics; *CSIT Release Notes* - changes in CSIT |release|, - added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description; *Documentation* - source - code documentation for Honeycomb performance tests. - #. **VPP Functional Tests** - VPP functional tests executed in virtual FD.io testbeds; *Overview* - tested virtual topologies, test coverage and naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added diff --git a/resources/tools/presentation/conf.py b/resources/tools/presentation/conf.py index c2e2ea037a..58dd56163a 100644 --- a/resources/tools/presentation/conf.py +++ b/resources/tools/presentation/conf.py @@ -14,7 +14,8 @@ # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. -# + + import os import sys @@ -35,7 +36,7 @@ extensions = ['sphinxcontrib.programoutput', # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] -# The suffix(es) of source filenames. +# The suffix(es) of source file names. # You can specify multiple suffix as a list of string: # source_suffix = ['.rst', '.md'] @@ -53,19 +54,19 @@ author = u'FD.io CSIT' # built documents. # # The short X.Y version. -#version = u'' +# version = u'' # The full version, including alpha/beta/rc tags. -#release = u'' +# release = u'' rst_epilog = """ -.. |release-1| replace:: rls1710 +.. |release-1| replace:: {prev_release} .. |srelease| replace:: {srelease} .. |vpp-release| replace:: VPP-{vpprelease} release -.. |vpp-release-1| replace:: VPP-17.10 release +.. |vpp-release-1| replace:: VPP-{vpp_prev_release} release .. |dpdk-release| replace:: DPDK {dpdkrelease} -.. |trex-release| replace:: TRex v2.35 -.. |virl-image-ubuntu| replace:: ubuntu-16.04.1_2017-10-21_2.0 -.. |virl-image-centos| replace:: centos-7.3-1611_2017-02-23_1.4 +.. |trex-release| replace:: TRex {trex_version} +.. |virl-image-ubuntu| replace:: {csit_ubuntu_ver} +.. |virl-image-centos| replace:: {csit_centos_ver} .. _pdf version of this report: https://docs.fd.io/csit/{release}/report/_static/archive/csit_{release}.pdf .. _tag documentation rst file: https://git.fd.io/csit/tree/docs/tag_documentation.rst?h={release} @@ -91,7 +92,16 @@ rst_epilog = """ .. _FD.io VPP compile job: https://jenkins.fd.io/view/vpp/job/vpp-merge-{srelease}-ubuntu1604/ .. _FD.io DPDK compile job: https://jenkins.fd.io/view/deb-dpdk/job/deb_dpdk-merge-{sdpdkrelease}-ubuntu1604/ .. _CSIT Testbed Setup: https://git.fd.io/csit/tree/resources/tools/testbed-setup/README.md?h={release} -""".format(release='rls1801', srelease='1801', vpprelease='18.01', dpdkrelease='17.11', sdpdkrelease='1711') +""".format(release='rls1804', + prev_release='rls1801', + srelease='1804', + vpprelease='18.04', + vpp_prev_release='18.01', + dpdkrelease='18.02', + sdpdkrelease='1802', + trex_version='v2.35', + csit_ubuntu_ver='csit-ubuntu-16.04.1_2018-03-07_2.1', + csit_centos_ver='csit-centos-7.4-1711_2018-03-20_1.9') # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages.