.. note::
Data sources for reported test results: i) FD.io test executor jobs
- `hc2vpp-csit-integration-1701-ubuntu1404
- <https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-integration-1701-ubuntu1404/>`_
+ `hc2vpp-csit-integration-1701-ubuntu1404/1
+ <https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-integration-1701-ubuntu1404/1>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
--- /dev/null
+HoneyComb Functional Results\r
+============================\r
+\r
+.. toctree::\r
+\r
+ honeycomb_functional_results\r
+\r
--- /dev/null
+Detailed Test Results
+=====================
+
+.. toctree::
+
+ vpp_performance_results/index
+ testpmd_performance_results/index
+ vpp_functional_results/index
+ honeycomb_functional_results/index
+ vpp_unittest_results/index
--- /dev/null
+Testpmd Performance Results\r
+===========================\r
+\r
+.. toctree::\r
+\r
+ testpmd_performance_results\r
--- /dev/null
+VPP Functional Results\r
+======================\r
+\r
+.. toctree::\r
+\r
+ vpp_functional_results\r
+\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-functional-1701-virl\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_\r
+ `csit-vpp-functional-1701-virl/18\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
--- /dev/null
+VPP Performance Results\r
+=======================\r
+\r
+.. toctree::\r
+\r
+ vpp_performance_results\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
.. note::\r
\r
Data sources for reported test results: i) FD.io test executor jobs\r
- `csit-vpp-perf-1701-all\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and\r
- `csit-vpp-perf-1701-long\r
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_\r
+ `csit-vpp-perf-1701-all/4\r
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_\r
, ii) archived FD.io jobs test result `output files\r
<../../_static/archive/>`_.\r
\r
--- /dev/null
+VPP Unit Test Results\r
+=====================\r
+\r
+.. toctree::\r
+\r
+ vpp_unittest_results\r
+\r
--- /dev/null
+CRUD Loopback\r
+`````````````\r
+::\r
+\r
+ ==============================================================================\r
+ CRUD Loopback\r
+ ==============================================================================\r
+ test_crud (test_interface_crud.TestLoopbackInterfaceCRUD) OK\r
+ test_down (test_interface_crud.TestLoopbackInterfaceCRUD) OK\r
+\r
+Flow-per-packet plugin\r
+``````````````````````\r
+::\r
+\r
+ ==============================================================================\r
+ Flow-per-packet plugin: test both L2 and IP4 reporting\r
+ ==============================================================================\r
+ Flow per packet L3 test OK\r
+\r
+DHCP\r
+````\r
+::\r
+\r
+ ==============================================================================\r
+ DHCP Test Case\r
+ ==============================================================================\r
+ DHCPv6 Proxy OK\r
+ DHCPv4 Proxy OK\r
+\r
+IP4 VRF Multi-instance\r
+``````````````````````\r
+::\r
+\r
+ ==============================================================================\r
+ IP4 VRF Multi-instance Test Case\r
+ ==============================================================================\r
+ IP4 VRF Multi-instance test 1 - create 5 BDs OK\r
+ IP4 VRF Multi-instance test 2 - delete 2 VRFs OK\r
+ IP4 VRF Multi-instance 3 - add 2 VRFs OK\r
+ IP4 VRF Multi-instance test 4 - delete 4 VRFs OK\r
+\r
+Load Balancer\r
+`````````````\r
+::\r
+\r
+ ==============================================================================\r
+ Load Balancer Test Case\r
+ ==============================================================================\r
+ Load Balancer IP4 GRE4 OK\r
+ Load Balancer IP4 GRE6 OK\r
+ Load Balancer IP6 GRE4 OK\r
+ Load Balancer IP6 GRE6 OK\r
+\r
+BFD IPv4\r
+````````\r
+\r
+::\r
+\r
+ ==============================================================================\r
+ Bidirectional Forwarding Detection (BFD)\r
+ ==============================================================================\r
+ verify session goes down after inactivity OK\r
+ hold BFD session up OK\r
+ immediately honor remote min rx reduction OK\r
+ large remote RequiredMinRxInterval OK\r
+ bring BFD session down OK\r
+ bring BFD session up OK\r
+ verify slow periodic control frames while session down OK\r
+ no packets when zero BFD RemoteMinRxInterval OK\r
+\r
+BFD IPv6\r
+````````\r
+::\r
+\r
+ ==============================================================================\r
+ Bidirectional Forwarding Detection (BFD) (IPv6)\r
+ ==============================================================================\r
+ hold BFD session up OK\r
+ bring BFD session up OK\r
+\r
+BFD API tests\r
+`````````````\r
+::\r
+\r
+ ==============================================================================\r
+ Bidirectional Forwarding Detection (BFD) - API\r
+ ==============================================================================\r
+ activate SHA1 authentication OK\r
+ create BFD session using non-existent SHA1 (negative case) OK\r
+ create a BFD session OK\r
+ create IPv6 BFD session OK\r
+ create a BFD session (SHA1) OK\r
+ add SHA1 keys OK\r
+ test_change_key (test_bfd.BFDAPITestCase) OK\r
+ deactivate SHA1 authentication OK\r
+ create the same BFD session twice (negative case) OK\r
+ create the same BFD session twice (negative case) (SHA1) OK\r
+ share single SHA1 key between multiple BFD sessions OK\r
+\r
+BFD authorization\r
+`````````````````\r
+::\r
+\r
+ ==============================================================================\r
+ Bidirectional Forwarding Detection (BFD) (changing auth)\r
+ ==============================================================================\r
+ change auth key without disturbing session state (delayed) OK\r
+ change auth key without disturbing session state (immediate) OK\r
+ turn auth off without disturbing session state (delayed) OK\r
+ turn auth off without disturbing session state (immediate) OK\r
+ turn auth on without disturbing session state (delayed) OK\r
+ turn auth on without disturbing session state (immediate) OK\r
+\r
+BFD authentication\r
+``````````````````\r
+::\r
+\r
+ ==============================================================================\r
+ Bidirectional Forwarding Detection (BFD) (SHA1 auth)\r
+ ==============================================================================\r
+ hold BFD session up OK\r
+ hold BFD session up - meticulous auth OK\r
+ session is not brought down by unauthenticated msg OK\r
+ session is not brought down by msg with non-existent key-id OK\r
+ session is not brought down by msg with wrong auth type OK\r
+ simulate remote peer restart and resynchronization OK\r
+ session is not kept alive by msgs with bad seq numbers OK\r
+ bring BFD session up OK\r
+\r
+IPv6 Tests\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ IPv6 Test Case\r
+ ==============================================================================\r
+ IPv6 FIB test OK\r
+ IPv6 Neighbour Solicitation Exceptions OK\r
+ IPv6 Router Solicitation Exceptions OK\r
+\r
+SPAN Test\r
+`````````\r
+::\r
+\r
+ ==============================================================================\r
+ SPAN Test Case\r
+ ==============================================================================\r
+ SPAN test OK\r
+\r
+GRE Tests\r
+`````````\r
+::\r
+\r
+ ==============================================================================\r
+ GRE Test Case\r
+ ==============================================================================\r
+ GRE tunnel Tests OK\r
+ GRE tunnel L2 Tests OK\r
+ GRE tunnel VRF Tests OK\r
+\r
+SNAT Test Cases\r
+```````````````\r
+::\r
+\r
+ ==============================================================================\r
+ SNAT Test Cases\r
+ ==============================================================================\r
+ SNAT dynamic translation test OK\r
+ SNAT hairpinning OK\r
+ SNAT multiple inside interfaces with overlapping address space OK\r
+ Acquire SNAT addresses from interface OK\r
+ Static mapping with addresses from interface OK\r
+ S-NAT IPFIX logging NAT addresses exhausted OK\r
+ S-NAT IPFIX logging NAT44 session created/delted OK\r
+ MAX translations per user - recycle the least recently used OK\r
+ SNAT multiple inside interfaces (non-overlapping address space) OK\r
+ S-NAT add pool addresses to FIB OK\r
+ SNAT 1:1 NAT initialized from inside network OK\r
+ SNAT 1:1 NAT initialized from outside network OK\r
+ SNAT 1:1 NAT VRF awareness OK\r
+ SNAT 1:1 NAT with port initialized from inside network OK\r
+ SNAT 1:1 NAT with port initialized from outside network OK\r
+\r
+L2XC Multi-instance\r
+```````````````````\r
+::\r
+\r
+ ==============================================================================\r
+ L2XC Multi-instance Test Case\r
+ ==============================================================================\r
+ L2XC Multi-instance test 1 - create 10 cross-connects OK\r
+ L2XC Multi-instance test 2 - delete 4 cross-connects OK\r
+ L2BD Multi-instance 3 - add new 4 cross-connects OK\r
+ L2XC Multi-instance test 4 - delete 10 cross-connects OK\r
+\r
+IPv4 Tests\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ IPv4 Test Case\r
+ ==============================================================================\r
+ IPv4 FIB test OK\r
+\r
+IPv4 FIB CRUD\r
+`````````````\r
+::\r
+\r
+ ==============================================================================\r
+ FIB - add/update/delete - ip4 routes\r
+ ==============================================================================\r
+ Add 1k routes OK\r
+ Delete 100 routes OK\r
+ Add 1k routes OK\r
+ Delete 1.5k routes OK\r
+\r
+IRB Tests\r
+`````````\r
+::\r
+\r
+ ==============================================================================\r
+ IRB Test Case\r
+ ==============================================================================\r
+ IPv4 IRB test 1 OK\r
+ IPv4 IRB test 2 OK\r
+\r
+MPLS Tests\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ MPLS Test Case\r
+ ==============================================================================\r
+ MPLS Local Label Binding test OK\r
+ MPLS Deagg OK\r
+ MPLS label imposition test OK\r
+ MPLS label swap tests OK\r
+ MPLS Tunnel Tests OK\r
+ MPLS V4 Explicit NULL test OK\r
+ MPLS V6 Explicit NULL test OK\r
+\r
+L2XC Tests\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ L2XC Test Case\r
+ ==============================================================================\r
+ L2XC dual-loop test OK\r
+ L2XC single-loop test OK\r
+\r
+Classifier\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ Classifier Test Case\r
+ ==============================================================================\r
+ IP ACL test OK\r
+ MAC ACL test OK\r
+ IP PBR test OK\r
+\r
+VXLAN Tests\r
+```````````\r
+::\r
+\r
+ ==============================================================================\r
+ VXLAN Test Case\r
+ ==============================================================================\r
+ Decapsulation test OK\r
+ Encapsulation test OK\r
+ Multicast flood test OK\r
+ Multicast receive test OK\r
+ Unicast flood test OK\r
+\r
+L2 FIB CRUD\r
+```````````\r
+::\r
+\r
+ ==============================================================================\r
+ L2 FIB Test Case\r
+ ==============================================================================\r
+ L2 FIB test 1 - program 100 MAC addresses OK\r
+ L2 FIB test 2 - delete 12 MAC entries OK\r
+ L2 FIB test 3 - program new 100 MAC addresses OK\r
+ L2 FIB test 4 - delete 160 MAC entries OK\r
+\r
+L2BD Tests\r
+``````````\r
+::\r
+\r
+ ==============================================================================\r
+ L2BD Test Case\r
+ ==============================================================================\r
+ L2BD MAC learning dual-loop test OK\r
+ L2BD MAC learning single-loop test OK\r
+++ /dev/null
-HoneyComb Functional Results VIRL\r
-=================================\r
-\r
-.. toctree::\r
-\r
- honeycomb_functional_results_virl\r
-\r
--- /dev/null
+CSIT Release Notes\r
+==================\r
+\r
+Changes in CSIT |release|\r
+-------------------------\r
+\r
+#. Naming change for all Honeycomb functional test suites in VIRL\r
+\r
+ - Honeycomb functional test case names stayed unchanged\r
+\r
+#. Added Honeycomb functional tests\r
+\r
+ - NSH_SFC\r
+ - LISP\r
+ - NAT\r
+ - SPAN\r
+\r
+Known Issues\r
+------------\r
+\r
+Here is the list of known issues in CSIT |release| for Honeycomb functional\r
+tests in VIRL:\r
+\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| # | Issue | Description |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 1 | Operational data for Vhost-user interfaces | Honeycomb Operational data reports Vhost-user interfaces |\r
+| | "server" flag | as client, even if they are server. |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 2 | Persistence of VxLAN tunnels | Configuration persistence often fails to restore |\r
+| | | Honeycomb's internal naming context for VxLAN interfaces. |\r
+| | | The interface is renamed to "vxlan_tunnel0" but is otherwise |\r
+| | | configured correctly. |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+\r
--- /dev/null
+Documentation\r
+=============\r
+\r
+`CSIT Honeycomb Functional Tests Documentation\r
+<https://docs.fd.io/csit/master/doc/tests.func.html>`_ contains detailed\r
+functional description and input parameters for each test case.\r
-HoneyComb Functional Tests VIRL
-===============================
+HoneyComb Functional Tests
+==========================
.. toctree::
Overview\r
========\r
\r
-Tested Topologies VIRL\r
-----------------------\r
+Tested Virtual Topologies\r
+-------------------------\r
\r
CSIT Honeycomb functional tests are executed on virtualized topologies created\r
using Virtual Internet Routing Lab (VIRL) simulation platform contributed by\r
to every node via dedicated virtual NICs and virtual links (not shown above\r
for clarity).\r
\r
-Honeycomb Functional Tests Overview\r
------------------------------------\r
+Functional Tests Coverage\r
+-------------------------\r
\r
The following Honeycomb functional test areas are included in the CSIT |release|\r
with results listed in this report:\r
Test cases involving network interfaces utilize the first two interfaces on\r
the DUT node.\r
\r
+Functional Tests Naming\r
+-----------------------\r
+\r
+CSIT |release| introduced a common structured naming convention for all\r
+performance and functional tests. This change was driven by substantially\r
+growing number and type of CSIT test cases. Firstly, the original practice did\r
+not always follow any strict naming convention. Secondly test names did not\r
+always clearly capture tested packet encapsulations, and the actual type or\r
+content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
+their locality were not captured either. These were but few reasons that drove\r
+the decision to change and define a new more complete and stricter test naming\r
+convention, and to apply this to all existing and new test cases.\r
+\r
+The new naming should be intuitive for majority of the tests. The complete\r
+description of CSIT test naming convention is provided on `CSIT test naming\r
+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
+\r
+Here few illustrative examples of the new naming usage for functional test\r
+suites:\r
+\r
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
+\r
+ - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
+ routed forwarding, functional tests.\r
+\r
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
+\r
+ - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
+ interfaces and one VM, functional tests.\r
+++ /dev/null
-CSIT Release Notes\r
-==================\r
-\r
-Changes in CSIT |release|\r
--------------------------\r
-\r
-#. Naming change for all Honeycomb functional test suites in VIRL\r
-\r
- - Honeycomb functional test case names stayed unchanged\r
-\r
-#. Added Honeycomb functional tests\r
-\r
- - NSH_SFC\r
- - LISP\r
- - NAT\r
- - SPAN\r
-\r
-Functional Tests Naming\r
------------------------\r
-\r
-CSIT |release| introduced a common structured naming convention for all\r
-performance and functional tests. This change was driven by substantially\r
-growing number and type of CSIT test cases. Firstly, the original practice did\r
-not always follow any strict naming convention. Secondly test names did not\r
-always clearly capture tested packet encapsulations, and the actual type or\r
-content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
-their locality were not captured either. These were but few reasons that drove\r
-the decision to change and define a new more complete and stricter test naming\r
-convention, and to apply this to all existing and new test cases.\r
-\r
-The new naming should be intuitive for majority of the tests. The complete\r
-description of CSIT test naming convention is provided on `CSIT test naming\r
-page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
-\r
-Here few illustrative examples of the new naming usage for functional test\r
-suites:\r
-\r
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
-\r
- - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
- routed forwarding, functional tests.\r
-\r
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
- P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
-\r
- - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
- Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
- interfaces and one VM, functional tests.\r
-\r
-\r
-Known Issues\r
-------------\r
-\r
-Here is the list of known issues in CSIT |release| for Honeycomb functional\r
-tests in VIRL:\r
-\r
-+-------------------------------------------------+-----------------------------------------------------------------+\r
-| Issue | Description |\r
-+-------------------------------------------------+-----------------------------------------------------------------+\r
-| Operational data for Vhost-user interfaces | Honeycomb Operational data reports Vhost-user interfaces |\r
-| "server" flag | as client, even if they are server. |\r
-+-------------------------------------------------+-----------------------------------------------------------------+\r
-| Persistence of VxLAN tunnels | Configuration persistence often fails to restore |\r
-| | Honeycomb's internal naming context for VxLAN interfaces. |\r
-| | The interface is renamed to "vxlan_tunnel0" but is otherwise |\r
-| | configured correctly. |\r
-+-------------------------------------------------+-----------------------------------------------------------------+\r
-\r
+++ /dev/null
-Documentation\r
-=============\r
-\r
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.func.html.\r
:numbered:
introduction/index
- vpp_performance_tests_hw/index
- testpmd_performance_tests_hw/index
- vpp_functional_tests_virl/index
- vpp_functional_tests_vpptest/index
- honeycomb_functional_tests_virl/index
- vpp_performance_results_hw/index
- testpmd_performance_results_hw/index
- vpp_functional_results_virl/index
- vpp_functional_results_vpptest/index
- honeycomb_functional_results_virl/index
-
+ vpp_performance_tests/index
+ testpmd_performance_tests/index
+ vpp_functional_tests/index
+ honeycomb_functional_tests/index
+ vpp_unit_tests/index
+ detailed_test_results/index
--- /dev/null
+General Notes\r
+=============\r
+\r
+All CSIT test results listed in this report are sourced and auto-generated\r
+from output.xml Robot Framework (RF) files resulting from LF FD.io Jenkins\r
+jobs execution against VPP-17.01 release artifacts. References are provided\r
+to the original LF FD.io Jenkins job results. However, as LF FD.io Jenkins\r
+infrastructure does not automatically archive all jobs (history record is\r
+provided for the last 30 days or 40 jobs only), additional references are\r
+provided to the RF result files that got archived in FD.io nexus online\r
+storage system.\r
+\r
+FD.io CSIT project currently covers multiple FD.io system and sub-system\r
+testing areas and this is reflected in this report, where each testing area\r
+is listed separately, as follows\r
+\r
+#. **VPP Performance Tests** - VPP performance are tests executed in physical\r
+ FD.io testbeds, focusing on VPP network data plane performance at this stage,\r
+ both for Phy-to-Phy (NIC-to-NIC) and Phy-to-VM-to-Phy (NIC-to-VM-to-NIC)\r
+ forwarding topologies. Tested across a range of NICs, 10GE and 40GE\r
+ interfaces, range of multi-thread and multi-core configurations. VPP\r
+ application runs in host user- mode. TRex is used as a traffic generator.\r
+\r
+#. **Testpmd Performance Tests** - VPP is using DPDK code to control and drive\r
+ the NICs and physical interfaces. Testpmd tests are used as a baseline to\r
+ profile the DPDK sub-system of VPP. Testpmd performance tests executed in\r
+ physical FD.io testbeds, focusing on Testpmd data plane performance for Phy-\r
+ to-Phy (NIC-to-NIC). Tests cover a range of NICs, 10GE and 40GE interfaces,\r
+ range of multi-thread and multi-core configurations. Testpmd application runs\r
+ in host user-mode. TRex is used as a traffic generator.\r
+\r
+#. **VPP Functional Tests** - VPP functional tests are executed in virtual\r
+ FD.io testbeds focusing on VPP packet processing functionality, including\r
+ network data plane and in -line control plane. Tests cover vNIC-to-vNIC\r
+ vNIC-to-VM-to-vNIC forwarding topologies. Scapy is used as a traffic\r
+ generator.\r
+\r
+#. **HoneyComb Functional Tests** - HoneyComb functional tests are executed in\r
+ virtual FD.io testbeds focusing on HoneyComb management and programming\r
+ functionality of VPP. Tests cover a range of CRUD operations executed\r
+ against VPP.\r
+\r
+CSIT |release| report does also include VPP unit test results. These tests\r
+have been developed within the FD.io VPP project, and not in CSIT, but they do\r
+complement tests done by CSIT. They are provided mainly as a reference, to\r
+give the reader a more complete view of automated testing executed against\r
+VPP-17.01 release.\r
+\r
+FD.io CSIT system is developed using two main coding platforms: Robot\r
+Framework (RF) and Python. CSIT |release| source code for the executed test\r
+suites is available in CSIT branch |release| in the directory\r
+"./tests/<name_of_the_test_suite>". A local copy of CSIT source code can be\r
+obtained by cloning CSIT git repository - "git clone\r
+https://gerrit.fd.io/r/csit". The CSIT testing virtual environment can be run\r
+on a local computer workstation (laptop, server) using Vagrant by following\r
+the instructions in `CSIT tutorials\r
+<https://wiki.fd.io/view/CSIT#Tutorials>`_.
\ No newline at end of file
+++ /dev/null
-General Remarks\r
-===============\r
-\r
-All CSIT test results listed in this report are sourced and auto-generated\r
-from output.xml Robot Framework (RF) files resulting from LF FD.io Jenkins\r
-jobs execution against VPP-17.01 release artifacts. References are provided\r
-to the original LF FD.io Jenkins job results. However, as LF FD.io Jenkins\r
-infrastructure does not automatically archive all jobs (history record is\r
-provided for the last 30 dayts or 40 jobs only), additional references are\r
-provided to the RF result files that got archived in FD.io nexus online\r
-storage system.\r
-\r
-CSIT |release| report does also include VPP make test results, mainly as\r
-a reference. Although these tests have not been developed FD.io CSIT project,\r
-but inthe FD.io VPP project, they have been provided in this report to give\r
-reader a more complete view of automated testing executed against VPP-17.01\r
-release, as they do complement tests done by CSIT.\r
.. toctree::
overview
+ general_notes
csit_test_naming
- general_remarks
Overview\r
========\r
\r
-This is the **F**\ast **D**\ata **I**/**O** Project (FD.io) **C**\ontinuous\r
-**S**\ystem **I**\ntegration and **T**\esting (CSIT) project report for CSIT\r
-|release| system testing of VPP-17.01 release. Initial sections of the report\r
-describe FD.io functional and performance tests and their continuous execution\r
-achieved in CSIT |release|. This is followed by high-level overview of FD.io\r
-test environments running in Linux Foundation (LF) FD.io Continuous\r
-Performance Labs), CSIT |release| release notes and result highlights.\r
-Subsequent sections list the functional and performance detailed test results\r
-generated by LF FD.io Jenkins system executing CSIT jobs running tests\r
-against the VPP-17.01 release and associated sub-systems (HoneyComb, DPDK).\r
-\r
-The exact structure of the report is as follows:\r
-\r
-#. ``Introduction`` - Includes: ``Overview`` - this section;\r
- ``CSIT Test Naming`` - describing CSIT naming convention for test suites and\r
- test cases, important to understand to correctly interpret the reported\r
- results; ``General Remarks`` - additional notes related to this report.\r
-#. ``VPP Performance Tests HW`` - Summary of VPP performance tests executed in\r
- physical Hardware testbed. Includes: ``Overview`` - containing topology\r
- description and list of executed tests; ``Performance Tests Naming`` - lists\r
- CSIT naming specifics for performance tests; ``CSIT Release Notes`` - with\r
- changes in CSIT |release| including major test additions, performance\r
- improvements, changes in testing environment and methodology and known\r
- issues; ``Performance Measurements Methodology`` - details testing\r
- methodology followed for multi-thread multi-core, packet throughput and\r
- latency, and KVM VM vhost tests; ``Report Addendum Targets`` - lists content\r
- to be added to this report as part of CSIT |release|.\r
-#. ``Testpmd Performance Tests HW`` - ...\r
-#. ``VPP Functional Tests VIRL`` - ...\r
-#. ``VPP Functional Tests VPPtest`` - ...\r
-#. ``HoneyComb Functional Tests VIRL`` - ..\r
-#. ``VPP Performance Results HW`` - ...\r
-#. ``Testpmd Performance Results HW`` - ...\r
-#. ``VPP Functional Results VIRL`` - ...\r
-#. ``VPP Functional Results VPPtest`` - ...\r
-#. ``HoneyComb Functional Results VIRL`` - ...\r
-\r
-FD.io CSIT system is developed using two main coding platforms: Robot\r
-Framework (RF) and Python. CSIT |release| source code for the executed test\r
-suites is available in CSIT branch |release| in the directory\r
-"./tests/<name_of_the_test_suite>". A local copy of CSIT source code can be\r
-obtained by cloning CSIT git repository - "git clone\r
-https://gerrit.fd.io/r/csit". The CSIT testing virtual environment can be run\r
-on a local computer workstation (laptop, server) using Vagrant by following\r
-the instructions in `CSIT tutorials\r
-<https://wiki.fd.io/view/CSIT#Tutorials>`_.\r
+This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous\r
+**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT\r
+|release| system testing of VPP-17.01 release.\r
+\r
+The report describes CSIT functional and performance tests and their\r
+continuous execution delivered in CSIT |release|. A high-level overview is\r
+provided for each CSIT test environment running in Linux Foundation (LF) FD.io\r
+Continuous Performance Labs. This is followed by summary of all executed tests\r
+against the VPP-17.01 release and associated sub-systems (HoneyComb, DPDK),\r
+CSIT |release| release notes, result highlights and known issues. More\r
+detailed description of each environment, pointers to CSIT test code\r
+documentation and detailed test resuls with links to the source data files are\r
+also provided.\r
+\r
+CSIT |release| report contains following main sections and sub-sections:\r
+\r
+#. **Introduction** - general introduction to CSIT project; *Overview* -\r
+ this section; *CSIT Test Naming* - CSIT general naming convention for test\r
+ suites and test cases, important to recognize the high-level test content\r
+ and interpret reported results; *General Notes* - general notes related to\r
+ this report.\r
+\r
+#. **VPP Performance Tests** - VPP performance tests executed in physical\r
+ FD.io testbeds; *Overview* - tested topologies, test coverage and naming\r
+ specifics, methodology for multi-core, packet throughput and latency, and\r
+ KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added\r
+ tests, performance changes, environment or methodology changes, known CSIT\r
+ issues, tests to be added; *Packet Throughput Graphs* and *Packet Latency\r
+ Graphs* - plotted NDR, PDR throughput and latency results from multiple\r
+ test job executions; *Test Environment* - environment description ;\r
+ *Documentation* - source code documentation for VPP performance tests.\r
+\r
+#. **Testpmd Performance Tests** - Testpmd performance tests executed in\r
+ physical FD.io testbeds; *Overview* - tested topologies, test coverage;\r
+ *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues;\r
+ *Tests to Be Added* - performance tests to be added in the next revision of\r
+ CSIT |release| report; *Packet Throughput Graphs* and *Packet Latency Graphs*\r
+ - plotted NDR, PDR throughput and latency results from multiple test job\r
+ executions; *Test Environment* - environment description; *Documentation* -\r
+ source code documentation for Testpmd performance tests.\r
+\r
+#. **VPP Functional Tests** - VPP functional tests executed in virtual\r
+ FD.io testbeds; *Overview* - tested virtual topologies, test coverage and\r
+ naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added\r
+ tests, environment or methodology changes, known CSIT issues, tests to be\r
+ added; *Test Environment* - environment description ; *Documentation* -\r
+ source code documentation for VPP functional tests.\r
+\r
+#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in\r
+ virtual FD.io testbeds; *Overview* - tested virtual topologies, test\r
+ coverage and naming specifics; *CSIT Release Notes* - changes in CSIT\r
+ |release|, added tests, environment or methodology changes, known CSIT issues,\r
+ tests to be added; *Test Environment* - environment description ;\r
+ *Documentation* - source code documentation for Honeycomb functional tests.\r
+\r
+#. **VPP Unit Tests** - refers to VPP functional unit tests executed as\r
+ part of vpp make test verify option within the FD.io VPP project; listed in\r
+ this report to give a more complete view about executed VPP functional tests;\r
+ *Overview* - short overview of unit test framework and executed tests;\r
+ *Documentation* - source code documentation of VPP unit tests.\r
+\r
+#. **Detailed Test Results** - auto-generated results from CSIT jobs\r
+ executions using CSIT Robot Framework output files as source data; *VPP\r
+ Performance Results*, *Testpmd Performance Results*, *VPP Functional\r
+ Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*.\r
+++ /dev/null
-Testpmd Performance Results HW\r
-==============================\r
-\r
-.. toctree::\r
-\r
- testpmd_performance_results_hw\r
--- /dev/null
+CSIT Release Notes
+==================
+
+Changes in CSIT |release|
+-------------------------
+
+#. Added Testpmd tests
+
+ - new NICs - Intel x520
+
+Tests to be Added - Additional NICs
+-----------------------------------
+
+Adding test cases with more NIC types. Once the results become available, they
+will be published as an addendum to the current version of CSIT |release|
+report.
--- /dev/null
+Documentation\r
+=============\r
+\r
+`CSIT Testpmd Performance Tests Documentation\r
+<https://docs.fd.io/csit/master/doc/tests.perf.html>`_ contains detailed\r
+functional description and input parameters for each test case.\r
-Testpmd Performance Tests HW
-============================
+Testpmd Performance Tests
+=========================
.. toctree::
Overview
========
-Tested Topologies HW
---------------------
+Tested Physical Topologies
+--------------------------
CSIT Testpmd performance tests are executed on physical baremetal servers hosted
by LF FD.io project. Testbed physical topology is shown in the figure below.
Detailed LF FD.io test bed specification and topology is described in
`wiki CSIT LF testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
-Testing Summary
----------------
+Performance Tests Coverage
+--------------------------
Performance tests are split into the two main categories:
testbed resources to grow, and will be adding complete set of performance
tests for all models of hardware to be executed regularly and(or)
continuously.
+
+Methodology: Multi-Thread and Multi-Core
+----------------------------------------
+
+**HyperThreading** - CSIT |release| performance tests are executed with SUT
+servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS
+settings). This is the simplest configuration used to establish baseline
+single-thread single-core SW packet processing and forwarding performance.
+Subsequent releases of CSIT will add performance tests with Intel
+HyperThreading Enabled (requires BIOS settings change and hard reboot).
+
+**Multi-core Test** - CSIT |release| multi-core tests are executed in the
+following Testpmd thread and core configurations:
+
+#. 1t1c - 1 Testpmd pmd thread on 1 CPU physical core.
+#. 2t2c - 2 Testpmd pmd threads on 2 CPU physical cores.
+#. 4t4c - 4 Testpmd pmd threads on 4 CPU physical cores.
+
+Note that in many tests running Testpmd reaches tested NIC I/O bandwidth
+or packets-per-second limit.
+
+Methodology: Packet Throughput
+------------------------------
+
+Following values are measured and reported for packet throughput tests:
+
+- NDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
+ (2x <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+
+- PDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
+ <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+ - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
+ lost at PDR rate>""
+
+- NDR and PDR are measured for the following L2 frame sizes:
+
+ - IPv4: 64B, 1518B, 9000B.
+
+
+Methodology: Packet Latency
+---------------------------
+
+TRex Traffic Generator (TG) is used for measuring latency of Testpmd DUTs.
+Reported latency values are measured using following methodology:
+
+- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
+ for each NDR throughput test and packet size (except IMIX).
+- TG sends dedicated latency streams, one per direction, each at the rate of
+ 10kpps at the prescribed packet size; these are sent in addition to the main
+ load streams.
+- TG reports min/avg/max latency values per stream direction, hence two sets
+ of latency values are reported per test case; future release of TRex is
+ expected to report latency percentiles.
+- Reported latency values are aggregate across two SUTs due to three node
+ topology used for all performance tests; for per SUT latency, reported value
+ should be divided by two.
+- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
+ FD.io labs used by CSIT project.
+- TRex setup introduces an always-on error of about 2*2usec per latency flow -
+ additonal Tx/Rx interface latency induced by TRex SW writing and reading
+ packet timestamps on CPU cores without HW acceleration on NICs closer to the
+ interface line.
\ No newline at end of file
+++ /dev/null
-CSIT Release Notes
-==================
-
-Changes in CSIT |release|
--------------------------
-
-#. Added Testpmd tests
-
- - new NICs - Intel x520
-
-Multi-Thread and Multi-Core Measurements
-----------------------------------------
-
-**HyperThreading** - CSIT |release| performance tests are executed with SUT
-servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS
-settings). This is the simplest configuration used to establish baseline
-single-thread single-core SW packet processing and forwarding performance.
-Subsequent releases of CSIT will add performance tests with Intel
-HyperThreading Enabled (requires BIOS settings change and hard reboot).
-
-**Multi-core Test** - CSIT |release| multi-core tests are executed in the
-following Testpmd thread and core configurations:
-
-#. 1t1c - 1 Testpmd pmd thread on 1 CPU physical core.
-#. 2t2c - 2 Testpmd pmd threads on 2 CPU physical cores.
-#. 4t4c - 4 Testpmd pmd threads on 4 CPU physical cores.
-
-Note that in many tests running Testpmd reaches tested NIC I/O bandwidth
-or packets-per-second limit.
-
-Packet Throughput Measurements
-------------------------------
-
-Following values are measured and reported for packet throughput tests:
-
-- NDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
- (2x <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
-
-- PDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
- <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
- - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
- lost at PDR rate>""
-
-- NDR and PDR are measured for the following L2 frame sizes:
-
- - IPv4: 64B, 1518B, 9000B.
-
-
-Packet Latency Measurements
----------------------------
-
-TRex Traffic Generator (TG) is used for measuring latency of Testpmd DUTs.
-Reported latency values are measured using following methodology:
-
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
- for each NDR throughput test and packet size (except IMIX).
-- TG sends dedicated latency streams, one per direction, each at the rate of
- 10kpps at the prescribed packet size; these are sent in addition to the main
- load streams.
-- TG reports min/avg/max latency values per stream direction, hence two sets
- of latency values are reported per test case; future release of TRex is
- expected to report latency percentiles.
-- Reported latency values are aggregate across two SUTs due to three node
- topology used for all performance tests; for per SUT latency, reported value
- should be divided by two.
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
- FD.io labs used by CSIT project.
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -
- additonal Tx/Rx interface latency induced by TRex SW writing and reading
- packet timestamps on CPU cores without HW acceleration on NICs closer to the
- interface line.
-
-
-Report Addendum Tests - Additional NICs
----------------------------------------
-
-Adding test cases with more NIC types. Once the results become available, they
-will be published as an addendum to the current version of CSIT |release|
-report.
+++ /dev/null
-Documentation\r
-=============\r
-\r
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.perf.html.\r
+++ /dev/null
-VPP Functional Results VIRL\r
-===========================\r
-\r
-.. toctree::\r
-\r
- functional_results_virl\r
-\r
+++ /dev/null
-DHCP - Client and Proxy\r
-```````````````````````\r
-\r
-to be added ..\r
-\r
-GRE Overlay Tunnels\r
-```````````````````\r
-\r
-to be added ..\r
-\r
-L2BD Ethernet Switching\r
-```````````````````````\r
-\r
-to be added ..\r
-\r
-L2XC Ethernet Switching\r
-```````````````````````\r
-\r
-to be added ..\r
-\r
-LISP Overlay Tunnels\r
-````````````````````\r
-\r
-to be added ..\r
+++ /dev/null
-VPP Functional Results VPPtest\r
-==============================\r
-\r
-.. toctree::\r
-\r
- functional_results_vpptest\r
-\r
--- /dev/null
+
+.. |br| raw:: html
+
+ <br />
+
+CSIT Release Notes
+==================
+
+Changes in CSIT |release|
+-------------------------
+
+#. Naming change for all VPP functional test suites in VIRL
+
+ - VPP functional test case names stayed unchanged
+
+#. VPP functional test environment changes
+
+ - upgrade to Ubuntu 16.04
+ - VM and vhost-user test environment optimizations
+
+#. Introduction of Centos tests
+
+#. Added VPP functional tests
+
+ - more VM vhost-user tests
+ - more LISP tests
+ - more IPSec crypto tests
+ - IPv4 and IPv6 Equal-Cost Multi-Path routing tests
+ - Telemetry:
+ - IPFIX tests
+ - SPAN tests
+
+Known Issues
+------------
+
+Here is the list of known issues in CSIT |release| for VPP functional tests in VIRL:
+
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| # | Issue | Description |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 1 | DHCPv4 client: Client responses to DHCPv4 OFFER | Client replies with DHCPv4 REQUEST message when received DHCPv4 |
+| | sent with different XID | OFFER message with different (wrong) XID. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 2 | Softwire - MAP-E: Incorrect calculation of IPv6 | IPv6 destination address is wrongly calculated in case that |
+| | destination address when IPv4 prefix is 0 | IPv4 prefix is equal to 0 and IPv6 prefix is less than 40. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 3 | Softwire - MAP-E: Map domain is created when | Map domain is created in case that the sum of suffix length of |
+| | incorrect parameters provided | IPv4 prefix and PSID length is greater than EA bits length. |
+| | | IPv6 destination address contains bits writen with PSID over |
+| | | the EA-bit length when IPv4 packet is sent. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 4 | IPv6 RA: Incorrect IPv6 destination address in | Wrong IPv6 destination address (ff02::1) is used in ICMPv6 |
+| | response to ICMPv6 Router Solicitation | Router Advertisement packet sent as a response to received |
+| | | ICMPv6 Router Solicitation packet. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 5 | IPFIX: IPv6_src key name reported instead of | The report contains IPv6_src key name instead of IPv6_dst when |
+| | IPv6_dst | classify session is configured with IPv6 destination address. |
+| | | Anyhow the correct IPv6 destination address is reported. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 6 | SPAN: Tx traffic is not mirrored | Tx traffic is not mirrored from SPAN source port to SPAN |
+| | | destination port. |br| |
+| | | NOTE: Fix is going to be merged in vpp stable/1701. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 7 | SPAN: Packet trace always contains local0 as | There is reported wrong destination port in the traffic trace: |
+| | destination port | |br| SPAN: mirrored GigabitEthernet0/5/0 -> local0 |br| |
+| | | The (Rx) traffic is mirrored to correct destination port. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+
+Tests to be Added - Centos
+--------------------------
+
+CSIT |release| added Centos functional test execution environment in FD.io
+VIRL testbeds.However due to the factors beyond CSIT project control execution
+of those tests within the LF FD.io test environment still need to complete.
+Once the results become available, they will be published as an addendum to
+the current version of CSIT |release| report.
--- /dev/null
+Documentation\r
+=============\r
+\r
+`CSIT VPP Functional Tests Documentation\r
+<https://docs.fd.io/csit/master/doc/tests.func.html>`_ contains detailed\r
+functional description and input parameters for each test case.\r
+\r
-VPP Functional Tests VIRL
-=========================
+VPP Functional Tests
+====================
.. toctree::
Overview\r
========\r
\r
-Tested Topologies VIRL\r
-----------------------\r
+Tested Virtual Topologies\r
+-------------------------\r
\r
CSIT VPP functional tests are executed on virtualized topologies created using\r
Virtual Internet Routing Lab (VIRL) simulation platform contributed by Cisco.\r
******************* | |********************\r
+-----------+\r
\r
-VPP Functional Tests Overview\r
------------------------------\r
+Functional Tests Coverage\r
+-------------------------\r
\r
-Following VPP functional test areas are included in the CSIT |release| with\r
+Following VPP functional test areas are covered in the CSIT |release| with\r
results listed in this report:\r
\r
- **DHCP - Client and Proxy** - Dynamic Host Control Protocol Client and Proxy\r
IPv6.\r
- **VLAN Tag Translation** - L2 VLAN tag translation 2to2, 2to1, 1to2, 1to1.\r
- **VXLAN Overlay Tunnels** - VXLAN tunneling for L2-over-IP, for IPv4, IPv6.\r
+\r
+Functional Tests Naming\r
+-----------------------\r
+\r
+CSIT |release| introduced a common structured naming convention for all\r
+performance and functional tests. This change was driven by substantially\r
+growing number and type of CSIT test cases. Firstly, the original practice did\r
+not always follow any strict naming convention. Secondly test names did not\r
+always clearly capture tested packet encapsulations, and the actual type or\r
+content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
+their locality were not captured either. These were but few reasons that drove\r
+the decision to change and define a new more complete and stricter test naming\r
+convention, and to apply this to all existing and new test cases.\r
+\r
+The new naming should be intuitive for majority of the tests. The complete\r
+description of CSIT test naming convention is provided on `CSIT test naming\r
+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
+\r
+Here few illustrative examples of the new naming usage for functional test\r
+suites:\r
+\r
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
+\r
+ - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline\r
+ routed forwarding, functional tests.\r
+\r
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
+\r
+ - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of\r
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost\r
+ interfaces and one VM, functional tests.\r
+++ /dev/null
-
-.. |br| raw:: html
-
- <br />
-
-CSIT Release Notes
-==================
-
-Changes in CSIT |release|
--------------------------
-
-#. Naming change for all VPP functional test suites in VIRL
-
- - VPP functional test case names stayed unchanged
-
-#. VPP functional test environment changes
-
- - upgrade to Ubuntu 16.04
- - VM and vhost-user test environment optimizations
-
-#. Introduction of Centos tests
-
-#. Added VPP functional tests
-
- - more VM vhost-user tests
- - more LISP tests
- - more IPSec crypto tests
- - IPv4 and IPv6 Equal-Cost Multi-Path routing tests
- - Telemetry:
- - IPFIX tests
- - SPAN tests
-
-Functional Tests Naming
------------------------
-
-CSIT |release| introduced a common structured naming convention for all
-performance and functional tests. This change was driven by substantially
-growing number and type of CSIT test cases. Firstly, the original practice did
-not always follow any strict naming convention. Secondly test names did not
-always clearly capture tested packet encapsulations, and the actual type or
-content of the tests. Thirdly HW configurations in terms of NICs, ports and
-their locality were not captured either. These were but few reasons that drove
-the decision to change and define a new more complete and stricter test naming
-convention, and to apply this to all existing and new test cases.
-
-The new naming should be intuitive for majority of the tests. The complete
-description of CSIT test naming convention is provided on `CSIT test naming
-page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-
-Here few illustrative examples of the new naming usage for functional test
-suites:
-
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
-
- - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
- routed forwarding, functional tests.
-
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
- P2V2P, NIC2VMchain2NIC, P2V2V2P**
-
- - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
- Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
- interfaces and one VM, functional tests.
-
-Known Issues
-------------
-
-Here is the list of known issues in CSIT |release| for VPP functional tests in VIRL:
-
-+-------------------------------------------------+-----------------------------------------------------------------+
-| ISSUE | DESCRIPTION |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| DHCPv4 client: Client responses to DHCPv4 OFFER | Client replies with DHCPv4 REQUEST message when received DHCPv4 |
-| sent with different XID | OFFER message with different (wrong) XID. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| Softwire - MAP-E: Incorrect calculation of IPv6 | IPv6 destination address is wrongly calculated in case that |
-| destination address when IPv4 prefix is 0 | IPv4 prefix is equal to 0 and IPv6 prefix is less than 40. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| Softwire - MAP-E: Map domain is created when | Map domain is created in case that the sum of suffix length of |
-| incorrect parameters provided | IPv4 prefix and PSID length is greater than EA bits length. |
-| | IPv6 destination address contains bits writen with PSID over |
-| | the EA-bit length when IPv4 packet is sent. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| IPv6 RA: Incorrect IPv6 destination address in | Wrong IPv6 destination address (ff02::1) is used in ICMPv6 |
-| response to ICMPv6 Router Solicitation | Router Advertisement packet sent as a response to received |
-| | ICMPv6 Router Solicitation packet. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| IPFIX: IPv6_src key name reported instead of | The report contains IPv6_src key name instead of IPv6_dst when |
-| IPv6_dst | classify session is configured with IPv6 destination address. |
-| | Anyhow the correct IPv6 destination address is reported. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| SPAN: Tx traffic is not mirrored | Tx traffic is not mirrored from SPAN source port to SPAN |
-| | destination port. |br| |
-| | NOTE: Fix is going to be merged in vpp stable/1701. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| SPAN: Packet trace always contains local0 as | There is reported wrong destination port in the traffic trace: |
-| destination port | |br| SPAN: mirrored GigabitEthernet0/5/0 -> local0 |br| |
-| | The (Rx) traffic is mirrored to correct destination port. |
-+-------------------------------------------------+-----------------------------------------------------------------+
+++ /dev/null
-Documentation\r
-=============\r
-\r
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.func.html.\r
+++ /dev/null
-Documentation\r
-=============\r
-\r
-Link\r
+++ /dev/null
-VPP Functional Tests VPPtest
-============================
-
-.. toctree::
-
- overview
- test_environment
- documentation
-
+++ /dev/null
-Overview\r
-========\r
-\r
-VPP Functional Test Topologies in VPPtest\r
------------------------------------------\r
-\r
-to be added ..\r
-\r
-VPP Functional Tests Overview\r
------------------------------\r
-\r
-to be added ..\r
+++ /dev/null
-Test Environment\r
-================\r
-\r
-VPPtest functional tests are currently executed in ...\r
-\r
-to be added\r
+++ /dev/null
-VPP Performance Results HW\r
-==========================\r
-\r
-.. toctree::\r
-\r
- performance_results_hw\r
- VM and vhost-user test environment optimizations\r
- addition of HW cryptodev devices in LFD FD.io physical testbed\r
\r
-\r
#. Added tests\r
\r
- - new NICs - Intel x710, Cisco VIC1385, Cisco VIC1227\r
- - more VM vhost-user tests\r
- - more LISP tests\r
+ - NICs\r
+\r
+ - Intel x710 (L2 Bridge Domain switched-forwarding, vhost-user)\r
+ - Cisco VIC1385 (L2 Bridge Domain switched-forwarding)\r
+ - Cisco VIC1227 (L2 Bridge Domain switched-forwarding)\r
+\r
+ - vhost-user tests with VM\r
+\r
+ - L2 Bridge Domain switched-forwarding with Intel x710 NIC\r
+ - L2 Bridge Domain switched-forwarding with VxLAN and Intel x520 NIC\r
+ - L2 Bridge Domain switched-forwarding with Intel xl710 NIC\r
\r
+ - Tests with VxLAN\r
\r
-Performance Tests Naming\r
+ - L2 Bridge Domain switched-forwarding with Intel x520 NIC\r
+ - L2 Bridge Domain switched-forwarding with vhost-user, VM and Intel x520\r
+ NIC\r
+\r
+Performance Improvements\r
------------------------\r
\r
-CSIT |release| introduced a common structured naming convention for all\r
-performance and functional tests. This change was driven by substantially\r
-growing number and type of CSIT test cases. Firstly, the original practice did\r
-not always follow any strict naming convention. Secondly test names did not\r
-always clearly capture tested packet encapsulations, and the actual type or\r
-content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
-their locality were not captured either. These were but few reasons that drove\r
-the decision to change and define a new more complete and stricter test naming\r
-convention, and to apply this to all existing and new test cases.\r
-\r
-The new naming should be intuitive for majority of the tests. The complete\r
-description of CSIT test naming convention is provided on `CSIT test naming wiki\r
-<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
-\r
-Here few illustrative examples of the new naming usage for performance test\r
-suites:\r
-\r
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
-\r
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
- PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*\r
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on\r
- Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching\r
- with MAC learning, NDR throughput discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE\r
- on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline\r
- switching with MAC learning, NDR throughput discovery.\r
- - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel\r
- x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.\r
- - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on\r
- Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput\r
- discovery.\r
-\r
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
- P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
-\r
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
- PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-\r
- VirtPortConfig-VMconfig-TestType*\r
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports\r
- of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain\r
- switching to/from two vhost interfaces and one VM, NDR throughput\r
- discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2\r
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
- switching to/from two vhost interfaces and one VM, NDR throughput\r
- discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2\r
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
- switching to/from four vhost interfaces and two VMs, NDR throughput\r
- discovery.\r
-\r
-Measured Performance Improvements\r
----------------------------------\r
-\r
-Substantial improvements in measured packet throughput have been observed\r
-in VPP-17.01 for the following CSIT |release| tests:\r
-\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |\r
-+===================+================================================================+===========+=================+======================+\r
-| L2XC | 10ge2p1x520:64B-1t1c-eth-l2xcbase-ndrdisc | 9.4 Mpps | 12.6..12.9 Mpps | 34..37% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| L2XC | 10ge2p1xl710:64B-1t1c-eth-l2xcbase-ndrdisc | 9.5 Mpps | 12.1..12.4 Mpps | 27..30% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| L2BD | 10ge2p1x520:64B-1t1c-eth-l2bdbasemaclrn-ndrdisc | 7.8 Mpps | 10.6 Mpps | 36% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| L2BD-vhost-VM | 10ge2p1x520:64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 2.8 Mpps | 600% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| L2XC-vhost-VM | 10ge2p1x520:64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 3.2 Mpps | 700% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 | 10ge2p1x520:64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 12% |\r
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-\r
-Non-drop rate search:\r
+Substantial improvements in measured packet throughput, with relative increase\r
+of double-digit percentage points, have been observed in a number of CSIT\r
+|release| tests listed below. Relative improvements are calculated against the\r
+test results in CSIT rls1609 report.\r
+\r
+NDR Throughput\r
+~~~~~~~~~~~~~~\r
+\r
+Non-Drop Rate Throughput discovery tests:\r
\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| L2BD-vhost-VM | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 2.7..2.8 Mpps | 575..600% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 11% |\r
+| IPv4 | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 12% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 COP | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc | 7.1 Mpps | 8.3..8.5 Mpps | 17..20% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc | 8.5 Mpps | 9.0 Mpps | 6% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 FIB 2M | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc | 8.5 Mpps | 7.8..8.1 Mpps | -8..-5% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 iAcl | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc | 6.9 Mpps | 7.6..7.8 Mpps | 10..13% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 Policer | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc | 6.9 Mpps | 7.4..7.6 Mpps | 7..10% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc | 4.4 Mpps | 4.8 Mpps | 9% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc | 0.3 Mpps | 2.6 Mpps | 767% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv6 | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc | 3.0 Mpps | 7.3..7.7 Mpps | 143..157% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv6 FIB 200k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc | 6.5 Mpps | 5.3..5.7 Mpps | -18..-12% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv6 FIB 20k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc | 6.9 Mpps | 6.5 Mpps | -6% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv6 FIB 2M | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc | 5.3 Mpps | 4.2 Mpps | -21% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
\r
-Partial drop rate search, LT = 0.5%:\r
+PDR Throughput\r
+~~~~~~~~~~~~~~\r
+\r
+Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |\r
+===================+=================================================================+===========+=================+======================+\r
| L2XC | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc | 9.4 Mpps | 12.7..12.9 Mpps | 35..37% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| L2XC | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-pdrdisc | no data | no data | |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| L2XC dot1ad | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc | 7.4 Mpps | 8.8..9.1 Mpps | 19..23% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| L2XC dot1q | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc | 7.5 Mpps | 8.8..9.0 Mpps | 17..20% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 COP | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc | 7.1 Mpps | 8.3..8.5 Mpps | 17..20% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc | 2.0 Mpps | 2.7 Mpps | 35% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+\r
+Measured improvements are in line with VPP code optimizations listed in\r
+`VPP-17.01 release notes\r
+<https://docs.fd.io/vpp/17.01/release_notes_1701.html>`_.\r
+\r
+Additionally, vhost-VM performance improvements are due to both VPP code\r
+optimizations as well as due to the FD.io CSIT Linux KVM test environment\r
+optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM\r
+vhost".\r
+\r
+\r
+Other Performance Changes\r
+-------------------------\r
+\r
+Other changes in measured packet throughput, with either minor relative\r
+increase or decrease, have been observed in a number of CSIT |release| tests\r
+listed below. Relative changes are calculated against the test results in CSIT\r
+rls1609 report.\r
+\r
+NDR Throughput\r
+~~~~~~~~~~~~~~\r
+\r
+Non-Drop Rate Throughput discovery tests:\r
+\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Change |\r
++===================+=================================================================+===========+=================+======================+\r
+| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc | 8.5 Mpps | 9.0 Mpps | 6% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv4 FIB 2M | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc | 8.5 Mpps | 7.8..8.1 Mpps | -8..-5% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv4 Policer | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc | 6.9 Mpps | 7.4..7.6 Mpps | 7..10% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc | 4.4 Mpps | 4.8 Mpps | 9% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv6 FIB 200k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc | 6.5 Mpps | 5.3..5.7 Mpps | -18..-12% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv6 FIB 20k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc | 6.9 Mpps | 6.5 Mpps | -6% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv6 FIB 2M | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc | 5.3 Mpps | 4.2 Mpps | -21% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+\r
+PDR Throughput\r
+~~~~~~~~~~~~~~\r
+\r
+Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
+\r
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
+| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Change |\r
++===================+=================================================================+===========+=================+======================+\r
| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc | 8.5 Mpps | 9.0 Mpps | 6% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc | 4.6 Mpps | 4.8 Mpps | 9% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
-| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc | 2.0 Mpps | 2.7 Mpps | 35% |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv6 | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc | 7.7 Mpps | 7.3..7.7 Mpps | -5..0% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |\r
| IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |\r
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
\r
-This is in line with VPP code optimizations listed in `VPP-17.01 release notes\r
-<https://docs.fd.io/vpp/17.01/release_notes_1701.html>`_.\r
-\r
-vhost-VM performance improvement is also due to Linux KVM test environment\r
-optimization for vhost-VM tests - see section below "VM vhost-user\r
-Throughput Measurements".\r
-\r
-Multi-Thread and Multi-Core Measurements\r
-----------------------------------------\r
-\r
-**HyperThreading** - CSIT |release| performance tests are executed with SUT\r
-servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS\r
-settings). This is the simplest configuration used to establish baseline\r
-single-thread single-core SW packet processing and forwarding performance.\r
-Subsequent releases of CSIT will add performance tests with Intel\r
-HyperThreading Enabled (requires BIOS settings change and hard reboot).\r
-\r
-**Multi-core Test** - CSIT |release| multi-core tests are executed in the\r
-following VPP thread and core configurations:\r
-\r
-#. 1t1c - 1 VPP worker thread on 1 CPU physical core.\r
-#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.\r
-#. 4t4c - 4 VPP threads on 4 CPU physical cores.\r
-\r
-Note that in quite a few test cases running VPP on 2 or 4 physical cores hits\r
-the tested NIC I/O bandwidth or packets-per-second limit.\r
-\r
-Packet Throughput Measurements\r
-------------------------------\r
-\r
-Following values are measured and reported for packet throughput tests:\r
-\r
-- NDR binary search per RFC2544:\r
-\r
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps\r
- (2x <per direction packets-per-second>)"\r
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
- second> Gbps (untagged)"\r
-\r
-- PDR binary search per RFC2544:\r
-\r
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x\r
- <per direction packets-per-second>)"\r
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
- second> Gbps (untagged)"\r
- - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets\r
- lost at PDR rate>""\r
-\r
-- NDR and PDR are measured for the following L2 frame sizes:\r
-\r
- - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.\r
- - IPv6: 78B, 1518B, 9000B.\r
-\r
-\r
-Packet Latency Measurements\r
----------------------------\r
-\r
-TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported\r
-latency values are measured using following methodology:\r
-\r
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)\r
- for each NDR throughput test and packet size (except IMIX).\r
-- TG sends dedicated latency streams, one per direction, each at the rate of\r
- 10kpps at the prescribed packet size; these are sent in addition to the main\r
- load streams.\r
-- TG reports min/avg/max latency values per stream direction, hence two sets\r
- of latency values are reported per test case; future release of TRex is\r
- expected to report latency percentiles.\r
-- Reported latency values are aggregate across two SUTs due to three node\r
- topology used for all performance tests; for per SUT latency, reported value\r
- should be divided by two.\r
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in\r
- FD.io labs used by CSIT project.\r
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -\r
- additonal Tx/Rx interface latency induced by TRex SW writing and reading\r
- packet timestamps on CPU cores without HW acceleration on NICs closer to the\r
- interface line.\r
-\r
-\r
-KVM VM vhost Measurements\r
--------------------------\r
-\r
-CSIT |release| introduced environment configuration changes to KVM Qemu vhost-\r
-user tests in order to more representatively measure VPP-17.01 performance in\r
-configurations with vhost-user interfaces and VMs.\r
-\r
-Current setup of CSIT FD.io performance lab is using tuned settings for more\r
-optimal performance of KVM Qemu:\r
+Known Issues\r
+------------\r
\r
-- Default Qemu virtio queue size of 256 descriptors.\r
-- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki\r
- page: https://wiki.fd.io/view/CSIT/VM-vhost-env-tuning.\r
+Here is the list of known issues in CSIT |release| for VPP performance tests:\r
\r
-Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance\r
-of VPP+VM system less sensitive to other Linux OS system tasks by reducing\r
-their interference on CPU cores that are designated for critical software\r
-tasks under test, namely VPP worker threads in host and Testpmd threads in\r
-guest dealing with data plan.\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| # | Issue | Description |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 1 | Sporadic IPv4 routed-forwarding NDR discovery | Suspected issue with DPDK IPv4 checksum calculation. |\r
+| | test failures for 1518B frame size | Observed frequency: sporadic, ca. 20% to 30% of test runs. |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 2 | Vic1385 and Vic1227 NICs jumbo frames test | Suspected issue with VIC DPDK drivers that do not support jumbo |\r
+| | failures (9000B) | frames (dropped rx-miss). Observed frequency: 100%. |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 3 | Vic1385 and Vic1227 performance | Low performance of NDR results. Big difference between NDR and |\r
+| | | PDR. . |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 4 | Sporadic NDR discovery test failures on x520 | Suspected issue with HW settings (BIOS, FW) in LF |\r
+| | | infrastructure. Issue can't be replicated outside LF. |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
+| 5 | Non-repeatible zig-zagging NDR throughput | Suspected NIC driver issue in multi-thread multi-core operation |\r
+| | results for multi-thread multi-core tests | affectin NDR throughput. Applies to XL710 and X710 NICs tested. |\r
+| | - 2t2c, 4t4c - for some tested NICs. | |\r
++---+-------------------------------------------------+-----------------------------------------------------------------+\r
\r
-Report Addendum Tests - Cryptodev\r
----------------------------------\r
+Tests to be Added - Cryptodev\r
+-----------------------------\r
\r
DPDK Cryptodev functionality support for both SW and HW crypto devices has\r
been introduced in VPP-17.01 release. CSIT functional and performance tests\r
still need to complete. Once the results become available, they will be\r
published as an addendum to the current version of CSIT |release| report.\r
\r
-Report Addendum Tests - Centos\r
-------------------------------\r
-\r
-CSIT |release| added Centos functional test execution environment in FD.io\r
-VIRL testbeds.However due to the factors beyond CSIT project control execution\r
-of those tests within the LF FD.io test environment still need to complete.\r
-Once the results become available, they will be published as an addendum to\r
-the current version of CSIT |release| report.\r
-\r
-Report Addendum Tests - SNAT44\r
-------------------------------\r
+Tests to be Added - SNAT44\r
+--------------------------\r
\r
VPP SNAT44 functionality has been introduced in VPP-17.01 release. CSIT\r
performance tests are still in development and integration into LFD FD.io test\r
environment. Once the tests are fully integrated and results become available,\r
they will be published as an addendum to the current version of CSIT |release|\r
report.\r
-\r
-Known Issues\r
-------------\r
-\r
-Here is the list of known issues in CSIT |release| for VPP performance tests in physical HW testbed:\r
-\r
-+---+-------------------------------------------------+-----------------------------------------------------------------+\r
-| # | Issue | Description |\r
-+---+-------------------------------------------------+-----------------------------------------------------------------+\r
-| 1 | Sporadic IPv4 routed-forwwarding NDR discovery | Suspected issue with DPDK IPv4 checksum calculation, VPP jira # |\r
-| | test failures for 1518B frame size | Observed frequency: sporadic, ca. 20% to 30% of test runs |\r
-+---+-------------------------------------------------+-----------------------------------------------------------------+\r
-| | | |\r
-+---+-------------------------------------------------+-----------------------------------------------------------------+\r
-| | | |\r
-+---+-------------------------------------------------+-----------------------------------------------------------------+\r
--- /dev/null
+Documentation\r
+=============\r
+\r
+`CSIT VPP Performance Tests Documentation\r
+<https://docs.fd.io/csit/master/doc/tests.perf.html>`_ contains detailed\r
+functional description and input parameters for each test case.\r
-VPP Performance Tests HW
-========================
+VPP Performance Tests
+=====================
.. toctree::
--- /dev/null
+Overview\r
+========\r
+\r
+Tested Physical Topologies\r
+--------------------------\r
+\r
+CSIT VPP performance tests are executed on physical baremetal servers hosted by LF\r
+FD.io project. Testbed physical topology is shown in the figure below.\r
+\r
+::\r
+\r
+ +------------------------+ +------------------------+\r
+ | | | |\r
+ | +------------------+ | | +------------------+ |\r
+ | | | | | | | |\r
+ | | <-----------------> | |\r
+ | | DUT1 | | | | DUT2 | |\r
+ | +--^---------------+ | | +---------------^--+ |\r
+ | | | | | |\r
+ | | SUT1 | | SUT2 | |\r
+ +------------------------+ +------------------^-----+\r
+ | |\r
+ | |\r
+ | +-----------+ |\r
+ | | | |\r
+ +------------------> TG <------------------+\r
+ | |\r
+ +-----------+\r
+\r
+SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two\r
+Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with\r
+two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a\r
+Device Under Test (DUT). TG runs TRex SW application as a packet Traffic\r
+Generator. Physical connectivity between SUTs and to TG is provided using\r
+different NIC models that need to be tested for performance. Currently\r
+installed and tested NIC models include:\r
+\r
+#. 2port10GE X520-DA2 Intel.\r
+#. 2port10GE X710 Intel.\r
+#. 2port10GE VIC1227 Cisco.\r
+#. 2port40GE VIC1385 Cisco.\r
+#. 2port40GE XL710 Intel.\r
+\r
+From SUT and DUT perspective, all performance tests involve forwarding packets\r
+between two physical Ethernet ports (10GE or 40GE). Due to the number of\r
+listed NIC models tested and available PCI slot capacity in SUT servers, in\r
+all of the above cases both physical ports are located on the same NIC. In\r
+some test cases this results in measured packet throughput being limited not\r
+by VPP DUT but by either the physical interface or the NIC capacity.\r
+\r
+Going forward CSIT project will be looking to add more hardware into FD.io\r
+performance labs to address larger scale multi-interface and multi-NIC\r
+performance testing scenarios.\r
+\r
+For test cases that require DUT (VPP) to communicate with VM over vhost-user\r
+interfaces, a VM is created on SUT1 and SUT2. DUT (VPP) test topology with VM\r
+is shown in the figure below including applicable packet flow thru the VM\r
+(marked in the figure with ``***``).\r
+\r
+::\r
+\r
+ +------------------------+ +------------------------+\r
+ | +----------+ | | +----------+ |\r
+ | | VM | | | | VM | |\r
+ | | ****** | | | | ****** | |\r
+ | +--^----^--+ | | +--^----^--+ |\r
+ | *| |* | | *| |* |\r
+ | +------v----v------+ | | +------v----v------+ |\r
+ | | * * |**|***********|**| * * | |\r
+ | | ***** *******<----------------->******* ***** | |\r
+ | | * DUT1 | | | | DUT2 * | |\r
+ | +--^---------------+ | | +---------------^--+ |\r
+ | *| | | |* |\r
+ | *| SUT1 | | SUT2 |* |\r
+ +------------------------+ +------------------^-----+\r
+ *| |*\r
+ *| |*\r
+ *| +-----------+ |*\r
+ *| | | |*\r
+ *+------------------> TG <------------------+*\r
+ ******************* | |********************\r
+ +-----------+\r
+\r
+For VM tests, packets are switched by DUT (VPP) twice, hence the\r
+throughput rates measured by TG (and listed in this report) must be multiplied\r
+by two to represent the actual DUT aggregate packet forwarding rate.\r
+\r
+Note that reported VPP performance results are specific to the SUT tested.\r
+Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with\r
+other CPUs are likely to yield different results. A good rule of thumb, that\r
+can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC,\r
+PCI-to-PCI) topology, is to expect the forwarding performance to be\r
+proportional to CPU core frequency, assuming CPU is the only limiting factor\r
+and all other SUT aspects equal to FD.io CSIT environment. The same rule of\r
+thumb can be also applied for Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) topology,\r
+but due to much higher dependency on very high frequency memory operations and\r
+sensitivity to Linux kernel scheduler settings and behaviour, this estimation\r
+may not always yield good enough accuracy.\r
+\r
+Detailed LF FD.io test bed specification and physical topology are described\r
+in `wiki CSIT LF FDio testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
+\r
+Performance Tests Coverage\r
+--------------------------\r
+\r
+Performance tests are split into the two main categories:\r
+\r
+- Throughput discovery - discovery of packet forwarding rate using binary search\r
+ in accordance to RFC2544.\r
+\r
+ - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;\r
+ followed by packet one-way latency measurements at 10%, 50% and 100% of\r
+ discovered NDR throughput.\r
+ - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss\r
+ currently set to 0.5%; followed by packet one-way latency measurements at\r
+ 100% of discovered PDR throughput.\r
+\r
+- Throughput verification - verification of packet forwarding rate against\r
+ previously discovered throughput rate. These tests are currently done against\r
+ 0.9 of reference NDR, with reference rates updated periodically.\r
+\r
+CSIT |release| includes following performance test suites, listed per NIC type:\r
+\r
+- 2port10GE X520-DA2 Intel\r
+\r
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad\r
+ VLAN tagged Ethernet frames.\r
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
+ with MAC learning; disabled MAC learning i.e. static MAC tests to be added.\r
+ - **IPv4** - IPv4 routed-forwarding.\r
+ - **IPv6** - IPv6 routed-forwarding.\r
+ - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.\r
+ - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.\r
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
+ interfaces in different switching modes incl. L2 Cross-Connect, L2\r
+ Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.\r
+ - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.\r
+ - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.\r
+ - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4,\r
+ IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.\r
+ - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.\r
+ - **QoS Policer** - ingress packet rate measuring, marking and limiting\r
+ (IPv4).\r
+\r
+- 2port40GE XL710 Intel\r
+\r
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.\r
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
+ with MAC learning.\r
+ - **IPv4** - IPv4 routed-forwarding.\r
+ - **IPv6** - IPv6 routed-forwarding.\r
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
+ interfaces in different switching modes incl. L2 Bridge-Domain.\r
+\r
+- 2port10GE X710 Intel\r
+\r
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
+ with MAC learning.\r
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
+ interfaces in different switching modes incl. L2 Bridge-Domain.\r
+\r
+- 2port10GE VIC1227 Cisco\r
+\r
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
+ with MAC learning.\r
+\r
+- 2port40GE VIC1385 Cisco\r
+\r
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
+ with MAC learning.\r
+\r
+Execution of performance tests takes time, especially the throughput discovery\r
+tests. Due to limited HW testbed resources available within FD.io labs hosted\r
+by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.\r
+Niantic) has been limited to few baseline tests. Over time we expect the HW\r
+testbed resources to grow, and will be adding complete set of performance\r
+tests for all models of hardware to be executed regularly and(or)\r
+continuously.\r
+\r
+Performance Tests Naming\r
+------------------------\r
+\r
+CSIT |release| introduced a common structured naming convention for all\r
+performance and functional tests. This change was driven by substantially\r
+growing number and type of CSIT test cases. Firstly, the original practice did\r
+not always follow any strict naming convention. Secondly test names did not\r
+always clearly capture tested packet encapsulations, and the actual type or\r
+content of the tests. Thirdly HW configurations in terms of NICs, ports and\r
+their locality were not captured either. These were but few reasons that drove\r
+the decision to change and define a new more complete and stricter test naming\r
+convention, and to apply this to all existing and new test cases.\r
+\r
+The new naming should be intuitive for majority of the tests. The complete\r
+description of CSIT test naming convention is provided on `CSIT test naming wiki\r
+<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
+\r
+Here few illustrative examples of the new naming usage for performance test\r
+suites:\r
+\r
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
+\r
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*\r
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on\r
+ Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching\r
+ with MAC learning, NDR throughput discovery.\r
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE\r
+ on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline\r
+ switching with MAC learning, NDR throughput discovery.\r
+ - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel\r
+ x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.\r
+ - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on\r
+ Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput\r
+ discovery.\r
+\r
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
+\r
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-\r
+ VirtPortConfig-VMconfig-TestType*\r
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports\r
+ of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain\r
+ switching to/from two vhost interfaces and one VM, NDR throughput\r
+ discovery.\r
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2\r
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
+ switching to/from two vhost interfaces and one VM, NDR throughput\r
+ discovery.\r
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2\r
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
+ switching to/from four vhost interfaces and two VMs, NDR throughput\r
+ discovery.\r
+\r
+Methodology: Multi-Thread and Multi-Core\r
+----------------------------------------\r
+\r
+**HyperThreading** - CSIT |release| performance tests are executed with SUT\r
+servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS\r
+settings). This is the simplest configuration used to establish baseline\r
+single-thread single-core SW packet processing and forwarding performance.\r
+Subsequent releases of CSIT will add performance tests with Intel\r
+HyperThreading Enabled (requires BIOS settings change and hard reboot).\r
+\r
+**Multi-core Test** - CSIT |release| multi-core tests are executed in the\r
+following VPP thread and core configurations:\r
+\r
+#. 1t1c - 1 VPP worker thread on 1 CPU physical core.\r
+#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.\r
+#. 4t4c - 4 VPP threads on 4 CPU physical cores.\r
+\r
+Note that in quite a few test cases running VPP on 2 or 4 physical cores hits\r
+the tested NIC I/O bandwidth or packets-per-second limit.\r
+\r
+Methodology: Packet Throughput\r
+------------------------------\r
+\r
+Following values are measured and reported for packet throughput tests:\r
+\r
+- NDR binary search per RFC2544:\r
+\r
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps\r
+ (2x <per direction packets-per-second>)"\r
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
+ second> Gbps (untagged)"\r
+\r
+- PDR binary search per RFC2544:\r
+\r
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x\r
+ <per direction packets-per-second>)"\r
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
+ second> Gbps (untagged)"\r
+ - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets\r
+ lost at PDR rate>""\r
+\r
+- NDR and PDR are measured for the following L2 frame sizes:\r
+\r
+ - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.\r
+ - IPv6: 78B, 1518B, 9000B.\r
+\r
+\r
+Methodology: Packet Latency\r
+---------------------------\r
+\r
+TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported\r
+latency values are measured using following methodology:\r
+\r
+- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)\r
+ for each NDR throughput test and packet size (except IMIX).\r
+- TG sends dedicated latency streams, one per direction, each at the rate of\r
+ 10kpps at the prescribed packet size; these are sent in addition to the main\r
+ load streams.\r
+- TG reports min/avg/max latency values per stream direction, hence two sets\r
+ of latency values are reported per test case; future release of TRex is\r
+ expected to report latency percentiles.\r
+- Reported latency values are aggregate across two SUTs due to three node\r
+ topology used for all performance tests; for per SUT latency, reported value\r
+ should be divided by two.\r
+- 1usec is the measurement accuracy advertised by TRex TG for the setup used in\r
+ FD.io labs used by CSIT project.\r
+- TRex setup introduces an always-on error of about 2*2usec per latency flow -\r
+ additonal Tx/Rx interface latency induced by TRex SW writing and reading\r
+ packet timestamps on CPU cores without HW acceleration on NICs closer to the\r
+ interface line.\r
+\r
+\r
+Methodology: KVM VM vhost\r
+-------------------------\r
+\r
+CSIT |release| introduced environment configuration changes to KVM Qemu vhost-\r
+user tests in order to more representatively measure VPP-17.01 performance in\r
+configurations with vhost-user interfaces and VMs.\r
+\r
+Current setup of CSIT FD.io performance lab is using tuned settings for more\r
+optimal performance of KVM Qemu:\r
+\r
+- Default Qemu virtio queue size of 256 descriptors.\r
+- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki\r
+ page: https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604.\r
+\r
+Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance\r
+of VPP+VM system less sensitive to other Linux OS system tasks by reducing\r
+their interference on CPU cores that are designated for critical software\r
+tasks under test, namely VPP worker threads in host and Testpmd threads in\r
+guest dealing with data plan.\r
+++ /dev/null
-Documentation\r
-=============\r
-\r
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.perf.html.\r
+++ /dev/null
-Overview\r
-========\r
-\r
-Tested Topologies HW\r
---------------------\r
-\r
-CSIT VPP performance tests are executed on physical baremetal servers hosted by LF\r
-FD.io project. Testbed physical topology is shown in the figure below.\r
-\r
-::\r
-\r
- +------------------------+ +------------------------+\r
- | | | |\r
- | +------------------+ | | +------------------+ |\r
- | | | | | | | |\r
- | | <-----------------> | |\r
- | | DUT1 | | | | DUT2 | |\r
- | +--^---------------+ | | +---------------^--+ |\r
- | | | | | |\r
- | | SUT1 | | SUT2 | |\r
- +------------------------+ +------------------^-----+\r
- | |\r
- | |\r
- | +-----------+ |\r
- | | | |\r
- +------------------> TG <------------------+\r
- | |\r
- +-----------+\r
-\r
-SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two\r
-Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with\r
-two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a\r
-Device Under Test (DUT). TG runs TRex SW application as a packet Traffic\r
-Generator. Physical connectivity between SUTs and to TG is provided using\r
-different NIC models that need to be tested for performance. Currently\r
-installed and tested NIC models include:\r
-\r
-#. 2port10GE X520-DA2 Intel.\r
-#. 2port10GE X710 Intel.\r
-#. 2port10GE VIC1227 Cisco.\r
-#. 2port40GE VIC1385 Cisco.\r
-#. 2port40GE XL710 Intel.\r
-\r
-Detailed LF FD.io test bed specification and physical topology are described\r
-in `wiki CSIT LF testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
-\r
-For test cases that require DUT (VPP) to communicate with VM over vhost-user\r
-interfaces, a VM is created on SUT1 and SUT2. DUT (VPP) test topology with VM\r
-is shown in the figure below including applicable packet flow thru the VM\r
-(marked in the figure with ``***``).\r
-\r
-::\r
-\r
- +------------------------+ +------------------------+\r
- | +----------+ | | +----------+ |\r
- | | VM | | | | VM | |\r
- | | ****** | | | | ****** | |\r
- | +--^----^--+ | | +--^----^--+ |\r
- | *| |* | | *| |* |\r
- | +------v----v------+ | | +------v----v------+ |\r
- | | * * |**|***********|**| * * | |\r
- | | ***** *******<----------------->******* ***** | |\r
- | | * DUT1 | | | | DUT2 * | |\r
- | +--^---------------+ | | +---------------^--+ |\r
- | *| | | |* |\r
- | *| SUT1 | | SUT2 |* |\r
- +------------------------+ +------------------^-----+\r
- *| |*\r
- *| |*\r
- *| +-----------+ |*\r
- *| | | |*\r
- *+------------------> TG <------------------+*\r
- ******************* | |********************\r
- +-----------+\r
-\r
-Note that for VM tests, packets are switched by DUT (VPP) twice, hence the\r
-throughput rates measured by TG (and listed in this report) must be multiplied\r
-by two to represent the actual DUT aggregate packet forwarding rate.\r
-\r
-VPP Performance Tests Overview\r
-------------------------------\r
-\r
-Performance tests are split into the two main categories:\r
-\r
-- Throughput discovery - discovery of packet forwarding rate using binary search\r
- in accordance to RFC2544.\r
-\r
- - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;\r
- followed by packet one-way latency measurements at 10%, 50% and 100% of\r
- discovered NDR throughput.\r
- - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss\r
- currently set to 0.5%; followed by packet one-way latency measurements at\r
- 100% of discovered PDR throughput.\r
-\r
-- Throughput verification - verification of packet forwarding rate against\r
- previously discovered throughput rate. These tests are currently done against\r
- 0.9 of reference NDR, with reference rates updated periodically.\r
-\r
-CSIT |release| includes following performance test suites, listed per NIC type:\r
-\r
-- 2port10GE X520-DA2 Intel\r
-\r
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad\r
- VLAN tagged Ethernet frames.\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning; disabled MAC learning i.e. static MAC tests to be added.\r
- - **IPv4** - IPv4 routed-forwarding.\r
- - **IPv6** - IPv6 routed-forwarding.\r
- - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.\r
- - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Cross-Connect, L2\r
- Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.\r
- - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.\r
- - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.\r
- - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4,\r
- IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.\r
- - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.\r
- - **QoS Policer** - ingress packet rate measuring, marking and limiting\r
- (IPv4).\r
-\r
-- 2port40GE XL710 Intel\r
-\r
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
- - **IPv4** - IPv4 routed-forwarding.\r
- - **IPv6** - IPv6 routed-forwarding.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Bridge-Domain.\r
-\r
-- 2port10GE X710 Intel\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Bridge-Domain.\r
-\r
-- 2port10GE VIC1227 Cisco\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
-\r
-- 2port40GE VIC1385 Cisco\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
-\r
-Execution of performance tests takes time, especially the throughput discovery\r
-tests. Due to limited HW testbed resources available within FD.io labs hosted\r
-by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.\r
-Niantic) has been limited to few baseline tests. Over time we expect the HW\r
-testbed resources to grow, and will be adding complete set of performance\r
-tests for all models of hardware to be executed regularly and(or)\r
-continuously.\r
-\r
--- /dev/null
+Documentation\r
+=============\r
+\r
+For complete description of the VPP Test Framework including anatomy of a test\r
+case and detailed documentation of existing VPP unit test cases please refer\r
+to the `VPP Unit Test Documentation <link>`_\r
--- /dev/null
+VPP Unit Tests
+==============
+
+.. toctree::
+
+ overview
+ documentation
+
--- /dev/null
+Overview\r
+========\r
+\r
+.. note::\r
+\r
+ This section includes an abbreviated version of the VPP Test Framework\r
+ overview maintained within the VPP project. Complete overview can be found\r
+ in `VPP unit test documentation <link>`_\r
+\r
+VPP Unit Test Framework\r
+-----------------------\r
+\r
+VPP Test Framework is used to ease writing, running and debugging unit tests\r
+for the VPP. It is based on python as a high level language to allow rapid\r
+test development. scapy\_ is used as a tool for creating and dissecting\r
+packets.\r
+\r
+VPP Test Framework does not send any packets to VPP directly. Traffic is\r
+instead injected using VPP packet-generator interfaces. Packets are written\r
+into a temporary .pcap file, which is then read by the VPP code with packets\r
+getting injected into the VPP processing nodes.\r
+\r
+Similarly, VPP does not send any packets to VPP Test Framework directly.\r
+Instead, VPP packet capture feature is used to capture and write packets to a\r
+temporary .pcap file, which is then read and analyzed by the VPP Test\r
+Framework.\r
+\r
+For complete description of the VPP Test Framework including anatomy of a test\r
+case and detailed documentation of existing VPP unit test cases please refer\r
+to the `VPP unit test documentation <link>`_\r
+\r
+Unit Tests Coverage\r
+-------------------\r
+\r
+Following VPP functional test areas are covered in VPP unit test code included\r
+in VPP rls1701 with results listed in this report:\r
+\r
+- CRUD Loopback - create, read, update, delete Loopback interfaces.\r
+- Flow-per-packet plugin - collect and report L2 and IP4 flow statistics.\r
+- DHCP - DHCPv4 and DHCPv6 Proxy.\r
+- IP4 VRF Multi-instance - create, read, update, delete and verify IPv4 VRFs.\r
+- Load Balancer - IP4 GRE4, IP4 GRE6, IP6 GRE4, IP6 GRE6\r
+- Bidirectional Forwarding Detection (BFD) IPv4 - baseline BFD session operation.\r
+- Bidirectional Forwarding Detection (BFD) IPv6 - baseline BFD session operation.\r
+- Bidirectional Forwarding Detection (BFD) - API tests.\r
+- Bidirectional Forwarding Detection (BFD) - changing authorization.\r
+- Bidirectional Forwarding Detection (BFD) - SHA1 authentication.\r
+- IPv6 Tests - baseline FIB operations, NS/RS exception handling.\r
+- SPAN Test - Switched Port Analyzer packet mirroring.\r
+- GRE Tests - GRE tunnel tests.\r
+- SNAT Test Cases - SNAT44 tests.\r
+- L2XC Multi-instance - L2 cross-connect multi-instance tests.\r
+- IPv4 Tests - baseline FIB tests.\r
+- IPv4 FIB CRUD - add/update/delete IPv4 routes.\r
+- IRB Tests - Integrated Routing and Bridging tests.\r
+- MPLS Tests - MPLS baseline tests.\r
+- L2XC Tests - L2 cross-connect baseline tests incl. single- and dual-loop.\r
+- Classifier - classification with IP ACL, MAC ACL, IP PBR.\r
+- VXLAN Tests - baseline VXLAN tunneling.\r
+- L2 FIB CRUD - add/update/delete L2 MAC entries.\r
+- L2BD Tests - L2 Bridge-Domain baseline tests incl. single- and dual-loop.\r
JEN_FILE_PERF='output_perf_data.xml'
JEN_JOB='csit-vpp-perf-1701-all'
-JEN_BUILD=(3 4 7)
+JEN_BUILD=(3 4 7 8)
for i in "${JEN_BUILD[@]}"; do
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
done
JEN_JOB='csit-vpp-perf-1701-long'
-JEN_BUILD=(2 4)
+JEN_BUILD=(2 4 5 6)
for i in "${JEN_BUILD[@]}"; do
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
done
JEN_JOB='csit-dpdk-perf-1701-all'
JEN_BUILD=(2 3)
for i in "${JEN_BUILD[@]}"; do
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_TESTPMD}/${JEN_JOB}-${i}.xml
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_TESTPMD}/${JEN_JOB}-${i}.xml
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
done
JEN_JOB='csit-vpp-functional-1701-virl'
JEN_BUILD=(18)
for i in "${JEN_BUILD[@]}"; do
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
done
JEN_URL='https://jenkins.fd.io/view/hc2vpp/job'
JEN_BUILD=(1)
for i in "${JEN_BUILD[@]}"; do
- wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
+ curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip
done
Code Review / csit.git / commitdiff