4 .. _test_environment_versioning:
9 CSIT test environment versioning has been introduced to track
10 modifications of the test environment.
12 Any benchmark anomalies (progressions, regressions) between releases of
13 a DUT application (e.g. VPP, DPDK), are determined by testing it in the
14 same test environment, to avoid test environment changes clouding the
17 A mirror approach is introduced to determine benchmarking anomalies due
18 to the test environment change. This is achieved by testing the same DUT
19 application version between releases of CSIT test system. This works
20 under the assumption that the behaviour of the DUT is deterministic
21 under the test conditions.
23 CSIT test environment versioning scheme ensures integrity of all the
24 test system components, including their HW revisions, compiled SW code
25 versions and SW source code, within a specific CSIT version. Components
26 included in the CSIT environment versioning include:
28 - **HW** Server hardware firmware and BIOS (motherboard, processsor,
29 NIC(s), accelerator card(s)), tracked in CSIT branch in
30 :file:`./docs/lab/<server_platform_name>_hw_bios_cfg.md`, e.g. `Xeon
32 <https://git.fd.io/csit/tree/docs/lab/testbeds_sm_skx_hw_bios_cfg.md#n556>`_.
33 - **Linux** Server Linux OS version and configuration, tracked in CSIT
34 Reports in `SUT Settings
35 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_
36 and `Pre-Test Server Calibration
37 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#id21>`_.
38 - **TRex** TRex Traffic Generator version, drivers and configuration
39 tracked in `TG Settings
40 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_.
41 - **CSIT** CSIT framework code tracked in CSIT release branches.
43 Following is the list of CSIT versions to date:
45 - Ver. 1 associated with CSIT rls1908 branch (`HW
46 <https://git.fd.io/csit/tree/docs/lab?h=rls1908>`_, `Linux
47 <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
49 <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
50 `CSIT <https://git.fd.io/csit/tree/?h=rls1908>`_).
51 - Ver. 2 associated with CSIT rls2001 branch (`HW
52 <https://git.fd.io/csit/tree/docs/lab?h=rls2001>`_, `Linux
53 <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
55 <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
56 `CSIT <https://git.fd.io/csit/tree/?h=rls2001>`_).
57 - Ver. 4 associated with CSIT rls2005 branch (`HW
58 <https://git.fd.io/csit/tree/docs/lab?h=rls2005>`_, `Linux
59 <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
61 <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
62 `CSIT <https://git.fd.io/csit/tree/?h=rls2005>`_).
63 - Ver. 5 associated with CSIT rls2009 branch (`HW
64 <https://git.fd.io/csit/tree/docs/lab?h=rls2009>`_, `Linux
65 <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
67 <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
68 `CSIT <https://git.fd.io/csit/tree/?h=rls2009>`_).
70 - The main change is TRex data-plane core resource adjustments:
71 `increase from 7 to 8 cores and pinning cores to interfaces <https://gerrit.fd.io/r/c/csit/+/28184>`_
72 for better TRex performance with symmetric traffic profiles.
73 - Ver. 6 associated with CSIT rls2101 branch (`HW
74 <https://git.fd.io/csit/tree/docs/lab?h=rls2101>`_, `Linux
75 <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
77 <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
78 `CSIT <https://git.fd.io/csit/tree/?h=rls2101>`_).
80 - The main change is TRex version upgrade:
81 `increase from 2.82 to 2.86 <https://gerrit.fd.io/r/c/csit/+/29980>`_.
82 - Ver. 7 associated with CSIT rls2106 branch (`HW
83 <https://git.fd.io/csit/tree/docs/lab?h=rls2106>`_, `Linux
84 <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
86 <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
87 `CSIT <https://git.fd.io/csit/tree/?h=rls2106>`_).
89 - TRex version upgrade:
90 `increase from 2.86 to 2.88 <https://gerrit.fd.io/r/c/csit/+/31652>`_.
92 `upgrade from 18.04 LTS to 20.04.2 LTS <https://gerrit.fd.io/r/c/csit/+/31290>`_.
93 - Ver. 8 associated with CSIT rls2110 branch (`HW
94 <https://git.fd.io/csit/tree/docs/lab?h=rls2110>`_, `Linux
95 <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
97 <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
98 `CSIT <https://git.fd.io/csit/tree/?h=rls2110>`_).
100 - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
101 matrix: `depends on testbed type <https://gerrit.fd.io/r/c/csit/+/33311>`_.
103 To identify performance changes due to VPP code development between previous
104 and current VPP release version, both have been tested in CSIT environment of
105 latest version and compared against each other. All substantial progressions and
106 regressions have been marked up with RCA analysis. See
107 :ref:`vpp_throughput_comparisons` and :ref:`vpp_known_issues`.
112 FD.io CSIT performance tests are executed in physical testbeds hosted by
113 :abbr:`LF (Linux Foundation)` for FD.io project. Two physical testbed
114 topology types are used:
116 - **3-Node Topology**: Consisting of two servers acting as SUTs
117 (Systems Under Test) and one server as TG (Traffic Generator), all
118 connected in ring topology.
119 - **2-Node Topology**: Consisting of one server acting as SUTs and one
120 server as TG both connected in ring topology.
122 Tested SUT servers are based on a range of processors including Intel
123 Intel Xeon Skylake-SP, Intel Xeon Cascade Lake-SP, Arm,
124 Intel Atom. More detailed description is provided in
125 :ref:`tested_physical_topologies`. Tested logical topologies are
126 described in :ref:`tested_logical_topologies`.
128 Server Specifications
129 ---------------------
131 Complete technical specifications of compute servers used in CSIT
132 physical testbeds are maintained in FD.io CSIT repository:
133 `FD.io CSIT testbeds - Xeon Cascade Lake`_,
134 `FD.io CSIT testbeds - Xeon Skylake, Arm, Atom`_.