4 .. _test_environment_versioning:
9 CSIT test environment versioning has been introduced to track
10 modifications of the test environment.
12 Any benchmark anomalies (progressions, regressions) between releases of
13 a DUT application (e.g. VPP, DPDK), are determined by testing it in the
14 same test environment, to avoid test environment changes clouding the
16 To beter distinguish impact of test environment changes,
17 we also execute tests without any SUT (just with TRex TG sending packets
18 over a link looping back to TG).
20 A mirror approach is introduced to determine benchmarking anomalies due
21 to the test environment change. This is achieved by testing the same DUT
22 application version between releases of CSIT test system. This works
23 under the assumption that the behaviour of the DUT is deterministic
24 under the test conditions.
26 CSIT test environment versioning scheme ensures integrity of all the
27 test system components, including their HW revisions, compiled SW code
28 versions and SW source code, within a specific CSIT version. Components
29 included in the CSIT environment versioning include:
31 - **HW** Server hardware firmware and BIOS (motherboard, processsor,
32 NIC(s), accelerator card(s)), tracked in CSIT branch in
33 :file:`./docs/lab/<server_platform_name>_hw_bios_cfg.md`, e.g. `Xeon
35 <https://git.fd.io/csit/tree/docs/lab/testbeds_sm_skx_hw_bios_cfg.md#n556>`_.
36 - **Linux** Server Linux OS version and configuration, tracked in CSIT
37 Reports in `SUT Settings
38 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_
39 and `Pre-Test Server Calibration
40 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#id21>`_.
41 - **TRex** TRex Traffic Generator version, drivers and configuration
42 tracked in `TG Settings
43 <https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_.
44 - **CSIT** CSIT framework code tracked in CSIT release branches.
46 Following is the list of CSIT versions to date:
48 - Ver. 1 associated with CSIT rls1908 branch (`HW
49 <https://git.fd.io/csit/tree/docs/lab?h=rls1908>`_, `Linux
50 <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
52 <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
53 `CSIT <https://git.fd.io/csit/tree/?h=rls1908>`_).
54 - Ver. 2 associated with CSIT rls2001 branch (`HW
55 <https://git.fd.io/csit/tree/docs/lab?h=rls2001>`_, `Linux
56 <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
58 <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
59 `CSIT <https://git.fd.io/csit/tree/?h=rls2001>`_).
60 - Ver. 4 associated with CSIT rls2005 branch (`HW
61 <https://git.fd.io/csit/tree/docs/lab?h=rls2005>`_, `Linux
62 <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
64 <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
65 `CSIT <https://git.fd.io/csit/tree/?h=rls2005>`_).
66 - Ver. 5 associated with CSIT rls2009 branch (`HW
67 <https://git.fd.io/csit/tree/docs/lab?h=rls2009>`_, `Linux
68 <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
70 <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
71 `CSIT <https://git.fd.io/csit/tree/?h=rls2009>`_).
73 - The main change is TRex data-plane core resource adjustments:
74 `increase from 7 to 8 cores and pinning cores to interfaces <https://gerrit.fd.io/r/c/csit/+/28184>`_
75 for better TRex performance with symmetric traffic profiles.
76 - Ver. 6 associated with CSIT rls2101 branch (`HW
77 <https://git.fd.io/csit/tree/docs/lab?h=rls2101>`_, `Linux
78 <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
80 <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
81 `CSIT <https://git.fd.io/csit/tree/?h=rls2101>`_).
83 - The main change is TRex version upgrade:
84 `increase from 2.82 to 2.86 <https://gerrit.fd.io/r/c/csit/+/29980>`_.
85 - Ver. 7 associated with CSIT rls2106 branch (`HW
86 <https://git.fd.io/csit/tree/docs/lab?h=rls2106>`_, `Linux
87 <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
89 <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
90 `CSIT <https://git.fd.io/csit/tree/?h=rls2106>`_).
92 - TRex version upgrade:
93 `increase from 2.86 to 2.88 <https://gerrit.fd.io/r/c/csit/+/31652>`_.
95 `upgrade from 18.04 LTS to 20.04.2 LTS <https://gerrit.fd.io/r/c/csit/+/31290>`_.
96 - Ver. 8 associated with CSIT rls2110 branch (`HW
97 <https://git.fd.io/csit/tree/docs/lab?h=rls2110>`_, `Linux
98 <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
100 <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
101 `CSIT <https://git.fd.io/csit/tree/?h=rls2110>`_).
103 - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
105 - Ver. 9 associated with CSIT rls2202 branch (`HW
106 <https://git.fd.io/csit/tree/docs/lab?h=rls2202>`_, `Linux
107 <https://s3-docs.fd.io/csit/rls2202/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
109 <https://s3-docs.fd.io/csit/rls2202/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
110 `CSIT <https://git.fd.io/csit/tree/?h=rls2202>`_).
112 - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
114 - Ver. 10 associated with CSIT rls2206 branch (`HW
115 <https://git.fd.io/csit/tree/docs/lab?h=rls2206>`_, `Linux
116 <https://s3-docs.fd.io/csit/rls2206/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
118 <https://s3-docs.fd.io/csit/rls2206/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
119 `CSIT <https://git.fd.io/csit/tree/?h=rls2206>`_).
121 - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
123 - Mellanox 556A series firmware upgrade based on DPDK compatibility
125 - Intel IceLake all core turbo frequency turned off. Current base frequency