docs/report/introduction/environment/intro.rst

   1 Test Environment
   2 ================
   3
   4 .. _test_environment_versioning:
   5
   6 Environment Versioning
   7 ----------------------
   8
   9 CSIT test environment versioning has been introduced to track
  10 modifications of the test environment.
  11
  12 Any benchmark anomalies (progressions, regressions) between releases of
  13 a DUT application (e.g. VPP, DPDK), are determined by testing it in the
  14 same test environment, to avoid test environment changes clouding the
  15 picture.
  16 To beter distinguish impact of test environment changes,
  17 we also execute tests without any SUT (just with TRex TG sending packets
  18 over a link looping back to TG).
  19
  20 A mirror approach is introduced to determine benchmarking anomalies due
  21 to the test environment change. This is achieved by testing the same DUT
  22 application version between releases of CSIT test system. This works
  23 under the assumption that the behaviour of the DUT is deterministic
  24 under the test conditions.
  25
  26 CSIT test environment versioning scheme ensures integrity of all the
  27 test system components, including their HW revisions, compiled SW code
  28 versions and SW source code, within a specific CSIT version. Components
  29 included in the CSIT environment versioning include:
  30
  31 - **HW** Server hardware firmware and BIOS (motherboard, processsor,
  32   NIC(s), accelerator card(s)), tracked in CSIT branch.
  33 - **Linux** Server Linux OS version and configuration, tracked in CSIT
  34   Reports.
  35 - **TRex** TRex Traffic Generator version, drivers and configuration
  36   tracked in TG Settings.
  37 - **CSIT** CSIT framework code tracked in CSIT release branches.
  38
  39 Following is the list of CSIT versions to date:
  40
  41 - Ver. 1 associated with CSIT rls1908 branch (`HW
  42   <https://git.fd.io/csit/tree/docs/lab?h=rls1908>`_, `Linux
  43   <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  44   `TRex
  45   <https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  46   `CSIT <https://git.fd.io/csit/tree/?h=rls1908>`_).
  47 - Ver. 2 associated with CSIT rls2001 branch (`HW
  48   <https://git.fd.io/csit/tree/docs/lab?h=rls2001>`_, `Linux
  49   <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  50   `TRex
  51   <https://docs.fd.io/csit/rls2001/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  52   `CSIT <https://git.fd.io/csit/tree/?h=rls2001>`_).
  53 - Ver. 4 associated with CSIT rls2005 branch (`HW
  54   <https://git.fd.io/csit/tree/docs/lab?h=rls2005>`_, `Linux
  55   <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  56   `TRex
  57   <https://docs.fd.io/csit/rls2005/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  58   `CSIT <https://git.fd.io/csit/tree/?h=rls2005>`_).
  59 - Ver. 5 associated with CSIT rls2009 branch (`HW
  60   <https://git.fd.io/csit/tree/docs/lab?h=rls2009>`_, `Linux
  61   <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  62   `TRex
  63   <https://docs.fd.io/csit/rls2009/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  64   `CSIT <https://git.fd.io/csit/tree/?h=rls2009>`_).
  65
  66   - The main change is TRex data-plane core resource adjustments:
  67     `increase from 7 to 8 cores and pinning cores to interfaces <https://gerrit.fd.io/r/c/csit/+/28184>`_
  68     for better TRex performance with symmetric traffic profiles.
  69 - Ver. 6 associated with CSIT rls2101 branch (`HW
  70   <https://git.fd.io/csit/tree/docs/lab?h=rls2101>`_, `Linux
  71   <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  72   `TRex
  73   <https://docs.fd.io/csit/rls2101/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  74   `CSIT <https://git.fd.io/csit/tree/?h=rls2101>`_).
  75
  76   - The main change is TRex version upgrade:
  77     `increase from 2.82 to 2.86 <https://gerrit.fd.io/r/c/csit/+/29980>`_.
  78 - Ver. 7 associated with CSIT rls2106 branch (`HW
  79   <https://git.fd.io/csit/tree/docs/lab?h=rls2106>`_, `Linux
  80   <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  81   `TRex
  82   <https://s3-docs.fd.io/csit/rls2106/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  83   `CSIT <https://git.fd.io/csit/tree/?h=rls2106>`_).
  84
  85   - TRex version upgrade: `increase from 2.86 to 2.88`_.
  86   - Ubuntu upgrade: `upgrade from 18.04 LTS to 20.04.2 LTS`_.
  87 - Ver. 8 associated with CSIT rls2110 branch (`HW
  88   <https://git.fd.io/csit/tree/docs/lab?h=rls2110>`_, `Linux
  89   <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  90   `TRex
  91   <https://s3-docs.fd.io/csit/rls2110/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
  92   `CSIT <https://git.fd.io/csit/tree/?h=rls2110>`_).
  93
  94   - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
  95     matrix.
  96 - Ver. 9 associated with CSIT rls2202 branch (`HW
  97   <https://git.fd.io/csit/tree/docs/lab?h=rls2202>`_, `Linux
  98   <https://s3-docs.fd.io/csit/rls2202/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
  99   `TRex
 100   <https://s3-docs.fd.io/csit/rls2202/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
 101   `CSIT <https://git.fd.io/csit/tree/?h=rls2202>`_).
 102
 103   - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
 104     matrix.
 105 - Ver. 10 associated with CSIT rls2206 branch (`HW
 106   <https://git.fd.io/csit/tree/docs/lab?h=rls2206>`_, `Linux
 107   <https://s3-docs.fd.io/csit/rls2206/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
 108   `TRex
 109   <https://s3-docs.fd.io/csit/rls2206/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
 110   `CSIT <https://git.fd.io/csit/tree/?h=rls2206>`_).
 111
 112   - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
 113     matrix.
 114   - Mellanox 556A series firmware upgrade based on DPDK compatibility
 115     matrix.
 116   - Intel IceLake all core turbo frequency turned off. Current base frequency
 117     is 2.6GHz.
 118
 119 - Ver. 11 associated with CSIT rls2210 branch (`HW
 120   <https://git.fd.io/csit/tree/docs/lab?h=rls2210>`_, `Linux
 121   <https://s3-docs.fd.io/csit/rls2210/report/vpp_performance_tests/test_environment.html#sut-settings-linux>`_,
 122   `TRex
 123   <https://s3-docs.fd.io/csit/rls2210/report/vpp_performance_tests/test_environment.html#tg-settings-trex>`_,
 124   `CSIT <https://git.fd.io/csit/tree/?h=rls2210>`_).
 125
 126   - Intel NIC 700/800 series firmware upgrade based on DPDK compatibility
 127     matrix.
 128   - Mellanox 556A series firmware upgrade based on DPDK compatibility
 129     matrix.
 130   - Ubuntu upgrade: `upgrade from 20.04.2 LTS to 22.04.1 LTS`_.
 131   - TRex version upgrade: `increase from 2.97 to 3.00`_.