X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Freport%2Fintroduction%2Fphysical_testbeds.rst;h=5a9b1edf97158826dbb84f1f4e6f679777dccee7;hb=3055ecfbb0b619318f7cfbbb18ba142968b4c617;hp=159728d8ee06131df29b29764ecbb825aec4bff2;hpb=25a8fe0aa594ca010b7f1aad449a2af2b6625bd7;p=csit.git diff --git a/docs/report/introduction/physical_testbeds.rst b/docs/report/introduction/physical_testbeds.rst index 159728d8ee..5a9b1edf97 100644 --- a/docs/report/introduction/physical_testbeds.rst +++ b/docs/report/introduction/physical_testbeds.rst @@ -1,7 +1,7 @@ .. _tested_physical_topologies: -Physical Testbeds -================= +Performance Physical Testbeds +============================= All :abbr:`FD.io (Fast Data Input/Ouput)` :abbr:`CSIT (Continuous System Integration and Testing)` performance test results included in this @@ -26,10 +26,10 @@ Two physical server topology types are used: Current FD.io production testbeds are built with SUT servers based on the following processor architectures: -- Intel Xeon: Skylake Platinum 8180, Haswell-SP E5-2699v3, - Cascade Lake Platinum 8280, Cascade Lake 6252N. -- Intel Atom: Denverton C3858. -- ARM: TaiShan 2280, hip07-d05. +- Intel Xeon: Cascadelake 6252N, Icelake 8358. +- Intel Atom: Denverton C3858, Snowridge P5362. +- Arm: TaiShan 2280, hip07-d05, Neoverse N1. +- AMD EPYC: Zen2 7532. Server SUT performance depends on server and processor type, hence results for testbeds based on different servers must be reported @@ -39,10 +39,93 @@ Complete technical specifications of compute servers used in CSIT physical testbeds are maintained in FD.io CSIT repository: https://git.fd.io/csit/tree/docs/lab/testbed_specifications.md. -Following is the description of existing production testbeds. +Physical NICs and Drivers +------------------------- + +SUT and TG servers are equipped with a number of different NIC models. + +VPP is performance tested on SUTs with the following NICs and drivers: + +#. 2p10GE: x550, x553 Intel (codename Niantic) + - DPDK Poll Mode Driver (PMD). +#. 4p10GE: x710-DA4 Intel (codename Fortville, FVL) + - DPDK PMD. + - AVF in PMD mode. + - AF_XDP in PMD mode. +#. 2p25GE: xxv710-DA2 Intel (codename Fortville, FVL) + - DPDK PMD. + - AVF in PMD mode. + - AF_XDP in PMD mode. +#. 4p25GE: xxv710-DA4 Intel (codename Fortville, FVL) + - DPDK PMD. + - AVF in PMD mode. + - AF_XDP in PMD mode. +#. 4p25GE: E822-CQDA4 Intel (codename Columbiaville, CVL) + - DPDK PMD. + - AVF in PMD mode. +#. 2p100GE: cx556a-edat Mellanox ConnectX5 + - RDMA_core in PMD mode. +#. 2p100GE: E810-2CQDA2 Intel (codename Columbiaville, CVL) + - DPDK PMD. + - AVF in PMD mode. + +DPDK applications, testpmd and l3fwd, are performance tested on the same +SUTs exclusively with DPDK drivers for all NICs. + +TRex running on TGs is using DPDK drivers for all NICs. + +VPP hoststack tests utilize ab (Apache HTTP server benchmarking tool) +running on TGs and using Linux drivers for all NICs. + +For more information see :ref:`vpp_test_environment` +and :ref:`dpdk_test_environment`. + +.. _physical_testbeds_2n_zn2: + +2-Node AMD EPYC Zen2 (2n-zn2) +----------------------------- + +One 2n-zn2 testbed in in operation in FD.io labs. It is built based on +two SuperMicro SuperMicro AS-1114S-WTRT servers, with SUT and TG servers +equipped with one AMD EPYC Zen2 7532 processor each (256 MB Cache, 2.40 +GHz, 32 cores). 2n-zn2 physical topology is shown below. -2-Node Xeon Cascade Lake (2n-clx) ---------------------------------- +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/introduction/}} + \includegraphics[width=0.90\textwidth]{testbed-2n-zn2} + \label{fig:testbed-2n-zn2} + \end{figure} + +.. only:: html + + .. figure:: testbed-2n-zn2.svg + :alt: testbed-2n-zn2 + :align: center + +SUT NICs: + +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. + +TG NICs: + +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. + +All AMD EPYC Zen2 7532 servers run with AMD SMT enabled, doubling the +number of logical cores exposed to Linux. + +.. _physical_testbeds_2n_clx: + +2-Node Xeon Cascadelake (2n-clx) +-------------------------------- Three 2n-clx testbeds are in operation in FD.io labs. Each 2n-clx testbed is built with two SuperMicro SYS-7049GP-TRT servers, SUTs are equipped with two @@ -67,35 +150,35 @@ Cache, 2.70 GHz, 28 cores). 2n-clx physical topology is shown below. :alt: testbed-2n-clx :align: center -SUT servers are populated with the following NIC models: +SUT NICs: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. -TG servers run T-Rex application and are populated with the following -NIC models: +TG NICs: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) -All Intel Xeon Cascade Lake servers run with Intel Hyper-Threading enabled, +All Intel Xeon Cascadelake servers run with Intel Hyper-Threading enabled, doubling the number of logical cores exposed to Linux. -2-Node Xeon Skylake (2n-skx) +.. _physical_testbeds_2n_icx: + +2-Node Xeon Icelake (2n-icx) ---------------------------- -Four 2n-skx testbeds are in operation in FD.io labs. Each 2n-skx testbed -is built with two SuperMicro SYS-7049GP-TRT servers, each in turn -equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 MB -Cache, 2.50 GHz, 28 cores). 2n-skx physical topology is shown below. +One 2n-icx testbed is in operation in FD.io labs. It is built with two +SuperMicro SYS-740GP-TNRT servers, each in turn equipped with two Intel Xeon +Platinum 8358 processors (48 MB Cache, 2.60 GHz, 32 cores). .. only:: latex @@ -104,46 +187,33 @@ Cache, 2.50 GHz, 28 cores). 2n-skx physical topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-2n-skx} - \label{fig:testbed-2n-skx} + \includegraphics[width=0.90\textwidth]{testbed-2n-icx} + \label{fig:testbed-2n-icx} \end{figure} .. only:: html - .. figure:: testbed-2n-skx.svg - :alt: testbed-2n-skx + .. figure:: testbed-2n-icx.svg + :alt: testbed-2n-icx :align: center -SUT servers are populated with the following NIC models: +SUT and TG NICs: -#. NIC-1: x710-DA4 4p10GE Intel. -#. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) -#. NIC-4: empty, future expansion. -#. NIC-5: empty, future expansion. -#. NIC-6: empty, future expansion. - -TG servers run T-Rex application and are populated with the following -NIC models: +#. NIC-1: xxv710-DA2 2p25GE Intel. +#. NIC-2: E810-2CQDA2 2p100GbE Intel (* to be added). +#. NIC-3: E810-CQDA4 4p100GbE Intel (* to be added). -#. NIC-1: x710-DA4 4p10GE Intel. -#. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) -#. NIC-4: empty, future expansion. -#. NIC-5: empty, future expansion. -#. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) +All Intel Xeon Icelake servers run with Intel Hyper-Threading enabled, +doubling the number of logical cores exposed to Linux. -All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, -doubling the number of logical cores exposed to Linux, with 56 logical -cores and 28 physical cores per processor socket. +.. _physical_testbeds_3n_icx: -3-Node Xeon Skylake (3n-skx) +3-Node Xeon Icelake (3n-icx) ---------------------------- -Two 3n-skx testbeds are in operation in FD.io labs. Each 3n-skx testbed -is built with three SuperMicro SYS-7049GP-TRT servers, each in turn -equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 MB -Cache, 2.50 GHz, 28 cores). 3n-skx physical topology is shown below. +One 3n-icx testbed is in operation in FD.io labs. It is built with three +SuperMicro SYS-740GP-TNRT servers, each in turn equipped with two Intel Xeon +Platinum 8358 processors (48 MB Cache, 2.60 GHz, 32 cores). .. only:: latex @@ -152,46 +222,35 @@ Cache, 2.50 GHz, 28 cores). 3n-skx physical topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-skx} - \label{fig:testbed-3n-skx} + \includegraphics[width=0.90\textwidth]{testbed-3n-icx} + \label{fig:testbed-3n-icx} \end{figure} .. only:: html - .. figure:: testbed-3n-skx.svg - :alt: testbed-3n-skx + .. figure:: testbed-3n-icx.svg + :alt: testbed-3n-icx :align: center -SUT1 and SUT2 servers are populated with the following NIC models: - -#. NIC-1: x710-DA4 4p10GE Intel. -#. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: empty, future expansion. -#. NIC-4: empty, future expansion. -#. NIC-5: empty, future expansion. -#. NIC-6: empty, future expansion. +SUT and TG NICs: -TG servers run T-Rex application and are populated with the following -NIC models: +#. NIC-1: xxv710-DA2 2p25GE Intel. +#. NIC-2: E810-2CQDA2 2p100GbE Intel (* to be added). +#. NIC-3: E810-CQDA4 4p100GbE Intel (* to be added). -#. NIC-1: x710-DA4 4p10GE Intel. -#. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: empty, future expansion. -#. NIC-4: empty, future expansion. -#. NIC-5: empty, future expansion. -#. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) +All Intel Xeon Icelake servers run with Intel Hyper-Threading enabled, +doubling the number of logical cores exposed to Linux. -All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, -doubling the number of logical cores exposed to Linux, with 56 logical -cores and 28 physical cores per processor socket. +.. _physical_testbeds_3n_alt: -3-Node Xeon Haswell (3n-hsw) ----------------------------- +3-Node ARM Altra (3n-alt) +--------------------------- -Three 3n-hsw testbeds are in operation in FD.io labs. Each 3n-hsw -testbed is built with three Cisco UCS-c240m3 servers, each in turn -equipped with two Intel Xeon Haswell-SP E5-2699v3 processors (45 MB -Cache, 2.3 GHz, 18 cores). 3n-hsw physical topology is shown below. +One 3n-tsh testbed is built with: i) one SuperMicro SYS-740GP-TNRT +server acting as TG and equipped with two Intel Xeon Icelake Platinum +8358 processors (80 MB Cache, 2.60 GHz, 32 cores), and ii) one Ampere +Altra server acting as SUT and equipped with two Q80-30 processors +(80* ARM Neoverse N1). 3n-alt physical topology is shown below. .. only:: latex @@ -200,47 +259,37 @@ Cache, 2.3 GHz, 18 cores). 3n-hsw physical topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-hsw} - \label{fig:testbed-3n-hsw} + \includegraphics[width=0.90\textwidth]{testbed-3n-alt} + \label{fig:testbed-3n-alt} \end{figure} .. only:: html - .. figure:: testbed-3n-hsw.svg - :alt: testbed-3n-hsw + .. figure:: testbed-3n-alt.svg + :alt: testbed-3n-alt :align: center -SUT1 and SUT2 servers are populated with the following NIC models: +SUT1 and SUT2 NICs: -#. NIC-1: VIC 1385 2p40GE Cisco. -#. NIC-2: NIC x520 2p10GE Intel. -#. NIC-3: empty. -#. NIC-4: NIC xl710-QDA2 2p40GE Intel. -#. NIC-5: NIC x710-DA2 2p10GE Intel. -#. NIC-6: QAT 8950 50G (Walnut Hill) Intel. +#. NIC-1: xl710-QDA2-2p40GE Intel. -TG servers run T-Rex application and are populated with the following -NIC models: +TG NICs: -#. NIC-1: NIC xl710-QDA2 2p40GE Intel. -#. NIC-2: NIC x710-DA2 2p10GE Intel. -#. NIC-3: empty. -#. NIC-4: NIC xl710-QDA2 2p40GE Intel. -#. NIC-5: NIC x710-DA2 2p10GE Intel. -#. NIC-6: NIC x710-DA2 2p10GE Intel. (For self-tests.) +#. NIC-1: xxv710-DA2-2p25GE Intel. +#. NIC-2: xl710-QDA2-2p40GE Intel. +#. NIC-3: e810-XXVDA4-4p25GE Intel. +#. NIC-4: e810-2CQDA2-2p100GE Intel. -All Intel Xeon Haswell servers run with Intel Hyper-Threading disabled, -making the number of logical cores exposed to Linux match the number of -18 physical cores per processor socket. +.. _physical_testbeds_3n_tsh: -2-Node Atom Denverton (2n-dnv) ------------------------------- +3-Node ARM TaiShan (3n-tsh) +--------------------------- -2n-dnv testbed is built with: i) one Intel S2600WFT server acting as TG -and equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 -MB Cache, 2.50 GHz, 28 cores), and ii) one SuperMicro SYS-E300-9A server -acting as SUT and equipped with one Intel Atom C3858 processor (12 MB -Cache, 2.00 GHz, 12 cores). 2n-dnv physical topology is shown below. +One 3n-tsh testbed is built with: i) one SuperMicro SYS-7049GP-TRT +server acting as TG and equipped with two Intel Xeon Skylake Platinum +8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one Huawei +TaiShan 2280 server acting as SUT and equipped with one hip07-d05 +processor (64* ARM Cortex-A72). 3n-tsh physical topology is shown below. .. only:: latex @@ -249,42 +298,37 @@ Cache, 2.00 GHz, 12 cores). 2n-dnv physical topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-2n-dnv} - \label{fig:testbed-2n-dnv} + \includegraphics[width=0.90\textwidth]{testbed-3n-tsh} + \label{fig:testbed-3n-tsh} \end{figure} .. only:: html - .. figure:: testbed-2n-dnv.svg - :alt: testbed-2n-dnv + .. figure:: testbed-3n-tsh.svg + :alt: testbed-3n-tsh :align: center -SUT server have four internal 10G NIC port: +SUT1 and SUT2 NICs: -#. P-1: x553 copper port. -#. P-2: x553 copper port. -#. P-3: x553 fiber port. -#. P-4: x553 fiber port. +#. NIC-1: connectx4 2p25GE Mellanox. +#. NIC-2: x520 2p10GE Intel. -TG server run T-Rex software traffic generator and are populated with the -following NIC models: +TG NICs: -#. NIC-1: x550-T2 2p10GE Intel. -#. NIC-2: x550-T2 2p10GE Intel. -#. NIC-3: x520-DA2 2p10GE Intel. -#. NIC-4: x520-DA2 2p10GE Intel. +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-3: xl710-QDA2 2p40GE Intel. -The 2n-dnv testbed is in operation in Intel SH labs. +.. _physical_testbeds_2n_tx2: -3-Node Atom Denverton (3n-dnv) ------------------------------- +2-Node ARM ThunderX2 (2n-tx2) +----------------------------- -One 3n-dnv testbed is built with: i) one SuperMicro SYS-7049GP-TRT +One 2n-tx2 testbed is built with: i) one SuperMicro SYS-7049GP-TRT server acting as TG and equipped with two Intel Xeon Skylake Platinum -8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one -SuperMicro SYS-E300-9A server acting as SUT and equipped with one Intel -Atom C3858 processor (12 MB Cache, 2.00 GHz, 12 cores). 3n-dnv physical -topology is shown below. +8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one Marvell +ThnderX2 9975 (28* ThunderX2) server acting as SUT and equipped with two +ThunderX2 ARMv8 CN9975 processors. 2n-tx2 physical topology is shown below. .. only:: latex @@ -293,34 +337,35 @@ topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-dnv} - \label{fig:testbed-3n-dnv} + \includegraphics[width=0.90\textwidth]{testbed-2n-tx2} + \label{fig:testbed-2n-tx2} \end{figure} .. only:: html - .. figure:: testbed-3n-dnv.svg - :alt: testbed-3n-dnv + .. figure:: testbed-2n-tx2.svg + :alt: testbed-2n-tx2 :align: center -SUT1 and SUT2 servers are populated with the following NIC models: +SUT NICs: -#. NIC-1: x553 2p10GE fiber Intel. -#. NIC-2: x553 2p10GE copper Intel. +#. NIC-1: xl710-QDA2 2p40GE Intel (not connected). +#. NIC-2: xl710-QDA2 2p40GE Intel. -TG servers run T-Rex application and are populated with the following -NIC models: +TG NICs: -#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-1: xl710-QDA2 2p40GE Intel. -3-Node ARM TaiShan (3n-tsh) ---------------------------- +.. _physical_testbeds_3n_snr: -One 3n-tsh testbed is built with: i) one SuperMicro SYS-7049GP-TRT -server acting as TG and equipped with two Intel Xeon Skylake Platinum -8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one Huawei -TaiShan 2280 server acting as SUT and equipped with one hip07-d05 -processor (64* ARM Cortex-A72). 3n-tsh physical topology is shown below. +3-Node Atom Snowridge (3n-snr) +------------------------------ + +One 3n-snr testbed is built with: i) one SuperMicro SYS-740GP-TNRT +server acting as TG and equipped with two Intel Xeon Icelake Platinum +8358 processors (48 MB Cache, 2.60 GHz, 32 cores), and ii) SUT equipped with +one Intel Atom P5362 processor (27 MB Cache, 2.20 GHz, 24 cores). 3n-snr +physical topology is shown below. .. only:: latex @@ -329,23 +374,20 @@ processor (64* ARM Cortex-A72). 3n-tsh physical topology is shown below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-tsh} - \label{fig:testbed-3n-tsh} + \includegraphics[width=0.90\textwidth]{testbed-3n-snr} + \label{fig:testbed-3n-snr} \end{figure} .. only:: html - .. figure:: testbed-3n-tsh.svg - :alt: testbed-3n-tsh + .. figure:: testbed-3n-snr.svg + :alt: testbed-3n-snr :align: center -SUT1 and SUT2 servers are populated with the following NIC models: +SUT1 and SUT2 NICs: -#. NIC-1: connectx4 2p25GE Mellanox. -#. NIC-2: x520 2p10GE Intel. +#. NIC-1: e822cq-DA4 4p25GE fiber Intel. -TG servers run T-Rex application and are populated with the following -NIC models: +TG NICs: -#. NIC-1: x710-DA4 4p10GE Intel. -#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-1: e810xxv-DA4 4p25GE Intel.