X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fintroduction%2Fphysical_testbeds.rst;h=8860638d4301e7e7598d1b24e7d04252f0d54767;hp=ba6cd393896c7f10d9a75b902d8238faf81a2878;hb=c5e84fbee876a45d3495cde6f4e2d8140cacbe5a;hpb=2c55fe4a2d74352090b64118858a3557e197aeee diff --git a/docs/report/introduction/physical_testbeds.rst b/docs/report/introduction/physical_testbeds.rst index ba6cd39389..8860638d43 100644 --- a/docs/report/introduction/physical_testbeds.rst +++ b/docs/report/introduction/physical_testbeds.rst @@ -4,38 +4,51 @@ Physical Testbeds ================= All :abbr:`FD.io (Fast Data Input/Ouput)` :abbr:`CSIT (Continuous System -Integration and Testing)` performance testing listed in this report are -executed on physical testbeds built with bare-metal servers hosted by -:abbr:`LF (Linux Foundation)` FD.io project. - -Two testbed topologies are used: - -- **3-Node Topology**: Consisting of two servers acting as SUTs - (Systems Under Test) and one server as TG (Traffic Generator), all - connected in ring topology. Used for executing all of the data plane - tests including overlay tunnels and IPSec tests. -- **2-Node Topology**: Consisting of one server acting as SUTs (Systems - Under Test) and one server as TG (Traffic Generator), both connected - in ring topology. Used for execution of tests without any overlay - tunnel encapsulations. Added in CSIT rls18.07. - -Current FD.io production testbeds are built with servers based on two -processor generations of Intel Xeons: Haswell-SP (E5-2699v3) and Skylake -(Platinum 8180). Testbeds built with servers based on Arm processors are -in the process of being added to FD.io production. - -Server SUT and DUT performance depends on server and processor type, -hence results for testbeds based on different servers must be reported +Integration and Testing)` performance test results included in this +report are executed on the physical testbeds hosted by :abbr:`LF (Linux +Foundation)` FD.io project, unless otherwise noted. + +Two physical server topology types are used: + +- **2-Node Topology**: Consists of one server acting as a System Under + Test (SUT) and one server acting as a Traffic Generator (TG), with + both servers connected into a ring topology. Used for executing tests + that require frame encapsulations supported by TG. + +- **3-Node Topology**: Consists of two servers acting as a Systems Under + Test (SUTs) and one server acting as a Traffic Generator (TG), with + all servers connected into a ring topology. Used for executing tests + that require frame encapsulations not supported by TG e.g. certain + overlay tunnel encapsulations and IPsec. Number of native Ethernet, + IPv4 and IPv6 encapsulation tests are also executed on these testbeds, + for comparison with 2-Node Topology. + +Current FD.io production testbeds are built with SUT servers based on +the following processor architectures: + +- Intel Xeon: Skylake Platinum 8180, Haswell-SP E5-2699v3, + Cascadelake Platinum 8280, Cascadelake 6252N. +- Intel Atom: Denverton C3858. +- ARM: TaiShan 2280, hip07-d05. + +Server SUT performance depends on server and processor type, hence +results for testbeds based on different servers must be reported separately, and compared if appropriate. -Following sections describe existing production testbed types. +Complete technical specifications of compute servers used in CSIT +physical testbeds are maintained in FD.io CSIT repository: +https://git.fd.io/csit/tree/docs/lab/testbed_specifications.md. -3-Node Xeon Haswell (3n-hsw) ----------------------------- +Following is the description of existing production testbeds. + +2-Node Xeon Cascadelake (2n-clx) +-------------------------------- -3n-hsw testbed is based on three Cisco UCS-c240m3 servers each equipped -with two Intel Xeon Haswell-SP E5-2699v3 2.3 GHz 18 core processors. -Physical testbed topology is depicted in a figure below. +Three 2n-clx testbeds are in operation in FD.io labs. Each 2n-clx testbed +is built with two SuperMicro SYS-7049GP-TRT servers, SUTs are equipped with two +Intel Xeon Gold 6252N processors (35.75 MB Cache, 2.30 GHz, 24 cores). +TGs are equiped with Intel Xeon Cascadelake Platinum 8280 processors (38.5 MB +Cache, 2.70 GHz, 28 cores). 2n-clx physical topology is shown below. .. only:: latex @@ -44,51 +57,45 @@ Physical testbed topology is depicted in a figure below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-hsw} - \label{fig:testbed-3n-hsw} + \includegraphics[width=0.90\textwidth]{testbed-2n-clx} + \label{fig:testbed-2n-clx} \end{figure} .. only:: html - .. figure:: testbed-3n-hsw.svg - :alt: testbed-3n-hsw + .. figure:: testbed-2n-clx.svg + :alt: testbed-2n-clx :align: center -SUT1 and SUT2 servers are populated with the following NIC models: +SUT servers are populated with the following NIC models: -#. NIC-1: VIC 1385 2p40GE Cisco. -#. NIC-2: NIC x520 2p10GE Intel. -#. NIC-3: empty. -#. NIC-4: NIC xl710-QDA2 2p40GE Intel. -#. NIC-5: NIC x710-DA2 2p10GE Intel. -#. NIC-6: QAT 8950 50G (Walnut Hill) Intel. +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-4: empty, future expansion. +#. NIC-5: empty, future expansion. +#. NIC-6: empty, future expansion. TG servers run T-Rex application and are populated with the following NIC models: -#. NIC-1: NIC xl710-QDA2 2p40GE Intel. -#. NIC-2: NIC x710-DA2 2p10GE Intel. -#. NIC-3: empty. -#. NIC-4: NIC xl710-QDA2 2p40GE Intel. -#. NIC-5: NIC x710-DA2 2p10GE Intel. -#. NIC-6: NIC x710-DA2 2p10GE Intel. (For self-tests.) - -All Intel Xeon Haswell servers run with Intel Hyper-Threading disabled, -making the number of logical cores exposed to Linux match the number of -18 physical cores per processor socket. - -Complete 3n-hsw testbeds specification is available on -`CSIT LF testbed `_ -wiki page. +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel. +#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-4: empty, future expansion. +#. NIC-5: empty, future expansion. +#. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) -Total of three 3n-hsw testbeds are in operation in FD.io labs. +All Intel Xeon Cascadelake servers run with Intel Hyper-Threading enabled, +doubling the number of logical cores exposed to Linux. -3-Node Xeon Skylake (3n-skx) +2-Node Xeon Skylake (2n-skx) ---------------------------- -3n-skx testbed is based on three SuperMicro SYS-7049GP-TRT servers each -equipped with two Intel Xeon Skylake Platinum 8180 2.5 GHz 28 core -processors. Physical testbed topology is depicted in a figure below. +Four 2n-skx testbeds are in operation in FD.io labs. Each 2n-skx testbed +is built with two SuperMicro SYS-7049GP-TRT servers, each in turn +equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 MB +Cache, 2.50 GHz, 28 cores). 2n-skx physical topology is shown below. .. only:: latex @@ -97,21 +104,21 @@ processors. Physical testbed topology is depicted in a figure below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-3n-skx} - \label{fig:testbed-3n-skx} + \includegraphics[width=0.90\textwidth]{testbed-2n-skx} + \label{fig:testbed-2n-skx} \end{figure} .. only:: html - .. figure:: testbed-3n-skx.svg - :alt: testbed-3n-skx + .. figure:: testbed-2n-skx.svg + :alt: testbed-2n-skx :align: center -SUT1 and SUT2 servers are populated with the following NIC models: +SUT servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: empty, future expansion. +#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -121,7 +128,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: empty, future expansion. +#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) @@ -130,18 +137,13 @@ All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, doubling the number of logical cores exposed to Linux, with 56 logical cores and 28 physical cores per processor socket. -Complete 3n-skx testbeds specification is available on -`CSIT LF lab extension `_ -wiki page. - -Total of two 3n-skx testbeds are in operation in FD.io labs. - -2-Node Xeon Skylake (2n-skx) +3-Node Xeon Skylake (3n-skx) ---------------------------- -2n-skx testbed is based on two SuperMicro SYS-7049GP-TRT servers each -equipped with two Intel Xeon Skylake Platinum 8180 2.5 GHz 28 core -processors. Physical testbed topology is depicted in a figure below. +Two 3n-skx testbeds are in operation in FD.io labs. Each 3n-skx testbed +is built with three SuperMicro SYS-7049GP-TRT servers, each in turn +equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 MB +Cache, 2.50 GHz, 28 cores). 3n-skx physical topology is shown below. .. only:: latex @@ -150,21 +152,21 @@ processors. Physical testbed topology is depicted in a figure below. \begin{figure}[H] \centering \graphicspath{{../_tmp/src/introduction/}} - \includegraphics[width=0.90\textwidth]{testbed-2n-skx} - \label{fig:testbed-2n-skx} + \includegraphics[width=0.90\textwidth]{testbed-3n-skx} + \label{fig:testbed-3n-skx} \end{figure} .. only:: html - .. figure:: testbed-2n-skx.svg - :alt: testbed-2n-skx + .. figure:: testbed-3n-skx.svg + :alt: testbed-3n-skx :align: center -SUT servers are populated with the following NIC models: +SUT1 and SUT2 servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -174,7 +176,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: mcx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) @@ -183,8 +185,167 @@ All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, doubling the number of logical cores exposed to Linux, with 56 logical cores and 28 physical cores per processor socket. -Complete 2n-skx testbed specification is available on -`CSIT/Testbeds: Xeon Skx, Arm, Atom -`_ wiki page. +3-Node Xeon Haswell (3n-hsw) +---------------------------- + +Three 3n-hsw testbeds are in operation in FD.io labs. Each 3n-hsw +testbed is built with three Cisco UCS-c240m3 servers, each in turn +equipped with two Intel Xeon Haswell-SP E5-2699v3 processors (45 MB +Cache, 2.3 GHz, 18 cores). 3n-hsw physical topology is shown below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/introduction/}} + \includegraphics[width=0.90\textwidth]{testbed-3n-hsw} + \label{fig:testbed-3n-hsw} + \end{figure} + +.. only:: html + + .. figure:: testbed-3n-hsw.svg + :alt: testbed-3n-hsw + :align: center -Total of four 2n-skx testbeds are in operation in FD.io labs. +SUT1 and SUT2 servers are populated with the following NIC models: + +#. NIC-1: VIC 1385 2p40GE Cisco. +#. NIC-2: NIC x520 2p10GE Intel. +#. NIC-3: empty. +#. NIC-4: NIC xl710-QDA2 2p40GE Intel. +#. NIC-5: NIC x710-DA2 2p10GE Intel. +#. NIC-6: QAT 8950 50G (Walnut Hill) Intel. + +TG servers run T-Rex application and are populated with the following +NIC models: + +#. NIC-1: NIC xl710-QDA2 2p40GE Intel. +#. NIC-2: NIC x710-DA2 2p10GE Intel. +#. NIC-3: empty. +#. NIC-4: NIC xl710-QDA2 2p40GE Intel. +#. NIC-5: NIC x710-DA2 2p10GE Intel. +#. NIC-6: NIC x710-DA2 2p10GE Intel. (For self-tests.) + +All Intel Xeon Haswell servers run with Intel Hyper-Threading disabled, +making the number of logical cores exposed to Linux match the number of +18 physical cores per processor socket. + +2-Node Atom Denverton (2n-dnv) +------------------------------ + +2n-dnv testbed is built with: i) one Intel S2600WFT server acting as TG +and equipped with two Intel Xeon Skylake Platinum 8180 processors (38.5 +MB Cache, 2.50 GHz, 28 cores), and ii) one SuperMicro SYS-E300-9A server +acting as SUT and equipped with one Intel Atom C3858 processor (12 MB +Cache, 2.00 GHz, 12 cores). 2n-dnv physical topology is shown below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/introduction/}} + \includegraphics[width=0.90\textwidth]{testbed-2n-dnv} + \label{fig:testbed-2n-dnv} + \end{figure} + +.. only:: html + + .. figure:: testbed-2n-dnv.svg + :alt: testbed-2n-dnv + :align: center + +SUT server have four internal 10G NIC port: + +#. P-1: x553 copper port. +#. P-2: x553 copper port. +#. P-3: x553 fiber port. +#. P-4: x553 fiber port. + +TG server run T-Rex software traffic generator and are populated with the +following NIC models: + +#. NIC-1: x550-T2 2p10GE Intel. +#. NIC-2: x550-T2 2p10GE Intel. +#. NIC-3: x520-DA2 2p10GE Intel. +#. NIC-4: x520-DA2 2p10GE Intel. + +The 2n-dnv testbed is in operation in Intel SH labs. + +3-Node Atom Denverton (3n-dnv) +------------------------------ + +One 3n-dnv testbed is built with: i) one SuperMicro SYS-7049GP-TRT +server acting as TG and equipped with two Intel Xeon Skylake Platinum +8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one +SuperMicro SYS-E300-9A server acting as SUT and equipped with one Intel +Atom C3858 processor (12 MB Cache, 2.00 GHz, 12 cores). 3n-dnv physical +topology is shown below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/introduction/}} + \includegraphics[width=0.90\textwidth]{testbed-3n-dnv} + \label{fig:testbed-3n-dnv} + \end{figure} + +.. only:: html + + .. figure:: testbed-3n-dnv.svg + :alt: testbed-3n-dnv + :align: center + +SUT1 and SUT2 servers are populated with the following NIC models: + +#. NIC-1: x553 2p10GE fiber Intel. +#. NIC-2: x553 2p10GE copper Intel. + +TG servers run T-Rex application and are populated with the following +NIC models: + +#. NIC-1: x710-DA4 4p10GE Intel. + +3-Node ARM TaiShan (3n-tsh) +--------------------------- + +One 3n-tsh testbed is built with: i) one SuperMicro SYS-7049GP-TRT +server acting as TG and equipped with two Intel Xeon Skylake Platinum +8180 processors (38.5 MB Cache, 2.50 GHz, 28 cores), and ii) one Huawei +TaiShan 2280 server acting as SUT and equipped with one hip07-d05 +processor (64* ARM Cortex-A72). 3n-tsh physical topology is shown below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/introduction/}} + \includegraphics[width=0.90\textwidth]{testbed-3n-tsh} + \label{fig:testbed-3n-tsh} + \end{figure} + +.. only:: html + + .. figure:: testbed-3n-tsh.svg + :alt: testbed-3n-tsh + :align: center + +SUT1 and SUT2 servers are populated with the following NIC models: + +#. NIC-1: connectx4 2p25GE Mellanox. +#. NIC-2: x520 2p10GE Intel. + +TG servers run T-Rex application and are populated with the following +NIC models: + +#. NIC-1: x710-DA4 4p10GE Intel. +#. NIC-2: xxv710-DA2 2p25GE Intel.