docs/report/introduction/methodology_mellanox_nic.rst

   1 Mellanox NIC
   2 ------------
   3
   4 Performance test results using Mellanox ConnectX5 2p100GE are reported for
   5 2-Node Xeon Cascade Lake physical testbed type present in FD.io labs. For
   6 description of physical testbeds used please refer to
   7 :ref:`tested_physical_topologies`.
   8
   9 Mellanox NIC settings
  10 ~~~~~~~~~~~~~~~~~~~~~
  11
  12 Mellanox ConnectX5 NIC settings are following recommendations from
  13 [DpdkPerformanceReport]_, [MellanoxDpdkGuide]_ and [MellanoxDpdkBits]_.
  14 Specifically:
  15
  16 - Flow Control OFF:
  17   ::
  18
  19       $ ethtool -A $netdev rx off tx off
  20
  21 - Change PCI MaxReadReq to 1024B for each port of each NIC:
  22   ::
  23
  24       $ setpci -s $PORT_PCI_ADDRESS 68.w=3BCD
  25
  26 - Set CQE COMPRESSION to "AGGRESSIVE":
  27   ::
  28
  29       $ mlxconfig -d $PORT_PCI_ADDRESS set CQE_COMPRESSION=1
  30
  31 Mellanox :abbr:`OFED (OpenFabrics Enterprise Distribution)` driver of version
  32 4.6-1.0.1.1 is installed and used to manage the NIC settings.
  33
  34 ::
  35
  36     $ sudo ethtool -i enp94s0f0
  37     driver: mlx5_core
  38     version: 4.6-1.0.1
  39     firmware-version: 16.25.1020 (MT_0000000009)
  40     expansion-rom-version:
  41     bus-info: 0000:5e:00.0
  42     supports-statistics: yes
  43     supports-test: yes
  44     supports-eeprom-access: no
  45     supports-register-dump: no
  46     supports-priv-flags: yes
  47
  48 ::
  49
  50     $ sudo lspci -vvvs 5e:00.0
  51     5e:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
  52             Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
  53             Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
  54             Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  55             Latency: 0, Cache Line Size: 32 bytes
  56             Interrupt: pin A routed to IRQ 37
  57             NUMA node: 0
  58             Region 0: Memory at 38fffe000000 (64-bit, prefetchable) [size=32M]
  59             Expansion ROM at c5e00000 [disabled] [size=1M]
  60             Capabilities: [60] Express (v2) Endpoint, MSI 00
  61                     DevCap:     MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
  62                             ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
  63                     DevCtl:     Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
  64                             RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
  65                             MaxPayload 256 bytes, MaxReadReq 4096 bytes
  66                     DevSta:     CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
  67                     LnkCap:     Port #0, Speed 16GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
  68                             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
  69                     LnkCtl:     ASPM Disabled; RCB 64 bytes Disabled- CommClk+
  70                             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  71                     LnkSta:     Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  72                     DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
  73                     DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  74                     LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
  75                              Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  76                              Compliance De-emphasis: -6dB
  77                     LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
  78                              EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
  79             Capabilities: [48] Vital Product Data
  80                     Product Name: CX556A - ConnectX-5 QSFP28
  81                     Read-only fields:
  82                             [PN] Part number: MCX556A-EDAT
  83                             [EC] Engineering changes: AA
  84                             [V2] Vendor specific: MCX556A-EDAT
  85                             [SN] Serial number: MT1945X00360
  86                             [V3] Vendor specific: f8d15ef7e701ea118000b8599ffe4aa8
  87                             [VA] Vendor specific: MLX:MODL=CX556A:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0
  88                             [V0] Vendor specific: PCIeGen4 x16
  89                             [RV] Reserved: checksum good, 2 byte(s) reserved
  90                     End
  91             Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
  92                     Vector table: BAR=0 offset=00002000
  93                     PBA: BAR=0 offset=00003000
  94             Capabilities: [c0] Vendor Specific Information: Len=18 <?>
  95             Capabilities: [40] Power Management version 3
  96                     Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
  97                     Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  98             Capabilities: [100 v1] Advanced Error Reporting
  99                     UESta:      DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
 100                     UEMsk:      DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
 101                     UESvrt:     DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
 102                     CESta:      RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
 103                     CEMsk:      RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
 104                     AERCap:     First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
 105             Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
 106                     ARICap:     MFVC- ACS-, Next Function: 1
 107                     ARICtl:     MFVC- ACS-, Function Group: 0
 108             Capabilities: [1c0 v1] #19
 109             Capabilities: [230 v1] Access Control Services
 110                     ACSCap:     SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
 111                     ACSCtl:     SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
 112             Capabilities: [320 v1] #27
 113             Capabilities: [370 v1] #26
 114             Capabilities: [420 v1] #25
 115             Kernel driver in use: mlx5_core
 116             Kernel modules: mlx5_core
 117
 118 TG and SUT settings
 119 ~~~~~~~~~~~~~~~~~~~
 120
 121 For the TG and SUT environment settings please refer to
 122 :ref:`_vpp_test_environment` and :ref:`_dpdk_test_environment`.
 123
 124 Links
 125 ~~~~~
 126
 127 .. [DpdkPerformanceReport] `DPDK 19.11 performance report <http://static.dpdk.org/doc/perf/DPDK_19_11_Mellanox_NIC_performance_report.pdf>`
 128 .. [MellanoxDpdkGuide] `Mellanox DPDK guide <https://www.mellanox.com/related-docs/prod_software/MLNX_DPDK_Quick_Start_Guide_v16.11_3.0.pdf>`
 129 .. [MellanoxDpdkBits] `Mellanox DPDK bits <https://community.mellanox.com/s/article/mellanox-dpdk>`