docs/lab/testbeds_sm_skx_hw_bios_cfg.md

   1 # SuperMicro Xeon Skylake Servers - Hardware and BIOS Configuration
   2
   3 1. [Linux lscpu TODO](#linux-lscpu-todo)
   4 1. [Linux dmidecode pci TODO](#linux-dmidecode-pci-todo)
   5 1. [Linux dmidecode memory TODO](#linux-dmidecode-memory-todo)
   6 1. [Xeon Skx Server BIOS Configuration](#xeon-skx-server-bios-configuration)
   7    1. [Boot Feature](#boot-feature)
   8    1. [CPU Configuration](#cpu-configuration)
   9       1. [Advanced Power Management Configuration](#advanced-power-management-configuration)
  10          1. [CPU P State Control](#cpu-p-state-control)
  11          1. [Hardware PM State Control](#hardware-pm-state-control)
  12          1. [CPU C State Control](#cpu-c-state-control)
  13          1. [Package C State Control](#package-c-state-control)
  14          1. [CPU T State Control](#cpu-t-state-control)
  15       1. [Chipset Configuration](#chipset-configuration)
  16          1. [North Bridge](#north-bridge)
  17          1. [UPI Configuration](#upi-configuration)
  18          1. [Memory Configuration](#memory-configuration)
  19          1. [IIO Configuration](#iio-configuration)
  20          1. [CPU1 Configuration](#cpu1-configuration)
  21          1. [CPU2 Configuration](#cpu2-configuration)
  22       1. [South Bridge](#south-bridge)
  23    1. [PCIe/PCI/PnP Configuration](#pciepcipnp-configuration)
  24    1. [ACPI Settings](#acpi-settings)
  25    1. [DMIDECODE](#dmidecode)
  26 1. [Xeon Skx Server Firmware Inventory](#xeon-skx-server-firmware-inventory)
  27
  28 ## Linux lscpu TODO
  29
  30 ## Linux dmidecode pci TODO
  31
  32 ## Linux dmidecode memory TODO
  33
  34 ## Xeon Skx Server BIOS Configuration
  35
  36 ### Boot Feature
  37
  38 ```
  39   |  Quiet Boot                                [Enabled]               |Boot option                  |
  40   |                                                                    |                             |
  41   |  Option ROM Messages                       [Force BIOS]            |                             |
  42   |  Bootup NumLock State                      [On]                    |                             |
  43   |  Wait For "F1" If Error                    [Enabled]               |                             |
  44   |  INT19 Trap Response                       [Immediate]             |                             |
  45   |  Re-try Boot                               [Disabled]              |                             |
  46   |  Install Windows 7 USB support             [Disabled]              |                             |
  47   |  Port 61h Bit-4 Emulation                  [Disabled]              |                             |
  48   |                                                                    |                             |
  49   |  Power Configuration                                               |                             |
  50   |  Watch Dog Function                        [Disabled]              |                             |
  51   |  Restore on AC Power Loss                  [Last State]            |                             |
  52   |  Power Button Function                     [Instant Off]           |                             |
  53   |  Throttle on Power Fail                    [Disabled]              |                             |
  54 ```
  55
  56 ### CPU Configuration
  57
  58 ```
  59   |  Processor Configuration                                           |Enables Hyper Threading      |
  60   |  --------------------------------------------------                |(Software Method to          |
  61   |  Processor BSP Revision                    50654 - SKX H0          |Enable/Disable Logical       |
  62   |  Processor Socket                          CPU1      |  CPU2       |Processor threads.           |
  63   |  Processor ID                              00050654* |  000506...  |                             |
  64   |  Processor Frequency                       2.500GHz  |  2.500GHz   |                             |
  65   |  Processor Max Ratio                            19H  |  19H        |                             |
  66   |  Processor Min Ratio                            0AH  |  0AH        |                             |
  67   |  Microcode Revision                        02000030                |                             |
  68   |  L1 Cache RAM                                  64KB  |      64KB   |                             |
  69   |  L2 Cache RAM                                1024KB  |    1024KB   |                             |
  70   |  L3 Cache RAM                               39424KB  |   39424KB   |                             |
  71   |  Processor 0 Version                                               |                             |
  72   |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
  73   |  Processor 1 Version                                               |                             |
  74   |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
  75   |                                                                    |                             |
  76   |  Hyper-Threading [ALL]                     [Enable]                |                             |
  77   |  Core Disable Bitmap(Hex)                  0                       |                             |
  78   |  Execute Disable Bit                       [Enable]                |                             |
  79   |  Intel Virtualization Technology           [Enable]                |                             |
  80   |  PPIN Control                              [Unlock/Enable]         |                             |
  81   |  Hardware Prefetcher                       [Enable]                |                             |
  82   |  Adjacent Cache Prefetch                   [Enable]                |                             |
  83   |  DCU Streamer Prefetcher                   [Enable]                |                             |
  84   |  DCU IP Prefetcher                         [Enable]                |                             |
  85   |  LLC Prefetch                              [Disable]               |                             |
  86   |  Extended APIC                             [Disable]               |                             |
  87   |  AES-NI                                    [Enable]                |                             |
  88   |> Advanced Power Management Configuration                           |                             |
  89 ```
  90
  91 #### Advanced Power Management Configuration
  92
  93 ```
  94   |  Advanced Power Management Configuration                           |Switch CPU Power Management  |
  95   |  --------------------------------------------------                |profile                      |
  96   |  Power Technology                          [Custom]                |                             |
  97   |  Power Performance Tuning                  [BIOS Controls EPB]     |                             |
  98   |  ENERGY_PERF_BIAS_CFG mode                 [Maximum Performance]   |                             |
  99   |> CPU P State Control                                               |                             |
 100   |> Hardware PM State Control                                         |                             |
 101   |> CPU C State Control                                               |                             |
 102   |> Package C State Control                                           |                             |
 103   |> CPU T State Control                                               |                             |
 104 ```
 105
 106 ##### CPU P State Control
 107
 108 ```
 109   |  CPU P State Control                                               |Enable/Disable EIST          |
 110   |                                                                    |(P-States)                   |
 111   |  SpeedStep (Pstates)                       [Disable]               |                             |
 112   |  EIST PSD Function                         [HW_ALL]                |                             |
 113 ```
 114
 115 ##### Hardware PM State Control
 116
 117 ```
 118   |  Hardware PM State Control                                         |Disable: Hardware chooses a  |
 119   |                                                                    |P-state based on OS Request  |
 120   |  Hardware P-States                         [Disable]               |(Legacy P-States)            |
 121   |                                                                    |Native Mode:Hardware         |
 122   |                                                                    |chooses a P-state based on   |
 123   |                                                                    |OS guidance                  |
 124   |                                                                    |Out of Band Mode:Hardware    |
 125   |                                                                    |autonomously chooses a       |
 126   |                                                                    |P-state (no OS guidance)     |
 127 ```
 128
 129 ##### CPU C State Control
 130
 131 ```
 132   |  CPU C State Control                                               |Autonomous Core C-State      |
 133   |                                                                    |Control                      |
 134   |  Autonomous Core C-State                   [Disable]               |                             |
 135   |  CPU C6 report                             [Disable]               |                             |
 136   |  Enhanced Halt State (C1E)                 [Disable]               |                             |
 137 ```
 138
 139 ##### Package C State Control
 140
 141 ```
 142   |  Package C State Control                                           |Package C State limit        |
 143   |                                                                    |                             |
 144   |  Package C State                           [C0/C1 state]           |                             |
 145 ```
 146
 147 ##### CPU T State Control
 148
 149 ```
 150   |  CPU T State Control                                               |Enable/Disable Software      |
 151   |                                                                    |Controlled T-States          |
 152   |  Software Controlled T-States              [Disable]               |                             |
 153 ```
 154
 155 #### Chipset Configuration
 156
 157 ```
 158   |  WARNING: Setting wrong values in below sections may cause         |North Bridge Parameters      |
 159   |           system to malfunction.                                   |                             |
 160   |> North Bridge                                                      |                             |
 161   |> South Bridge                                                      |                             |
 162 ```
 163
 164 ##### North Bridge
 165
 166 ```
 167   |> UPI Configuration                                                 |Displays and provides        |
 168   |> Memory Configuration                                              |option to change the UPI     |
 169   |> IIO Configuration                                                 |Settings                     |
 170 ```
 171
 172 ##### UPI Configuration
 173
 174 ```
 175   |  UPI Configuration                                                 |Choose Topology Precedence   |
 176   |  --------------------------------------------------                |to degrade features if       |
 177   |  Number of CPU                             2                       |system options are in        |
 178   |  Number of Active UPI Link                 3                       |conflict or choose Feature   |
 179   |  Current UPI Link Speed                    Fast                    |Precedence to degrade        |
 180   |  Current UPI Link Frequency                10.4 GT/s               |topology if system options   |
 181   |  UPI Global MMIO Low Base / Limit          90000000 / FBFFFFFF     |are in conflict.             |
 182   |  UPI Global MMIO High Base / Limit         0000000000000000 / ...  |                             |
 183   |  UPI Pci-e Configuration Base / Size       80000000 / 10000000     |                             |
 184   |  Degrade Precedence                        [Topology Precedence]   |                             |
 185   |  Link L0p Enable                           [Disable]               |                             |
 186   |  Link L1 Enable                            [Disable]               |                             |
 187   |  IO Directory Cache (IODC)                 [Auto]                  |                             |
 188   |  SNC                                       [Disable]               |                             |
 189   |  XPT Prefetch                              [Disable]               |                             |
 190   |  KTI Prefetch                              [Enable]                |                             |
 191   |  Local/Remote Threshold                    [Auto]                  |                             |
 192   |  Stale AtoS                                [Disable]               |                             |
 193   |  LLC dead line alloc                       [Enable]                |                             |
 194   |  Isoc Mode                                 [Auto]                  |                             |
 195 ```
 196
 197 ##### Memory Configuration
 198
 199 ```
 200   |                                                                    |POR - Enforces Plan Of       |
 201   |  --------------------------------------------------                |Record restrictions for      |
 202   |  Integrated Memory Controller (iMC)                                |DDR4 frequency and voltage   |
 203   |  --------------------------------------------------                |programming. Disable -       |
 204   |                                                                    |Disables this feature.       |
 205   |  Enforce POR                               [Disable]               |                             |
 206   |  Memory Frequency                          [2666]                  |                             |
 207   |  Data Scrambling for NVMDIMM               [Auto]                  |                             |
 208   |  Data Scrambling for DDR4                  [Auto]                  |                             |
 209   |  tCCD_L Relaxation                         [Auto]                  |                             |
 210   |  Memory tRWSR Relaxation                   [Enable]                |                             |
 211   |  2X REFRESH                                [Auto]                  |                             |
 212   |  Page Policy                               [Auto]                  |                             |
 213   |  IMC Interleaving                          [2-way Interleave]      |                             |
 214   |> Memory Topology                                                   |                             |
 215   |> Memory RAS Configuration                                          |                             |
 216 ```
 217
 218 ##### IIO Configuration
 219
 220 ```
 221   |  IIO Configuration                                                 |Expose IIO DFX devices and   |
 222   |  --------------------------------------------------                |other CPU devices like PMON  |
 223   |                                                                    |                             |
 224   |  EV DFX Features                           [Disable]               |                             |
 225   |> CPU1 Configuration                                                |                             |
 226   |> CPU2 Configuration                                                |                             |
 227   |> IOAT Configuration                                                |                             |
 228   |> Intel. VT for Directed I/O (VT-d)                                 |                             |
 229   |> Intel. VMD technology                                             |                             |
 230   |                                                                    |                             |
 231   |   IIO-PCIE Express Global Options                                  |                             |
 232   |  ========================================                          |                             |
 233   |  PCI-E Completion Timeout Disable          [No]                    |                             |
 234 ```
 235
 236 ##### CPU1 Configuration
 237
 238 ```
 239   |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
 240   |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
 241   |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
 242   |> CPU1 SLOT2 PCI-E 3.0 X16                                          |                             |
 243   |> CPU1 SLOT4 PCI-E 3.0 X16                                          |                             |
 244   |> CPU1 SLOT9 PCI-E 3.0 X16                                          |                             |
 245 ```
 246
 247 ##### CPU2 Configuration
 248
 249 ```
 250   |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
 251   |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
 252   |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
 253   |> CPU2 SLOT6 PCI-E 3.0 X16                                          |                             |
 254   |> CPU2 SLOT8 PCI-E 3.0 X16                                          |                             |
 255   |> CPU2 SLOT10 PCI-E 3.0 X16                                         |                             |
 256 ```
 257
 258 #### South Bridge
 259
 260 ```
 261   |                                                                    |Enables Legacy USB support.  |
 262   |  USB Module Version                        17                      |AUTO option disables legacy  |
 263   |                                                                    |support if no USB devices    |
 264   |  USB Devices:                                                      |are connected. DISABLE       |
 265   |        1 Keyboard, 1 Mouse, 1 Hub                                  |option will keep USB         |
 266   |                                                                    |devices available only for   |
 267   |  Legacy USB Support                        [Enabled]               |EFI applications.            |
 268   |  XHCI Hand-off                             [Disabled]              |                             |
 269   |  Port 60/64 Emulation                      [Enabled]               |                             |
 270   |  PCIe PLL SSC                              [Disable]               |                             |
 271   |  Real USB Wake Up                          [Enabled]               |                             |
 272   |  Front USB Wake Up                         [Enabled]               |                             |
 273   |                                                                    |                             |
 274   |  Azalia                                    [Auto]                  |                             |
 275   |    Azalia PME Enable                       [Disabled]              |                             |
 276 ```
 277
 278 ### PCIe/PCI/PnP Configuration
 279
 280 ```
 281   |  PCI Bus Driver Version                    A5.01.12                |Enables or Disables 64bit    |
 282   |                                                                    |capable Devices to be        |
 283   |  PCI Devices Common Settings:                                      |Decoded in Above 4G Address  |
 284   |  Above 4G Decoding                         [Enabled]               |Space (Only if System        |
 285   |  SR-IOV Support                            [Enabled]               |Supports 64 bit PCI          |
 286   |  MMIO High Base                            [56T]                   |Decoding).                   |
 287   |  MMIO High Granularity Size                [256G]                  |                             |
 288   |  Maximum Read Request                      [Auto]                  |                             |
 289   |  MMCFG Base                                [2G]                    |                             |
 290   |  NVMe Firmware Source                      [Vendor Defined Fi...]  |                             |
 291   |  VGA Priority                              [Onboard]               |                             |
 292   |  CPU1 SLOT2 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
 293   |  CPU1 SLOT4 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
 294   |  CPU2 SLOT6 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
 295   |  CPU2 SLOT8 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
 296   |  CPU1 SLOT9 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
 297   |  CPU2 SLOT10 PCI-E 3.0 X16 OPROM           [Legacy]                |                             |
 298   |  CPU2 SLOT11 PCI-E 3.0 X4(IN X8) OPROM     [Legacy]                |                             |
 299   |  M.2 CONNECTOR OPROM                       [Legacy]                |                             |
 300   |  Onboard LAN1 Option ROM                   [Legacy]                |                             |
 301   |  Onboard LAN2 Option ROM                   [Disabled]              |                             |
 302   |  Onboard Video Option ROM                  [Legacy]                |                             |
 303   |> Network Stack Configuration                                       |                             |
 304 ```
 305
 306 ### ACPI Settings
 307
 308 ```
 309   |  ACPI Settings                                                     |Enable or Disable Non        |
 310   |                                                                    |uniform Memory Access        |
 311   |  NUMA                                      [Enabled]               |(NUMA).                      |
 312   |  WHEA Support                              [Enabled]               |                             |
 313   |  High Precision Event Timer                [Enabled]               |                             |
 314   |  ACPI Sleep State                          [S3 (Suspend to RAM)]   |                             |
 315 ```
 316
 317 ### DMIDECODE
 318
 319 ```
 320   # dmidecode 3.1
 321   Getting SMBIOS data from sysfs.
 322   SMBIOS 3.1.1 present.
 323   Table at 0x000E89C0.
 324
 325   Handle 0x0000, DMI type 0, 26 bytes
 326   BIOS Information
 327         Vendor: American Megatrends Inc.
 328         Version: 2.0
 329         Release Date: 11/29/2017
 330         Address: 0xF0000
 331         Runtime Size: 64 kB
 332         ROM Size: 64 MB
 333         Characteristics:
 334                 PCI is supported
 335                 BIOS is upgradeable
 336                 BIOS shadowing is allowed
 337                 Boot from CD is supported
 338                 Selectable boot is supported
 339                 BIOS ROM is socketed
 340                 EDD is supported
 341                 5.25"/1.2 MB floppy services are supported (int 13h)
 342                 3.5"/720 kB floppy services are supported (int 13h)
 343                 3.5"/2.88 MB floppy services are supported (int 13h)
 344                 Print screen service is supported (int 5h)
 345                 Serial services are supported (int 14h)
 346                 Printer services are supported (int 17h)
 347                 ACPI is supported
 348                 USB legacy is supported
 349                 BIOS boot specification is supported
 350                 Targeted content distribution is supported
 351                 UEFI is supported
 352         BIOS Revision: 5.12
 353
 354   Handle 0x0001, DMI type 1, 27 bytes
 355   System Information
 356         Manufacturer: Supermicro
 357         Product Name: SYS-7049GP-TRT
 358         Version: 0123456789
 359         Serial Number: S291427X8332242
 360         UUID: 00000000-0000-0000-0000-AC1F6B8A8DB6
 361         Wake-up Type: Power Switch
 362         SKU Number: To be filled by O.E.M.
 363         Family: To be filled by O.E.M.
 364
 365   Handle 0x0002, DMI type 2, 15 bytes
 366   Base Board Information
 367         Manufacturer: Supermicro
 368         Product Name: X11DPG-QT
 369         Version: 1.02
 370         Serial Number: VM183S014930
 371         Asset Tag: To be filled by O.E.M.
 372         Features:
 373                 Board is a hosting board
 374                 Board is replaceable
 375         Location In Chassis: To be filled by O.E.M.
 376         Chassis Handle: 0x0003
 377         Type: Motherboard
 378         Contained Object Handles: 0
 379
 380   Handle 0x0003, DMI type 3, 22 bytes
 381   Chassis Information
 382         Manufacturer: Supermicro
 383         Type: Other
 384         Lock: Not Present
 385         Version: 0123456789
 386         Serial Number: C7470KH06A20167
 387         Asset Tag: To be filled by O.E.M.
 388         Boot-up State: Safe
 389         Power Supply State: Safe
 390         Thermal State: Safe
 391         Security Status: None
 392         OEM Information: 0x00000000
 393
 394   Handle 0x0050, DMI type 4, 48 bytes
 395   Processor Information
 396         Socket Designation: CPU1
 397         Type: Central Processor
 398         Family: Xeon
 399         Manufacturer: Intel(R) Corporation
 400         ID: 54 06 05 00 FF FB EB BF
 401         Signature: Type 0, Family 6, Model 85, Stepping 4
 402         Flags:
 403                 FPU (Floating-point unit on-chip)
 404                 VME (Virtual mode extension)
 405                 DE (Debugging extension)
 406                 PSE (Page size extension)
 407                 TSC (Time stamp counter)
 408                 MSR (Model specific registers)
 409                 PAE (Physical address extension)
 410                 MCE (Machine check exception)
 411                 CX8 (CMPXCHG8 instruction supported)
 412                 APIC (On-chip APIC hardware supported)
 413                 SEP (Fast system call)
 414                 MTRR (Memory type range registers)
 415                 PGE (Page global enable)
 416                 MCA (Machine check architecture)
 417                 CMOV (Conditional move instruction supported)
 418                 PAT (Page attribute table)
 419                 PSE-36 (36-bit page size extension)
 420                 CLFSH (CLFLUSH instruction supported)
 421                 DS (Debug store)
 422                 ACPI (ACPI supported)
 423                 MMX (MMX technology supported)
 424                 FXSR (FXSAVE and FXSTOR instructions supported)
 425                 SSE (Streaming SIMD extensions)
 426                 SSE2 (Streaming SIMD extensions 2)
 427                 SS (Self-snoop)
 428                 HTT (Multi-threading)
 429                 TM (Thermal monitor supported)
 430                 PBE (Pending break enabled)
 431         Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
 432         Voltage: 1.6 V
 433         External Clock: 100 MHz
 434         Max Speed: 4000 MHz
 435         Current Speed: 2500 MHz
 436         Status: Populated, Enabled
 437         Upgrade: Other
 438         L1 Cache Handle: 0x004D
 439         L2 Cache Handle: 0x004E
 440         L3 Cache Handle: 0x004F
 441         Serial Number: Not Specified
 442         Asset Tag: UNKNOWN
 443         Part Number: Not Specified
 444         Core Count: 28
 445         Core Enabled: 28
 446         Thread Count: 56
 447         Characteristics:
 448                 64-bit capable
 449                 Multi-Core
 450                 Hardware Thread
 451                 Execute Protection
 452                 Enhanced Virtualization
 453                 Power/Performance Control
 454
 455
 456   Handle 0x0054, DMI type 4, 48 bytes
 457   Processor Information
 458         Socket Designation: CPU2
 459         Type: Central Processor
 460         Family: Xeon
 461         Manufacturer: Intel(R) Corporation
 462         ID: 54 06 05 00 FF FB EB BF
 463         Signature: Type 0, Family 6, Model 85, Stepping 4
 464         Flags:
 465                 FPU (Floating-point unit on-chip)
 466                 VME (Virtual mode extension)
 467                 DE (Debugging extension)
 468                 PSE (Page size extension)
 469                 TSC (Time stamp counter)
 470                 MSR (Model specific registers)
 471                 PAE (Physical address extension)
 472                 MCE (Machine check exception)
 473                 CX8 (CMPXCHG8 instruction supported)
 474                 APIC (On-chip APIC hardware supported)
 475                 SEP (Fast system call)
 476                 MTRR (Memory type range registers)
 477                 PGE (Page global enable)
 478                 MCA (Machine check architecture)
 479                 CMOV (Conditional move instruction supported)
 480                 PAT (Page attribute table)
 481                 PSE-36 (36-bit page size extension)
 482                 CLFSH (CLFLUSH instruction supported)
 483                 DS (Debug store)
 484                 ACPI (ACPI supported)
 485                 MMX (MMX technology supported)
 486                 FXSR (FXSAVE and FXSTOR instructions supported)
 487                 SSE (Streaming SIMD extensions)
 488                 SSE2 (Streaming SIMD extensions 2)
 489                 SS (Self-snoop)
 490                 HTT (Multi-threading)
 491                 TM (Thermal monitor supported)
 492                 PBE (Pending break enabled)
 493         Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
 494         Voltage: 1.6 V
 495         External Clock: 100 MHz
 496         Max Speed: 4000 MHz
 497         Current Speed: 2500 MHz
 498         Status: Populated, Enabled
 499         Upgrade: Other
 500         L1 Cache Handle: 0x0051
 501         L2 Cache Handle: 0x0052
 502         L3 Cache Handle: 0x0053
 503         Serial Number: Not Specified
 504         Asset Tag: UNKNOWN
 505         Part Number: Not Specified
 506         Core Count: 28
 507         Core Enabled: 28
 508         Thread Count: 56
 509         Characteristics:
 510                 64-bit capable
 511                 Multi-Core
 512                 Hardware Thread
 513                 Execute Protection
 514                 Enhanced Virtualization
 515                 Power/Performance Control
 516 ```
 517
 518 ## Xeon Skx Server Firmware Inventory
 519
 520 ```
 521 Host.           IPMI IP.      BIOS. CPLD.     Aptio SU.   CPU Microcode.  PCI Bus.   ME Operation FW.    X710 Firmware.            XXV710 Firmware.          i40e.
 522 s1-t11-sut1.    10.30.50.47.  2.1.  03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 523 s2-t12-sut1.    10.30.50.48.  2.1.  03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 524 s3-t21-sut1.    10.30.50.41.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 525 s4-t21-tg1.     10.30.50.42.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 526 s5-t22-sut1.    10.30.50.49.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 527 s6-t22-tg1.     10.30.50.50.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 528 s7-t23-sut1.    10.30.50.51.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 529 s8-t23-tg1.     10.30.50.52.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 530 s9-t24-sut1.    10.30.50.53.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 531 s10-t24-tg1.    10.30.50.54.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 532 s11-t31-sut1.   10.30.50.43.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 533 s12-t31-sut2.   10.30.50.44.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 534 s13-t31-tg1.    10.30.50.45.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 535 s14-t32-sut1.   10.30.50.55.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 536 s15-t32-sut2.   10.30.50.56.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 537 s16-t32-tg1.    10.30.50.57.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 538 s19-t33t34-tg1. 10.30.50.46.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
 539 ```