4 :email: <hhaim@cisco.com>
8 :web_server_url: http://trex-tgn.cisco.com/trex
9 :local_web_server_url: csi-wiki-01:8181/trex
12 include::trex_ga.asciidoc[]
17 === A word on traffic generators
19 Traditionally, routers have been tested using commercial traffic generators, while performance
20 typically has been measured using packets per second (PPS) metrics. As router functionality and
21 services became more complex, stateful traffic generators now need to provide more realistic traffic scenarios.
23 Advantages of realistic traffic generators:
25 * Accurate performance metrics.
26 * Discovering bottlenecks in realistic traffic scenarios.
28 ==== Current Challenges:
30 * *Cost*: Commercial stateful traffic generators are very expensive.
31 * *Scale*: Bandwidth does not scale up well with feature complexity.
32 * *Standardization*: Lack of standardization of traffic patterns and methodologies.
33 * *Flexibility*: Commercial tools do not allow agility when flexibility and changes are needed.
37 * High capital expenditure (capex) spent by different teams.
38 * Testing in low scale and extrapolation became a common practice. This is non-ideal and fails to indicate bottlenecks that appear in real-world scenarios.
39 * Teams use different benchmark methodologies, so results are not standardized.
40 * Delays in development and testing due to dependence on testing tool features.
41 * Resource and effort investment in developing different ad hoc tools and test methodologies.
45 TRex addresses these problems through an innovative and extendable software implementation and by leveraging standard and open software and x86/UCS hardware.
47 * Generates and analyzes L4-7 traffic. In one package, provides capabilities of commercial L7 tools.
48 * Stateful traffic generator based on pre-processing and smart replay of real traffic templates.
49 * Generates and *amplifies* both client and server side traffic.
50 * Customized functionality can be added.
51 * Scales to 200Gb/sec for one UCS (using Intel 40Gb/sec NICs).
53 * Self-contained package that can be easily installed and deployed.
54 * Virtual interface support enables TRex to be used in a fully virtual environment without physical NICs. Example use cases:
57 // Which LaaS is this? Location as a service? Linux?
58 ** TRex on your laptop
63 [options="header",cols="1^,1^"]
65 |Cisco UCS Platform | Intel NIC
66 | image:images/ucs200_2.png[title="generator"] | image:images/Intel520.png[title="generator"]
69 === Purpose of this guide
71 This guide explains the use of TRex internals and the use of TRex together with Cisco ASR1000 Series routers. The examples illustrate novel traffic generation techniques made possible by TRex.
73 == Download and installation
75 === Hardware recommendations
77 TRex operates in a Linux application environment, interacting with Linux kernel modules.
78 TRex curretly works on x86 architecture and can operate well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex.
81 =====================================
82 A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
83 =====================================
86 =====================================
87 Not all supported DPDK interfaces are supported by TRex
88 =====================================
91 .Preferred UCS hardware
92 [options="header",cols="1,3"]
95 | UCS C220 M3/M4 | *Preferred Low-End*. Supports up to 40Gb/sec with 540-D2. With newer Intel NIC (recommended), supports 80Gb/sec with 1RU. See table below describing components.
96 | UCS C200| Early UCS model.
97 | UCS C210 M2 | Supports up to 40Gb/sec PCIe3.0.
98 | UCS C240 M3/M4 | *Preferred, High-End* Supports up to 200Gb/sec. 6x XL710 NICS (PCIex8) or 2xFM10K (PCIex16). See table below describing components.
99 | UCS C260M2 | Supports up to 30Gb/sec (limited by V2 PCIe).
102 .Low-End UCS C220 M4 - Internal components
103 [options="header",cols="1,2",width="60%"]
105 | Components | Details
106 | CPU | 2x E5-2620 @ 2.0 GHz.
107 | CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
108 | Memory | 2x4 banks f.or each CPU. Total of 32GB in 8 banks.
112 .High-End C240 M4 - Internal components
113 [options="header",cols="1,2",width="60%"]
115 | Components | Details
116 | CPU | 2x E5-2667 @ 3.20 GHz.
117 | PCIe | 1x Riser PCI expansion card option A PID UCSC-PCI-1A-240M4 enables 2 PCIex16.
118 | CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
119 | Memory | 2x4 banks for each CPU. Total of 32GB in 8 banks.
121 | Riser 1/2 | both left and right should support x16 PCIe. Right (Riser1) should be from option A x16 and Left (Riser2) should be x16. need to order both
125 [options="header",cols="1,1,4",width="90%"]
127 | Chipset | Bandwidth (Gb/sec) | Example
128 | Intel I350 | 1 | Intel 4x1GE 350-T4 NIC
129 | Intel 82599 | 10 | Cisco part ID:N2XX-AIPCI01 Intel x520-D2, Intel X520 Dual Port 10Gb SFP+ Adapter
130 | Intel X710 | 10 | Cisco part ID:UCSC-PCIE-IQ10GF link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+], *Preferred* support per stream stats in hardware link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L]
131 | Intel XL710 | 40 | Cisco part ID:UCSC-PCIE-ID40GF, link:https://en.wikipedia.org/wiki/QSFP[QSFP+] (copper/optical)
132 | Intel FM10420 | 25/100 | QSFP28, by Silicom link:http://www.silicom-usa.com/100_Gigabit_Dual_Port_Fiber_Ethernet_PCI_Express_PE3100G2DQiR_96[Silicom PE3100G2DQiR_96] (*in development*)
133 | Mellanox ConnectX-4 | 25/40/50/56/100 | QSFP28, link:http://www.mellanox.com/page/products_dyn?product_family=201&[ConnectX-4] link:http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf[ConnectX-4-brief] (copper/optical) supported from v2.11 more details xref:connectx_support[TRex Support]
134 | Mellanox ConnectX-5 | 25/40/50/56/100 | Not supported yet
135 | Cisco 1300 series | 40 | QSFP+, VIC 1380, VIC 1385, VIC 1387 see more xref:ciscovic_support[TRex Support]
137 VMXNET3 (see notes) | VMware paravirtualized | Connect using VMware vSwitch
138 | E1000 | paravirtualized | VMware/KVM/VirtualBox
139 | Virtio | paravirtualized | KVM
142 // in table above, is it correct to list "paravirtualized" as chipset? Also, what is QSFP28? It does not appear on the lined URL. Clarify: is Intel X710 the preferred NIC?
145 [options="header",cols="2,1,1,1",width="90%"]
147 | link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+] | Intel Ethernet Converged X710-DAX | Silicom link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[PE310G4i71L] (Open optic) | 82599EB 10-Gigabit
148 | link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-SR] | Does not work | [green]*works* | [green]*works*
149 | link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-LR] | Does not work | [green]*works* | [green]*works*
150 | link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-H10GB-CU1M]| [green]*works* | [green]*works* | [green]*works*
151 | link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-AOC1M] | [green]*works* | [green]*works* | [green]*works*
155 =====================================
156 Intel X710 NIC (example: FH X710DA4FHBLK) operates *only* with Intel SFP+. For open optic, use the link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L] NIC.
157 =====================================
159 // clarify above table and note
161 .XL710 NIC base QSFP+ support
162 [options="header",cols="1,1,1",width="90%"]
164 | link:https://en.wikipedia.org/wiki/QSFP[QSFP+] | Intel Ethernet Converged XL710-QDAX | Silicom link:http://www.silicom-usa.com/Dual_Port_Fiber_40_Gigabit_Ethernet_PCI_Express_Server_Adapter_PE340G2Qi71_83[PE340G2Qi71] Open optic
165 | QSFP+ SR4 optics | APPROVED OPTICS [green]*works*, Cisco QSFP-40G-SR4-S does *not* work | Cisco QSFP-40G-SR4-S [green]*works*
166 | QSFP+ LR-4 Optics | APPROVED OPTICS [green]*works*, Cisco QSFP-40G-LR4-S does *not* work | Cisco QSFP-40G-LR4-S [green]*works*
167 | QSFP Active Optical Cables (AoC) | Cisco QSFP-H40G-AOC [green]*works* | Cisco QSFP-H40G-AOC [green]*works*
168 | QSFP+ Intel Ethernet Modular Optics | N/A | N/A
169 | QSFP+ DA twin-ax cables | N/A | N/A
170 | Active QSFP+ Copper Cables | Cisco QSFP-4SFP10G-CU [green]*works* | Cisco QSFP-4SFP10G-CU [green]*works*
174 =====================================
175 For Intel XL710 NICs, Cisco SR4/LR QSFP+ does not operate. Use Silicom with Open Optic.
176 =====================================
179 .ConnectX-4 NIC base QSFP28 support (100gb)
180 [options="header",cols="1,2",width="90%"]
182 | link:https://en.wikipedia.org/wiki/QSFP[QSFP28] | ConnectX-4
183 | QSFP28 SR4 optics | N/A
184 | QSFP28 LR-4 Optics | N/A
185 | QSFP28 (AoC) | Cisco QSFP-100G-AOCxM [green]*works*
186 | QSFP28 DA twin-ax cables | Cisco QSFP-100G-CUxM [green]*works*
189 .Cisco VIC NIC base QSFP+ support
190 [options="header",cols="1,2",width="90%"]
192 | link:https://en.wikipedia.org/wiki/QSFP[QSFP+] | Intel Ethernet Converged XL710-QDAX
193 | QSFP+ SR4 optics | N/A
194 | QSFP+ LR-4 Optics | N/A
195 | QSFP Active Optical Cables (AoC) | Cisco QSFP-H40G-AOC [green]*works*
196 | QSFP+ Intel Ethernet Modular Optics | N/A
197 | QSFP+ DA twin-ax cables | N/A | N/A
198 | Active QSFP+ Copper Cables | N/A
202 // clarify above table and note. let's discuss.
203 .FM10K QSFP28 support
204 [options="header",cols="1,1",width="70%"]
210 // do we want to show "todo"? maybe "pending"
214 =====================================
215 * Intel SFP+ 10Gb/sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+.
216 // above, replace "only one" with "only mode"?
217 * For operating high speed throughput (example: several Intel XL710 40Gb/sec), use different link:https://en.wikipedia.org/wiki/Non-uniform_memory_access[NUMA] nodes for different NICs. +
218 To verify NUMA and NIC topology: `lstopo (yum install hwloc)` +
219 To display CPU info, including NUMA node: `lscpu` +
220 NUMA usage xref:numa-example[example]
221 * For Intel XL710 NICs, verify that the NVM is v5.04 . xref:xl710-firmware[Info].
222 ** `> sudo ./t-rex-64 -f cap2/dns.yaml -d 0 *-v 6* --nc | grep NVM` +
223 `PMD: FW 5.0 API 1.5 NVM 05.00.04 eetrack 800013fc`
224 =====================================
226 // above, maybe rename the bullet points "NIC usage notes"? should we create a subsection for NICs? Maybe it would be under "2.1 Hardware recommendations" as a subsection.
229 .Sample order for recommended low-end Cisco UCSC-C220-M3S with 4x10Gb ports
230 [options="header",cols="1,1",width="70%"]
232 | Component | Quantity
234 | UCS-CPU-E5-2650 | 2
235 | UCS-MR-1X041RY-A | 8
239 | SFS-250V-10A-IS | 1
247 // should table above say "low-end Cisco UCS C220 M3S" instead of "low-end USCS-C220-M3S"?
249 NOTE: Purchase the 10Gb/sec SFP+ separately. Cisco would be fine with TRex (but not for plain Linux driver).
250 // does note above mean "TRex operates with 10Gb/sec SFP+ components, but plain Linux does not provide drivers."? if so, how does purchasing separately solve this? where do they get drivers?
254 ==== Supported versions
256 Supported Linux versions:
258 * Fedora 20-23, 64-bit kernel (not 32-bit)
259 * Ubuntu 14.04.1 LTS, 64-bit kernel (not 32-bit)
260 * Ubuntu 16.xx LTS, 64-bit kernel (not 32-bit)
262 NOTE: Additional OS version may be supported by compiling the necessary drivers.
264 To check whether a kernel is 64-bit, verify that the ouput of the following command is `x86_64`.
275 ISO images for supported Linux releases can be downloaded from:
277 .Supported Linux ISO image links
278 [options="header",cols="1^,2^",width="50%"]
279 |======================================
280 | Distribution | SHA256 Checksum
281 | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-DVD.iso[Fedora 20]
282 | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-CHECKSUM[Fedora 20 CHECKSUM]
283 | link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-DVD-x86_64-21.iso[Fedora 21]
284 | link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-21-x86_64-CHECKSUM[Fedora 21 CHECKSUM]
285 | link:http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-desktop-amd64.iso[Ubuntu 14.04.1]
286 | http://old-releases.ubuntu.com/releases/14.04.1/SHA256SUMS[Ubuntu 14.04* CHECKSUMs]
287 | link:http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso[Ubuntu 16.04.1]
288 | http://releases.ubuntu.com/16.04.1/SHA256SUMS[Ubuntu 16.04* CHECKSUMs]
290 |======================================
292 For Fedora downloads...
294 * Select a mirror close to your location: +
295 https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora +
296 Choose: "Fedora Linux http" -> releases -> <version number> -> Server -> x86_64 -> iso -> Fedora-Server-DVD-x86_64-<version number>.iso
298 * Verify the checksum of the downloaded file matches the linked checksum values with the `sha256sum` command. Example:
302 $sha256sum Fedora-18-x86_64-DVD.iso
303 91c5f0aca391acf76a047e284144f90d66d3d5f5dcd26b01f368a43236832c03 #<1>
305 <1> Should be equal to the link:https://en.wikipedia.org/wiki/SHA-2[SHA-256] values described in the linked checksum files.
310 Ask your lab admin to install the Linux using CIMC, assign an IP, and set the DNS. Request the sudo or super user password to enable you to ping and SSH.
312 xref:fedora21_example[Example of installing Fedora 21 Server]
315 =====================================
316 * To use TRex, you should have sudo on the machine or the root password.
317 * Upgrading the linux Kernel using `yum upgrade` requires building the TRex drivers.
318 * In Ubuntu 16, auto-updater is enabled by default. It's advised to turn it off as with update of Kernel need to compile again the DPDK .ko file. +
319 Command to remove it: +
320 > sudo apt-get remove unattended-upgrades
321 =====================================
323 ==== Verify Intel NIC installation
325 Use `lspci` to verify the NIC installation.
327 Example 4x 10Gb/sec TRex configuration (see output below):
329 * I350 management port
331 * 4x Intel Ethernet Converged Network Adapter model x520-D2 (82599 chipset)
335 $[root@trex]lspci | grep Ethernet
336 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) #<1>
337 01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) #<2>
338 03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) #<3>
339 03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
340 82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
341 82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
345 <3> 10Gb/sec traffic ports (Intel 82599EB)
347 === Obtaining the TRex package
349 Connect using `ssh` to the TRex machine and execute the commands described below.
351 NOTE: Prerequisite: *$WEB_URL* is *{web_server_url}* or *{local_web_server_url}* (Cisco internal)
358 $wget --no-cache $WEB_URL/release/latest
363 Bleeding edge version:
366 $wget --no-cache $WEB_URL/release/be_latest
369 To obtain a specific version, do the following:
372 $wget --no-cache $WEB_URL/release/vX.XX.tar.gz #<1>
375 <1> X.XX = Version number
377 == First time Running
379 === Configuring for loopback
381 Before connecting TRex to your DUT, it is strongly advised to verify that TRex and the NICs work correctly in loopback. +
382 To get best performance, it is advised to loopback interfaces on the same NUMA (controlled by the same physical processor). If you do not know how to check this, you can ignore this advice for now. +
385 =====================================================================
386 If you are using 10Gbs NIC based on Intel 520-D2 NICs, and you loopback ports on the same NIC, using SFP+, it might not sync, and you will fail to get link up. +
387 We checked many types of SFP+ (Intel/Cisco/SR/LR) and it worked for us. +
388 If you still encounter link issues, you can either try to loopback interfaces from different NICs, or use link:http://www.fiberopticshare.com/tag/cisco-10g-twinax[Cisco twinax copper cable].
389 =====================================================================
392 image:images/loopback_example.png[title="Loopback example"]
394 ==== Identify the ports
398 $>sudo ./dpdk_setup_ports.py -s
400 Network devices using DPDK-compatible driver
401 ============================================
403 Network devices using kernel driver
404 ===================================
405 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
406 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
407 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
408 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
409 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active* #<2>
411 Other network devices
412 =====================
416 <1> If you did not run any DPDK application, you will see list of interfaces binded to the kernel, or not binded at all.
417 <2> Interface marked as 'active' is the one used by your ssh connection. *Never* put it in TRex config file.
419 Choose ports to use and follow the instructions in the next section to create configuration file.
421 ==== Creating minimum configuration file
423 Default configuration file name is: `/etc/trex_cfg.yaml`.
425 You can copy basic configuration file from cfg folder
429 $cp cfg/simple_cfg.yaml /etc/trex_cfg.yaml
432 Then, edit the configuration file and put your interface's and IP addresses details.
441 #List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
442 interfaces : ["03:00.0", "03:00.1"] #<1>
443 port_info : # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
449 <1> You need to edit this line to match the interfaces you are using.
450 Notice that all NICs you are using should have the same type. You cannot mix different NIC types in one config file. For more info, see link:http://trex-tgn.cisco.com/youtrack/issue/trex-201[trex-201].
452 You can find xref:trex_config[here] full list of configuration file options.
454 === Script for creating config file
456 To help starting with basic configuration file that suits your needs, there a script that can automate this process.
457 The script helps you getting started, and you can then edit the file and add advanced options from xref:trex_config[here]
459 There are two ways to run the script. Interactively (script will pormpt you for parameters), or providing all parameters
460 using command line options.
462 ==== Interactive mode
466 sudo ./dpdk_setup_ports.py -i
469 You will see a list of available interfaces with their related information +
470 Just follow the instructions to get basic config file.
472 ==== Specifying input arguments using command line options
474 First, run this command to see the list of all interfaces and their related information:
478 sudo ./dpdk_setup_ports.py -t
481 * In case of *Loopback* and/or only *L1-L2 Switches* on the way, you do not need to provide IPs or destination MACs. +
482 The script Will assume the following interface connections: 0↔1, 2↔3 etc. +
487 sudo ./dpdk_setup_ports.py -c <TRex interface 0> <TRex interface 1> ...
490 * In case of *Router* (or other next hop device, such as *L3 Switch*), you should specify the TRex IPs and default gateways, or
491 MACs of the router as described below.
493 .Additional arguments to creating script (dpdk_setup_ports.py -c)
494 [options="header",cols="2,5,3",width="100%"]
496 | Arg | Description | Example
497 | -c | Create a configuration file by specified interfaces (PCI address or Linux names: eth1 etc.) | -c 03:00.1 eth1 eth4 84:00.0
498 | --dump | Dump created config to screen. |
499 | -o | Output the config to this file. | -o /etc/trex_cfg.yaml
500 | --dest-macs | Destination MACs to be used per each interface. Specify this option if you want MAC based config instead of IP based one. You must not set it together with --ip and --def_gw | --dest-macs 11:11:11:11:11:11 22:22:22:22:22:22
501 | --ip | List of IPs to use for each interface. If this option and --dest-macs is not specified, script assumes loopback connections (0↔1, 2↔3 etc.) | --ip 1.2.3.4 5.6.7.8
502 |--def-gw | List of default gateways to use for each interface. If --ip given, you must provide --def_gw as well | --def-gw 3.4.5.6 7.8.9.10
503 | --ci | Cores include: White list of cores to use. Make sure there is enough for each NUMA. | --ci 0 2 4 5 6
504 | --ce | Cores exclude: Black list of cores to exclude. Make sure there will be enough for each NUMA. | --ci 10 11 12
505 | --no-ht | No HyperThreading: Use only one thread of each Core in created config yaml. |
506 | --prefix | Advanced option: prefix to be used in TRex config in case of parallel instances. | --prefix first_instance
507 | --zmq-pub-port | Advanced option: ZMQ Publisher port to be used in TRex config in case of parallel instances. | --zmq-pub-port 4000
508 | --zmq-rpc-port | Advanced option: ZMQ RPC port to be used in TRex config in case of parallel instances. | --zmq-rpc-port
509 | --ignore-numa | Advanced option: Ignore NUMAs for config creation. Use this option only if you have to, as it might reduce performance. For example, if you have pair of interfaces at different NUMAs |
512 === Configuring ESXi for running TRex
514 To get best performance, it is advised to run TRex on bare metal hardware, and not use any kind of VM.
515 Bandwidth on VM might be limited, and IPv6 might not be fully supported.
516 Having said that, there are sometimes benefits for running on VM. +
518 * Virtual NICs can be used to bridge between TRex and NICs not supported by TRex. +
519 * If you already have VM installed, and do not require high performance. +
521 1. Click the host machine, enter Configuration -> Networking.
523 a. One of the NICs should be connected to the main vSwitch network to get an "outside" connection, for the TRex client and ssh: +
524 image:images/vSwitch_main.png[title="vSwitch_main"]
526 b. Other NICs that are used for TRex traffic should be in distinguish vSwitch: +
527 image:images/vSwitch_loopback.png[title="vSwitch_loopback"]
529 2. Right-click guest machine -> Edit settings -> Ensure the NICs are set to their networks: +
530 image:images/vSwitch_networks.png[title="vSwitch_networks"]
533 =====================================================================
534 Before version 2.10, the following command did not function as expected:
537 sudo ./t-rex-64 -f cap2/dns.yaml *--lm 1 --lo* -l 1000 -d 100
539 The vSwitch did not "know" where to route the packet. Was solved on version 2.10 when TRex started to support ARP.
540 =====================================================================
542 * Pass-through is the way to use directly the NICs from host machine inside the VM. Has no limitations except the NIC/hardware itself. The only difference via bare-metal OS is occasional spikes of latency (~10ms). Passthrough settings cannot be saved to OVA.
544 1. Click on the host machine. Enter Configuration -> Advanced settings -> Edit. Mark the desired NICs. Reboot the ESXi to apply. +
545 image:images/passthrough_marking.png[title="passthrough_marking"]
547 2. Right click on guest machine. Edit settings -> Add -> *PCI device* -> Choose the NICs one by one. +
548 image:images/passthrough_adding.png[title="passthrough_adding"]
550 === Configuring for running with router (or other L3 device) as DUT
552 You can follow link:trex_config_guide.html[this] presentation for an example of how to configure router as DUT.
556 When all is set, use the following command to start basic TRex run for 10 seconds
557 (it will use the default config file name /etc/trex_cfg.yaml):
560 $sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 10 -l 1000
563 If successful, the output will be similar to the following:
567 $ sudo ./t-rex-64 -f cap2/dns.yaml -d 10 -l 1000
568 Starting TRex 2.09 please wait ...
569 zmq publisher at: tcp://*:4500
570 number of ports found : 4
573 link : link : Link Up - speed 10000 Mbps - full-duplex <1>
577 link : link : Link Up - speed 10000 Mbps - full-duplex
581 link : link : Link Up - speed 10000 Mbps - full-duplex
585 link : link : Link Up - speed 10000 Mbps - full-duplex
589 -Per port stats table
590 ports | 0 | 1 | 2 | 3
591 -------------------------------------------------------------------------------------
592 opackets | 1003 | 1003 | 1002 | 1002
593 obytes | 66213 | 66229 | 66132 | 66132
594 ipackets | 1003 | 1003 | 1002 | 1002
595 ibytes | 66225 | 66209 | 66132 | 66132
596 ierrors | 0 | 0 | 0 | 0
597 oerrors | 0 | 0 | 0 | 0
598 Tx Bw | 217.09 Kbps | 217.14 Kbps | 216.83 Kbps | 216.83 Kbps
600 -Global stats enabled
601 Cpu Utilization : 0.0 % <2> 29.7 Gb/core <3>
602 Platform_factor : 1.0
603 Total-Tx : 867.89 Kbps <4>
604 Total-Rx : 867.86 Kbps <5>
605 Total-PPS : 1.64 Kpps
608 Expected-PPS : 2.00 pps <6>
609 Expected-CPS : 1.00 cps <7>
610 Expected-BPS : 1.36 Kbps <8>
612 Active-flows : 0 <9> Clients : 510 Socket-util : 0.0000 %
613 Open-flows : 1 <10> Servers : 254 Socket : 1 Socket/Clients : 0.0
614 drop-rate : 0.00 bps <11>
615 current time : 5.3 sec
616 test duration : 94.7 sec
618 -Latency stats enabled
619 Cpu Utilization : 0.2 % <12>
620 if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window
621 | , , check, , latency(usec),latency (usec) ,(usec) ,
622 --------------------------------------------------------------------------------------------------
623 0 | 1002, 1002, 0, 0, 51 , 69, 0 | 0 69 67 <13>
624 1 | 1002, 1002, 0, 0, 53 , 196, 0 | 0 196 53
625 2 | 1002, 1002, 0, 0, 54 , 71, 0 | 0 71 69
626 3 | 1002, 1002, 0, 0, 53 , 193, 0 | 0 193 52
628 <1> Link must be up for TRex to work.
629 <2> Average CPU utilization of transmitters threads. For best results it should be lower than 80%.
630 <3> Gb/sec generated per core of DP. Higher is better.
631 <4> Total Tx must be the same as Rx at the end of the run
632 <5> Total Rx must be the same as Tx at the end of the run
633 <6> Expected number of packets per second (calculated without latency packets).
634 <7> Expected number of connections per second (calculated without latency packets).
635 <8> Expected number of bits per second (calculated without latency packets).
636 <9> Number of TRex active "flows". Could be different than the number of router flows, due to aging issues. Usualy the TRex number of active flows is much lower than that of the router because the router ages flows slower.
637 <10> Total number of TRex flows opened since startup (including active ones, and ones already closed).
639 <12> Rx and latency thread CPU utilization.
640 <13> Tx_ok on port 0 should equal Rx_ok on port 1, and vice versa.
642 More statistics information:
644 *socket*:: Same as the active flows.
646 *Socket/Clients*:: Average of active flows per client, calculated as active_flows/#clients.
648 *Socket-util*:: Estimation of number of L4 ports (sockets) used per client IP. This is approximately (100*active_flows/#clients)/64K, calculated as (average active flows per client*100/64K). Utilization of more than 50% means that TRex is generating too many flows per single client, and that more clients must be added in the generator config.
649 // clarify above, especially the formula
651 *Max window*:: Momentary maximum latency for a time window of 500 msec. There are few numbers shown per port.
652 The newest number (last 500msec) is on the right. Oldest on the left. This can help identifying spikes of high latency clearing after some time. Maximum latency is the total maximum over the entire test duration. To best understand this,
653 run TRex with latency option (-l) and watch the results with this section in mind.
655 *Platform_factor*:: There are cases in which we duplicate the traffic using splitter/switch and we would like all numbers displayed by TRex to be multiplied by this factor, so that TRex counters will match the DUT counters.
657 WARNING: If you don't see rx packets, revisit your MAC address configuration.
659 include::trex_book_basic.asciidoc[]
663 === VLAN Trunk support
667 The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation when the traffic profile is asymmetric. Example: Asymmetric SFR profile.
668 This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces.
669 This requires TRex to send the traffic on two VLANs, as described below.
674 vlan : { enable : 1 , vlan0 : 100 , vlan1 : 200 }
682 vlan : { enable : 1 , vlan0 : 100 , vlan1 : 200 } <1>
684 <1> Enable VLAN feature, vlan0==100 , vlan1==200
686 *Problem definition:*::
688 Scenario: TRex with two ports and an SFR traffic profile.
690 .Without VLAN/sub interfaces
693 0 ( client) -> [ ] - 1 ( server)
695 Without VLAN support the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% is from port 1 (server). Port 1 become the bottlneck (10Gb/s limit) before port 0.
697 .With VLAN/sub interfaces
700 port 0 ( client VLAN0) <-> | | <-> port 1 ( server-VLAN0)
701 port 0 ( server VLAN1) <-> | | <-> port 1 ( client-VLAN1)
704 In this case both ports have the same amount of traffic.
706 *Router configuation:*::
710 interface TenGigabitEthernet1/0/0 <1>
711 mac-address 0000.0001.0000
717 interface TenGigabitEthernet1/0/0.100
718 encapsulation dot1Q 100 <2>
719 ip address 11.77.11.1 255.255.255.0
720 ip nbar protocol-discovery
721 ip policy route-map vlan_100_p1_to_p2 <3>
723 interface TenGigabitEthernet1/0/0.200
724 encapsulation dot1Q 200 <4>
725 ip address 11.88.11.1 255.255.255.0
726 ip nbar protocol-discovery
727 ip policy route-map vlan_200_p1_to_p2 <5>
729 interface TenGigabitEthernet1/1/0
730 mac-address 0000.0001.0000
735 interface TenGigabitEthernet1/1/0.100
736 encapsulation dot1Q 100
737 ip address 22.77.11.1 255.255.255.0
738 ip nbar protocol-discovery
739 ip policy route-map vlan_100_p2_to_p1
741 interface TenGigabitEthernet1/1/0.200
742 encapsulation dot1Q 200
743 ip address 22.88.11.1 255.255.255.0
744 ip nbar protocol-discovery
745 ip policy route-map vlan_200_p2_to_p1
748 arp 11.77.11.12 0000.0001.0000 ARPA <6>
749 arp 22.77.11.12 0000.0001.0000 ARPA
751 route-map vlan_100_p1_to_p2 permit 10 <7>
752 set ip next-hop 22.77.11.12
754 route-map vlan_100_p2_to_p1 permit 10
755 set ip next-hop 11.77.11.12
758 route-map vlan_200_p1_to_p2 permit 10
759 set ip next-hop 22.88.11.12
761 route-map vlan_200_p2_to_p1 permit 10
762 set ip next-hop 11.88.11.12
765 <1> Disable the IP on the main port it is important.
766 // above, clarify what's important
768 <3> PBR configuration
770 <5> PBR configuration
771 <6> TRex destination port MAC address
772 <7> PBR configuration rules
774 === Static source MAC address setting
776 With this feature, TRex replaces the source MAC address with the client IP address.
778 Note: This feature was requested by the Cisco ISG group.
784 mac_override_by_ip : true
792 mac_override_by_ip : true <1>
794 <1> In this case, the client side MAC address looks like this:
795 SRC_MAC = IPV4(IP) + 00:00
799 Support for IPv6 includes:
801 1. Support for pcap files containing IPv6 packets
802 2. Ability to generate IPv6 traffic from pcap files containing IPv4 packets
803 The following command line option enables this feature: `--ipv6`
804 The keywords (`src_ipv6` and `dst_ipv6`) specify the most significant 96 bits of the IPv6 address - for example:
808 src_ipv6 : [0xFE80,0x0232,0x1002,0x0051,0x0000,0x0000]
809 dst_ipv6 : [0x2001,0x0DB8,0x0003,0x0004,0x0000,0x0000]
812 The IPv6 address is formed by placing what would typically be the IPv4
813 address into the least significant 32 bits and copying the value provided
814 in the src_ipv6/dst_ipv6 keywords into the most signficant 96 bits.
815 If src_ipv6 and dst_ipv6 are not specified, the default
816 is to form IPv4-compatible addresses (most signifcant 96 bits are zero).
818 There is support for all plugins.
823 $sudo ./t-rex-64 -f cap2l/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30 --ipv6
828 * TRex cannot generate both IPv4 and IPv6 traffic.
829 * The `--ipv6` switch must be specified even when using pcap file containing only IPv6 packets.
832 *Router configuration:*::
836 interface TenGigabitEthernet1/0/0
837 mac-address 0000.0001.0000
839 ip address 11.11.11.11 255.255.255.0
840 ip policy route-map p1_to_p2
843 ipv6 address 2001:DB8:1111:2222::1/64 <1>
844 ipv6 policy route-map ipv6_p1_to_p2 <2>
848 ipv6 unicast-routing <3>
850 ipv6 neighbor 3001::2 TenGigabitEthernet0/1/0 0000.0002.0002 <4>
851 ipv6 neighbor 2001::2 TenGigabitEthernet0/0/0 0000.0003.0002
853 route-map ipv6_p1_to_p2 permit 10 <5>
854 set ipv6 next-hop 2001::2
856 route-map ipv6_p2_to_p1 permit 10
857 set ipv6 next-hop 3001::2
861 asr1k(config)#ipv6 route 4000::/64 2001::2
862 asr1k(config)#ipv6 route 5000::/64 3001::2
866 <3> Enable IPv6 routing
867 <4> MAC address setting. Should be TRex MAC.
871 === Client clustering configuration
872 TRex supports testing complex topologies, with more than one DUT, using a feature called "client clustering".
873 This feature allows specifying the distribution of clients TRex emulates.
875 Let's look at the following topology:
878 image:images/topology.png[title="Client Clustering",width=850]
880 We have two clusters of DUTs.
881 Using config file, you can partition TRex emulated clients to groups, and define
882 how they will be spread between the DUT clusters.
884 Group configuration includes:
888 * Initiator side configuration. - These are the parameters affecting packets sent from client side.
889 * Responder side configuration. - These are the parameters affecting packets sent from server side.
892 It is important to understand that this is *complimentary* to the client generator
893 configured per profile - it only defines how the clients will be spread between clusters.
895 Let's look at an example.
897 We have a profile defining client generator.
905 clients_start : "16.0.0.1"
906 clients_end : "16.0.0.255"
907 servers_start : "48.0.0.1"
908 servers_end : "48.0.0.255"
909 dual_port_mask : "1.0.0.0"
911 - name: cap2/dns.pcap
918 We want to create two clusters with 4 and 3 devices respectively.
919 We also want to send *80%* of the traffic to the upper cluster and *20%* to the lower cluster.
920 We can specify to which DUT the packet will be sent by MAC address or IP. We will present a MAC
921 based example, and then see how to change to be IP based.
923 We will create the following cluster configuration file.
928 # Client configuration example file
929 # The file must contain the following fields
931 # 'vlan' - if the entire configuration uses VLAN,
932 # each client group must include vlan
935 # 'groups' - each client group must contain range of IPs
936 # and initiator and responder section
937 # 'count' represents the number of different DUTs
941 # 'true' means each group must contain VLAN configuration. 'false' means no VLAN config allowed.
946 - ip_start : 16.0.0.1
950 dst_mac : "00:00:00:01:00:00"
953 dst_mac : "00:00:00:02:00:00"
957 - ip_start : 16.0.0.205
961 dst_mac : "00:00:01:00:00:00"
965 dst_mac : "00:00:02:00:00:00"
971 The above configuration will divide the generator range of 255 clients to two clusters. The range
972 of IPs in all groups in the client config file together, must cover the entire range of client IPs
973 from the traffic profile file.
975 MACs will be allocated incrementally, with a wrap around after count addresses.
979 *Initiator side: (packets with source in 16.x.x.x net)*
981 * 16.0.0.1 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00 vlan: 100
982 * 16.0.0.2 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01 vlan: 100
983 * 16.0.0.3 -> 48.x.x.x - dst_mac: 00:00:00:01:00:02 vlan: 100
984 * 16.0.0.4 -> 48.x.x.x - dst_mac: 00:00:00:01:00:03 vlan: 100
985 * 16.0.0.5 -> 48.x.x.x - dst_mac: 00:00:00:01:00:00 vlan: 100
986 * 16.0.0.6 -> 48.x.x.x - dst_mac: 00:00:00:01:00:01 vlan: 100
988 *responder side: (packets with source in 48.x.x.x net)*
990 * 48.x.x.x -> 16.0.0.1 - dst_mac(from responder) : "00:00:00:02:00:00" , vlan:200
991 * 48.x.x.x -> 16.0.0.2 - dst_mac(from responder) : "00:00:00:02:00:01" , vlan:200
994 This means that the DUT MACs of each cluster has to be changed to be sequential. Other option is to
995 specify instead of ``dst_mac'', ip address, using ``next_hop''. +
996 For example, config file first group will look like:
999 - ip_start : 16.0.0.1
1013 In this case, TRex will try to resolve using ARP requests the addresses
1014 1.1.1.1, 1.1.1.2, 1.1.1.3, 1.1.1.4 (and the range 2.2.2.1-2.2.2.4). If not all IPs are resolved,
1015 TRex will exit with an error message. ``src_ip'' will be used for sending gratitues ARP, and
1016 for filling relevant fields in ARP request. If no ``src_ip'' given, TRex will look for source
1017 IP in the relevant port section in the platform config file (/etc/trex_cfg.yaml). If none is found, TRex
1018 will exit with an error message. +
1019 If client config file is given, the ``dest_mac'' and ``default_gw'' parameters from the platform config
1024 It is important to understand that the ip to MAC coupling (both with MAC based config or IP based)
1025 is done at the beginning and never changes. Meaning, for example, for the MAC case, packets
1026 with source IP 16.0.0.2 will always have VLAN 100 and dst MAC 00:00:00:01:00:01.
1027 Packets with destination IP 16.0.0.2 will always have VLAN 200 and dst MAC "00:00:00:02:00:01.
1028 This way, you can predict exactly which packet (and how many packets) will go to each DUT.
1034 sudo ./t-rex-64 -f cap2/dns.yaml --client_cfg my_cfg.yaml
1039 TRex can learn dynamic NAT/PAT translation. To enable this feature add `--learn-mode <mode>` to the command line.
1040 To learn the NAT translation, TRex must embed information describing the flow a packet belongs to, in the first
1041 packet of each flow. This can be done in two different methods, depending on the chosen <mode>.
1045 Flow info is embedded in the ACK of the first TCP SYN.
1046 In this mode, there is a limitation that bidirectional UDP templates (for example, DNS) are not supported.
1047 This mode was developed for testing NAT with firewalls (which usually do not work with mode 2).
1048 In this mode, TRex also learn and compensate for TCP sequence number randomization that might be done by the DUT.
1049 TRex can learn and compensate for seq num randomization in both directions of the connection.
1053 Flow info is added in a special IPv4 option header (8 bytes long 0x10 id). The option is added only to the first packet in the flow.
1054 This mode does not work with DUTs that drop packets with IP options (for example, Cisco ASA firewall).
1058 This is like mode 1, with the only change being that TRex does not learn the seq num randomization in the server->client direction.
1059 This mode can give much better connections per second performance than mode 1 (still, for all existing firewalls, mode 1 cps rate is more than enough).
1063 *simple HTTP traffic*
1067 $sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4 -l 1000 -d 100000 -m 30 --learn-mode 1
1070 *SFR traffic without bundling/ALG support*
1074 $sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundling.yaml -c 4 -l 1000 -d 100000 -m 10 --learn-mode 2
1077 *NAT terminal counters:*::
1081 -Global stats enabled
1082 Cpu Utilization : 0.6 % 33.4 Gb/core
1083 Platform_factor : 1.0
1084 Total-Tx : 3.77 Gbps NAT time out : 917 <1> (0 in wait for syn+ack) <5>
1085 Total-Rx : 3.77 Gbps NAT aged flow id: 0 <2>
1086 Total-PPS : 505.72 Kpps Total NAT active: 163 <3> (12 waiting for syn) <6>
1087 Total-CPS : 13.43 Kcps Total NAT opened: 82677 <4>
1089 <1> Number of connections for which TRex had to send the next packet in the flow, but did not learn the NAT translation yet. Should be 0. Usually, value different than 0 is seen if the DUT drops the flow (probably because it can't handle the number of connections)
1090 <2> Number of flows for which when we got the translation info, flow was aged out already. Non 0 value here should be very rare. Can occur only when there is huge latency in the DUT input/output queue.
1091 <3> Number of flows for which we sent the first packet, but did not learn the NAT translation yet. Value seen depends on the connection per second rate and round trip time.
1092 <4> Total number of translations over the lifetime of the TRex instance. May be different from the total number of flows if template is uni-directional (and consequently does not need translation).
1093 <5> Out of the timed out flows, how many were timed out while waiting to learn the TCP seq num randomization of the server->client from the SYN+ACK packet (Seen only in --learn-mode 1)
1094 <6> Out of the active NAT sessions, how many are waiting to learn the client->server translation from the SYN packet (others are waiting for SYN+ACK from server) (Seen only in --learn-mode 1)
1096 *Configuration for Cisco ASR1000 Series:*::
1098 This feature was tested with the following configuration and sfr_delay_10_1g_no_bundling. yaml traffic profile.
1099 Client address range is 16.0.0.1 to 16.0.0.255
1103 interface TenGigabitEthernet1/0/0 <1>
1104 mac-address 0000.0001.0000
1106 ip address 11.11.11.11 255.255.255.0
1107 ip policy route-map p1_to_p2
1112 interface TenGigabitEthernet1/1/0
1113 mac-address 0000.0001.0000
1115 ip address 11.11.11.11 255.255.255.0
1116 ip policy route-map p1_to_p2
1120 ip nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0 <4>
1122 ip nat inside source list 7 pool my overload
1123 access-list 7 permit 16.0.0.0 0.0.0.255 <5>
1125 ip nat inside source list 8 pool my overload <6>
1126 access-list 8 permit 17.0.0.0 0.0.0.255
1128 <1> Must be connected to TRex Client port (router inside port)
1131 <4> Pool of outside address with overload
1132 <5> Match TRex YAML client range
1133 <6> In case of dual port TRex
1135 // verify 1 and 5 above; rephrased
1140 . The IPv6-IPv6 NAT feature does not exist on routers, so this feature can work only with IPv4.
1141 . Does not support NAT64.
1142 . Bundling/plugin is not fully supported. Consequently, sfr_delay_10.yaml does not work. Use sfr_delay_10_no_bundling.yaml instead.
1145 =====================================================================
1146 * `--learn-verify` is a TRex debug mechanism for testing the TRex learn mechanism.
1147 * Need to run it when DUT is configured without NAT. It will verify that the inside_ip==outside_ip and inside_port==outside_port.
1148 =====================================================================
1150 === Flow order/latency verification
1152 In normal mode (without this feature enabled), received traffic is not checked by software. Hardware (Intel NIC) testing for dropped packets occurs at the end of the test. The only exception is the Latency/Jitter packets.
1153 This is one reason that with TRex, you *cannot* check features that terminate traffic (for example TCP Proxy).
1154 To enable this feature, add `--rx-check <sample>` to the command line options, where <sample> is the sample rate.
1155 The number of flows that will be sent to the software for verification is (1/(sample_rate). For 40Gb/sec traffic you can use a sample rate of 1/128. Watch for Rx CPU% utilization.
1159 This feature changes the TTL of the sampled flows to 255 and expects to receive packets with TTL 254 or 255 (one routing hop). If you have more than one hop in your setup, use `--hops` to change it to a higher value. More than one hop is possible if there are number of routers betwean TRex client side and TRex server side.
1162 This feature ensures that:
1164 * Packets get out of DUT in order (from each flow perspective).
1165 * There are no packet drops (no need to wait for the end of the test). Without this flag, you must wait for the end of the test in order to identify packet drops, because there is always a difference between TX and Rx, due to RTT.
1171 $sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30 --rx-check 128
1176 Cpu Utilization : 0.1 % <1>
1177 if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window
1178 | , , check, , latency(usec),latency (usec) ,(usec) ,
1179 --------------------------------------------------------------------------------
1180 0 | 1002, 1002, 2501, 0, 61 , 70, 3 | 60
1181 1 | 1002, 1002, 2012, 0, 56 , 63, 2 | 50
1182 2 | 1002, 1002, 2322, 0, 66 , 74, 5 | 68
1183 3 | 1002, 1002, 1727, 0, 58 , 68, 2 | 52
1185 Rx Check stats enabled <2>
1186 -------------------------------------------------------------------------------------------
1187 rx check: avg/max/jitter latency, 94 , 744, 49 | 252 287 309 <3>
1189 active flows: <6> 10, fif: <5> 308, drop: 0, errors: 0 <4>
1190 -------------------------------------------------------------------------------------------
1192 <1> CPU% of the Rx thread. If it is too high, *increase* the sample rate.
1193 <2> Rx Check section. For more detailed info, press 'r' during the test or at the end of the test.
1194 <3> Average latency, max latency, jitter on the template flows in microseconds. This is usually *higher* than the latency check packet because the feature works more on this packet.
1195 <4> Drop counters and errors counter should be zero. If not, press 'r' to see the full report or view the report at the end of the test.
1196 <5> fif - First in flow. Number of new flows handled by the Rx thread.
1197 <6> active flows - number of active flows handled by rx thread
1199 .Press R to Display Full Report
1210 0 0 0 0 1041 0 0 0 0 0 0 0 0 min_delta : 10 usec
1213 max_d_time : 1041 usec
1214 sliding_average : 1 usec <2>
1219 tempate_id_ 0 , errors: 0, jitter: 61 <3>
1220 tempate_id_ 1 , errors: 0, jitter: 0
1221 tempate_id_ 2 , errors: 0, jitter: 0
1222 tempate_id_ 3 , errors: 0, jitter: 0
1223 tempate_id_ 4 , errors: 0, jitter: 0
1224 tempate_id_ 5 , errors: 0, jitter: 0
1225 tempate_id_ 6 , errors: 0, jitter: 0
1226 tempate_id_ 7 , errors: 0, jitter: 0
1227 tempate_id_ 8 , errors: 0, jitter: 0
1228 tempate_id_ 9 , errors: 0, jitter: 0
1229 tempate_id_10 , errors: 0, jitter: 0
1230 tempate_id_11 , errors: 0, jitter: 0
1231 tempate_id_12 , errors: 0, jitter: 0
1232 tempate_id_13 , errors: 0, jitter: 0
1233 tempate_id_14 , errors: 0, jitter: 0
1234 tempate_id_15 , errors: 0, jitter: 0
1242 <1> Errors, if any, shown here
1243 <2> Low pass filter on the active average of latency events
1244 <3> Error per template info
1246 // IGNORE: this line added to help rendition. Without this line, the "Notes and Limitations" section below does not appear.
1248 *Notes and Limitations:*::
1250 ** To receive the packets TRex does the following:
1251 *** Changes the TTL to 0xff and expects 0xFF (loopback) or oxFE (route). (Use `--hop` to configure this value.)
1252 *** Adds 24 bytes of metadata as ipv4/ipv6 option header.
1253 // clarify "ipv4/ipv6 option header" above
1257 === Traffic YAML (parameter of -f option)
1259 ==== Global Traffic YAML section
1263 - duration : 10.0 <1>
1265 distribution : "seq"
1266 clients_start : "16.0.0.1"
1267 clients_end : "16.0.0.255"
1268 servers_start : "48.0.0.1"
1269 servers_end : "48.0.0.255"
1270 clients_per_gb : 201
1272 dual_port_mask : "1.0.0.0"
1275 mac : [0x00,0x00,0x00,0x01,0x00,0x00] <3>
1277 cap_ipg_min : 30 <5>
1278 cap_override_ipg : 200 <6>
1279 vlan : { enable : 1 , vlan0 : 100 , vlan1 : 200 } <7>
1280 mac_override_by_ip : true <8>
1282 <1> Test duration (seconds). Can be overridden using the `-d` option.
1283 <2> See the link:trex_manual.html#_clients_servers_ip_allocation_scheme[generator] section.
1284 // what does note 2 mean? see somewhere else? isn't this simply the generator section?
1285 <3> Default source/destination MAC address. The configuration YAML can override this.
1286 <4> true (default) indicates that the IPG is taken from the cap file (also taking into account cap_ipg_min and cap_override_ipg if they exist). false indicates that IPG is taken from per template section.
1287 <5> The following two options can set the min ipg in microseconds: (if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg} )
1288 <6> Value to override (microseconds), as described in note above.
1289 <7> Enable vlan feature. See xref:trex_vlan[trex_vlan section] for info.
1290 <8> Enable MAC address replacement by client IP.
1292 ==== Per template section
1293 // clarify "per template"
1297 - name: cap2/dns.pcap <1>
1302 server_addr : "48.0.0.7" <6>
1303 one_app_server : true <7>
1306 <1> The name of the template pcap file. Can be relative path from the t-rex-64 image directory, or an absolute path. The pcap file should include only one flow. (Exception: in case of plug-ins).
1307 <2> Connection per second. This is the value that will be used if specifying -m 1 from command line (giving -m x will multiply this
1308 <3> If the global section of the YAML file includes `cap_ipg : false`, this line sets the inter-packet gap in microseconds.
1309 <4> Should be set to the same value as ipg (microseconds).
1310 <5> Default value: w=1. This indicates to the IP generator how to generate the flows. If w=2, two flows from the same template will be generated in a burst (more for HTTP that has burst of flows).
1311 <6> If `one_app_server` is set to true, then all templates will use the same server.
1312 <7> If the same server address is required, set this value to true.
1316 === Configuration YAML (parameter of --cfg option)
1318 anchor:trex_config[]
1320 The configuration file, in YAML format, configures TRex behavior, including:
1322 - IP address or MAC address for each port (source and destination).
1323 - Masked interfaces, to ensure that TRex does not try to use the management ports as traffic ports.
1324 - Changing the zmq/telnet TCP port.
1326 You specify which config file to use by adding --cfg <file name> to the command line arguments. +
1327 If no --cfg given, the default `/etc/trex_cfg.yaml` is used. +
1328 Configuration file examples can be found in the `$TREX_ROOT/scripts/cfg` folder.
1330 ==== Basic Configurations
1334 - port_limit : 2 #mandatory <1>
1335 version : 2 #mandatory <2>
1336 interfaces : ["03:00.0", "03:00.1"] #mandatory <3>
1337 #enable_zmq_pub : true #optional <4>
1338 #zmq_pub_port : 4500 #optional <5>
1339 #prefix : setup1 #optional <6>
1340 #limit_memory : 1024 #optional <7>
1342 port_bandwidth_gb : 10 #optional <9>
1343 port_info : # set eh mac addr mandatory
1344 - default_gw : 1.1.1.1 # port 0 <10>
1345 dest_mac : '00:00:00:01:00:00' # Either default_gw or dest_mac is mandatory <10>
1346 src_mac : '00:00:00:02:00:00' # optional <11>
1347 ip : 2.2.2.2 # optional <12>
1348 vlan : 15 # optional <13>
1349 - dest_mac : '00:00:00:03:00:00' # port 1
1350 src_mac : '00:00:00:04:00:00'
1351 - dest_mac : '00:00:00:05:00:00' # port 2
1352 src_mac : '00:00:00:06:00:00'
1353 - dest_mac : [0x0,0x0,0x0,0x7,0x0,0x01] # port 3 <14>
1354 src_mac : [0x0,0x0,0x0,0x8,0x0,0x02] # <14>
1356 <1> Number of ports. Should be equal to the number of interfaces listed in 3. - mandatory
1357 <2> Must be set to 2. - mandatory
1358 <3> List of interfaces to use. Run `sudo ./dpdk_setup_ports.py --show` to see the list you can choose from. - mandatory
1359 <4> Enable the ZMQ publisher for stats data, default is true.
1360 <5> ZMQ port number. Default value is good. If running two TRex instances on the same machine, each should be given distinct number. Otherwise, can remove this line.
1361 <6> If running two TRex instances on the same machine, each should be given distinct name. Otherwise, can remove this line. ( Passed to DPDK as --file-prefix arg)
1362 <7> Limit the amount of packet memory used. (Passed to dpdk as -m arg)
1363 <8> Number of threads (cores) TRex will use per interface pair ( Can be overridden by -c command line option )
1364 <9> The bandwidth of each interface in Gbs. In this example we have 10Gbs interfaces. For VM, put 1. Used to tune the amount of memory allocated by TRex.
1365 <10> TRex need to know the destination MAC address to use on each port. You can specify this in one of two ways: +
1366 Specify dest_mac directly. +
1367 Specify default_gw (since version 2.10). In this case (only if no dest_mac given), TRex will issue ARP request to this IP, and will use
1368 the result as dest MAC. If no dest_mac given, and no ARP response received, TRex will exit.
1370 <11> Source MAC to use when sending packets from this interface. If not given (since version 2.10), MAC address of the port will be used.
1371 <12> If given (since version 2.10), TRex will issue gratitues ARP for the ip + src MAC pair on appropriate port. In stateful mode,
1372 gratitues ARP for each ip will be sent every 120 seconds (Can be changed using --arp-refresh-period argument).
1373 <13> If given, gratitues ARP and ARP request will be sent using the given VLAN tag.
1374 <14> Old MAC address format. New format is supported since version v2.09.
1377 =========================================================================================
1378 If you use version earlier than 2.10, or choose to omit the ``ip''
1379 and have mac based configuration, be aware that TRex will not send any
1380 gratitues ARP and will not answer ARP requests. In this case, you must configure static
1381 ARP entries pointing to TRex port on your DUT. For an example config, you can look
1382 xref:trex_config[here].
1383 =========================================================================================
1385 To find out which interfaces (NIC ports) can be used, perform the following:
1389 $>sudo ./dpdk_setup_ports.py --show
1391 Network devices using DPDK-compatible driver
1392 ============================================
1394 Network devices using kernel driver
1395 ===================================
1396 0000:02:00.0 '82545EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio *Active* #<1>
1397 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
1398 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1399 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1400 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb
1402 Other network devices
1403 =====================
1406 <1> We see that 02:00.0 is active (our management port).
1407 <2> All other NIC ports (03:00.0, 03:00.1, 13:00.0, 13:00.1) can be used.
1409 minimum configuration file is:
1416 interfaces : ["03:00.0","03:00.1","13:00.1","13:00.0"]
1419 ==== Memory section configuration
1421 The memory section is optional. It is used when there is a need to tune the amount of memory used by TRex packet manager.
1422 Default values (from the TRex source code), are usually good for most users. Unless you have some unusual needs, you can
1423 eliminate this section.
1429 interfaces : ["03:00.0","03:00.1"]
1437 traffic_mbuf_64 : 16380 <3>
1438 traffic_mbuf_128 : 8190
1439 traffic_mbuf_256 : 8190
1440 traffic_mbuf_512 : 8190
1441 traffic_mbuf_1024 : 8190
1442 traffic_mbuf_2048 : 4096
1443 dp_flows : 1048576 <4>
1444 global_flows : 10240 <5>
1446 <1> Memory section header
1447 <2> Numbers of memory buffers allocated for packets in transit, per port pair. Numbers are specified per packet size.
1448 <3> Numbers of memory buffers allocated for holding the part of the packet which is remained unchanged per template.
1449 You should increase numbers here, only if you have very large amount of templates.
1450 <4> Number of TRex flow objects allocated (To get best performance they are allocated upfront, and not dynamically).
1451 If you expect more concurrent flows than the default (1048576), enlarge this.
1452 <5> Number objects TRex allocates for holding NAT ``in transit'' connections. In stateful mode, TRex learn NAT
1453 translation by looking at the address changes done by the DUT to the first packet of each flow. So, these are the
1454 number of flows for which TRex sent the first flow packet, but did not learn the translation yet. Again, default
1455 here (10240) should be good. Increase only if you use NAT and see issues.
1458 ==== Platform section configuration
1460 The platform section is optional. It is used to tune the performance and allocate the cores to the right NUMA
1461 a configuration file now has the folowing struct to support multi instance
1466 interfaces : ["03:00.0","03:00.1"]
1470 master_thread_id : 0 <2>
1471 latency_thread_id : 5 <3>
1474 threads : [1,2,3,4] <6>
1476 <1> Platform section header.
1477 <2> Hardware thread_id for control thread.
1478 <3> Hardware thread_id for RX thread.
1479 <4> ``dual_if'' section defines info for interface pairs (according to the order in ``interfaces'' list).
1480 each section, starting with ``- socket'' defines info for different interface pair.
1481 <5> The NUMA node from which memory will be allocated for use by the interface pair.
1482 <6> Hardware threads to be used for sending packets for the interface pair. Threads are pinned to cores, so specifying threads
1483 actually determines the hardware cores.
1485 *Real example:* anchor:numa-example[]
1487 We connected 2 Intel XL710 NICs close to each other on the motherboard. They shared the same NUMA:
1489 image:images/same_numa.png[title="2_NICSs_same_NUMA"]
1491 CPU utilization was very high ~100%, with c=2 and c=4 the results were same.
1493 Then, we moved the cards to different NUMAs:
1495 image:images/different_numa.png[title="2_NICSs_different_NUMAs"]
1498 We added configuration to the /etc/trex_cfg.yaml:
1502 master_thread_id : 0
1503 latency_thread_id : 8
1506 threads : [1, 2, 3, 4, 5, 6, 7]
1508 threads : [9, 10, 11, 12, 13, 14, 15]
1510 This gave best results: with *\~98 Gb/s* TX BW and c=7, CPU utilization became *~21%*! (40% with c=4)
1512 === Command line options
1516 *--allow-coredump*::
1517 Allow creation of core dump.
1519 *--arp-refresh-period <num>*::
1520 Period in seconds between sending of gratuitous ARP for our addresses. Value of 0 means ``never send``.
1523 Number of hardware threads to use per interface pair. Use at least 4 for TRex 40Gbs. +
1524 TRex uses 2 threads for inner needs. Rest of the threads can be used. Maximum number here, can be number of free threads
1525 divided by number of interface pairs. +
1526 For virtual NICs on VM, we always use one thread per interface pair.
1528 *--cfg <file name>*::
1529 TRex configuration file to use. See relevant manual section for all config file options.
1531 *--checksum-offload*::
1532 Enable IP, TCP and UDP tx checksum offloading, using DPDK. This requires all used interfaces to support this.
1534 *--client_cfg <file>*::
1535 YAML file describing clients configuration. Look link:trex_manual.html#_client_clustering_configuration[here] for details.
1538 Duration of the test in seconds.
1541 Same as `-p`, but change the src/dst IP according to the port. Using this, you will get all the packets of the
1542 same flow from the same port, and with the same src/dst IP. +
1543 It will not work good with NBAR as it expects all clients ip to be sent from same direction.
1546 Specify traffic YAML configuration file to use. Mandatory option for stateful mode.
1549 Provide number of hops in the setup (default is one hop). Relevant only if the Rx check is enabled.
1550 Look link:trex_manual.html#_flow_order_latency_verification[here] for details.
1553 I/O mode. Possible values: 0 (silent), 1 (normal), 2 (short).
1556 Convert templates to IPv6 mode.
1559 Run ``warm up'' traffic for num seconds before starting the test. This is needed if TRex is connected to switch running
1560 spanning tree. You want the switch to see traffic from all relevant source MAC addresses before starting to send real
1561 data. Traffic sent is the same used for the latency test (-l option) +
1562 Current limitation (holds for TRex version 1.82): does not work properly on VM.
1565 In parallel to the test, run latency check, sending packets at rate/sec from each interface.
1567 *--learn-mode <mode>*::
1568 Learn the dynamic NAT translation. Look link:trex_manual.html#_nat_support[here] for details.
1571 Used for testing the NAT learning mechanism. Do the learning as if DUT is doing NAT, but verify that packets
1572 are not actually changed.
1574 *--limit-ports <port num>*::
1575 Limit the number of ports used. Overrides the ``port_limit'' from config file.
1577 *--lm <hex bit mask>*::
1578 Mask specifying which ports will send traffic. For example, 0x1 - Only port 0 will send. 0x4 - only port 2 will send.
1579 This can be used to verify port connectivity. You can send packets from one port, and look at counters on the DUT.
1582 Latency only - Send only latency packets. Do not send packets from the templates/pcap files.
1585 Rate multiplier. TRex will multiply the CPS rate of each template by num.
1588 If set, will terminate exacly at the end of the specified duration.
1589 This provides faster, more accurate TRex termination.
1590 By default (without this option), TRex waits for all flows to terminate gracefully. In case of a very long flow, termination might prolong.
1592 *--no-flow-control-change*::
1593 Prevents TRex from changing flow control. By default (without this option), TRex disables flow control at startup for all cards, except for the Intel XL710 40G card.
1595 *--no-key*:: Daemon mode, don't get input from keyboard.
1597 *--no-watchdog*:: Disable watchdog.
1600 Send all packets of the same flow from the same direction. For each flow, TRex will randomly choose between client port and
1601 server port, and send all the packets from this port. src/dst IPs keep their values as if packets are sent from two ports.
1602 Meaning, we get on the same port packets from client to server, and from server to client. +
1603 If you are using this with a router, you can not relay on routing rules to pass traffic to TRex, you must configure policy
1604 based routes to pass all traffic from one DUT port to the other. +
1607 Platform factor. If the setup includes splitter, you can multiply all statistic number displayed by TRex by this factor, so that they will match the DUT counters.
1610 Disable ZMQ monitor's publishers.
1612 *--rx-check <sample rate>*::
1613 Enable Rx check module. Using this, each thread randomly samples 1/sample_rate of the flows and checks packet order, latency, and additional statistics for the sampled flows.
1614 Note: This feature works on the RX thread.
1616 *-v <verbosity level>*::
1617 Show debug info. Value of 1 shows debug info on startup. Value of 3, shows debug info during run at some cases. Might slow down operation.
1619 *--vlan*:: Relevant only for stateless mode with Intel 82599 10G NIC.
1620 When configuring flow stat and latency per stream rules, assume all streams uses VLAN.
1622 *-w <num seconds>*::
1623 Wait additional time between NICs initialization and sending traffic. Can be useful if DUT needs extra setup time. Default is 1 second.
1625 ifndef::backend-docbook[]
1628 endif::backend-docbook[]
1634 The TRex simulator is a linux application (no DPDK needed) that can run on any Linux (it can also run on TRex machine itself).
1635 you can create output pcap file from input of traffic YAML.
1643 $./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1
1645 -- loading cap file avl/delay_10_http_get_0.pcap
1646 -- loading cap file avl/delay_10_http_post_0.pcap
1647 -- loading cap file avl/delay_10_https_0.pcap
1648 -- loading cap file avl/delay_10_http_browsing_0.pcap
1649 -- loading cap file avl/delay_10_exchange_0.pcap
1650 -- loading cap file avl/delay_10_mail_pop_0.pcap
1651 -- loading cap file avl/delay_10_mail_pop_1.pcap
1652 -- loading cap file avl/delay_10_mail_pop_2.pcap
1653 -- loading cap file avl/delay_10_oracle_0.pcap
1654 -- loading cap file avl/delay_10_rtp_160k_full.pcap
1655 -- loading cap file avl/delay_10_rtp_250k_full.pcap
1656 -- loading cap file avl/delay_10_smtp_0.pcap
1657 -- loading cap file avl/delay_10_smtp_1.pcap
1658 -- loading cap file avl/delay_10_smtp_2.pcap
1659 -- loading cap file avl/delay_10_video_call_0.pcap
1660 -- loading cap file avl/delay_10_sip_video_call_full.pcap
1661 -- loading cap file avl/delay_10_citrix_0.pcap
1662 -- loading cap file avl/delay_10_dns_0.pcap
1663 id,name , tps, cps,f-pkts,f-bytes, duration, Mb/sec, MB/sec, c-flows, PPS,total-Mbytes-duration,errors,flows #<2>
1664 00, avl/delay_10_http_get_0.pcap ,404.52,404.52, 44 , 37830 , 0.17 , 122.42 , 15.30 , 67 , 17799 , 2 , 0 , 1
1665 01, avl/delay_10_http_post_0.pcap ,404.52,404.52, 54 , 48468 , 0.21 , 156.85 , 19.61 , 85 , 21844 , 2 , 0 , 1
1666 02, avl/delay_10_https_0.pcap ,130.87,130.87, 96 , 91619 , 0.22 , 95.92 , 11.99 , 29 , 12564 , 1 , 0 , 1
1667 03, avl/delay_10_http_browsing_0.pcap ,709.89,709.89, 37 , 34425 , 0.13 , 195.50 , 24.44 , 94 , 26266 , 2 , 0 , 1
1668 04, avl/delay_10_exchange_0.pcap ,253.81,253.81, 43 , 9848 , 1.57 , 20.00 , 2.50 , 400 , 10914 , 0 , 0 , 1
1669 05, avl/delay_10_mail_pop_0.pcap ,4.76,4.76, 20 , 5603 , 0.17 , 0.21 , 0.03 , 1 , 95 , 0 , 0 , 1
1670 06, avl/delay_10_mail_pop_1.pcap ,4.76,4.76, 114 , 101517 , 0.25 , 3.86 , 0.48 , 1 , 543 , 0 , 0 , 1
1671 07, avl/delay_10_mail_pop_2.pcap ,4.76,4.76, 30 , 15630 , 0.19 , 0.60 , 0.07 , 1 , 143 , 0 , 0 , 1
1672 08, avl/delay_10_oracle_0.pcap ,79.32,79.32, 302 , 56131 , 6.86 , 35.62 , 4.45 , 544 , 23954 , 0 , 0 , 1
1673 09, avl/delay_10_rtp_160k_full.pcap ,2.78,8.33, 1354 , 1232757 , 61.24 , 27.38 , 3.42 , 170 , 3759 , 0 , 0 , 3
1674 10, avl/delay_10_rtp_250k_full.pcap ,1.98,5.95, 2069 , 1922000 , 61.38 , 30.48 , 3.81 , 122 , 4101 , 0 , 0 , 3
1675 11, avl/delay_10_smtp_0.pcap ,7.34,7.34, 22 , 5618 , 0.19 , 0.33 , 0.04 , 1 , 161 , 0 , 0 , 1
1676 12, avl/delay_10_smtp_1.pcap ,7.34,7.34, 35 , 18344 , 0.21 , 1.08 , 0.13 , 2 , 257 , 0 , 0 , 1
1677 13, avl/delay_10_smtp_2.pcap ,7.34,7.34, 110 , 96544 , 0.27 , 5.67 , 0.71 , 2 , 807 , 0 , 0 , 1
1678 14, avl/delay_10_video_call_0.pcap ,11.90,11.90, 2325 , 2532577 , 36.56 , 241.05 , 30.13 , 435 , 27662 , 3 , 0 , 1
1679 15, avl/delay_10_sip_video_call_full.pcap ,29.35,58.69, 1651 , 120315 , 24.56 , 28.25 , 3.53 , 721 , 48452 , 0 , 0 , 2
1680 16, avl/delay_10_citrix_0.pcap ,43.62,43.62, 272 , 84553 , 6.23 , 29.51 , 3.69 , 272 , 11866 , 0 , 0 , 1
1681 17, avl/delay_10_dns_0.pcap ,1975.02,1975.02, 2 , 162 , 0.01 , 2.56 , 0.32 , 22 , 3950 , 0 , 0 , 1
1683 00, sum ,4083.86,93928.84, 8580 , 6413941 , 0.00 , 997.28 , 124.66 , 2966 , 215136 , 12 , 0 , 23
1691 Total : 8.89 Mbytes 159% util #<1>
1694 <1> the memory usage of the templates
1695 <2> CSV for all the templates
1698 === firmware update to XL710/X710
1699 anchor:xl710-firmware[]
1701 To upgrade the firmware follow this
1703 ==== Download the driver
1705 *Download driver i40e from link:https://downloadcenter.intel.com/download/24411/Network-Adapter-Driver-for-PCI-E-40-Gigabit-Network-Connections-under-Linux-[here]
1706 *Build the kernel module
1710 $tar -xvzf i40e-1.3.47
1713 $sudo insmod i40e.ko
1717 ==== Bind the NIC to Linux
1719 In this stage we bind the NIC to Linux (take it from DPDK)
1723 $sudo ./dpdk_nic_bind.py --status # show the ports
1725 Network devices using DPDK-compatible driver
1726 ============================================
1727 0000:02:00.0 'Device 1583' drv=igb_uio unused= #<1>
1728 0000:02:00.1 'Device 1583' drv=igb_uio unused= #<2>
1729 0000:87:00.0 'Device 1583' drv=igb_uio unused=
1730 0000:87:00.1 'Device 1583' drv=igb_uio unused=
1732 $sudo dpdk_nic_bind.py -u 02:00.0 02:00.1 #<3>
1734 $sudo dpdk_nic_bind.py -b i40e 02:00.0 02:00.1 #<4>
1736 $ethtool -i p1p2 #<5>
1740 firmware-version: 4.24 0x800013fc 0.0.0 #<6>
1741 bus-info: 0000:02:00.1
1742 supports-statistics: yes
1744 supports-eeprom-access: yes
1745 supports-register-dump: yes
1746 supports-priv-flags: yes
1750 $lspci -s 02:00.0 -vvv #<7>
1754 <1> XL710 ports that need to unbind from DPDK
1755 <2> XL710 ports that need to unbind from DPDK
1756 <3> Unbind from DPDK using this command
1757 <4> Bind to linux to i40e driver
1758 <5> Show firmware version throw linux driver
1765 Download NVMUpdatePackage.zip from Intel site link:http://downloadcenter.intel.com/download/24769/NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series[here]
1766 It includes the utility `nvmupdate64e`
1772 $sudo ./nvmupdate64e
1775 You might need a power cycle and to run this command a few times to get the latest firmware
1777 ==== QSFP+ support for XL710
1779 see link:https://www.google.co.il/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjJhPSH3b3LAhUp7nIKHSkACUYQFggaMAA&url=http%3A%2F%2Fwww.intel.co.id%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Frelease-notes%2Fxl710-ethernet-controller-feature-matrix.pdf&usg=AFQjCNFhwozfz-XuKGMOy9_MJDbetw15Og&sig2=ce7YU9F9Et6xf6KvqSFBxg&bvm=bv.116636494,d.bGs[QSFP+ support] for QSFP+ support and Firmware requirement for XL710
1782 === TRex with ASA 5585
1784 When running TRex aginst ASA 5585, you have to notice following things:
1786 * ASA can't forward ipv4 options, so there is a need to use --learn-mode 1 (or 3) in case of NAT. In this mode, bidirectional UDP flows are not supported.
1787 --learn-mode 1 support TCP sequence number randomization in both sides of the connection (client to server and server client). For this to work, TRex must learn
1788 the translation of packets from both sides, so this mode reduce the amount of connections per second TRex can generate (The number is still high enough to test
1789 any existing firewall). If you need higher cps rate, you can use --learn-mode 3. This mode handles sequence number randomization on client->server side only.
1790 * Latency should be tested using ICMP with `--l-pkt-mode 2`
1792 ==== ASA 5585 sample configuration
1796 ciscoasa# show running-config
1800 : Serial Number: JAD194801KX
1801 : Hardware: ASA5585-SSP-10, 6144 MB RAM, CPU Xeon 5500 series 2000 MHz, 1 CPU (4 cores)
1806 enable password 8Ry2YjIyt7RRXU24 encrypted
1807 passwd 2KFQnbNIdI.2KYOU encrypted
1810 interface Management0/0
1814 ip address 10.56.216.106 255.255.255.0
1816 interface TenGigabitEthernet0/8
1819 ip address 15.0.0.1 255.255.255.0
1821 interface TenGigabitEthernet0/9
1824 ip address 40.0.0.1 255.255.255.0
1826 boot system disk0:/asa952-smp-k8.bin
1829 logging asdm informational
1834 no monitor-interface service-module
1835 icmp unreachable rate-limit 1 burst-size 1
1836 no asdm history enable
1837 arp outside 40.0.0.2 90e2.baae.87d1
1838 arp inside 15.0.0.2 90e2.baae.87d0
1840 no arp permit-nonconnected
1841 route management 0.0.0.0 0.0.0.0 10.56.216.1 1
1842 route inside 16.0.0.0 255.0.0.0 15.0.0.2 1
1843 route outside 48.0.0.0 255.0.0.0 40.0.0.2 1
1844 timeout xlate 3:00:00
1845 timeout pat-xlate 0:00:30
1846 timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
1847 timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
1848 timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
1849 timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
1850 timeout tcp-proxy-reassembly 0:01:00
1851 timeout floating-conn 0:00:00
1852 user-identity default-domain LOCAL
1854 http 192.168.1.0 255.255.255.0 management
1855 no snmp-server location
1856 no snmp-server contact
1857 crypto ipsec security-association pmtu-aging infinite
1858 crypto ca trustpool policy
1859 telnet 0.0.0.0 0.0.0.0 management
1861 ssh stricthostkeycheck
1863 ssh key-exchange group dh-group1-sha1
1866 tls-proxy maximum-session 1000
1868 threat-detection basic-threat
1869 threat-detection statistics access-list
1870 no threat-detection statistics tcp-intercept
1871 dynamic-access-policy-record DfltAccessPolicy
1873 class-map icmp-class
1874 match default-inspection-traffic
1875 class-map inspection_default
1876 match default-inspection-traffic
1879 policy-map type inspect dns preset_dns_map
1881 message-length maximum client auto
1882 message-length maximum 512
1883 policy-map icmp_policy
1886 policy-map global_policy
1887 class inspection_default
1888 inspect dns preset_dns_map
1904 service-policy global_policy global
1905 service-policy icmp_policy interface outside
1906 prompt hostname context
1908 jumbo-frame reservation
1910 no call-home reporting anonymous
1915 ==== TRex commands example
1917 Using these commands the configuration is:
1919 1. NAT learn mode (TCP-ACK)
1920 2. Delay of 1 second at start up (-k 1). It was added because ASA drops the first packets.
1921 3. Latency is configured to ICMP reply mode (--l-pkt-mode 2).
1927 $sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 1000 --learn-mode 1 -k 1
1930 This is more realistic traffic for enterprise (we removed from SFR file the bidirectional UDP traffic templates, which (as described above), are not supported in this mode).
1932 *Enterprise profile:*::
1935 $sudo ./t-rex-64 -f avl/sfr_delay_10_1g_asa_nat.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 4 --learn-mode 1 -k 1
1942 -Per port stats table
1944 -----------------------------------------------------------------------------------------
1945 opackets | 106347896 | 118369678
1946 obytes | 33508291818 | 118433748567
1947 ipackets | 118378757 | 106338782
1948 ibytes | 118434305375 | 33507698915
1951 Tx Bw | 656.26 Mbps | 2.27 Gbps
1953 -Global stats enabled
1954 Cpu Utilization : 18.4 % 31.7 Gb/core
1955 Platform_factor : 1.0
1956 Total-Tx : 2.92 Gbps NAT time out : 0 #<1> (0 in wait for syn+ack) #<1>
1957 Total-Rx : 2.92 Gbps NAT aged flow id: 0 #<1>
1958 Total-PPS : 542.29 Kpps Total NAT active: 163 (12 waiting for syn)
1959 Total-CPS : 8.30 Kcps Nat_learn_errors: 0
1961 Expected-PPS : 539.85 Kpps
1962 Expected-CPS : 8.29 Kcps
1963 Expected-BPS : 2.90 Gbps
1965 Active-flows : 7860 Clients : 255 Socket-util : 0.0489 %
1966 Open-flows : 3481234 Servers : 5375 Socket : 7860 Socket/Clients : 30.8
1967 drop-rate : 0.00 bps #<1>
1968 current time : 425.1 sec
1969 test duration : 574.9 sec
1971 -Latency stats enabled
1972 Cpu Utilization : 0.3 %
1973 if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window
1974 | , , check, , latency(usec),latency (usec) ,(usec) ,
1975 ----------------------------------------------------------------------------------------------------------------
1976 0 | 420510, 420495, 0, 1, 58 , 1555, 14 | 240 257 258 258 219 930 732 896 830 472 190 207 729
1977 1 | 420496, 420509, 0, 1, 51 , 1551, 13 | 234 253 257 258 214 926 727 893 826 468 187 204 724
1979 <1> These counters should be zero
1981 anchor:fedora21_example[]
1983 === Fedora 21 Server installation
1985 Download the .iso file from link above, boot with it using Hypervisor or CIMC console. +
1986 Troubleshooting -> install in basic graphics mode
1988 * In packages selection, choose:
1990 ** C Development Tools and Libraries
1992 ** Development Tools
1996 * Set Ethernet configuration if needed
1998 * Use default hard-drive partitions, reclaim space if needed
2000 * After installation, edit file /etc/selinux/config +
2005 systemctl disable firewalld
2007 * Edit file /etc/yum.repos.d/fedora-updates.repo +
2013 === Configure Linux host as network emulator
2015 There are lots of Linux tutorials on the web, so this will not be full tutorial, only highlighting some key points. Commands
2016 were checked on Ubuntu system.
2020 1. TRex Client side network is 16.0.0.x
2021 2. TRex Server side network is 48.0.0.x
2022 3. Linux Client side network eth0 is configured with IPv4 as 172.168.0.1
2023 4. Linux Server side network eth1 is configured with IPv4 as 10.0.0.1
2028 TRex-0 (16.0.0.1->48.0.0.1 ) <-->
2030 ( 172.168.0.1/255.255.0.0)-eth0 [linux] -( 10.0.0.1/255.255.0.0)-eth1
2032 <--> TRex-1 (16.0.0.1<-48.0.0.1)
2037 ==== Enable forwarding
2038 One time (will be discarded after reboot): +
2042 echo 1 > /proc/sys/net/ipv4/ip_forward
2044 To make this permanent, add the following line to the file /etc/sysctl.conf: +
2046 net.ipv4.ip_forward=1
2049 ==== Add static routes
2050 Example if for the default TRex networks, 48.0.0.0 and 16.0.0.0.
2052 Routing all traffic from 48.0.0.0 to the gateway 10.0.0.100
2055 route add -net 48.0.0.0 netmask 255.255.0.0 gw 10.0.0.100
2058 Routing all traffic from 16.0.0.0 to the gateway 172.168.0.100
2061 route add -net 16.0.0.0 netmask 255.255.0.0 gw 172.168.0.100
2063 If you use stateless mode, and decide to add route only in one direction, remember to disable reverse path check. +
2064 For example, to disable on all interfaces:
2067 for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
2072 Alternatively, you can edit /etc/network/interfaces, and add something like this for both ports connected to TRex.
2073 This will take effect, only after restarting networking (rebooting the machine in an alternative also).
2076 iface eth1 inet static
2080 broadcast 16.255.255.255
2081 ... same for 48.0.0.0
2084 ==== Add static ARP entries
2087 sudo arp -s 10.0.0.100 <Second TRex port MAC>
2088 sudo arp -s 172.168.0.100 <TRex side the NICs are not visible to ifconfig, run:
2092 === Mellanox ConnectX-4 support
2094 anchor:connectx_support[]
2096 Mellanox ConnectX-4 adapter family supports 100/56/40/25/10 Gb/s Ethernet speeds.
2097 Its DPDK support is a bit different from Intel DPDK support, more information can be found link:http://dpdk.org/doc/guides/nics/mlx5.html[DPDK support].
2098 Intel NICs does not require kernel drivers (except dpdk igb_uio which already supported) while ConnectX-4 works on top of Infinibad API (verbs) and require a kernel modules/user space libs.
2099 This means that it is required to install OFED package to be able to work with the NIC.
2100 Installing OFED is the simplest way to make it work (trying to install part of the package can work too but didn't work for us).
2101 The advantage of this model that you can control it throw Linux driver (ethtol can still work, you will be able to ifconfig it).
2102 The disadvantage is the OFED dependency.
2108 The following distro were tested with TRex and OFED, others might work
2112 The following distro were tested and did *not* work for us
2114 * Fedora 21 (3.17.4-301.fc21.x86_64)
2115 * Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64) -- crash when RSS was enabled link:https://trex-tgn.cisco.com/youtrack/issue/trex-294[MLX RSS issue]
2119 The information was taken from link:http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers[Install OFED]
2121 * Download 3.4-2 OFED tar for your distro
2124 =====================================
2125 it must be version *MLNX_OFED_LINUX-3.4-2*
2126 =====================================
2129 =====================================
2130 Make sure you have an internet connection without firewalls for HTTPS/HTTP - required by yum/apt-get
2131 =====================================
2136 $md5sum md5sum MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.2-x86_64.tgz
2137 58b9fb369d7c62cedbc855661a89a9fd MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.2-x86_64.tgz
2143 $tar -xvzf MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.2-x86_64.tgz
2144 $cd MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.2-x86_64
2150 $sudo ./mlnxofedinstall
2153 Log: /tmp/ofed.build.log
2154 Logs dir: /tmp/MLNX_OFED_LINUX.10406.logs
2156 Below is the list of MLNX_OFED_LINUX packages that you have chosen
2157 (some may have been added by the installer due to package dependencies):
2160 mlnx-ofed-kernel-utils
2161 mlnx-ofed-kernel-dkms
2195 infiniband-diags-compat
2227 This program will install the MLNX_OFED_LINUX package on your machine.
2228 Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
2229 Do you want to continue?[y/N]:y
2231 Checking SW Requirements...
2233 One or more required packages for installing MLNX_OFED_LINUX are missing.
2234 Attempting to install the following missing packages:
2235 autotools-dev tcl debhelper dkms tk8.4 libgfortran3 graphviz chrpath automake dpatch flex bison autoconf quilt m4 tcl8.4 libltdl-dev pkg-config pytho
2236 bxml2 tk swig gfortran libnl1
2240 Removing old packages...
2241 Installing new packages
2242 Installing ofed-scripts-3.4...
2243 Installing mlnx-ofed-kernel-utils-3.4...
2244 Installing mlnx-ofed-kernel-dkms-3.4...
2246 Removing old packages...
2247 Installing new packages
2248 Installing ofed-scripts-3.4...
2249 Installing mlnx-ofed-kernel-utils-3.4...
2250 Installing mlnx-ofed-kernel-dkms-3.4...
2251 Installing iser-dkms-1.8.1...
2252 Installing srp-dkms-1.6.1...
2253 Installing mlnx-sdp-dkms-3.4...
2254 Installing mlnx-rds-dkms-3.4...
2255 Installing mlnx-nfsrdma-dkms-3.4...
2256 Installing libibverbs1-1.2.1mlnx1...
2257 Installing ibverbs-utils-1.2.1mlnx1...
2258 Installing libibverbs-dev-1.2.1mlnx1...
2259 Installing libibverbs1-dbg-1.2.1mlnx1...
2260 Installing libmlx4-1-1.2.1mlnx1...
2261 Installing libmlx4-dev-1.2.1mlnx1...
2262 Installing libmlx4-1-dbg-1.2.1mlnx1...
2263 Installing libmlx5-1-1.2.1mlnx1...
2264 Installing libmlx5-dev-1.2.1mlnx1...
2265 Installing libmlx5-1-dbg-1.2.1mlnx1...
2266 Installing libibumad-1.3.10.2.MLNX20150406.966500d...
2267 Installing libibumad-static-1.3.10.2.MLNX20150406.966500d...
2268 Installing libibumad-devel-1.3.10.2.MLNX20150406.966500d...
2269 Installing ibacm-1.2.1mlnx1...
2270 Installing ibacm-dev-1.2.1mlnx1...
2271 Installing librdmacm1-1.1.0mlnx...
2272 Installing librdmacm-utils-1.1.0mlnx...
2273 Installing librdmacm-dev-1.1.0mlnx...
2274 Installing mstflint-4.5.0...
2275 Installing ibdump-4.0.0...
2276 Installing libibmad-1.3.12.MLNX20160814.4f078cc...
2277 Installing libibmad-static-1.3.12.MLNX20160814.4f078cc...
2278 Installing libibmad-devel-1.3.12.MLNX20160814.4f078cc...
2279 Installing libopensm-4.8.0.MLNX20160906.32a95b6...
2280 Installing opensm-4.8.0.MLNX20160906.32a95b6...
2281 Installing opensm-doc-4.8.0.MLNX20160906.32a95b6...
2282 Installing libopensm-devel-4.8.0.MLNX20160906.32a95b6...
2283 Installing infiniband-diags-1.6.6.MLNX20160814.999c7b2...
2284 Installing infiniband-diags-compat-1.6.6.MLNX20160814.999c7b2...
2285 Installing mft-4.5.0...
2286 Installing kernel-mft-dkms-4.5.0...
2287 Installing libibcm1-1.0.5mlnx2...
2288 Installing libibcm-dev-1.0.5mlnx2...
2289 Installing perftest-3.0...
2290 Installing ibutils2-2.1.1...
2291 Installing libibdm1-1.5.7.1...
2292 Installing ibutils-1.5.7.1...
2293 Installing cc-mgr-1.0...
2294 Installing ar-mgr-1.0...
2295 Installing dump-pr-1.0...
2296 Installing ibsim-0.6...
2297 Installing ibsim-doc-0.6...
2298 Installing knem-dkms-1.1.2.90mlnx1...
2299 Installing mxm-3.5.220c57f...
2300 Installing fca-2.5.2431...
2301 Installing sharp-1.1.1.MLNX20160915.8763a35...
2302 Installing hcoll-3.6.1228...
2303 Installing openmpi-1.10.5a1...
2304 Installing mpitests-3.2.18...
2305 Installing knem-1.1.2.90mlnx1...
2306 Installing rds-tools-2.0.7...
2307 Installing libdapl2-2.1.9mlnx...
2308 Installing dapl2-utils-2.1.9mlnx...
2309 Installing libdapl-dev-2.1.9mlnx...
2310 Installing srptools-1.0.3...
2311 Installing mlnx-ethtool-4.2...
2312 Installing libsdp1-1.1.108...
2313 Installing libsdp-dev-1.1.108...
2314 Installing sdpnetstat-1.60...
2315 Selecting previously unselected package mlnx-fw-updater.
2316 (Reading database ... 70592 files and directories currently installed.)
2317 Preparing to unpack .../mlnx-fw-updater_3.4-1.0.0.0_amd64.deb ...
2318 Unpacking mlnx-fw-updater (3.4-1.0.0.0) ...
2319 Setting up mlnx-fw-updater (3.4-1.0.0.0) ...
2321 Added RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
2323 Attempting to perform Firmware update...
2324 Querying Mellanox devices firmware ...
2328 Device Type: ConnectX4
2329 Part Number: MCX416A-CCA_Ax
2330 Description: ConnectX-4 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; ROHS R6
2332 PCI Device Name: 03:00.0
2333 Base GUID: 248a07030014fc60
2334 Base MAC: 0000248a0714fc60
2335 Versions: Current Available
2336 FW 12.16.1006 12.17.1010
2337 PXE 3.4.0812 3.4.0903
2339 Status: Update required
2342 Found 1 device(s) requiring firmware update...
2344 Device #1: Updating FW ... Done
2346 Restart needed for updates to take effect.
2347 Log File: /tmp/MLNX_OFED_LINUX.16084.logs/fw_update.log
2348 Please reboot your system for the changes to take effect.
2350 03:00.0 Ethernet controller: Mellanox Technologies MT27620 Family
2352 PCI Link Speed: 8GT/s
2355 03:00.1 Ethernet controller: Mellanox Technologies MT27620 Family
2357 PCI Link Speed: 8GT/s
2359 Installation passed successfully
2360 To load the new driver, run:
2361 /etc/init.d/openibd restart
2377 transport: InfiniBand (0)
2378 fw_ver: 12.17.1010 << 12.17.00
2379 node_guid: 248a:0703:0014:fc61
2380 sys_image_guid: 248a:0703:0014:fc60
2382 vendor_part_id: 4115
2384 board_id: MT_2150110033
2388 state: PORT_DOWN (1)
2390 active_mtu: 1024 (3)
2394 link_layer: Ethernet
2397 transport: InfiniBand (0)
2399 node_guid: 248a:0703:0014:fc60
2400 sys_image_guid: 248a:0703:0014:fc60
2402 vendor_part_id: 4115
2404 board_id: MT_2150110033
2408 state: PORT_DOWN (1)
2410 active_mtu: 1024 (3)
2414 link_layer: Ethernet
2422 mlx5_0 port 1 ==> eth6 (Down)
2423 mlx5_1 port 1 ==> eth7 (Down)
2430 $sudo ./dpdk_setup_ports.py -t
2431 +----+------+---------++---------------------------------------------
2432 | ID | NUMA | PCI || Name | Driver |
2433 +====+======+=========++===============================+===========+=
2434 | 0 | 0 | 06:00.0 || VIC Ethernet NIC | enic |
2435 +----+------+---------++-------------------------------+-----------+-
2436 | 1 | 0 | 07:00.0 || VIC Ethernet NIC | enic |
2437 +----+------+---------++-------------------------------+-----------+-
2438 | 2 | 0 | 0a:00.0 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe |
2439 +----+------+---------++-------------------------------+-----------+-
2440 | 3 | 0 | 0a:00.1 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe |
2441 +----+------+---------++-------------------------------+-----------+-
2442 | 4 | 0 | 0d:00.0 || Device 15d0 | |
2443 +----+------+---------++-------------------------------+-----------+-
2444 | 5 | 0 | 10:00.0 || I350 Gigabit Network Connectio| igb |
2445 +----+------+---------++-------------------------------+-----------+-
2446 | 6 | 0 | 10:00.1 || I350 Gigabit Network Connectio| igb |
2447 +----+------+---------++-------------------------------+-----------+-
2448 | 7 | 1 | 85:00.0 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe |
2449 +----+------+---------++-------------------------------+-----------+-
2450 | 8 | 1 | 85:00.1 || 82599ES 10-Gigabit SFI/SFP+ Ne| ixgbe |
2451 +----+------+---------++-------------------------------+-----------+-
2452 | 9 | 1 | 87:00.0 || MT27700 Family [ConnectX-4] | mlx5_core | #<1>
2453 +----+------+---------++-------------------------------+-----------+-
2454 | 10 | 1 | 87:00.1 || MT27700 Family [ConnectX-4] | mlx5_core | #<2>
2455 +----+------+---------++---------------------------------------------
2457 <1> ConnectX-4 port 0
2458 <2> ConnectX-4 port 1
2461 .Config file example
2464 ### Config file generated by dpdk_setup_ports.py ###
2468 interfaces: ['87:00.0', '87:00.1']
2477 latency_thread_id: 1
2480 threads: [8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31]
2486 ==== TRex specific implementation details
2488 TRex uses flow director filter to steer specific packets to specific queues.
2489 To support that we change IPv4.TOS/Ipv6.TC LSB to *1* to be steered. So latency packets will have this bit turn on (not only for ConnectX-4)
2490 Watch out, In case DUT will clear this bit (change the TOS with LSB==0, e.g. 0x3->0x2) packets won't be forward to TRex.
2492 ==== Which NIC to buy?
2494 NIC with two ports will work better from performance prospective, so it is better to have MCX456A-ECAT(dual 100gb port) and *not* the MCX455A-ECAT (single 100gb port).
2496 ==== Limitation/Issues
2498 * Stateless per stream statistic is not supported yet
2499 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-260[64B performance issue]
2500 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-261[Latency issue]
2501 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-262[Statful RX out of order]
2502 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-273[Fedora 21 & OFED 3.4.1]
2505 ==== Performance Cycles/Packet ConnectX-4 vs Intel XL710
2507 For version TRex v2.11, this is the comparison results between XL710 and ConnectX-4 for various scenarios
2509 .Stateless MPPS/Core [Perlimeniary]
2510 image:images/xl710_vs_mlx5_64b.png[title="Stateless 64B"]
2512 .Stateless Gb/Core [Perlimeniary]
2513 image:images/xl710_vs_mlx5_var_size.png[title="Stateless variable size packet"]
2517 1. For Stateless 64B profiles ConnectX-4 cost 50-90% more cycles per packet (it is actually even more because there is the TRex scheduler overhead) - it means that in the worst case scenario you will need x2 CPU for the same total MPPS
2518 2. For Stateless/Stateful 256B profiles, ConnectX-4 cost half of the cycles per packets. ConnectX-4 probably can handle in a better way chained mbuf (scatter gather).
2519 3. In Average Stateful senario ConnectX-4 will be slightly better.
2520 4. MLX5 can reach ~90MPPS while XL710 limited to 35MPPS
2524 =====================================
2525 There is a task to automate the production of this reports
2526 =====================================
2528 ==== Troubleshooting
2530 * Before running TRex make sure the commands `ibv_devinfo` and `ibdev2netdev` present the NICS
2531 * `ifconfig` should work too, you need to be able to ping from those ports
2532 * run TRex server with '-v 7' for example `$sudo ./t-rex-64 -i -v 7`
2535 === Cisco VIC support
2537 anchor:ciscovic_support[]
2539 * Supported from TRex version v2.11
2540 * Only 1300 series Cisco adapter
2541 * Firmware version 2.0(13) for UCS C-series servers, GA in Febuary 2017.
2542 * Firmware version 3.1(2) for blade servers supports more filtering capabilities.
2543 * The feature can be enabled via Cisco CIMC or USCM with the 'advanced filters' radio button. When enabled, the these additional flow director modes are available:
2544 RTE_ETH_FLOW_NONFRAG_IPV4_OTHER
2545 RTE_ETH_FLOW_NONFRAG_IPV4_SCTP
2546 RTE_ETH_FLOW_NONFRAG_IPV6_UDP
2547 RTE_ETH_FLOW_NONFRAG_IPV6_TCP
2548 RTE_ETH_FLOW_NONFRAG_IPV6_SCTP
2549 RTE_ETH_FLOW_NONFRAG_IPV6_OTHER
2552 ==== Limitation/Issues
2554 * Stateless per stream statistic is not supported yet
2555 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-265[flow-director]
2556 * link:https://trex-tgn.cisco.com/youtrack/issue/trex-272[QSFP+ issue]
Code Review / trex.git / blob