3 l4fwd is a sample application to demonstrate and test TLDK TCP/UDP
4 functionalities. Depending on configuration it can do simple send, recv or
5 both over opened TCP/UDP streams. Also it implements ability to do TCP/UDP
6 packet forwarding between different streams, so it is possible to use the
7 l4fwd application as a TCP/UDP proxy.
9 The l4fwd application logically is divided into two parts, Back End (BE) and
14 BE is responsible for:
15 - RX over DPDK ports and feed them into TCP/UDP TLDK context(s)
18 - retrieve packets ready to be send out from TCP/UDP TLDK context(s) and TX
19 them over destined DPDK port.
21 - Multiple RX/TX queues per port are supported by RSS. Right now the number
22 of TX is same as the number of RX queue.
24 - Each BE lcore can serve multiple DPDK ports, TLDK TCP/UDP contexts.
26 BE configuration record format:
28 port=<uint>,addr=<ipv4/ipv6>,masklen=<uint>,mac=<ether><mtu>
30 port - DPDK port id to be used to send packets to the destination.
31 It is an mandatory option.
32 addr - destination network address. It is an mandatory option.
33 masklen - destination network prefix length. It is an mandatory option.
34 mac - destination Ethernet address. It is an mandatory option.
35 mtu - MTU to be used on that port (= application data size + L2/L3/L4
36 headers sizes, default=1514). It is an optional option.
38 Below are some example of BE entries
40 port=0,masklen=16,addr=192.168.0.0,mac=01:de:ad:be:ef:01
41 port=0,addr=2001:4860:b002::,masklen=64,mac=01:de:ad:be:ef:01
43 These examples are also available in be.cfg file.
47 FE is responsible for:
48 - to open configured TCP/UDP streams and perform send/recv over them.
49 These streams can belong to different TCP/UDP contexts.
51 Each lcore can act as BE and/or FE.
53 In UDP mode the application can reassemble input fragmented IP packets, and
54 fragment outgoing IP packets (if destination MTU is less then packet size).
56 FE configuration record format:
58 lcore=<uint>,op=<"rx|tx|echo|fwd|rxtx">,\
59 laddr=<ip>,lport=<uint16>,raddr=<ip>,rport=<uint16>,\
60 [txlen=<uint>,fwladdr=<ip>,fwlport=<uint16>,fwraddr=<ip>,fwrport=<uint16>,\
61 belcore=<uint>,rxlen=<uint>]
63 lcore - EAL lcore to manage that stream(s) in the FE. It is an mandatory
65 belcore - EAL lcore to manage that stream(s) in the BE. It is an optional
66 option. lcore and belcore can specify the same cpu core.
67 op - operation to perform on that stream:
68 "rx" - do receive only on that stream.
69 "tx" - do send only on that stream.
70 "echo" - mimic recvfrom(..., &addr);sendto(..., &addr);
72 "fwd" - forward packets between streams.
73 "rxtx" - Receive/reply transactions on stream.
74 It is an mandatory option.
75 laddr - local address for the stream to open. It is an mandatory option.
76 lport - local port for the stream to open. It is an mandatory option.
77 raddr - remote address for the stream to open. It is an mandatory option.
78 rport - remote port for the stream to open. It is an mandatory option.
79 txlen - data length sending in each packet.
80 (mandatory for "tx" & "rxtx" modes only).
81 rxlen - Expected response length (mandatory for "rxtx" mode only).
82 fwladdr - local address for the forwarding stream(s) to open
83 (mandatory for "fwd" mode only).
84 fwlport - local port for the forwarding stream(s) to open
85 (mandatory for "fwd" mode only).
86 fwraddr - remote address for the forwarding stream(s) to open
87 (mandatory for "fwd" mode only).
88 fwrport - remote port for the forwarding stream(s) to open
89 (mandatory for "fwd" mode only).
91 Below are some example of FE entries
93 lcore=3,op=echo,laddr=192.168.1.233,lport=0x8000,raddr=0.0.0.0,rport=0
95 lcore=3,op=tx,laddr=192.168.1.233,lport=0x8001,raddr=192.168.1.56,\
98 lcore=3,op=rx,laddr=::,lport=0x200,raddr=::,rport=0,txlen=72
100 lcore=3,op=fwd,laddr=0.0.0.0,lport=11211,raddr=0.0.0.0,rport=0,\
101 fwladdr=::,fwlport=0,fwraddr=2001:4860:b002::56,fwrport=11211
103 These examples are also available in fe.cfg file with some more explanation.
105 1.3 Configuration files format
107 - each record on a separate line.
108 - lines started with '#' are treated as comments.
109 - empty lines (containing whitespace chars only) are ignored.
110 - kvargs style format for each record.
111 - each FE record correspond to at least one stream to be opened
112 (could be multiple streams in case of op="fwd").
113 - each BE record define a ipv4/ipv6 destination.
117 DPDK libraries (16.11 or higher)
119 Back-End (BE) configuration file
120 Front-End(FE) configuration file
124 l4fwd <DPDK EAL parameters> -- \
125 -P | --promisc /* promiscuous mode enabled. */ \
126 -R | --rbufs <num> /* max recv buffers per stream. */ \
127 -S | --sbufs <num> /* max send buffers per stream. */ \
128 -s | --streams <num> /* streams to open per context. */ \
129 -b | --becfg <filename> /* backend configuration file. */ \
130 -f | --fecfg <filename> /* frontend configuration file. */ \
131 -U | --udp /* run the app to handle UDP streams only. */ \
132 -T | --tcp /* run the app to handle TCP streams only. */ \
133 -L | --listen /* open TCP streams in server mode (listen). */ \
134 -a | --enable-arp /* enable arp responses (request not supported) */ \
135 -v | --verbose /* different level of verbose mode */ \
136 -H | --hash <string> /* hash algorithm i.e. siphash or jhash to be */ \
137 /* used to generate the sequence number. */ \
138 -K | --seckey <string> /* 16 character long secret key used by */ \
139 /* hash algorithms to generate the */ \
140 /* sequence number. */ \
141 <port0_params> <port1_params> ... <portN_params>
143 Note that: options -U and -T cannot be used together.
144 Option -L can be used only with option -T.
146 portX_params: port=<uint>,lcore=<uint>[-<uint>],[lcore=<uint>[-<uint>],]\
147 [rx_offload=<uint>,tx_offload=<uint>,mtu=<uint>,ipv4=<ipv4>,ipv6=<ipv6>]
149 portX_params are used to configure the particular DPDK device
150 (rte_ethdev port), and specify BE lcore that will handle RX/TX from/to the
151 device and manage BE part of corresponding TCP/UDP context.
152 Multiple BE lcore can be specified.
154 port - DPDK port id (RSS are supported when multiple lcores are
155 specified for a port). It is an mandatory option.
156 lcore - EAL lcore id to handle IO over that port (rx_burst/tx_burst).
157 several ports can be managed by the same lcore, and same port
158 can be managed by more than one lcore.
159 It is an mandatory option. At least one lcore option has to be
160 specified. lcore range can be specified in one lcore option.
161 e.g. lcore=2-3,lcore=6 will enable lcores 2, 3, and 6 to
163 rx_offload - RX HW offload capabilities to enable/use on this port.
164 (bitmask of DEV_RX_OFFLOAD_* values). It is an optional option.
165 tx_offload - TX HW offload capabilities to enable/use on this port.
166 (bitmask of DEV_TX_OFFLOAD_* values).
167 mtu - MTU to be used on that port (= application data size + L2/L3/L4
168 headers sizes, default=1514).
169 ipv4 - ipv4 address to assign to that port.
170 ipv6 - ipv6 address to assign to that port.
172 At least one of ipv4/ipv6 values have to be specified for each port.
176 If multiple lcore is specified per DPDK port, the following RSS hash will
177 be enabled on that port:
178 ETH_RSS_UDP, or ETH_RSS_TCP
180 The RSS queue qid will handle the stream according to the TCP/UDP source
181 ports of the stream. The qid can be calculated as below
183 qid = (src_port % power_of_2(n)) % n
185 where n is number of lcore used to mane the DPDK port.
191 +----------------------------+ +-------------------------------+
192 | TLDK Box | | Linux Box |
194 | port 0 +----------------+ port 0 |
195 | 192.168.1.1 | | 192.168.1.2 |
196 | 2001:4860:b002::1 | | 2001:4860:b002::2 |
197 | AA:BB:CC:DD:EE:F1 | | AA:BB:CC:DD:EE:F2 |
198 +----------------------------+ +-------------------------------+
200 4.2 UDP, "rx" mode, IPv4-only, Single core
202 This example shows receiving data from a IPv4 stream. The TLDK UDP server
203 runs on single core where both BE and FE run on cpu core 3.
205 be.cfg file contains:
207 port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
209 fe.cfg file contains (udp server listening to port 6000):
211 lcore=3,op=rx,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
213 run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
215 l4fwd --lcores='3' -w 01:00.0 -- \
216 --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
217 --becfg be.cfg -U port=0,lcore=3,ipv4=192.168.1.1
219 This will create TLDK UDP context on lcore=3 (BE lcore) to manage
220 DPDK port 0. The port 0 will have IPv4 address 192.168.1.1.
221 All the streams will be in server mode and also managed by lcore 3.
223 4.3 UDP, "echo" mode, IPv6-only, Multicore
225 This example shows receiving data from a IPv6 stream and sending the data
226 back through the same IPv6 stream. The TLDK UDP server runs on multicore
227 where BE runs on cpu core 2 and FE runs on cpu core 3.
229 be.cfg file contains:
231 port=0,masklen=64,addr=2001:4860:b002::,mac=AA:BB:CC:DD:EE:F2
233 fe.cfg file contains (udp server listening to port 6000):
235 lcore=3,op=rx,laddr=2001:4860:b002::1,lport=6000,raddr=::,rport=0
237 run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
239 l4fwd --lcores='2,3' -w 01:00.0 -- \
240 --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
241 --becfg be.cfg -U port=0,lcore=2,ipv6=2001:4860:b002::1
243 This will create TLDK UDP context on lcore=2 (BE lcore) to manage
244 DPDK port 0. The port 0 will have IPv4 address 2001:4860:b002::1.
245 All the streams will be in server mode and managed by lcore 3 (FE lcore).
246 In this case, the UDP server will send the incoming data back to the sender.
248 4.4 TCP, "echo" mode, IPv4-only, Multicore, RX-Offload
250 This example shows receiving data from a IPv4 stream and sending the data
251 back through the same IPv4 stream. The TLDK TCP server runs on multicore
252 where BE runs on cpu core 2 and FE runs on cpu core 3. The BE also uses
253 receive offload features of the NIC.
255 be.cfg file contains:
257 port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
259 fe.cfg file contains (tcp server listening to port 6000):
261 lcore=3,op=echo,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
263 run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
265 l4fwd --lcores='2,3' -w 01:00.0 -- \
266 --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
267 --becfg be.cfg -T -L port=0,lcore=2,rx_offload=0xf,tx_offload=0,\
270 This will create TLDK TCP context on lcore=2 (BE lcore) to manage
271 DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
272 DPDK RX HW offloads will be enabled on that port.
273 DEV_RX_OFFLOAD_VLAN_STRIP,
274 DEV_RX_OFFLOAD_IPV4_CKSUM,
275 DEV_RX_OFFLOAD_UDP_CKSUM,
276 DEV_RX_OFFLOAD_TCP_CKSUM
277 No HW TX offloads will be enabled.
278 All the streams will be in server mode and managed by lcore 3 (FE core).
279 In this case, the TCP server will send the incoming data back to the sender.
281 4.5 TCP, "fwd" (proxy) mode, IPv4-to-IPv6, Multi-core, RX-Offload
283 This example shows receiving data from a IPv4 stream and forwarding the
284 data to a IPv6 stream. The TLDK TCP server runs on multicore
285 where BE runs on cpu core 2 and FE runs on cpu core 3. The BE also uses
286 receive offload features of the NIC.
288 be.cfg file contains:
290 port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
292 fe.cfg file contains (tcp server listening to port 6000):
294 lcore=3,op=fwd,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0,\
295 rladdr=::,lport=0,raddr=2001:4860:b002::2,rport=7000
297 run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
299 l4fwd --lcores='2,3' -w 01:00.0 -- \
300 --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
301 --becfg be.cfg -T -L port=0,lcore=2,rx_offload=0xf,tx_offload=0,\
302 ipv4=192.168.1.1,ipv6=2001:4860:b002::1
304 This will create TLDK TCP context on lcore=2 (BE lcore) to manage
305 DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
306 DPDK RX HW offloads will be enabled on that port.
307 DEV_RX_OFFLOAD_VLAN_STRIP,
308 DEV_RX_OFFLOAD_IPV4_CKSUM,
309 DEV_RX_OFFLOAD_UDP_CKSUM,
310 DEV_RX_OFFLOAD_TCP_CKSUM
311 No HW TX offloads will be enabled.
312 All the streams will be in server mode and managed by lcore 3 (FE core).
313 In this case, the IPv4 TCP server will forward the incoming data to the IPv6
314 TCP server 2001:4860:b002::2 listening to port 7000.
316 4.6 TCP, "echo" mode, RSS, IPv4-only, Multicore, RX-Offload
318 This example shows receiving data from a IPv4 stream and sending the data
319 back through the same IPv4 stream. The TLDK TCP server runs on multicore
320 where BE runs on cpu cores 1-2 and FE runs on cpu core 3. As BE runs on
321 multicore, Receive Side Scaling (RSS) feature will be automatically enabled.
322 The BE also uses receive offload features of the NIC.
324 be.cfg file contains:
326 port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
328 fe.cfg file contains (tcp server listening to port 6000):
330 lcore=3,op=echo,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
331 lcore=3,op=echo,laddr=192.168.1.1,lport=6001,raddr=0.0.0.0,rport=0
333 run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
335 l4fwd --lcores='1,2,3' -w 01:00.0 -- \
336 --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
337 --becfg be.cfg -T -L port=0,lcore="1-2",rx_offload=0xf,tx_offload=0,\
340 This will create TLDK TCP context on lcore=1-2 (BE lcore) to manage
341 DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
342 DPDK RX HW offloads will be enabled on that port.
343 DEV_RX_OFFLOAD_VLAN_STRIP,
344 DEV_RX_OFFLOAD_IPV4_CKSUM,
345 DEV_RX_OFFLOAD_UDP_CKSUM,
346 DEV_RX_OFFLOAD_TCP_CKSUM
347 No HW TX offloads will be enabled.
348 All the streams will be in server mode and managed by lcore 3 (FE core).
349 In this case, the TCP server will send the incoming data back to the sender.
351 As RSS is enabled, all the packets with destination port 6000 and 6001 will
352 be managed by HW queue 0 and queue 1 respectively. Please note that RSS
353 is not supported on the interface when both IPv4 and IPv6 are enabled.
354 Only one of IPv4 or IPv6 has to be enabled in the port.