1 ## How to do VPP Packet Tracing in Kubernetes
3 This document describes the steps to do *manual* packet tracing (capture) using
4 VPP in Kubernetes. Contiv/VPP also ships with a simple bash script
5 [vpptrace.sh](https://github.com/contiv/vpp/blob/master/scripts/vpptrace.sh),
6 which allows to *continuously* trace and
7 *filter* packets incoming through a given set of interface types.
8 Documentation for vpptrace.sh is available [here](https://github.com/contiv/vpp/blob/master/docs/VPPTRACE.md).
11 More information about VPP packet tracing is in:
13 * <https://wiki.fd.io/view/VPP/Command-line_Interface_(CLI)_Guide#packet_tracer>
14 * <https://wiki.fd.io/view/VPP/How_To_Use_The_Packet_Generator_and_Packet_Tracer>
15 * <https://wiki.fd.io/view/VPP/Tutorial_Routing_and_Switching>
17 #### SSH into the Node
18 Perform the following commands to SSH into the node:
21 cd vpp/vagrant/vagrant-scripts/
22 vagrant ssh k8s-worker1
25 #### Check the VPP Graph Nodes (Input and Output Queues)
27 The following content shows what is running on VPP, via the `show run` command
30 vagrant@k8s-worker1:~$ sudo vppctl
32 __/ __/ _ \ (_)__ | | / / _ \/ _ \
33 _/ _// // / / / _ \ | |/ / ___/ ___/
34 /_/ /____(_)_/\___/ |___/_/ /_/
37 Time 1026791.9, average vectors/node 1.12, last 128 main loops 0.00 per node 0.00
38 vector rates in 1.6459e-4, out 1.5485e-4, drop 1.3635e-5, punt 0.0000e0
39 Name State Calls Vectors Suspends Clocks Vectors/Call
40 GigabitEthernet0/8/0-output active 56 69 0 1.34e3 1.23
41 GigabitEthernet0/8/0-tx active 54 67 0 8.09e5 1.24
42 acl-plugin-fa-cleaner-process event wait 0 0 1 2.84e4 0.00
43 admin-up-down-process event wait 0 0 1 4.59e3 0.00
44 api-rx-from-ring any wait 0 0 3316292 1.24e5 0.00
45 arp-input active 3 3 0 2.53e5 1.00
46 bfd-process event wait 0 0 1 5.94e3 0.00
47 cdp-process any wait 0 0 145916 1.36e4 0.00
48 dhcp-client-process any wait 0 0 10268 3.65e4 0.00
49 dns-resolver-process any wait 0 0 1027 5.86e4 0.00
50 dpdk-input polling 8211032318951 93 0 1.48e13 0.00
51 dpdk-ipsec-process done 1 0 0 2.10e5 0.00
52 dpdk-process any wait 0 0 342233 9.86e6 0.00
53 error-drop active 12 14 0 6.67e3 1.17
54 ethernet-input active 60 74 0 5.81e3 1.23
55 fib-walk any wait 0 0 513322 1.59e4 0.00
56 flow-report-process any wait 0 0 1 1.45e3 0.00
57 flowprobe-timer-process any wait 0 0 1 6.34e3 0.00
58 ikev2-manager-process any wait 0 0 1026484 1.18e4 0.00
59 interface-output active 2 2 0 3.23e3 1.00
60 ioam-export-process any wait 0 0 1 1.98e3 0.00
61 ip-route-resolver-process any wait 0 0 10268 3.02e4 0.00
62 ip4-arp active 1 1 0 1.49e4 1.00
63 ip4-input active 223 248 0 3.39e3 1.11
64 ip4-load-balance active 106 132 0 5.34e3 1.25
65 ip4-local active 86 92 0 2.46e3 1.07
66 ip4-local-end-of-arc active 86 92 0 1.00e3 1.07
67 ip4-lookup active 223 248 0 3.31e3 1.11
68 ip4-rewrite active 190 222 0 1.92e3 1.17
69 ip4-udp-lookup active 86 92 0 3.76e3 1.07
70 ip6-drop active 6 7 0 2.29e3 1.17
71 ip6-icmp-neighbor-discovery-ev any wait 0 0 1026484 1.13e4 0.00
72 ip6-input active 6 7 0 3.33e3 1.17
73 l2-flood active 2 2 0 4.42e3 1.00
74 l2-fwd active 138 157 0 2.13e3 1.14
75 l2-input active 140 159 0 2.41e3 1.14
76 l2-learn active 86 92 0 3.64e4 1.07
77 l2-output active 54 67 0 3.05e3 1.24
78 l2fib-mac-age-scanner-process event wait 0 0 85 5.01e4 0.00
79 lisp-retry-service any wait 0 0 513322 1.62e4 0.00
80 lldp-process event wait 0 0 1 5.02e4 0.00
81 loop0-output active 54 67 0 1.66e3 1.24
82 loop0-tx active 54 0 0 2.49e3 0.00
83 memif-process event wait 0 0 1 1.70e4 0.00
84 nat-det-expire-walk done 1 0 0 3.79e3 0.00
85 nat44-classify active 171 183 0 2.49e3 1.07
86 nat44-hairpinning active 86 92 0 1.80e3 1.07
87 nat44-in2out active 171 183 0 4.45e3 1.07
88 nat44-in2out-slowpath active 171 183 0 3.98e3 1.07
89 nat44-out2in active 52 65 0 1.28e4 1.25
90 nat64-expire-walk any wait 0 0 102677 5.95e4 0.00
91 nat64-expire-worker-walk interrupt wa 102676 0 0 7.39e3 0.00
92 send-garp-na-process event wait 0 0 1 1.28e3 0.00
93 startup-config-process done 1 0 1 4.19e3 0.00
94 tapcli-0-output active 1 1 0 6.97e3 1.00
95 tapcli-0-tx active 1 1 0 7.32e4 1.00
96 tapcli-1-output active 57 63 0 1.66e3 1.11
97 tapcli-1-tx active 57 63 0 1.35e5 1.11
98 tapcli-2-output active 28 28 0 3.26e3 1.00
99 tapcli-2-tx active 28 28 0 4.06e5 1.00
100 tapcli-rx interrupt wa 62 76 0 6.58e4 1.23
101 udp-ping-process any wait 0 0 1 1.79e4 0.00
102 unix-cli-127.0.0.1:43282 active 2 0 455 1.26e15 0.00
103 unix-epoll-input polling 8010763239 0 0 8.17e2 0.00
104 vhost-user-process any wait 0 0 1 1.96e3 0.00
105 vhost-user-send-interrupt-proc any wait 0 0 1 3.85e3 0.00
106 vpe-link-state-process event wait 0 0 8 9.79e4 0.00
107 vpe-oam-process any wait 0 0 503263 1.21e4 0.00
108 vxlan-gpe-ioam-export-process any wait 0 0 1 2.91e3 0.00
109 vxlan4-encap active 54 67 0 3.55e3 1.24
110 vxlan4-input active 86 92 0 3.79e3 1.07
111 wildcard-ip4-arp-publisher-pro event wait 0 0 1 6.44e3 0.00
114 `tapcli-rx` above is the node-level input queue for incoming packets into all the pods on the node. There is one `tapcli-rx` input queue for every node.
116 The following are the input and output queues for each pod and the node:
127 Each pod and node has two queues, one for rx (`tapcli-X-output`), and one for tx (`tapcli-X-tx`). The above output is with two `nginx` pods in kubernetes.
129 #### Clear Existing VPP Packet Trace
130 Enter the following command:
135 #### How to Turn on VPP Packet Tracing
136 Enter the following commands:
139 vpp# trace add <input or output queue name> <number of packets to capture>
141 vpp# trace add dpdk-input 1000
143 vpp# trace add tapcli-rx 1000
146 #### Send Traffic to the Pods
148 Open another terminal, SSH into the master node, refer the documentation in `vpp/vagrant/README.md` and send traffic to the two `nginx` pods using `wget`.
151 cd vpp/vagrant/vagrant-scripts/
152 vagrant ssh k8s-master
154 vagrant@k8s-master:~$ kubectl get pods -o wide
155 NAME READY STATUS RESTARTS AGE IP NODE
156 nginx-8586cf59-768qw 1/1 Running 0 11d 10.1.2.3 k8s-worker1
157 nginx-8586cf59-d27h2 1/1 Running 0 11d 10.1.2.2 k8s-worker1
159 vagrant@k8s-master:~$ wget 10.1.2.2
160 --2018-02-08 16:46:01-- http://10.1.2.2/
161 Connecting to 10.1.2.2:80... connected.
162 HTTP request sent, awaiting response... 200 OK
163 Length: 612 [text/html]
164 Saving to: ‘index.html’
165 index.html 100%[=========================================================>] 612 --.-KB/s in 0.004s
166 2018-02-08 16:46:01 (162 KB/s) - ‘index.html’ saved [612/612]
168 vagrant@k8s-master:~$ wget 10.1.2.3
169 --2018-02-08 16:46:02-- http://10.1.2.3/
170 Connecting to 10.1.2.3:80... connected.
171 HTTP request sent, awaiting response... 200 OK
172 Length: 612 [text/html]
173 Saving to: ‘index.html.1’
174 index.html.1 100%[=========================================================>] 612 --.-KB/s in 0.004s
175 2018-02-08 16:46:02 (143 KB/s) - ‘index.html.1’ saved [612/612]
178 #### Check the Packets Captured by VPP
180 Back in the first terminal, check the packets captured by VPP.
188 21:34:51:476110: tapcli-rx
190 21:34:51:476115: ethernet-input
191 IP4: 00:00:00:00:00:02 -> 02:fe:72:95:66:c7
192 21:34:51:476117: ip4-input
193 TCP: 10.1.2.3 -> 172.30.1.2
194 tos 0x00, ttl 64, length 52, checksum 0x6fb4
195 fragment id 0x11ec, flags DONT_FRAGMENT
197 seq. 0x5db741c8 ack 0x709defa7
198 flags 0x11 FIN ACK, tcp header: 32 bytes
199 window 235, checksum 0x55c3
200 21:34:51:476118: nat44-out2in
201 NAT44_OUT2IN: sw_if_index 6, next index 1, session index -1
202 21:34:51:476120: ip4-lookup
203 fib 0 dpo-idx 23 flow hash: 0x00000000
204 TCP: 10.1.2.3 -> 172.30.1.2
205 tos 0x00, ttl 64, length 52, checksum 0x6fb4
206 fragment id 0x11ec, flags DONT_FRAGMENT
208 seq. 0x5db741c8 ack 0x709defa7
209 flags 0x11 FIN ACK, tcp header: 32 bytes
210 window 235, checksum 0x55c3
211 21:34:51:476121: ip4-load-balance
212 fib 0 dpo-idx 23 flow hash: 0x00000000
213 TCP: 10.1.2.3 -> 172.30.1.2
214 tos 0x00, ttl 64, length 52, checksum 0x6fb4
215 fragment id 0x11ec, flags DONT_FRAGMENT
217 seq. 0x5db741c8 ack 0x709defa7
218 flags 0x11 FIN ACK, tcp header: 32 bytes
219 window 235, checksum 0x55c3
220 21:34:51:476122: ip4-rewrite
221 tx_sw_if_index 3 dpo-idx 5 : ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800 flow hash: 0x00000000
222 00000000: 1a2b3c4d5e011a2b3c4d5e0208004500003411ec40003f0670b40a010203ac1e
223 00000020: 01020050e43e5db741c8709defa7801100eb55c300000101080a0f4b
224 21:34:51:476123: loop0-output
226 IP4: 1a:2b:3c:4d:5e:02 -> 1a:2b:3c:4d:5e:01
227 TCP: 10.1.2.3 -> 172.30.1.2
228 tos 0x00, ttl 63, length 52, checksum 0x70b4
229 fragment id 0x11ec, flags DONT_FRAGMENT
231 seq. 0x5db741c8 ack 0x709defa7
232 flags 0x11 FIN ACK, tcp header: 32 bytes
233 window 235, checksum 0x55c3
234 21:34:51:476124: l2-input
235 l2-input: sw_if_index 3 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02
236 21:34:51:476125: l2-fwd
237 l2-fwd: sw_if_index 3 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02 bd_index 1
238 21:34:51:476125: l2-output
239 l2-output: sw_if_index 4 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02 data 08 00 45 00 00 34 11 ec 40 00 3f 06
240 21:34:51:476126: vxlan4-encap
241 VXLAN encap to vxlan_tunnel0 vni 10
242 21:34:51:476126: ip4-load-balance
243 fib 4 dpo-idx 22 flow hash: 0x00000103
244 UDP: 192.168.16.2 -> 192.168.16.1
245 tos 0x00, ttl 254, length 102, checksum 0x1b33
248 length 82, checksum 0x0000
249 21:34:51:476127: ip4-rewrite
250 tx_sw_if_index 1 dpo-idx 4 : ipv4 via 192.168.16.1 GigabitEthernet0/8/0: 080027b2610908002733fb6f0800 flow hash: 0x00000103
251 00000000: 080027b2610908002733fb6f08004500006600000000fd111c33c0a81002c0a8
252 00000020: 10015f0012b5005200000800000000000a001a2b3c4d5e011a2b3c4d
253 21:34:51:476127: GigabitEthernet0/8/0-output
255 IP4: 08:00:27:33:fb:6f -> 08:00:27:b2:61:09
256 UDP: 192.168.16.2 -> 192.168.16.1
257 tos 0x00, ttl 253, length 102, checksum 0x1c33
260 length 82, checksum 0x0000
261 21:34:51:476128: GigabitEthernet0/8/0-tx
262 GigabitEthernet0/8/0 tx queue 0
263 buffer 0xfa7f: current data -50, length 116, free-list 0, clone-count 0, totlen-nifb 0, trace 0x20
264 l2-hdr-offset 0 l3-hdr-offset 14
265 PKT MBUF: port 255, nb_segs 1, pkt_len 116
266 buf_len 2176, data_len 116, ol_flags 0x0, data_off 78, phys_addr 0x569ea040
267 packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
268 IP4: 08:00:27:33:fb:6f -> 08:00:27:b2:61:09
269 UDP: 192.168.16.2 -> 192.168.16.1
270 tos 0x00, ttl 253, length 102, checksum 0x1c33
273 length 82, checksum 0x0000
276 In the above captured packet, we can see:
278 * Input queue name `tapcli-rx`
279 * Pod's IP address `10.1.2.3`
280 * IP address of the master node `172.30.1.2`, which sent the `wget` traffic to the two pods
281 * HTTP port `80`, destination port and TCP protocol (`TCP: 80 -> 58430`)
282 * NAT queue name `nat44-out2in`
283 * VXLAN VNI ID `VXLAN encap to vxlan_tunnel0 vni 10`
284 * VXLAN UDP port `4789`
285 * IP address of `GigabitEthernet0/8/0` interface (`192.168.16.2`)
286 * Packet on the outgoing queue `GigabitEthernet0/8/0-tx`
288 #### Find IP Addresses of GigabitEthernet and the Tap Interfaces
289 Enter the following commands to find the IP addresses and Tap interfaces:
292 vpp# show int address
293 GigabitEthernet0/8/0 (up):
297 L2 bridge bd-id 1 idx 1 shg 0 bvi
306 L2 bridge bd-id 1 idx 1 shg 0
309 #### Other Useful VPP CLIs
311 Enter the following commands to see additional information about VPP:
315 Name Idx State Counter Count
316 GigabitEthernet0/8/0 1 up rx packets 138
323 local0 0 down drops 1
324 loop0 3 up rx packets 137
330 tapcli-0 2 up rx packets 8
336 tapcli-1 5 up rx packets 56
342 tapcli-2 6 up rx packets 42
348 vxlan_tunnel0 4 up rx packets 137
354 Name Idx Link Hardware
355 GigabitEthernet0/8/0 1 up GigabitEthernet0/8/0
356 Ethernet address 08:00:27:33:fb:6f
357 Intel 82540EM (e1000)
358 carrier up full duplex speed 1000 mtu 9216
359 rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
374 Ethernet address 1a:2b:3c:4d:5e:02
375 tapcli-0 2 up tapcli-0
376 Ethernet address 02:fe:95:07:df:9c
377 tapcli-1 5 up tapcli-1
378 Ethernet address 02:fe:3f:5f:0f:9a
379 tapcli-2 6 up tapcli-2
380 Ethernet address 02:fe:72:95:66:c7
381 vxlan_tunnel0 4 up vxlan_tunnel0
384 vpp# show bridge-domain
385 BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term BVI-Intf
386 1 1 1 off on on on on off loop0
388 vpp# show bridge-domain 1 detail
389 BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term BVI-Intf
390 1 1 1 off on on on on off loop0
392 Interface If-idx ISN SHG BVI TxFlood VLAN-Tag-Rewrite
394 vxlan_tunnel0 4 1 0 - * none
396 vpp# show l2fib verbose
397 Mac-Address BD-Idx If-Idx BSN-ISN Age(min) static filter bvi Interface-Name
398 1a:2b:3c:4d:5e:02 1 3 0/0 - * - * loop0
399 1a:2b:3c:4d:5e:01 1 4 1/1 - - - - vxlan_tunnel0
400 L2FIB total/learned entries: 2/1 Last scan time: 0.0000e0sec Learn limit: 4194304
403 ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:[src:(nil):2, src:adjacency:3, src:default-route:1, ]
406 [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:21 to:[0:0]]
407 [0] [@5]: ipv4 via 172.30.2.2 tapcli-0: def35b93961902fe9507df9c0800
410 [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
411 [0] [@0]: dpo-drop ip4
414 [@0]: dpo-load-balance: [proto:ip4 index:24 buckets:1 uRPF:29 to:[0:0]]
415 [0] [@10]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]]
416 [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800
419 [@0]: dpo-load-balance: [proto:ip4 index:27 buckets:1 uRPF:12 to:[78:5641]]
420 [0] [@5]: ipv4 via 10.1.2.2 tapcli-1: 00000000000202fe3f5f0f9a0800
423 [@0]: dpo-load-balance: [proto:ip4 index:29 buckets:1 uRPF:32 to:[58:4184]]
424 [0] [@5]: ipv4 via 10.1.2.3 tapcli-2: 00000000000202fe729566c70800
427 [@0]: dpo-load-balance: [proto:ip4 index:26 buckets:1 uRPF:31 to:[0:0]]
428 [0] [@2]: dpo-receive: 10.2.1.2 on tapcli-1
431 [@0]: dpo-load-balance: [proto:ip4 index:28 buckets:1 uRPF:33 to:[0:0]]
432 [0] [@2]: dpo-receive: 10.2.1.3 on tapcli-2
435 [@0]: dpo-load-balance: [proto:ip4 index:25 buckets:1 uRPF:29 to:[98:23234]]
436 [0] [@10]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]]
437 [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800
440 [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:15 to:[0:0]]
441 [0] [@0]: dpo-drop ip4
444 [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:14 to:[0:0]]
445 [0] [@4]: ipv4-glean: tapcli-0
448 [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:19 to:[0:0]]
449 [0] [@2]: dpo-receive: 172.30.2.1 on tapcli-0
452 [@0]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:18 to:[0:0]]
453 [0] [@5]: ipv4 via 172.30.2.2 tapcli-0: def35b93961902fe9507df9c0800
456 [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:17 to:[0:0]]
457 [0] [@0]: dpo-drop ip4
460 [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]
461 [0] [@0]: dpo-drop ip4
464 [@0]: dpo-load-balance: [proto:ip4 index:22 buckets:1 uRPF:34 to:[0:0] via:[100:28290]]
465 [0] [@5]: ipv4 via 192.168.16.1 GigabitEthernet0/8/0: 080027b2610908002733fb6f0800
468 [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:30 to:[0:0]]
469 [0] [@4]: ipv4-glean: GigabitEthernet0/8/0
472 [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[137:16703]]
473 [0] [@2]: dpo-receive: 192.168.16.2 on GigabitEthernet0/8/0
476 [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]
477 [0] [@0]: dpo-drop ip4
480 [@0]: dpo-load-balance: [proto:ip4 index:19 buckets:1 uRPF:23 to:[0:0]]
481 [0] [@0]: dpo-drop ip4
484 [@0]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]]
485 [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800
488 [@0]: dpo-load-balance: [proto:ip4 index:18 buckets:1 uRPF:22 to:[0:0]]
489 [0] [@4]: ipv4-glean: loop0
492 [@0]: dpo-load-balance: [proto:ip4 index:21 buckets:1 uRPF:27 to:[0:0]]
493 [0] [@2]: dpo-receive: 192.168.30.2 on loop0
496 [@0]: dpo-load-balance: [proto:ip4 index:20 buckets:1 uRPF:25 to:[0:0]]
497 [0] [@0]: dpo-drop ip4
500 [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
501 [0] [@0]: dpo-drop ip4
504 [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
505 [0] [@0]: dpo-drop ip4
508 [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
509 [0] [@0]: dpo-drop ip4