8 Although every situation is different, this section describes how to
9 collect data which will help make efficient use of everyone's time
10 when dealing with vpp bugs.
12 Before you press the Jira button to create a bug report - or email
13 vpp-dev@lists.fd.io - please ask yourself whether there's enough
14 information for someone else to understand and to reproduce the issue
15 given a reasonable amount of effort. **Unicast emails to maintainers,
16 committers, and the project PTL are strongly discouraged.**
18 A good strategy for clear-cut bugs: file a detailed Jira ticket, and
19 then send a short description of the issue to vpp-dev@lists.fd.io,
20 perhaps from the Jira ticket description. It's fine to send email to
21 vpp-dev@lists.fd.io to ask a few questions **before** filing Jira tickets.
23 Data to include in bug reports
24 ==============================
26 Image version and operating environment
27 ---------------------------------------
29 Please make sure to include the vpp image version and command-line arguments.
31 .. code-block:: console
34 # vppctl show version verbose cmdline
35 Version: v18.07-rc0~509-gb9124828
37 Compile host: vppbuild
38 Compile date: Fri Jul 13 09:05:37 EDT 2018
39 Compile location: /scratch/vpp-showversion
42 Command line arguments:
43 /scratch/vpp-showversion/build-root/install-vpp_debug-native/vpp/bin/vpp
47 With respect to the operating environment: if misbehavior involving a
48 specific VM / container / bare-metal environment is involved, please
49 describe the environment in detail:
51 * Linux Distro (e.g. Ubuntu 18.04.2 LTS, CentOS-7, etc.)
52 * NIC type(s) (ixgbe, i40e, enic, etc. etc.), vhost-user, tuntap
53 * NUMA configuration if applicable
55 Please note the CPU architecture (x86_86, aarch64), and hardware platform.
57 When practicable, please report issues against released software, or
58 unmodified master/latest software.
63 Every situation is different. If the issue involves a sequence of
64 debug CLI command, please enable CLI command logging, and send the
65 sequence involved. Note that the debug CLI is a developer's tool -
66 **no warranty express or implied** - and that we may choose not to fix
69 Please include "show error" [error counter] output. It's often helpful
70 to "clear error", send a bit of traffic, then "show error"
71 particularly when running vpp on noisy networks.
73 Please include ip4 / ip6 / mpls FIB contents ("show ip fib", "show ip6
74 fib", "show mpls fib", "show mpls tunnel").
76 Please include "show hardware", "show interface", and "show interface
79 Here is a consolidated set of commands that are generally useful
80 before/after sending traffic. Before sending traffic:
82 .. code-block:: console
85 vppctl clear interface
89 Send some traffic and then issue the following commands.
91 .. code-block:: console
93 vppctl show version verbose
95 vppctl show interface address
100 Here are some protocol specific show commands that may also make
101 sense. Only include those features which have been configured.
103 .. code-block:: console
106 vppctl show bridge-domain
109 vppctl show ip neighbors
112 vppctl show ip6 neighbors
115 vppctl show mpls tunnel
120 Please include a crisp description of the network topology, including
121 L2 / IP / MPLS / segment-routing addressing details. If you expect
122 folks to reproduce and debug issues, this is a must.
124 At or above a certain level of topological complexity, it becomes
125 problematic to reproduce the original setup.
130 If you capture packet tracer output which seems relevant, please include it.
132 .. code-block:: console
134 vppctl trace add dpdk-input 100 # or similar
138 .. code-block:: console
142 Capturing post-mortem data
143 ==========================
145 It should go without saying, but anyhow: **please put post-mortem data
146 in obvious, accessible places.** Time wasted trying to acquire
147 accounts, credentials, and IP addresses simply delays problem
150 Please remember to add post-mortem data location information to Jira
156 The vpp signal handler typically writes a certain amount of data in
157 /var/log/syslog before exiting. Make sure to check for evidence, e.g
158 via "grep /usr/bin/vpp /var/log/syslog" or similar.
163 If the issue involves a sequence of control-plane API messages - even
164 a very long sequence - please enable control-plane API
165 tracing. Control-plane API post-mortem traces end up in
166 /tmp/api_post_mortem.<pid>.
168 Please remember to put post-mortem binary api traces in accessible
171 These API traces are especially helpful in cases where the vpp engine
172 is throwing traffic on the floor, e.g. for want of a default route or
175 Make sure to leave the default stanza "... api-trace { on } ... " in
176 the vpp startup configuration file /etc/vpp/startup.conf, or to
177 include it in the command line arguments passed by orchestration
183 Production systems, as well as long-running pre-production soak-test
184 systems, **must** arrange to collect core images. There are various
185 ways to configure core image capture, including e.g. the Ubuntu
186 "corekeeper" package. In a pinch, the following very basic sequence
187 will capture usable vpp core files in /tmp/dumps.
189 .. code-block:: console
191 # mkdir -p /tmp/dumps
192 # sysctl -w debug.exception-trace=1
193 # sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
194 # ulimit -c unlimited
195 # echo 2 > /proc/sys/fs/suid_dumpable
197 If you start VPP from systemd, you also need to edit
198 /lib/systemd/system/vpp.service and uncomment the "LimitCORE=infinity"
199 line before restarting VPP.
201 Vpp core files often appear enormous, but they are invariably
202 sparse. Gzip compresses them to manageable sizes. A multi-GByte
203 corefile often compresses to 10-20 Mbytes.
205 When decompressing a vpp core file, we suggest using "dd" as shown to
206 create a sparse, uncompressed core file:
208 .. code-block:: console
210 $ zcat vpp_core.gz | dd conv=sparse of=vpp_core
212 Please remember to put compressed core files in accessible places.
214 Make sure to leave the default stanza "... unix { ... full-coredump
215 ... } ... " in the vpp startup configuration file
216 /etc/vpp/startup.conf, or to include it in the command line arguments
217 passed by orchestration software.
219 Core files from Private Images
220 ==============================
222 Core files from private images require special handling. If it's
223 necessary to go that route, copy the **exact** Debian packages (or
224 RPMs) which correspond to the core file to the same public place as
225 the core file. A no-excuses-allowed, hard-and-fast requirement.
229 .. code-block:: console
231 libvppinfra_<version>_<arch>.deb # vppinfra library
232 libvppinfra-dev_<version>_<arch>.deb # vppinfra library development pkg
233 vpp_<version>_<arch>.deb # the vpp executable
234 vpp-dbg_<version>_<arch>.deb # debug symbols
235 vpp-dev_<version>_<arch>.deb # vpp development pkg
236 vpp-lib_<version>_<arch>.deb # shared libraries
237 vpp-plugin-core_<version>_<arch>.deb # core plugins
238 vpp-plugin-dpdk_<version>_<arch>.deb # dpdk plugin
240 For reference, please include git commit-ID, branch, and git repo
241 information [for repos other than gerrit.fd.io] in the Jira ticket.
243 Note that git commit-ids are crypto sums of the head [latest]
244 **merged** patch. They say **nothing whatsoever** about local
245 workspace modifications, branching, or the git repo in question.
247 Even given a byte-for-byte identical source tree, it's easy to build
248 dramatically different binary artifacts. All it takes is a different
252 On-the-fly Core File Compression
253 --------------------------------
255 Depending on operational requirements, it's possible to compress
256 corefiles as they are generated. Please note that it takes several
257 seconds' worth of wall-clock time to compress a vpp core file on the
258 fly, during which all packet processing activities are suspended.
260 To create compressed core files on the fly, create the following
261 script, e.g. in /usr/local/bin/compressed_corefiles, owned by root,
264 .. code-block:: console
267 exec /bin/gzip -f - >"/tmp/dumps/core-$1.$2.gz"
269 Adjust the kernel core file pattern as shown:
271 .. code-block:: console
273 sysctl -w kernel.core_pattern="|/usr/local/bin/compressed_corefiles %e %t"
278 Bottom line: please follow core file handling instructions to the
279 letter. It's not complicated. Simply copy the exact Debian packages or
280 RPMs which correspond to core files to accessible locations.
282 If we go through the setup process only to discover that the image and
283 core files don't match, it will simply delay resolution of the issue;
284 to say nothing of irritating the person who just wasted their time.