8 Although every situation is different, this page describes how to
9 collect data which will help make efficient use of everyone's time
10 when dealing with vpp bugs.
12 Before you press the Jira button to create a bug report - or email
13 vpp-dev@lists.fd.io - please ask yourself whether there's enough
14 information for someone else to understand and possibly to reproduce
15 the issue given a reasonable amount of effort. **Unicast emails to
16 maintainers, committers, and the project PTL are strongly discouraged.**
18 A good strategy for clear-cut bugs: file a detailed Jira ticket, and
19 then send a short description of the issue to vpp-dev@lists.fd.io,
20 perhaps from the Jira ticket description. It's fine to send email to
21 vpp-dev@lists.fd.io to ask a few questions **before** filing Jira tickets.
23 Data to include in bug reports
24 ==============================
26 Image version and operating environment
27 ---------------------------------------
29 Please make sure to include the vpp image version and command-line arguments.
31 .. code-block:: console
34 # vppctl show version verbose cmdline
35 Version: v18.07-rc0~509-gb9124828
37 Compile host: vppbuild
38 Compile date: Fri Jul 13 09:05:37 EDT 2018
39 Compile location: /scratch/vpp-showversion
42 Command line arguments:
43 /scratch/vpp-showversion/build-root/install-vpp_debug-native/vpp/bin/vpp
47 With respect to the operating environment: if misbehavior involving a
48 specific VM / container / bare-metal environment is involved, please
49 describe the environment in detail:
51 * Linux Distro (e.g. Ubuntu 14.04.3 LTS, CentOS-7, etc.)
52 * NIC type(s) (ixgbe, i40e, enic, etc. etc.), vhost-user, tuntap
53 * NUMA configuration if applicable
55 Please note the CPU architecture (x86_86, aarch64), and hardware platform.
57 When practicable, please report issues against released software, or
58 unmodified master/latest software.
63 Every situation is different. If the issue involves a sequence of debug CLI command, please enable CLI command logging, and send the sequence involved. Note that the debug CLI is a developer's tool - **no warranty express or implied** - and that we may choose not to fix debug CLI bugs.
65 Please include "show error" [error counter] output. It's often helpful to "clear error", send a bit of traffic, then "show error" particularly when running vpp on a noisy networks.
67 Please include ip4 / ip6 / mpls FIB contents ("show ip fib", "show ip6 fib", "show mpls fib", "show mpls tunnel").
69 Please include "show hardware", "show interface", and "show interface address" output
71 Here is a consolidated set of commands that are generally useful before/after sending traffic. Before sending traffic.
73 .. code-block:: console
76 vppctl clear interface
80 Send some traffic and then issue the following commands.
82 .. code-block:: console
84 vppctl show version verbose
86 vppctl show hardware address
91 Here are some protocol specific show commands that may also make
92 sense. Only include those features which have been configured.
94 .. code-block:: console
97 vppctl show bridge-domain
103 vppctl show ip6 neighbors
106 vppctl show mpls tunnel
111 Please include a crisp description of the network topology, including
112 L2 / IP / MPLS / segment-routing addressing details. If you expect
113 folks to reproduce and debug issues, this is a must.
115 At or above a certain level of topological complexity, it becomes
116 problematic to reproduce the original setup.
121 If you capture packet tracer output which seems relevant, please include it.
123 .. code-block:: console
125 vppctl trace add dpdk-input 100 # or similar
129 .. code-block:: console
133 Capturing post-mortem data
134 ==========================
136 It should go without saying, but anyhow: **please put post-mortem data
137 in obvious, accessible places.** Time wasted trying to acquire
138 accounts, credentials, and IP addresses simply delays problem
141 Please remember to add post-mortem data location information to Jira
147 The vpp signal handler typically writes a certain amount of data in
148 /var/log/syslog before exiting. Make sure to check for evidence, e.g
149 via "grep /usr/bin/vpp /var/log/syslog" or similar.
154 If the issue involves a sequence of control-plane API messages - even
155 a very long sequence - please enable control-plane API
156 tracing. Control-plane API post-mortem traces end up in
157 /tmp/api_post_mortem.<pid>.
159 Please remember to put post-mortem binary api traces in accessible
162 These API traces are especially helpful in cases where the vpp engine
163 is throwing traffic on the floor, e.g. for want of a default route or
166 Make sure to leave the default stanza "... api-trace { on } ... " in
167 the vpp startup configuration file /etc/vpp/startup.conf, or to
168 include it in the command line arguments passed by orchestration
174 Production systems, as well as long-running pre-production soak-test
175 systems, **must** arrange to collect core images. There are various
176 ways to configure core image capture, including e.g. the Ubuntu
177 "corekeeper" package. In a pinch, the following very basic sequence
178 will capture usable vpp core files in /tmp/dumps.
180 .. code-block:: console
182 # mkdir -p /tmp/dumps
183 # sysctl -w debug.exception-trace=1
184 # sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
185 # ulimit -c unlimited
186 # echo 2 > /proc/sys/fs/suid_dumpable
188 Vpp core files often appear enormous. Gzip typically compresses them
189 to manageable sizes. A multi-GByte corefile often compresses to 10-20
192 Please remember to put compressed core files in accessible places.
194 Make sure to leave the default stanza "... unix { ... full-coredump
195 ... } ... " in the vpp startup configuration file
196 /etc/vpp/startup.conf, or to include it in the command line arguments
197 passed by orchestration software.
199 Core files from private, modified images are discouraged. If it's
200 necessary to go that route, please copy the **exact** Debian
201 packages (or RPMs) corresponding to the core file to the same public
202 place as the core file. In particular.
204 * vpp_<version>_<arch>.deb # the vpp executable
205 * vpp-dbg_<version>_<arch>.deb # debug symbols
206 * vpp-dev_<version>_<arch>.deb # development package
207 * vpp-lib_<version>_<arch>.deb # shared libraries
208 * vpp-plugins_<version>_<arch>.deb # plugins
210 Please include the full commit-ID the Jira ticket.
212 If we go through the setup process only to discover that the image and
213 core files don't match, it will simply delay resolution of the
214 issue. And it will annoy the heck out of the engineer who just wasted
215 their time. Exact means **exact**, not "oh, gee, I added a few lines
216 of debug scaffolding since then..."