5 Data to include in bug reports
6 ==============================
8 Image version and operating environment
9 ---------------------------------------
11 Please make sure to include the vpp image version and command-line arguments.
13 .. code-block:: console
16 # vppctl show version verbose cmdline
17 Version: v18.07-rc0~509-gb9124828
19 Compile host: vppbuild
20 Compile date: Fri Jul 13 09:05:37 EDT 2018
21 Compile location: /scratch/vpp-showversion
24 Command line arguments:
25 /scratch/vpp-showversion/build-root/install-vpp_debug-native/vpp/bin/vpp
29 With respect to the operating environment: if misbehavior involving a
30 specific VM / container / bare-metal environment is involved, please
31 describe the environment in detail:
33 * Linux Distro (e.g. Ubuntu 18.04.2 LTS, CentOS-7, etc.)
34 * NIC type(s) (ixgbe, i40e, enic, etc. etc.), vhost-user, tuntap
35 * NUMA configuration if applicable
37 Please note the CPU architecture (x86_86, aarch64), and hardware platform.
39 When practicable, please report issues against released software, or
40 unmodified master/latest software.
45 Every situation is different. If the issue involves a sequence of
46 debug CLI command, please enable CLI command logging, and send the
47 sequence involved. Note that the debug CLI is a developer's tool -
48 **no warranty express or implied** - and that we may choose not to fix
51 Please include "show error" [error counter] output. It's often helpful
52 to "clear error", send a bit of traffic, then "show error"
53 particularly when running vpp on noisy networks.
55 Please include ip4 / ip6 / mpls FIB contents ("show ip fib", "show ip6
56 fib", "show mpls fib", "show mpls tunnel").
58 Please include "show hardware", "show interface", and "show interface
61 Here is a consolidated set of commands that are generally useful
62 before/after sending traffic. Before sending traffic:
64 .. code-block:: console
67 vppctl clear interface
71 Send some traffic and then issue the following commands.
73 .. code-block:: console
75 vppctl show version verbose
77 vppctl show interface address
82 Here are some protocol specific show commands that may also make
83 sense. Only include those features which have been configured.
85 .. code-block:: console
88 vppctl show bridge-domain
91 vppctl show ip neighbors
94 vppctl show ip6 neighbors
97 vppctl show mpls tunnel
102 Please include a crisp description of the network topology, including
103 L2 / IP / MPLS / segment-routing addressing details. If you expect
104 folks to reproduce and debug issues, this is a must.
106 At or above a certain level of topological complexity, it becomes
107 problematic to reproduce the original setup.
112 If you capture packet tracer output which seems relevant, please include it.
114 .. code-block:: console
116 vppctl trace add dpdk-input 100 # or similar
120 .. code-block:: console
124 Capturing post-mortem data
125 ==========================
127 It should go without saying, but anyhow: **please put post-mortem data
128 in obvious, accessible places.** Time wasted trying to acquire
129 accounts, credentials, and IP addresses simply delays problem
132 Please remember to add post-mortem data location information to Jira
138 The vpp signal handler typically writes a certain amount of data in
139 /var/log/syslog before exiting. Make sure to check for evidence, e.g
140 via "grep /usr/bin/vpp /var/log/syslog" or similar.
145 If the issue involves a sequence of control-plane API messages - even
146 a very long sequence - please enable control-plane API
147 tracing. Control-plane API post-mortem traces end up in
148 /tmp/api_post_mortem.<pid>.
150 Please remember to put post-mortem binary api traces in accessible
153 These API traces are especially helpful in cases where the vpp engine
154 is throwing traffic on the floor, e.g. for want of a default route or
157 Make sure to leave the default stanza "... api-trace { on } ... " in
158 the vpp startup configuration file /etc/vpp/startup.conf, or to
159 include it in the command line arguments passed by orchestration
165 Production systems, as well as long-running pre-production soak-test
166 systems, **must** arrange to collect core images. There are various
167 ways to configure core image capture, including e.g. the Ubuntu
168 "corekeeper" package. In a pinch, the following very basic sequence
169 will capture usable vpp core files in /tmp/dumps.
171 .. code-block:: console
173 # mkdir -p /tmp/dumps
174 # sysctl -w debug.exception-trace=1
175 # sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
176 # ulimit -c unlimited
177 # echo 2 > /proc/sys/fs/suid_dumpable
179 If you start VPP from systemd, you also need to edit
180 /lib/systemd/system/vpp.service and uncomment the "LimitCORE=infinity"
181 line before restarting VPP.
183 Vpp core files often appear enormous, but they are invariably
184 sparse. Gzip compresses them to manageable sizes. A multi-GByte
185 corefile often compresses to 10-20 Mbytes.
187 When decompressing a vpp core file, we suggest using "dd" as shown to
188 create a sparse, uncompressed core file:
190 .. code-block:: console
192 $ zcat vpp_core.gz | dd conv=sparse of=vpp_core
194 Please remember to put compressed core files in accessible places.
196 Make sure to leave the default stanza "... unix { ... full-coredump
197 ... } ... " in the vpp startup configuration file
198 /etc/vpp/startup.conf, or to include it in the command line arguments
199 passed by orchestration software.
201 Core files from Private Images
202 ==============================
204 Core files from private images require special handling. If it's
205 necessary to go that route, copy the **exact** Debian packages (or
206 RPMs) which correspond to the core file to the same public place as
207 the core file. A no-excuses-allowed, hard-and-fast requirement.
211 .. code-block:: console
213 libvppinfra_<version>_<arch>.deb # vppinfra library
214 libvppinfra-dev_<version>_<arch>.deb # vppinfra library development pkg
215 vpp_<version>_<arch>.deb # the vpp executable
216 vpp-dbg_<version>_<arch>.deb # debug symbols
217 vpp-dev_<version>_<arch>.deb # vpp development pkg
218 vpp-lib_<version>_<arch>.deb # shared libraries
219 vpp-plugin-core_<version>_<arch>.deb # core plugins
220 vpp-plugin-dpdk_<version>_<arch>.deb # dpdk plugin
222 For reference, please include git commit-ID, branch, and git repo
223 information [for repos other than gerrit.fd.io] in the Jira ticket.
225 Note that git commit-ids are crypto sums of the head [latest]
226 **merged** patch. They say **nothing whatsoever** about local
227 workspace modifications, branching, or the git repo in question.
229 Even given a byte-for-byte identical source tree, it's easy to build
230 dramatically different binary artifacts. All it takes is a different
234 On-the-fly Core File Compression
235 --------------------------------
237 Depending on operational requirements, it's possible to compress
238 corefiles as they are generated. Please note that it takes several
239 seconds' worth of wall-clock time to compress a vpp core file on the
240 fly, during which all packet processing activities are suspended.
242 To create compressed core files on the fly, create the following
243 script, e.g. in /usr/local/bin/compressed_corefiles, owned by root,
246 .. code-block:: console
249 exec /bin/gzip -f - >"/tmp/dumps/core-$1.$2.gz"
251 Adjust the kernel core file pattern as shown:
253 .. code-block:: console
255 sysctl -w kernel.core_pattern="|/usr/local/bin/compressed_corefiles %e %t"
260 Bottom line: please follow core file handling instructions to the
261 letter. It's not complicated. Simply copy the exact Debian packages or
262 RPMs which correspond to core files to accessible locations.
264 If we go through the setup process only to discover that the image and
265 core files don't match, it will simply delay resolution of the issue;
266 to say nothing of irritating the person who just wasted their time.