4 As of this writing, the vpp source tree includes over 1,000 unit test
5 vectors. Best practices prior to pushing patches for code review: make
6 sure that all of the "make test" test vectors pass.
8 We attempt to maintain the top-level "make test-help" command so that it
9 accurately describes all of the "make test" options.
14 Basic test run, all test vectors, single-vpp instance, optimized image:
20 10-way parallel basic test run:
23 $ make TEST_JOBS=10 test
26 Run a specific test suite (mpls, in this case):
29 $ make TEST=test_mpls test
32 Run a specifc test suite, debug image, pause prior to running the test
33 suite; attach to the vpp image in gdb:
36 $ make TEST=xxx DEBUG=gdb test-debug
39 Detailed Documentation
40 ----------------------
42 Current "make test-help" output:
47 test - build and run (basic) functional tests
48 test-debug - build and run (basic) functional tests (debug build)
49 test-all - build and run functional and extended tests
50 test-all-debug - build and run functional and extended tests (debug build)
51 retest - run functional tests
52 retest-debug - run functional tests (debug build)
53 retest-all - run functional and extended tests
54 retest-all-debug - run functional and extended tests (debug build)
55 test-cov - generate code coverage report for test framework
56 test-gcov - build and run functional tests (gcov build)
57 test-wipe - wipe (temporary) files generated by unit tests
58 test-wipe-cov - wipe code coverage report for test framework
59 test-wipe-doc - wipe documentation for test framework
60 test-wipe-papi - rebuild vpp_papi sources
61 test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage
62 test-shell - enter shell with test environment
63 test-shell-debug - enter shell with test environment (debug build)
64 test-checkstyle - check PEP8 compliance for test framework
65 test-refresh-deps - refresh the Python dependencies for the tests
67 Arguments controlling test runs:
68 V=[0|1|2] - set test verbosity level
69 0=ERROR, 1=INFO, 2=DEBUG
70 TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
71 MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
72 CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
73 FAILFAST=[0|1] - fail fast if 1, complete all tests if 0
74 TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
75 RETRIES=<n> - retry failed tests <n> times
76 DEBUG=<type> - set VPP debugging kind
77 DEBUG=core - detect coredump and load it in gdb on crash
78 DEBUG=gdb - allow easy debugging by printing VPP PID
79 and waiting for user input before running
80 and tearing down a testcase
81 DEBUG=gdbserver - run gdb inside a gdb server, otherwise
83 DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
85 STEP=[yes|no] - ease debugging by stepping through a testcase
86 SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
87 EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
88 TEST=<filter> - filter the set of tests:
89 by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
90 by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
91 by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
92 e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
93 TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
94 TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
95 TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
96 TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
98 VARIANT=<variant> - specify which march node variant to unit test
99 e.g. VARIANT=skx test the skx march variants
100 e.g. VARIANT=icl test the icl march variants
102 COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
103 e.g. COREDUMP_SIZE=4g
104 COREDUMP_SIZE=unlimited
105 COREDUMP_COMPRESS=1 - compress core files if not debugging them
106 EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
107 EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
108 EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
110 PROFILE=1 - enable profiling of test framework via cProfile module
111 PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
112 PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
114 TEST_DEBUG=1 - turn on debugging of the test framework itself (expert)
116 SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI
118 RND_SEED=seed - Seed RND with given seed
120 Starting VPP in GDB for use with DEBUG=attach:
122 test-start-vpp-in-gdb - start VPP in gdb (release)
123 test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
125 Arguments controlling VPP in GDB runs:
127 VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
128 VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead
129 VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line
131 Creating test documentation
132 test-doc - generate documentation for test framework
133 test-wipe-doc - wipe documentation for test framework
135 Creating test code coverage report
136 test-cov - generate code coverage report for test framework
137 test-wipe-cov - wipe code coverage report for test framework
140 test-checkstyle - check PEP8 compliance