1 .. _unittest: https://docs.python.org/2/library/unittest.html
2 .. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase
3 .. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError
4 .. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest
5 .. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/
6 .. _scapy: http://www.secdev.org/projects/scapy/
7 .. _logging: https://docs.python.org/2/library/logging.html
8 .. _process: https://docs.python.org/2/library/multiprocessing.html#the-process-class
9 .. _pipes: https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Pipe
10 .. _managed: https://docs.python.org/2/library/multiprocessing.html#managers
12 .. |vtf| replace:: VPP Test Framework
24 The goal of the |vtf| is to ease writing, running and debugging
25 unit tests for the VPP. For this, python was chosen as a high level language
26 allowing rapid development with scapy_ providing the necessary tool for creating
27 and dissecting packets.
29 Anatomy of a test case
30 ######################
32 Python's unittest_ is used as the base framework upon which the VPP test
33 framework is built. A test suite in the |vtf| consists of multiple classes
34 derived from `VppTestCase`, which is itself derived from TestCase_.
35 The test class defines one or more test functions, which act as test cases.
37 Function flow when running a test case is:
39 1. `setUpClass <VppTestCase.setUpClass>`:
40 This function is called once for each test class, allowing a one-time test
41 setup to be executed. If this functions throws an exception,
42 none of the test functions are executed.
43 2. `setUp <VppTestCase.setUp>`:
44 The setUp function runs before each of the test functions. If this function
45 throws an exception other than AssertionError_ or SkipTest_, then this is
46 considered an error, not a test failure.
48 This is the guts of the test case. It should execute the test scenario
49 and use the various assert functions from the unittest framework to check
50 necessary. Multiple test_<name> methods can exist in a test case.
51 4. `tearDown <VppTestCase.tearDown>`:
52 The tearDown function is called after each test function with the purpose
53 of doing partial cleanup.
54 5. `tearDownClass <VppTestCase.tearDownClass>`:
55 Method called once after running all of the test functions to perform
61 Each test case has a logger automatically created for it, stored in
62 'logger' property, based on logging_. Use the logger's standard methods
63 debug(), info(), error(), ... to emit log messages to the logger.
65 All the log messages go always into a log file in temporary directory
68 To control the messages printed to console, specify the V= parameter.
72 make test # minimum verbosity
73 make test V=1 # moderate verbosity
74 make test V=2 # maximum verbosity
76 Parallel test execution
77 #######################
79 |vtf| test suites can be run in parallel. Each test suite is executed
80 in a separate process spawned by Python multiprocessing process_.
82 The results from child test suites are sent to parent through pipes_, which are
83 aggregated and summarized at the end of the run.
85 Stdout, stderr and logs logged in child processes are redirected to individual
86 parent managed_ queues. The data from these queues are then emitted to stdout
87 of the parent process in the order the test suites have finished. In case there
88 are no finished test suites (such as at the beginning of the run), the data
89 from last started test suite are emitted in real time.
91 To enable parallel test run, specify the number of parallel processes:
95 make test TEST_JOBS=n # at most n processes will be spawned
96 make test TEST_JOBS=auto # chosen based on the number of cores
97 # and the size of shared memory
99 Test temporary directory and VPP life cycle
100 ###########################################
102 Test separation is achieved by separating the test files and vpp instances.
103 Each test creates a temporary directory and it's name is used to create
104 a shared memory prefix which is used to run a VPP instance.
105 The temporary directory name contains the testcase class name for easy
106 reference, so for testcase named 'TestVxlan' the directory could be named
107 e.g. vpp-unittest-TestVxlan-UNUP3j.
108 This way, there is no conflict between any other VPP instances running
109 on the box and the test VPP. Any temporary files created by the test case
110 are stored in this temporary test directory.
112 The test temporary directory holds the following interesting files:
114 * log.txt - this contains the logger output on max verbosity
115 * pg*_in.pcap - last injected packet stream into VPP, named after the interface,
116 so for pg0, the file will be named pg0_in.pcap
117 * pg*_out.pcap - last capture file created by VPP for interface, similarly,
118 named after the interface, so for e.g. pg1, the file will be named
120 * history files - whenever the capture is restarted or a new stream is added,
121 the existing files are rotated and renamed, soo all the pcap files
122 are always saved for later debugging if needed
123 * core - if vpp dumps a core, it'll be stored in the temporary directory
124 * vpp_stdout.txt - file containing output which vpp printed to stdout
125 * vpp_stderr.txt - file containing output which vpp printed to stderr
127 *NOTE*: existing temporary directories named vpp-unittest-* are automatically
128 removed when invoking 'make test*' or 'make retest*' to keep the temporary
134 Virtualenv_ is a python module which provides a means to create an environment
135 containing the dependencies required by the |vtf|, allowing a separation
136 from any existing system-wide packages. |vtf|'s Makefile automatically
137 creates a virtualenv_ inside build-root and installs the required packages
138 in that environment. The environment is entered whenever executing a test
139 via one of the make test targets.
144 Most unit tests do some kind of packet manipulation - sending and receiving
145 packets between VPP and virtual hosts connected to the VPP. Referring
146 to the sides, addresses, etc. is always done as if looking from the VPP side,
149 * *local_* prefix is used for the VPP side.
150 So e.g. `local_ip4 <VppInterface.local_ip4>` address is the IPv4 address
151 assigned to the VPP interface.
152 * *remote_* prefix is used for the virtual host side.
153 So e.g. `remote_mac <VppInterface.remote_mac>` address is the MAC address
154 assigned to the virtual host connected to the VPP.
156 Automatically generated addresses
157 #################################
159 To send packets, one needs to typically provide some addresses, otherwise
160 the packets will be dropped. The interface objects in |vtf| automatically
161 provide addresses based on (typically) their indexes, which ensures
162 there are no conflicts and eases debugging by making the addressing scheme
165 The developer of a test case typically doesn't need to work with the actual
166 numbers, rather using the properties of the objects. The addresses typically
167 come in two flavors: '<address>' and '<address>n' - note the 'n' suffix.
168 The former address is a Python string, while the latter is translated using
169 socket.inet_pton to raw format in network byte order - this format is suitable
170 for passing as an argument to VPP APIs.
172 e.g. for the IPv4 address assigned to the VPP interface:
174 * local_ip4 - Local IPv4 address on VPP interface (string)
175 * local_ip4n - Local IPv4 address - raw, suitable as API parameter.
177 These addresses need to be configured in VPP to be usable using e.g.
178 `VppInterface.config_ip4` API. Please see the documentation to
179 `VppInterface` for more details.
181 By default, there is one remote address of each kind created for L3:
182 remote_ip4 and remote_ip6. If the test needs more addresses, because it's
183 simulating more remote hosts, they can be generated using
184 `generate_remote_hosts` API and the entries for them inserted into the ARP
185 table using `configure_ipv4_neighbors` API.
187 Packet flow in the |vtf|
188 ########################
190 Test framework -> VPP
191 ~~~~~~~~~~~~~~~~~~~~~
193 |vtf| doesn't send any packets to VPP directly. Traffic is instead injected
194 using packet-generator interfaces, represented by the `VppPGInterface` class.
195 Packets are written into a temporary .pcap file, which is then read by the VPP
196 and the packets are injected into the VPP world.
198 To add a list of packets to an interface, call the `VppPGInterface.add_stream`
199 method on that interface. Once everything is prepared, call `pg_start` method to
200 start the packet generator on the VPP side.
202 VPP -> test framework
203 ~~~~~~~~~~~~~~~~~~~~~
205 Similarly, VPP doesn't send any packets to |vtf| directly. Instead, packet
206 capture feature is used to capture and write traffic to a temporary .pcap file,
207 which is then read and analyzed by the |vtf|.
209 The following APIs are available to the test case for reading pcap files.
211 * `VppPGInterface.get_capture`: this API is suitable for bulk & batch
212 style of test, where a list of packets is prepared & sent, then the
213 received packets are read and verified. The API needs the number of
214 packets which are expected to be captured (ignoring filtered
215 packets - see below) to know when the pcap file is completely
216 written by the VPP. If using packet infos for verifying packets,
217 then the counts of the packet infos can be automatically used by
218 `VppPGInterface.get_capture` to get the proper count (in this case
219 the default value None can be supplied as expected_count or ommitted
221 * `VppPGInterface.wait_for_packet`: this API is suitable for
222 interactive style of test, e.g. when doing session management,
223 three-way handshakes, etc. This API waits for and returns a single
224 packet, keeping the capture file in place and remembering
225 context. Repeated invocations return following packets (or raise
226 Exception if timeout is reached) from the same capture file (=
227 packets arriving on the same interface).
229 *NOTE*: it is not recommended to mix these APIs unless you understand
230 how they work internally. None of these APIs rotate the pcap capture
231 file, so calling e.g. `VppPGInterface.get_capture` after
232 `VppPGInterface.wait_for_packet` will return already read packets. It
233 is safe to switch from one API to another after calling
234 `VppPGInterface.enable_capture` as that API rotates the capture file.
236 Automatic filtering of packets:
237 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
239 Both APIs (`VppPGInterface.get_capture` and
240 `VppPGInterface.wait_for_packet`) by default filter the packet
241 capture, removing known uninteresting packets from it - these are IPv6
242 Router Advertisments and IPv6 Router Alerts. These packets are
243 unsolicitated and from the point of |vtf| are random. If a test wants
244 to receive these packets, it should specify either None or a custom
245 filtering function as the value to the 'filter_out_fn' argument.
247 Common API flow for sending/receiving packets:
248 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
250 We will describe a simple scenario, where packets are sent from pg0 to pg1
251 interface, assuming that the interfaces were created using
252 `create_pg_interfaces` API.
254 1. Create a list of packets for pg0::
257 packets = create_packets(src=self.pg0, dst=self.pg1,
260 2. Add that list of packets to the source interface::
262 self.pg0.add_stream(packets)
264 3. Enable capture on the destination interface::
266 self.pg1.enable_capture()
268 4. Start the packet generator::
272 5. Wait for capture file to appear and read it::
274 capture = self.pg1.get_capture(expected_count=packet_count)
276 6. Verify packets match sent packets::
278 self.verify_capture(send=packets, captured=capture)
280 Test framework objects
281 ######################
283 The following objects provide VPP abstraction and provide a means to do
284 common tasks easily in the test cases.
286 * `VppInterface`: abstract class representing generic VPP interface
287 and contains some common functionality, which is then used by derived classes
288 * `VppPGInterface`: class representing VPP packet-generator interface.
289 The interface is created/destroyed when the object is created/destroyed.
290 * `VppSubInterface`: VPP sub-interface abstract class, containing common
291 functionality for e.g. `VppDot1QSubint` and `VppDot1ADSubint` classes
293 How VPP APIs/CLIs are called
294 ############################
296 Vpp provides python bindings in a python module called vpp-papi, which the test
297 framework installs in the virtual environment. A shim layer represented by
298 the `VppPapiProvider` class is built on top of the vpp-papi, serving these
301 1. Automatic return value checks:
302 After each API is called, the return value is checked against the expected
303 return value (by default 0, but can be overridden) and an exception
304 is raised if the check fails.
305 2. Automatic call of hooks:
307 a. `before_cli <Hook.before_cli>` and `before_api <Hook.before_api>` hooks
308 are used for debug logging and stepping through the test
309 b. `after_cli <Hook.after_cli>` and `after_api <Hook.after_api>` hooks
310 are used for monitoring the vpp process for crashes
311 3. Simplification of API calls:
312 Many of the VPP APIs take a lot of parameters and by providing sane defaults
313 for these, the API is much easier to use in the common case and the code is
314 more readable. E.g. ip_add_del_route API takes ~25 parameters, of which
315 in the common case, only 3 are needed.
320 Some interesting utility methods are:
322 * `ppp`: 'Pretty Print Packet' - returns a string containing the same output
323 as Scapy's packet.show() would print
324 * `ppc`: 'Pretty Print Capture' - returns a string containing printout of
325 a capture (with configurable limit on the number of packets printed from it)
328 *NOTE*: Do not use Scapy's packet.show() in the tests, because it prints
329 the output to stdout. All output should go to the logger associated with
332 Example: how to add a new test
333 ##############################
335 In this example, we will describe how to add a new test case which tests
336 basic IPv4 forwarding.
338 1. Add a new file called test_ip4_fwd.py in the test directory, starting
341 from framework import VppTestCase
342 from scapy.layers.l2 import Ether
343 from scapy.packet import Raw
344 from scapy.layers.inet import IP, UDP
345 from random import randint
347 2. Create a class inherited from the VppTestCase::
349 class IP4FwdTestCase(VppTestCase):
350 """ IPv4 simple forwarding test case """
352 3. Add a setUpClass function containing the setup needed for our test to run::
355 def setUpClass(self):
356 super(IP4FwdTestCase, self).setUpClass()
357 self.create_pg_interfaces(range(2)) # create pg0 and pg1
358 for i in self.pg_interfaces:
359 i.admin_up() # put the interface up
360 i.config_ip4() # configure IPv4 address on the interface
361 i.resolve_arp() # resolve ARP, so that we know VPP MAC
363 4. Create a helper method to create the packets to send::
365 def create_stream(self, src_if, dst_if, count):
367 for i in range(count):
368 # create packet info stored in the test case instance
369 info = self.create_packet_info(src_if, dst_if)
370 # convert the info into packet payload
371 payload = self.info_to_payload(info)
372 # create the packet itself
373 p = (Ether(dst=src_if.local_mac, src=src_if.remote_mac) /
374 IP(src=src_if.remote_ip4, dst=dst_if.remote_ip4) /
375 UDP(sport=randint(1000, 2000), dport=5678) /
377 # store a copy of the packet in the packet info
379 # append the packet to the list
382 # return the created packet list
385 5. Create a helper method to verify the capture::
387 def verify_capture(self, src_if, dst_if, capture):
389 for packet in capture:
393 # convert the payload to packet info object
394 payload_info = self.payload_to_info(packet[Raw])
395 # make sure the indexes match
396 self.assert_equal(payload_info.src, src_if.sw_if_index,
397 "source sw_if_index")
398 self.assert_equal(payload_info.dst, dst_if.sw_if_index,
399 "destination sw_if_index")
400 packet_info = self.get_next_packet_info_for_interface2(
404 # make sure we didn't run out of saved packets
405 self.assertIsNotNone(packet_info)
406 self.assert_equal(payload_info.index, packet_info.index,
408 saved_packet = packet_info.data # fetch the saved packet
409 # assert the values match
410 self.assert_equal(ip.src, saved_packet[IP].src,
412 # ... more assertions here
413 self.assert_equal(udp.sport, saved_packet[UDP].sport,
416 self.logger.error(ppp("Unexpected or invalid packet:",
419 remaining_packet = self.get_next_packet_info_for_interface2(
423 self.assertIsNone(remaining_packet,
424 "Interface %s: Packet expected from interface "
425 "%s didn't arrive" % (dst_if.name, src_if.name))
427 6. Add the test code to test_basic function::
429 def test_basic(self):
431 # create the packet stream
432 packets = self.create_stream(self.pg0, self.pg1, count)
433 # add the stream to the source interface
434 self.pg0.add_stream(packets)
435 # enable capture on both interfaces
436 self.pg0.enable_capture()
437 self.pg1.enable_capture()
438 # start the packet generator
440 # get capture - the proper count of packets was saved by
441 # create_packet_info() based on dst_if parameter
442 capture = self.pg1.get_capture()
443 # assert nothing captured on pg0 (always do this last, so that
444 # some time has already passed since pg_start())
445 self.pg0.assert_nothing_captured()
447 self.verify_capture(self.pg0, self.pg1, capture)
449 7. Run the test by issuing 'make test' or, to run only this specific
450 test, issue 'make test TEST=test_ip4_fwd'.