1 .. _unittest: https://docs.python.org/2/library/unittest.html
2 .. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase
3 .. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError
4 .. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest
5 .. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/
6 .. _scapy: http://www.secdev.org/projects/scapy/
7 .. _logging: https://docs.python.org/2/library/logging.html
9 .. |vtf| replace:: VPP Test Framework
17 The goal of the |vtf| is to ease writing, running and debugging
18 unit tests for the VPP. For this, python was chosen as a high level language
19 allowing rapid development with scapy_ providing the necessary tool for creating
20 and dissecting packets.
22 Anatomy of a test case
23 ######################
25 Python's unittest_ is used as the base framework upon which the VPP test
26 framework is built. A test suite in the |vtf| consists of multiple classes
27 derived from `VppTestCase`, which is itself derived from TestCase_.
28 The test class defines one or more test functions, which act as test cases.
30 Function flow when running a test case is:
32 1. `setUpClass <VppTestCase.setUpClass>`:
33 This function is called once for each test class, allowing a one-time test
34 setup to be executed. If this functions throws an exception,
35 none of the test functions are executed.
36 2. `setUp <VppTestCase.setUp>`:
37 The setUp function runs before each of the test functions. If this function
38 throws an exception other than AssertionError_ or SkipTest_, then this is
39 considered an error, not a test failure.
41 This is the guts of the test case. It should execute the test scenario
42 and use the various assert functions from the unittest framework to check
43 necessary. Multiple test_<name> methods can exist in a test case.
44 4. `tearDown <VppTestCase.tearDown>`:
45 The tearDown function is called after each test function with the purpose
46 of doing partial cleanup.
47 5. `tearDownClass <VppTestCase.tearDownClass>`:
48 Method called once after running all of the test functions to perform
54 Each test case has a logger automatically created for it, stored in
55 'logger' property, based on logging_. Use the logger's standard methods
56 debug(), info(), error(), ... to emit log messages to the logger.
58 All the log messages go always into a log file in temporary directory
61 To control the messages printed to console, specify the V= parameter.
65 make test # minimum verbosity
66 make test V=1 # moderate verbosity
67 make test V=2 # maximum verbosity
69 Test temporary directory and VPP life cycle
70 ###########################################
72 Test separation is achieved by separating the test files and vpp instances.
73 Each test creates a temporary directory and it's name is used to create
74 a shared memory prefix which is used to run a VPP instance.
75 The temporary directory name contains the testcase class name for easy
76 reference, so for testcase named 'TestVxlan' the directory could be named
77 e.g. vpp-unittest-TestVxlan-UNUP3j.
78 This way, there is no conflict between any other VPP instances running
79 on the box and the test VPP. Any temporary files created by the test case
80 are stored in this temporary test directory.
82 The test temporary directory holds the following interesting files:
84 * log.txt - this contains the logger output on max verbosity
85 * pg*_in.pcap - last injected packet stream into VPP, named after the interface,
86 so for pg0, the file will be named pg0_in.pcap
87 * pg*_out.pcap - last capture file created by VPP for interface, similarly,
88 named after the interface, so for e.g. pg1, the file will be named
90 * history files - whenever the capture is restarted or a new stream is added,
91 the existing files are rotated and renamed, soo all the pcap files
92 are always saved for later debugging if needed
93 * core - if vpp dumps a core, it'll be stored in the temporary directory
94 * vpp_stdout.txt - file containing output which vpp printed to stdout
95 * vpp_stderr.txt - file containing output which vpp printed to stderr
97 *NOTE*: existing temporary directories named vpp-unittest-* are automatically
98 removed when invoking 'make test*' or 'make retest*' to keep the temporary
104 Virtualenv_ is a python module which provides a means to create an environment
105 containing the dependencies required by the |vtf|, allowing a separation
106 from any existing system-wide packages. |vtf|'s Makefile automatically
107 creates a virtualenv_ inside build-root and installs the required packages
108 in that environment. The environment is entered whenever executing a test
109 via one of the make test targets.
114 Most unit tests do some kind of packet manipulation - sending and receiving
115 packets between VPP and virtual hosts connected to the VPP. Referring
116 to the sides, addresses, etc. is always done as if looking from the VPP side,
119 * *local_* prefix is used for the VPP side.
120 So e.g. `local_ip4 <VppInterface.local_ip4>` address is the IPv4 address
121 assigned to the VPP interface.
122 * *remote_* prefix is used for the virtual host side.
123 So e.g. `remote_mac <VppInterface.remote_mac>` address is the MAC address
124 assigned to the virtual host connected to the VPP.
126 Automatically generated addresses
127 #################################
129 To send packets, one needs to typically provide some addresses, otherwise
130 the packets will be dropped. The interface objects in |vtf| automatically
131 provide addresses based on (typically) their indexes, which ensures
132 there are no conflicts and eases debugging by making the addressing scheme
135 The developer of a test case typically doesn't need to work with the actual
136 numbers, rather using the properties of the objects. The addresses typically
137 come in two flavors: '<address>' and '<address>n' - note the 'n' suffix.
138 The former address is a Python string, while the latter is translated using
139 socket.inet_pton to raw format in network byte order - this format is suitable
140 for passing as an argument to VPP APIs.
142 e.g. for the IPv4 address assigned to the VPP interface:
144 * local_ip4 - Local IPv4 address on VPP interface (string)
145 * local_ip4n - Local IPv4 address - raw, suitable as API parameter.
147 These addresses need to be configured in VPP to be usable using e.g.
148 `config_ip4` API. Please see the documentation to `VppInterface` for more
151 By default, there is one remote address of each kind created for L3:
152 remote_ip4 and remote_ip6. If the test needs more addresses, because it's
153 simulating more remote hosts, they can be generated using
154 `generate_remote_hosts` API and the entries for them inserted into the ARP
155 table using `configure_ipv4_neighbors` API.
157 Packet flow in the |vtf|
158 ########################
160 Test framework -> VPP
161 ~~~~~~~~~~~~~~~~~~~~~
163 |vtf| doesn't send any packets to VPP directly. Traffic is instead injected
164 using packet-generator interfaces, represented by the `VppPGInterface` class.
165 Packets are written into a temporary .pcap file, which is then read by the VPP
166 and the packets are injected into the VPP world.
168 To add a list of packets to an interface, call the `add_stream` method on that
169 interface. Once everything is prepared, call `pg_start` method to start
170 the packet generator on the VPP side.
172 VPP -> test framework
173 ~~~~~~~~~~~~~~~~~~~~~
175 Similarly, VPP doesn't send any packets to |vtf| directly. Instead, packet
176 capture feature is used to capture and write traffic to a temporary .pcap file,
177 which is then read and analyzed by the |vtf|.
179 The following APIs are available to the test case for reading pcap files.
181 * `get_capture`: this API is suitable for bulk & batch style of test, where
182 a list of packets is prepared & sent, then the received packets are read
183 and verified. The API needs the number of packets which are expected to
184 be captured (ignoring filtered packets - see below) to know when the pcap
185 file is completely written by the VPP. If using packet infos for verifying
186 packets, then the counts of the packet infos can be automatically used
187 by `get_capture` to get the proper count (in this case the default value
188 None can be supplied as expected_count or ommitted altogether).
189 * `wait_for_packet`: this API is suitable for interactive style of test,
190 e.g. when doing session management, three-way handsakes, etc. This API waits
191 for and returns a single packet, keeping the capture file in place
192 and remembering context. Repeated invocations return following packets
193 (or raise Exception if timeout is reached) from the same capture file
194 (= packets arriving on the same interface).
196 *NOTE*: it is not recommended to mix these APIs unless you understand how they
197 work internally. None of these APIs rotate the pcap capture file, so calling
198 e.g. `get_capture` after `wait_for_packet` will return already read packets.
199 It is safe to switch from one API to another after calling `enable_capture`
200 as that API rotates the capture file.
202 Automatic filtering of packets:
203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
205 Both APIs (`get_capture` and `wait_for_packet`) by default filter the packet
206 capture, removing known uninteresting packets from it - these are IPv6 Router
207 Advertisments and IPv6 Router Alerts. These packets are unsolicitated
208 and from the point of |vtf| are random. If a test wants to receive these
209 packets, it should specify either None or a custom filtering function
210 as the value to the 'filter_out_fn' argument.
212 Common API flow for sending/receiving packets:
213 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
215 We will describe a simple scenario, where packets are sent from pg0 to pg1
216 interface, assuming that the interfaces were created using
217 `create_pg_interfaces` API.
219 1. Create a list of packets for pg0::
222 packets = create_packets(src=self.pg0, dst=self.pg1,
225 2. Add that list of packets to the source interface::
227 self.pg0.add_stream(packets)
229 3. Enable capture on the destination interface::
231 self.pg1.enable_capture()
233 4. Start the packet generator::
237 5. Wait for capture file to appear and read it::
239 capture = self.pg1.get_capture(expected_count=packet_count)
241 6. Verify packets match sent packets::
243 self.verify_capture(send=packets, captured=capture)
245 Test framework objects
246 ######################
248 The following objects provide VPP abstraction and provide a means to do
249 common tasks easily in the test cases.
251 * `VppInterface`: abstract class representing generic VPP interface
252 and contains some common functionality, which is then used by derived classes
253 * `VppPGInterface`: class representing VPP packet-generator interface.
254 The interface is created/destroyed when the object is created/destroyed.
255 * `VppSubInterface`: VPP sub-interface abstract class, containing common
256 functionality for e.g. `VppDot1QSubint` and `VppDot1ADSubint` classes
258 How VPP APIs/CLIs are called
259 ############################
261 Vpp provides python bindings in a python module called vpp-papi, which the test
262 framework installs in the virtual environment. A shim layer represented by
263 the `VppPapiProvider` class is built on top of the vpp-papi, serving these
266 1. Automatic return value checks:
267 After each API is called, the return value is checked against the expected
268 return value (by default 0, but can be overridden) and an exception
269 is raised if the check fails.
270 2. Automatic call of hooks:
272 a. `before_cli <Hook.before_cli>` and `before_api <Hook.before_api>` hooks
273 are used for debug logging and stepping through the test
274 b. `after_cli <Hook.after_cli>` and `after_api <Hook.after_api>` hooks
275 are used for monitoring the vpp process for crashes
276 3. Simplification of API calls:
277 Many of the VPP APIs take a lot of parameters and by providing sane defaults
278 for these, the API is much easier to use in the common case and the code is
279 more readable. E.g. ip_add_del_route API takes ~25 parameters, of which
280 in the common case, only 3 are needed.
285 Some interesting utility methods are:
287 * `ppp`: 'Pretty Print Packet' - returns a string containing the same output
288 as Scapy's packet.show() would print
289 * `ppc`: 'Pretty Print Capture' - returns a string containing printout of
290 a capture (with configurable limit on the number of packets printed from it)
293 *NOTE*: Do not use Scapy's packet.show() in the tests, because it prints
294 the output to stdout. All output should go to the logger associated with
297 Example: how to add a new test
298 ##############################
300 In this example, we will describe how to add a new test case which tests
301 basic IPv4 forwarding.
303 1. Add a new file called test_ip4_fwd.py in the test directory, starting
306 from framework import VppTestCase
307 from scapy.layers.l2 import Ether
308 from scapy.packet import Raw
309 from scapy.layers.inet import IP, UDP
310 from random import randint
312 2. Create a class inherited from the VppTestCase::
314 class IP4FwdTestCase(VppTestCase):
315 """ IPv4 simple forwarding test case """
317 2. Add a setUpClass function containing the setup needed for our test to run::
320 def setUpClass(self):
321 super(IP4FwdTestCase, self).setUpClass()
322 self.create_pg_interfaces(range(2)) # create pg0 and pg1
323 for i in self.pg_interfaces:
324 i.admin_up() # put the interface up
325 i.config_ip4() # configure IPv4 address on the interface
326 i.resolve_arp() # resolve ARP, so that we know VPP MAC
328 3. Create a helper method to create the packets to send::
330 def create_stream(self, src_if, dst_if, count):
332 for i in range(count):
333 # create packet info stored in the test case instance
334 info = self.create_packet_info(src_if, dst_if)
335 # convert the info into packet payload
336 payload = self.info_to_payload(info)
337 # create the packet itself
338 p = (Ether(dst=src_if.local_mac, src=src_if.remote_mac) /
339 IP(src=src_if.remote_ip4, dst=dst_if.remote_ip4) /
340 UDP(sport=randint(1000, 2000), dport=5678) /
342 # store a copy of the packet in the packet info
344 # append the packet to the list
347 # return the created packet list
350 4. Create a helper method to verify the capture::
352 def verify_capture(self, src_if, dst_if, capture):
354 for packet in capture:
358 # convert the payload to packet info object
359 payload_info = self.payload_to_info(str(packet[Raw]))
360 # make sure the indexes match
361 self.assert_equal(payload_info.src, src_if.sw_if_index,
362 "source sw_if_index")
363 self.assert_equal(payload_info.dst, dst_if.sw_if_index,
364 "destination sw_if_index")
365 packet_info = self.get_next_packet_info_for_interface2(
369 # make sure we didn't run out of saved packets
370 self.assertIsNotNone(packet_info)
371 self.assert_equal(payload_info.index, packet_info.index,
373 saved_packet = packet_info.data # fetch the saved packet
374 # assert the values match
375 self.assert_equal(ip.src, saved_packet[IP].src,
377 # ... more assertions here
378 self.assert_equal(udp.sport, saved_packet[UDP].sport,
381 self.logger.error(ppp("Unexpected or invalid packet:",
384 remaining_packet = self.get_next_packet_info_for_interface2(
388 self.assertIsNone(remaining_packet,
389 "Interface %s: Packet expected from interface "
390 "%s didn't arrive" % (dst_if.name, src_if.name))
392 5. Add the test code to test_basic function::
394 def test_basic(self):
396 # create the packet stream
397 packets = self.create_stream(self.pg0, self.pg1, count)
398 # add the stream to the source interface
399 self.pg0.add_stream(packets)
400 # enable capture on both interfaces
401 self.pg0.enable_capture()
402 self.pg1.enable_capture()
403 # start the packet generator
405 # get capture - the proper count of packets was saved by
406 # create_packet_info() based on dst_if parameter
407 capture = self.pg1.get_capture()
408 # assert nothing captured on pg0 (always do this last, so that
409 # some time has already passed since pg_start())
410 self.pg0.assert_nothing_captured()
412 self.verify_capture(self.pg0, self.pg1, capture)
414 6. Run the test by issuing 'make test'.
417 |vtf| module documentation
418 ##########################