1 Container-based network simulation
2 ==================================
4 The "make test" framework provides a good way to test individual
5 features. However, when testing several features at once - or
6 validating nontrivial configurations - it may prove difficult or
7 impossible to use the unit-test framework.
9 This note explains how to set up lxc/lxd, and a 5-container testbed to
10 test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation
13 OS / Distro test results
14 ------------------------
16 This setup has been tested on an Ubuntu 18.04 LTS system. If you're
17 feeling adventurous, the same scenario also worked on a recent Ubuntu
18 20.04 "preview" daily build.
20 Other distros may work fine, or not at all.
25 If you need to use a proxy server e.g. from a lab system, you'll
26 probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and
27 https_proxy in /etc/environment. Directly setting variables in the
28 environment doesn't work. The lxd snap _daemon_ needs the proxy settings,
29 not the user interface.
34 HTTP_PROXY=http://my.proxy.server:8080
35 HTTPS_PROXY=http://my.proxy.server:4333
36 http_proxy=http://my.proxy.server:8080
37 https_proxy=http://my.proxy.server:4333
40 Install and configure lxd
41 -------------------------
43 Install the lxd snap. The lxd snap is up to date, as opposed to the
44 results of "sudo apt-get install lxd".
51 "lxd init" asks several questions. With the exception of the storage
52 pool, take the defaults. To match the configs shown below, create a
53 storage pool named "vpp." Storage pools of type "zfs" and "files" have
54 been tested successfully.
56 zfs is more space-efficient. "lxc copy" is infinitely faster with
57 zfs. The path for the zfs storage pool is under /var. Do not replace
58 it with a symbolic link, unless you want to rebuild all of your
59 containers from scratch. Ask me how I know that.
61 Create three network segments
62 -----------------------------
67 # lxc network create dora
68 # lxc network create internet
69 # lxc network create swan
72 We'll explain the test topology in a bit. Stay tuned.
74 Set up the default container profile
75 ------------------------------------
77 Execute "lxc profile edit default", and install the following
78 configuration. Note that the "shared" directory should mount your vpp
79 workspaces. With that trick, you can edit code from any of the
80 containers, run vpp without installing it, etc.
84 description: Default LXD profile
116 Set up the network configurations
117 ---------------------------------
119 Edit the fake "internet" backbone:
122 # lxc network edit internet
125 Install the ip addresses shown below, to avoid having to rebuild the vpp
126 and host configuration:
130 ipv4.address: 10.26.68.1/24
131 ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50
145 Repeat the process with the "dora" and "swan" networks, using these
148 ### dora network configuration
152 ipv4.address: 10.166.14.1/24
153 ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50
166 ### swan network configuration
170 ipv4.address: 10.219.188.1/24
171 ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50
185 Create a "master" container image
186 ---------------------------------
188 The master container image should be set up so that you can
189 build vpp, ssh into the container, edit source code, run gdb, etc.
191 Make sure that e.g. public key auth ssh works.
194 # lxd launch ubuntu:18.04 dora
197 dora# cd /scratch/my-vpp-workspace
198 dora# apt-get install make ssh
199 dora# make install-dep
204 Mark the container image privileged. If you forget this step, you'll
205 trip over a netlink error (-11) aka EAGAIN when you try to roll in the
209 # lxc config set dora security.privileged "true"
212 Duplicate the "master" container image
213 --------------------------------------
215 To avoid having to configure N containers, be sure that the master
216 container image is fully set up before you help it have children:
219 # lxc copy dora dorahost
221 # lxc copy dora swanhost
222 # lxc copy dora dhcpserver # optional, to test ipv6 prefix delegation
228 See below for a handly script which executes lxc commands across the
229 current set of running containers. I call it "lxc-foreach," feel free
230 to call the script Ishmael if you like.
236 <issues "lxc start" for each container in the list>
239 After a few seconds, use this one to open an ssh connection to each
240 container. The ssh command parses the output of "lxc info," which
241 displays container ip addresses.
253 export containers="dora dorahost swan swanhost dhcpserver"
255 if [ x$1 = "x" ] ; then
260 if [ $1 = "ssh" ] ; then
263 inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3`
264 if [ x$inet = "x" ] ; then
267 gnome-terminal --command "/usr/bin/ssh $inet"
285 Finally, we're ready to describe a test topology. First, a picture:
288 ===+======== management lan/bridge lxdbr0 (dhcp) ===========+===
294 +------+ eth1 eth1 +------+
295 | dora | 10.26.88.100 <= internet bridge => 10.26.88.101 | swan |
297 eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0
299 | ("dora" bridge) | ("swan" bridge) |
302 eth2 10.166.14.3 | eth3 10.219.188.3
303 +----------+ | +----------+
304 | dorahost | | | dorahost |
305 +----------+ | +----------+
306 eth0 (management lan) <========+========> eth0 (management lan)
309 ### Test topology discussion
311 This topology is suitable for testing almost any tunnel encap/decap
312 scenario. The two containers "dorahost" and "swanhost" are end-stations
313 connected to two vpp instances running on "dora" and "swan".
315 We leverage the Linux end-station network stacks to generate traffic
318 The so-called "internet" bridge models the public internet. The "dora" and
319 "swan" bridges connect vpp instances to local hosts
324 The end-station Linux configurations set up the eth2 and eth3 ip
325 addresses shown above, and add tunnel routes to the opposite
326 end-station networks.
328 ### dorahost configuration
331 ifconfig eth2 10.166.14.3/24 up
332 route add -net 10.219.188.0/24 gw 10.166.14.2
335 ### swanhost configuration
338 sudo ifconfig eth3 10.219.188.3/24 up
339 sudo route add -net 10.166.14.0/24 gw 10.219.188.2
345 Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
348 ### dora configuration
353 comment { "internet" }
354 create host-interface name eth1
355 set int ip address host-eth1 10.26.68.100/24
356 set int ip6 table host-eth1 0
357 set int state host-eth1 up
359 comment { default route via swan }
360 ip route add 0.0.0.0/0 via 10.26.68.101
362 comment { "dora-private-net" }
363 create host-interface name eth2
364 bvi create instance 0
365 set int l2 bridge bvi0 1 bvi
366 set int ip address bvi0 10.166.14.2/24
367 set int state bvi0 up
368 set int l2 bridge host-eth2 1
369 set int state host-eth2 up
372 nat44 add interface address host-eth1
373 set interface nat44 in host-eth2 out host-eth1
374 nat44 add identity mapping external host-eth1 udp 500
375 nat44 add identity mapping external host-eth1 udp 4500
376 comment { nat44 untranslated subnet 10.219.188.0/24 }
378 comment { responder profile }
379 ikev2 profile add swan
380 ikev2 profile set swan udp-encap
381 ikev2 profile set swan auth rsa-sig cert-file /scratch/setups/doracert.pem
382 set ikev2 local key /scratch/setups/swankey.pem
383 ikev2 profile set swan id local fqdn swan.barachs.net
384 ikev2 profile set swan id remote fqdn broiler2.barachs.net
385 ikev2 profile set swan traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
386 ikev2 profile set swan traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
387 create ipip tunnel src 10.26.68.100 dst 10.26.68.101
388 ikev2 profile set swan tunnel ipip0
390 comment { ipv6 prefix delegation }
391 ip6 nd address autoconfig host-eth1 default-route
392 dhcp6 client host-eth1
393 dhcp6 pd client host-eth1 prefix group hgw
394 set ip6 address bvi0 prefix group hgw ::2/56
395 ip6 nd address autoconfig bvi0 default-route
396 ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180
398 set int mtu packet 1390 ipip0
399 set int unnum ipip0 use host-eth1
400 ip route add 10.219.188.0/24 via ipip0
403 ### swan configuration
408 comment { "internet" }
409 create host-interface name eth1
410 comment { set dhcp client intfc host-eth1 hostname swan }
411 set int ip address host-eth1 10.26.68.101/24
412 set int state host-eth1 up
414 comment { default route via "internet gateway" }
415 comment { ip route add 0.0.0.0/0 via 10.26.68.1 }
417 comment { "swan-private-net" }
418 create host-interface name eth3
419 bvi create instance 0
420 set int l2 bridge bvi0 1 bvi
421 set int ip address bvi0 10.219.188.2/24
422 set int state bvi0 up
423 set int l2 bridge host-eth3 1
424 set int state host-eth3 up
426 nat44 add interface address host-eth1
427 set interface nat44 in bvi0 out host-eth1
428 nat44 add identity mapping external host-eth1 udp 500
429 nat44 add identity mapping external host-eth1 udp 4500
430 comment { nat44 untranslated subnet 10.166.14.0/24 }
432 comment { initiator profile }
433 ikev2 profile add dora
434 ikev2 profile set dora udp-encap
435 ikev2 profile set dora auth rsa-sig cert-file /scratch/setups/swancert.pem
436 set ikev2 local key /scratch/setups/dorakey.pem
437 ikev2 profile set dora id local fqdn broiler2.barachs.net
438 ikev2 profile set dora id remote fqdn swan.barachs.net
440 ikev2 profile set dora traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
441 ikev2 profile set dora traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
443 ikev2 profile set dora responder host-eth1 10.26.68.100
444 ikev2 profile set dora ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048
445 ikev2 profile set dora esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256
446 ikev2 profile set dora sa-lifetime 3600 10 5 0
448 create ipip tunnel src 10.26.68.101 dst 10.26.68.100
449 ikev2 profile set dora tunnel ipip0
450 ikev2 initiate sa-init dora
452 set int mtu packet 1390 ipip0
453 set int unnum ipip0 use host-eth1
454 ip route add 10.166.14.0/24 via ipip0
457 IKEv2 certificate setup
458 -----------------------
460 In both of the vpp configurations, you'll see "/scratch/setups/xxx.pem"
461 mentioned. These certificates are used in the ikev2 key exchange.
463 Here's how to generate the certificates:
466 openssl req -x509 -nodes -newkey rsa:4096 -keyout dorakey.pem -out doracert.pem -days 3560
467 openssl x509 -text -noout -in doracert.pem
468 openssl req -x509 -nodes -newkey rsa:4096 -keyout swankey.pem -out swancert.pem -days 3560
469 openssl x509 -text -noout -in swancert.pem
472 Make sure that the "dora" and "swan" configurations point to the certificates.
477 If you need an ipv6 dhcp server to test ipv6 prefix delegation,
478 create the "dhcpserver" container as shown above.
480 Install the "isc-dhcp-server" Debian package:
483 sudo apt-get install isc-dhcp-server
486 ### /etc/dhcp/dhcpd6.conf
488 Edit the dhcpv6 configuration and add an ipv6 subnet with prefix
489 delegation. For example:
492 subnet6 2001:db01:0:1::/64 {
493 range6 2001:db01:0:1::1 2001:db01:0:1::9;
494 prefix6 2001:db01:0:100:: 2001:db01:0:200::/56;
498 Add an ipv6 address on eth1, which is connected to the "internet"
499 bridge, and start the dhcp server. I use the following trivial bash
500 script, which runs the dhcp6 server in the foreground and produces
505 ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true
506 dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf
509 The "|| true" bit keeps going if eth1 already has the indicated ipv6
512 Container / Host Interoperation
513 -------------------------------
515 Host / container interoperation is highly desirable. If the host and a
516 set of containers don't run the same distro _and distro version_, it's
517 reasonably likely that the glibc versions won't match. That, in turn,
518 makes vpp binaries built in one environment fail in the other.
520 Trying to install multiple versions of glibc - especially at the host
521 level - often ends very badly and is _not recommended_. It's not just
522 glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version
525 Fortunately, it's reasonable easy to build lxd container images based on
526 specific Ubuntu or Debian versions.
528 ### Create a custom root filesystem image
530 First, install the "debootstrap" tool:
533 sudo apt-get install debootstrap
536 Make a temp directory, and use debootstrap to populate it. In this
537 example, we create an Ubuntu 20.04 (focal fossa) base image:
541 # debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu
544 To tinker with the base image (if desired):
552 Make a compressed tarball of the base image:
555 # tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot .
558 Create a "metadata.yaml" file which describes the base image:
561 architecture: "x86_64"
562 # To get current date in Unix time, use `date +%s` command
563 creation_date: 1458040200
565 architecture: "x86_64"
566 description: "My custom Focal Fossa image"
571 Make a compressed tarball of metadata.yaml:
574 # tar zcf metadata.tar.gz metadata.yaml
577 Import the image into lxc / lxd:
580 $ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base
583 ### Create a container which uses the customized base image:
586 $ lxc launch focal-base focaltest
587 $ lxc exec focaltest bash
590 The next several steps should be executed in the container, in the
591 bash shell spun up by "lxc exec..."
593 ### Configure container networking
595 In the container, create /etc/netplan/50-cloud-init.yaml:
605 Use "cat > /etc/netplan/50-cloud-init.yaml", and cut-'n-paste if your
606 favorite text editor is AWOL.
608 Apply the configuration:
614 At this point, eth0 should have an ip address, and you should see
615 a default route with "route -n".
619 Again, in the container, set up /etc/apt/sources.list via cut-'n-paste
620 from a recently update "focal fossa" host. Something like so:
623 deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
624 deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
625 deb http://us.archive.ubuntu.com/ubuntu/ focal universe
626 deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
627 deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
628 deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
629 deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
630 deb http://security.ubuntu.com/ubuntu focal-security main restricted
631 deb http://security.ubuntu.com/ubuntu focal-security universe
632 deb http://security.ubuntu.com/ubuntu focal-security multiverse
635 "apt-get update" and "apt-install" should produce reasonable results.
636 Suggest "apt-get install make git".
638 At this point, you can use the "/scratch" sharepoint (or similar) to
639 execute "make install-dep install-ext-deps" to set up the container
640 with the vpp toolchain; proceed as desired.