--- /dev/null
+Container-based network simulation
+==================================
+
+The "make test" framework provides a good way to test individual
+features. However, when testing several features at once - or
+validating nontrivial configurations - it may prove difficult or
+impossible to use the unit-test framework.
+
+This note explains how to set up lxc/lxd, and a 5-container testbed to
+test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation
+scenario.
+
+OS / Distro test results
+========================
+
+This setup has been tested on an Ubuntu 18.04 LTS system. If you're
+feeling adventurous, the same scenario also worked on a recent Ubuntu
+20.04 "preview" daily build.
+
+Other distros may work fine, or not at all.
+
+Proxy Server
+============
+
+If you need to use a proxy server e.g. from a lab system, you'll
+probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and
+https_proxy in /etc/environment. Directly setting variables in the
+environment doesn't work. The lxd snap _daemon_ needs the proxy settings,
+not the user interface.
+
+Something like so:
+
+```
+ HTTP_PROXY=http://my.proxy.server:8080
+ HTTPS_PROXY=http://my.proxy.server:4333
+ http_proxy=http://my.proxy.server:8080
+ https_proxy=http://my.proxy.server:4333
+```
+
+Install and configure lxd
+=========================
+
+Install the lxd snap. The lxd snap is up to date, as opposed to the
+results of "sudo apt-get install lxd".
+
+```
+ # snap install lxd
+ # lxd init
+```
+
+"lxd init" asks several questions. With the exception of the storage
+pool, take the defaults. To match the configs shown below, create a
+storage pool named "vpp." Storage pools of type "zfs" and "files" have
+been tested successfully.
+
+zfs is more space-efficient. "lxc copy" is infinitely faster with
+zfs. The path for the zfs storage pool is under /var. Do not replace
+it with a symbolic link, unless you want to rebuild all of your
+containers from scratch. Ask me how I know that.
+
+Create three network segments
+=============================
+
+Aka, linux bridges.
+
+```
+ # lxc network create dora
+ # lxc network create internet
+ # lxc network create swan
+```
+
+We'll explain the test topology in a bit. Stay tuned.
+
+Set up the default container profile
+====================================
+
+Execute "lxc profile edit default", and install the following
+configuration. Note that the "shared" directory should mount your vpp
+workspaces. With that trick, you can edit code from any of the
+containers, run vpp without installing it, etc.
+
+```
+ config: {}
+ description: Default LXD profile
+ devices:
+ eth0:
+ name: eth0
+ network: lxdbr0
+ type: nic
+ eth1:
+ name: eth1
+ nictype: bridged
+ parent: internet
+ type: nic
+ eth2:
+ name: eth2
+ nictype: bridged
+ parent: dora
+ type: nic
+ eth3:
+ name: eth3
+ nictype: bridged
+ parent: swan
+ type: nic
+ root:
+ path: /
+ pool: vpp
+ type: disk
+ shared:
+ path: /scratch
+ source: /scratch
+ type: disk
+ name: default
+```
+
+Set up the network configurations
+=================================
+
+Edit the fake "internet" backbone:
+
+```
+ # lxc network edit internet
+```
+
+Install the ip addresses shown below, to avoid having to rebuild the vpp
+and host configuration:
+
+```
+ config:
+ ipv4.address: 10.26.68.1/24
+ ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: internet
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+```
+
+Repeat the process with the "dora" and "swan" networks, using these
+configurations:
+
+### dora network configuration
+
+```
+ config:
+ ipv4.address: 10.166.14.1/24
+ ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: dora
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+```
+### swan network configuration
+
+```
+ config:
+ ipv4.address: 10.219.188.1/24
+ ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: swan
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+```
+
+Create a "master" container image
+=================================
+
+The master container image should be set up so that you can
+build vpp, ssh into the container, edit source code, run gdb, etc.
+
+Make sure that e.g. public key auth ssh works.
+
+```
+ # lxd launch ubuntu:18.04 dora
+ <spew>
+ # lxc exec dora bash
+ dora# cd /scratch/my-vpp-workspace
+ dora# apt-get install make ssh
+ dora# make install-dep
+ dora# exit
+ # lxc stop dora
+```
+
+Mark the container image privileged. If you forget this step, you'll
+trip over a netlink error (-11) aka EAGAIN when you try to roll in the
+vpp configurations.
+
+```
+ # lxc config set dora security.privileged "true"
+```
+
+Duplicate the "master" container image
+======================================
+
+To avoid having to configure N containers, be sure that the master
+container image is fully set up before you help it have children:
+
+```
+ # lxc copy dora dorahost
+ # lxc copy dora swan
+ # lxc copy dora swanhost
+ # lxc copy dora dhcpserver # optional, to test ipv6 prefix delegation
+```
+
+Install handy script
+====================
+
+See below for a handly script which executes lxc commands across the
+current set of running containers. I call it "lxc-foreach," feel free
+to call the script Ishmael if you like.
+
+Examples:
+
+```
+ $ lxc-foreach start
+ <issues "lxc start" for each container in the list>
+```
+
+After a few seconds, use this one to open an ssh connection to each
+container. The ssh command parses the output of "lxc info," which
+displays container ip addresses.
+
+```
+ $ lxc-foreach ssh
+```
+
+Here's the script:
+
+```
+ #!/bin/bash
+
+ set -u
+ export containers="dora dorahost swan swanhost dhcpserver"
+
+ if [ x$1 = "x" ] ; then
+ echo missing command
+ exit 1
+ fi
+
+ if [ $1 = "ssh" ] ; then
+ for c in $containers
+ do
+ inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3`
+ if [ x$inet = "x" ] ; then
+ echo $c not started
+ else
+ gnome-terminal --command "/usr/bin/ssh $inet"
+ fi
+ done
+ exit 0
+ fi
+
+ for c in $containers
+ do
+ echo lxc $1 $c
+ lxc $1 $c
+ done
+
+ exit 0
+```
+
+Test topology
+=============
+
+Finally, we're ready to describe a test topology. First, a picture:
+
+```
+ ===+======== management lan/bridge lxdbr0 (dhcp) ===========+===
+ | | |
+ | | |
+ | | |
+ v | v
+ eth0 | eth0
+ +------+ eth1 eth1 +------+
+ | dora | 10.26.88.100 <= internet bridge => 10.26.88.101 | swan |
+ +------+ +------+
+ eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0
+ | | |
+ | ("dora" bridge) | ("swan" bridge) |
+ | | |
+ v | v
+ eth2 10.166.14.3 | eth3 10.219.188.3
+ +----------+ | +----------+
+ | dorahost | | | dorahost |
+ +----------+ | +----------+
+ eth0 (management lan) <========+========> eth0 (management lan)
+```
+
+### Test topology discussion
+
+This topology is suitable for testing almost any tunnel encap/decap
+scenario. The two containers "dorahost" and "swanhost" are end-stations
+connected to two vpp instances running on "dora" and "swan".
+
+We leverage the Linux end-station network stacks to generate traffic
+of all sorts.
+
+The so-called "internet" bridge models the public internet. The "dora" and
+"swan" bridges connect vpp instances to local hosts
+
+End station configs
+===================
+
+The end-station Linux configurations set up the eth2 and eth3 ip
+addresses shown above, and add tunnel routes to the opposite
+end-station networks.
+
+### dorahost configuration
+
+```
+ ifconfig eth2 10.166.14.3/24 up
+ route add -net 10.219.188.0/24 gw 10.166.14.2
+```
+
+### swanhost configuration
+
+```
+ sudo ifconfig eth3 10.219.188.3/24 up
+ sudo route add -net 10.166.14.0/24 gw 10.219.188.2
+```
+
+VPP configs
+===========
+
+Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
+the "dora" config.
+
+### dora configuration
+
+```
+ set term pag off
+
+ comment { "internet" }
+ create host-interface name eth1
+ set int ip address host-eth1 10.26.68.100/24
+ set int ip6 table host-eth1 0
+ set int state host-eth1 up
+
+ comment { default route via swan }
+ ip route add 0.0.0.0/0 via 10.26.68.101
+
+ comment { "dora-private-net" }
+ create host-interface name eth2
+ bvi create instance 0
+ set int l2 bridge bvi0 1 bvi
+ set int ip address bvi0 10.166.14.2/24
+ set int state bvi0 up
+ set int l2 bridge host-eth2 1
+ set int state host-eth2 up
+
+
+ nat44 add interface address host-eth1
+ set interface nat44 in host-eth2 out host-eth1
+ nat44 add identity mapping external host-eth1 udp 500
+ nat44 add identity mapping external host-eth1 udp 4500
+ comment { nat44 untranslated subnet 10.219.188.0/24 }
+
+ comment { responder profile }
+ ikev2 profile add swan
+ ikev2 profile set swan udp-encap
+ ikev2 profile set swan auth rsa-sig cert-file /scratch/setups/doracert.pem
+ set ikev2 local key /scratch/setups/swankey.pem
+ ikev2 profile set swan id local fqdn swan.barachs.net
+ ikev2 profile set swan id remote fqdn broiler2.barachs.net
+ ikev2 profile set swan traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+ ikev2 profile set swan traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+ create ipip tunnel src 10.26.68.100 dst 10.26.68.101
+ ikev2 profile set swan tunnel ipip0
+
+ comment { ipv6 prefix delegation }
+ ip6 nd address autoconfig host-eth1 default-route
+ dhcp6 client host-eth1
+ dhcp6 pd client host-eth1 prefix group hgw
+ set ip6 address bvi0 prefix group hgw ::2/56
+ ip6 nd address autoconfig bvi0 default-route
+ ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180
+
+ set int mtu packet 1390 ipip0
+ set int unnum ipip0 use host-eth1
+ ip route add 10.219.188.0/24 via ipip0
+```
+
+### swan configuration
+
+```
+ set term pag off
+
+ comment { "internet" }
+ create host-interface name eth1
+ comment { set dhcp client intfc host-eth1 hostname swan }
+ set int ip address host-eth1 10.26.68.101/24
+ set int state host-eth1 up
+
+ comment { default route via "internet gateway" }
+ comment { ip route add 0.0.0.0/0 via 10.26.68.1 }
+
+ comment { "swan-private-net" }
+ create host-interface name eth3
+ bvi create instance 0
+ set int l2 bridge bvi0 1 bvi
+ set int ip address bvi0 10.219.188.2/24
+ set int state bvi0 up
+ set int l2 bridge host-eth3 1
+ set int state host-eth3 up
+
+ nat44 add interface address host-eth1
+ set interface nat44 in bvi0 out host-eth1
+ nat44 add identity mapping external host-eth1 udp 500
+ nat44 add identity mapping external host-eth1 udp 4500
+ comment { nat44 untranslated subnet 10.166.14.0/24 }
+
+ comment { initiator profile }
+ ikev2 profile add dora
+ ikev2 profile set dora udp-encap
+ ikev2 profile set dora auth rsa-sig cert-file /scratch/setups/swancert.pem
+ set ikev2 local key /scratch/setups/dorakey.pem
+ ikev2 profile set dora id local fqdn broiler2.barachs.net
+ ikev2 profile set dora id remote fqdn swan.barachs.net
+
+ ikev2 profile set dora traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+ ikev2 profile set dora traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+
+ ikev2 profile set dora responder host-eth1 10.26.68.100
+ ikev2 profile set dora ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048
+ ikev2 profile set dora esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256
+ ikev2 profile set dora sa-lifetime 3600 10 5 0
+
+ create ipip tunnel src 10.26.68.101 dst 10.26.68.100
+ ikev2 profile set dora tunnel ipip0
+ ikev2 initiate sa-init dora
+
+ set int mtu packet 1390 ipip0
+ set int unnum ipip0 use host-eth1
+ ip route add 10.166.14.0/24 via ipip0
+```
+
+IKEv2 certificate setup
+=======================
+
+In both of the vpp configurations, you'll see "/scratch/setups/xxx.pem"
+mentioned. These certificates are used in the ikev2 key exchange.
+
+Here's how to generate the certificates:
+
+```
+ openssl req -x509 -nodes -newkey rsa:4096 -keyout dorakey.pem -out doracert.pem -days 3560
+ openssl x509 -text -noout -in doracert.pem
+ openssl req -x509 -nodes -newkey rsa:4096 -keyout swankey.pem -out swancert.pem -days 3560
+ openssl x509 -text -noout -in swancert.pem
+```
+
+Make sure that the "dora" and "swan" configurations point to the certificates.
+
+DHCPv6 server setup
+===================
+
+If you need an ipv6 dhcp server to test ipv6 prefix delegation,
+create the "dhcpserver" container as shown above.
+
+Install the "isc-dhcp-server" Debian package:
+
+```
+ sudo apt-get install isc-dhcp-server
+```
+
+### /etc/dhcp/dhcpd6.conf
+
+Edit the dhcpv6 configuration and add an ipv6 subnet with prefix
+delegation. For example:
+
+```
+ subnet6 2001:db01:0:1::/64 {
+ range6 2001:db01:0:1::1 2001:db01:0:1::9;
+ prefix6 2001:db01:0:100:: 2001:db01:0:200::/56;
+ }
+```
+
+Add an ipv6 address on eth1, which is connected to the "internet"
+bridge, and start the dhcp server. I use the following trivial bash
+script, which runs the dhcp6 server in the foreground and produces
+dhcp traffic spew:
+
+```
+ #!/bin/bash
+ ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true
+ dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf
+```
+
+The "|| true" bit keeps going if eth1 already has the indicated ipv6
+address.
Code Review / vpp.git / commitdiff