Preface

One day I wondered: would it be possible to create a fully virtual IPsec tunnel? Without using any physical network appliances? On a single Linux VM? This led me to go into the rabbit hole of Linux network namespaces and virtual interfaces, the results of which I would like to share in this post. I believe you will be amazed what the networking stack in Linux can do just as I was when I learned about this.

The path I decided to take was to use a lightweight Linux distribution - I chose Alpine Linux - and set up a VM in which the IPsec lab will reside. After which, utilize the well known Libreswan project for the software implementation of the IPsec protocol. Pieces of the project are clear, now the question remains, how to connect them together.

Before we can start to ponder about this, logically the first step will be to setup the Alpine VM, and for that I chose QEMU. Because why not (VirtualBox and VMware are boring).

Keep in mind: this post assumes we are working in a Unix-like environment as the host machine with QEMU installed.

Preparing the environment - Alpine Linux VM in QEMU

Getting the ISO file for Alpine Linux is as simple as going to https://www.alpinelinux.org/downloads/ and selecting the appropriate version.

(ISO file contains Linux disk that bundles the entire operating system, required bootloader and other important utilities into a single, ready-to-install file)

The one that is interesting for us today will be Virtual x86_64. Pressing the green x86_64 button will start the download which shouldn’t take long as being a minimal image it is way smaller (roughly 67 MB for the whole ISO file) compared to other fully featured Linux distros.

Now with the ISO file acquired, we will need to create a virtual hard drive onto which we will be able to install our OS. To do that QEMU includes a useful utility called - qemu-img - which allows to do many kinds of operations on virtual disks, including creating one with the format of our choosing. The most popular one for QEMU virtual disks is QCOW2 thus using it is a good idea.

# Creating a new alpine.qcow2 virtual disk with 20 GB capacity
qemu-img create -f qcow2 alpine.qcow2 20G

If no output was printed to the console this means our virtual hard drive was created successfully and we can confirm that by running ls and noticing a new file has appeared - alpine.qcow2.

To clarify, we used 20G (20 Gigabytes) as our disk size but this doesn’t mean this is how much space will be taken by this file on the physical disk. By default QCOW2 files like the one we created right now is dynamically allocated - meaning it will grow in size as the disk is being used and the 20G we set is the maximum size the disk is allowed to grow so don’t worry about losing this amount of disk space for a half-empty virtual hard drive.

Now it’s time to boot our Virtual Machine using the ISO file we downloaded. For that we will use the qemu-system-x86_64 command which is a 64 bit x86 variant of the QEMU virtualization software (64 bit x86 is the CPU architecture our ISO file is designed for so this is the one we need to use but as a side note QEMU supports many different CPU architectures, I believe the most amount of CPU architectures in the industry). To run our VM I propose we use the following command:

qemu-system-x86_64 -cdrom <PATH-TO-THE-ISO-FILE> -m 512 -nic user -drive file=alpine.qcow2 -enable-kvm

This should open a new QEMU window and start the Alpine boot up sequence. Once you are greeted with a “Welcome to Alpine Linux” and a prompt to enter login, type root and after pressing enter you should now see a prompt waiting for your further commands. Now we can proceed with the installation process using a nifty little utility - setup-alpine.

Upon executing setup-alpine you will notice an interactive menu which will help us setup Alpine easily and quickly. Here is a quick guide regarding the options we will use for the installation:

  • Keymap: us
  • Keymap variant: us
  • Hostname: alpinelinux
  • Interface: we will use the default as detected by Alpine (in my case eth0) - press enter
  • IP Address: default is dhcp and this is what we will use - press enter
  • Manual Network Configuration: No - press enter
  • Root Password: any password you will remember (proceed to retype it again when prompted)
  • Timezone: Select one per your region
  • Proxy: none
  • NTP client: default - busybox
  • APK Mirror: by default it will find and use fastest mirror - press enter
  • User: in this article we will stick to the root user for simplicity - choose no
  • SSH server: openssh
  • Allow root ssh login: prohibit-password
  • Enter ssh key for root: none (at this point want to clarify that with this setup - prohibit password login for SSH root access, no SSH keys setup and no users other than root existing on the system we will not be able to ssh onto this machine but given that we will live fully inside of the QEMU window for the rest of this article this will not be a problem for us)
  • Disk & Install: here we need to choose which disk to use for the installation, in my case it is sda described as 21.5 GB ATA QEMU HARDDISK
  • How would you like to use the disk?: sys (we will use one partition for both system data and user data)
  • Erase the above disk: make sure the proper disk is selected and select y to begin the installation process

The installation will conclude with a message printed on the screen “Installation is complete. Please reboot.”. Congratulations! Alpine Linux is now installed on our virtual hard drive - alpine.qcow2.

The next step will be to boot into our freshly installed OS and prepare it with necessary software which we will need later.

To do that we first gonna power off our machine - execute poweroff command which will do just that. Before starting the machine again we need to change the arguments for qemu-system-x86_64. With the OS installed we don’t need the ISO anymore so we are going to stop QEMU from loading it by omitting the -cdrom <ISO> argument. The modified startup command from now will be:

qemu-system-x86_64 -m 512 -nic user -drive file=alpine.qcow2 -enable-kvm

At this stage I think I owe you a quick explanation of what are we actually doing with this command.

  • Firstly we have the -m 512 argument which tells QEMU to assign 512 MB of RAM for the Virtual Machine (VM)
  • -nic user option enables "User Mode Networking", which acts as a built-in virtual router. It provides the guest OS (Alpine) with immediate outbound internet access and an automatic IP via DHCP without requiring root privileges or complex host configuration
  • -drive file=alpine.qcow2 specifies the disk image file that will act as the VM’s hard drive, where the operating system is being installed
  • -enable-kvm tells QEMU to use the host's KVM (Kernel-based Virtual Machine) features for hardware acceleration, drastically improving performance by running the guest code directly on the host CPU without full emulation

Updated command will again pop up the QEMU window as before but this time it will proceed to boot the just installed Alpine Linux from the virtual hard drive not the ISO image. Prompted for login username enter root and this time a password will be required, the same one you chose during the installation.

Welcome to Alpine!

The last thing we need to do before we can get started is to install the necessary software. In order to do that we first need to switch the repository used as not everything we need can be found in the default Alpine repository.

(A quick side-note - software repository in this context is a URL defining a server which provides a set of software we can download from it and install on our machine).

The repository we need is called edge and the installation process is straight forward (described in the following Alpine Linux documentation page: https://wiki.alpinelinux.org/wiki/Include:Upgrading_to_Edge) and boils down to editing the /etc/apk/repositories file and pointing the system to use the edge repository URLs instead of the default ones listed already in this file. Here is a command I prepared which will replace that file with the content expected when edge repository is to be used:

echo "#/media/cdrom/apks
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community
@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" > /etc/apk/repositories

To confirm if the above command worked, execute apk update and notice if the command output includes edge in the URLs and if the last line printed says OK: X distinct packages available.

Now with the edge repository configured we can finally install the packages we need. One more command and we are done:

apk add iputils iproute2 iperf3 tmux tcpdump libreswan
  • iputils - providing us with the ping utility for probing if the network connection with a chosen endpoint is working correctly
  • iproute2 - collection of utilities for controlling TCP/IP networking with the ip command
  • iperf3 - network bandwidth measurement tools we will use to test our IPsec tunnel
  • tmux - terminal multiplexer, in our environment it will allow us to have two terminal windows opened in one QEMU instance for working with more than one command at once
  • tcpdump - command-line packet analysis toolkit for observing packet traffic on a specified network interface
  • libreswan - last but not least, the most important package, Libreswan (https://libreswan.org/), self described as a free software implementation of the most widely supported and standardized VPN protocol using "IPsec" and the Internet Key Exchange ("IKE"), in short this software provides the IPsec implementation we will use to create our IPsec tunnel

It’s time for the fun part. Actually setting everything up and connecting all the pieces together. First I would like to quickly overview the setup we will create and the steps we will take so you understand what do we want to achieve.

Using magical Linux networking capabilities combined with Libreswan IPsec implementation we will create two network namespaces and connect them using an IPsec tunnel. Libreswan will need to be configured to use the network interfaces, after which we will confirm our IPsec tunnel is running correctly and proceed to do some performance testing and network simulations but let’s not get ahead of ourselves.

Creating and configuring the Linux Network Namespaces

So what is a Linux Network Namespace you may ask?

Linux Network Namespaces can be described in the following way: In a standard Linux installation, the operating system has a single routing table and a single set of network interfaces across the entire system. Network Namespaces allow us to create separate, isolated instances of the entire network stack. Each Network Namespace operates independently and can have its own:

  • Network interfaces
  • IP addresses
  • Routing tables
  • Default gateways

The key concept in Linux Network Namespaces is isolation - one namespace is completely walled off from other namespaces, for example the “global” or “default” namespace but also custom ones. As an example, an IP address assigned inside a namespace will not be visible outside of the namespace and routing rules in one namespace do not affect others.

To connect a namespace to the outside world, Linux uses virtual ethernet cables - “veth pairs”. These act as a virtual network cables connecting two ends together. We can use them to create a communication bridge between two namespaces, either between a global namespace and a custom namespace or, for example, to create a tunnel between two custom namespaces.

One last characteristic of Linux Network Namespaces I would like to bring up right now is the possibility to not only assign virtual interfaces into a namespace but it is also possible to assign an actual physical network card (NIC) directly to a specific namespace. While we will not be using this feature today, it is beneficial to understand what can and cannot be done using namespaces.

With the above explanation and a basic understanding of Linux Network Namespaces we can proceed with our networking configuration and the logical first step will be to create two of the aforementioned namespaces. To do that, the ip netns add <NAME> command can be used (netns stands for network namespace) and the names I chose for our namespaces for today will be node-a and node-b giving us these two commands we will run:

# Create two namespaces - node-a and node-b
ip netns add node-a
ip netns add node-b

Quickly confirm that the namespaces were created successfully using:

# List custom network namespaces
ip netns list

# Output
node-b
node-a

To connect the namespaces together we will use an aforementioned virtual ethernet cable (veth). The process of doing that starts with creating such veth cable (which is characterized by two names for both ends of a cable, on our case: veth-a and veth-b) after which we can assign both ends of such cable into two namespaces using ip link commands like this:

# Create a VETH pair (the "cable")
ip link add veth-a type veth peer name veth-b

# Assigning both cable ends to our namespaces
ip link set veth-a netns node-a
ip link set veth-b netns node-b

Before any communication can happen between these namespaces we need to configure IP addressing which as you know is a base for any network communication using the IP protocol. Without any IP addresses specified, our networking stack will not know how to forward any packets through our virtual cable.

# IP configuration for the veth cable in our namespaces
ip netns exec node-a ip addr add 10.0.0.1/24 dev veth-a
ip netns exec node-a ip link set veth-a up
ip netns exec node-b ip addr add 10.0.0.2/24 dev veth-b
ip netns exec node-b ip link set veth-b up

Executing the above will assign an 10.0.0.1 IPv4 address into veth-a cable within our node-a namespace. Similarly, 10.0.0.2 IP is being used for veth-b within namespace node-b.

With this configured we can test our config and try to send any packets between our namespaces. The ping utility is the easiest to use and solely designed for that purpose. (additionally, ip addr will let us confirm our IP configuration was applied)

# Confirm connectivity from node-a towards node-b
ip netns exec node-a ip addr
ip netns exec node-a ping 10.0.0.2

# Confirm connectivity from node-b towards node-a
ip netns exec node-b ip addr
ip netns exec node-b ping 10.0.0.1

Quick clarification: you may have noticed a pattern here. ip netns exec node-a/node-b XYZ. With this syntax we can choose to execute any commands in the context of a specified networking namespace.

The pings should go through without problems which means our virtual networks can communicate. Cool!

Configure IPsec Tunnel via Libreswan

With our network setup configured, the next stage will be to setup the IPsec tunnel itself. As you may remember, libreswan is the software we are going to use to define and configure the tunnel.

To create a new tunnel, a configuration for such needs to be added to the Libreswan IPsec config file located under /etc/ipsec.d/tunnel.conf. Open this file with the text editor or your choice and follow along:

#/etc/ipsec.d/tunnel.conf

conn my-ipsec-tunnel
    left=10.0.0.1
    leftsubnet=192.168.1.0/24
    right=10.0.0.2
    rightsubnet=192.168.2.0/24
    authby=secret
    auto=start
  • conn my-ipsec-tunnel defines a new connection profile (tunnel) with our chosen name.
  • left and right identify the two IPsec peers (their IP addresses) and leftsubnet and rightsubnet defines the protected subnets (traffic selectors) behind each peer. Traffic between these subnets is what the tunnel will encrypt/decrypt.
  • authby=secret specifies that a secret pre-shared key - PSK (which we will define later) will be used for authentication
  • Finally, auto=start means the tunnel will be initiated automatically as soon as the IPsec service is started

In the context of our network namespaces, the above gives us:

  • Namespace node-a (IP: 10.0.0.1) is the “left” peer with internal subnet of 192.168.1.0/24
  • Namespace node-b (IP: 10.0.0.2) is the “right” peer with internal subnet of 192.168.2.0/24

Add Dummy Interfaces (The "Private" Networks) to our network namespaces

To make the protected subnets locally reachable in each namespace, we will add some dummy network interfaces (loopback like devices that route traffic without actually transmitting it) and assign them subnet IPs so the kernel can route the traffic and IPsec policies can match it.

Starting with node-a namespace, we will add the dummy0 interface with type dummy and assign it the 192.168.1.1 (part of node-a’s 192.168.1.0/24 internal subnet) IPv4 address after which we can mark it as up:

ip netns exec node-a ip link add dummy0 type dummy
ip netns exec node-a ip addr add 192.168.1.1/24 dev dummy0
ip netns exec node-a ip link set dummy0 up

Same applies for node-b namespace, for which the IPv4 address will be 192.168.2.1 (included in the 192.168.2.0/24 internal subnet for node-b):

ip netns exec node-b ip link add dummy0 type dummy
ip netns exec node-b ip addr add 192.168.2.1/24 dev dummy0
ip netns exec node-b ip link set dummy0 up

Adding routes for the traffic

To ensure the Linux Kernel routes packets correctly inside our custom namespaces, we need to add routes that tell it how to reach the remote protected subnet, which in our environment means sending traffic for the other side’s 192.168.X.0/24 network to the peer’s veth IP address (over the veth link).

# Tell Node A that the 192.168.2.0 network is reachable via its veth-a interface
ip netns exec node-a ip route add 192.168.2.0/24 dev veth-a

# Tell Node B that the 192.168.1.0 network is reachable via its veth-b interface
ip netns exec node-b ip route add 192.168.1.0/24 dev veth-b

Initialize NSS Database

The Libreswan IKE daemon (called pluto, we will get to that a little bit later) uses the Mozilla Network Security Services ("NSS") crypto library for all cryptographic functions during the IKE negotiation.

NSS is a userspace library for cryptographic operations. NSS does not handle the IPsec crypto operations themselves though; these are handled separately by NETKEY or the KLIPS kernel modules.

Libreswan uses NSS to store private keys and X.509 certificates. By default, the NSS database is being initialized by the package installer but in our environment with custom network namespaces we need to initialize the NSS databases by ourselves.

Because we run two independent pluto daemons on one host, we create separate NSS DBs (and run dirs) per instance.

# Create the directories if they don't exist
mkdir -p /etc/ipsec.d/node-a /etc/ipsec.d/node-b

# Initialize the NSS databases
ipsec initnss --nssdir /etc/ipsec.d/node-a
ipsec initnss --nssdir /etc/ipsec.d/node-b

To learn more about the usage of NSS in the Libreswan project visit the Libreswan wiki page hosted under the following link:

Initialize Secrets

In our IPsec tunnel configuration file (/etc/ipsec.d/tunnel.conf) we have defined that our authentication method will be a “secret”. This secret needs to be defined before a full authentication flow can be achieved.

The configuration of such is done by modifying a different file - this time the file system path for this is /etc/ipsec.d/*.secrets, which is self descriptive as well. Here the asterisk symbol signifies a placeholder for the name of the specific IPsec tunnel configuration.

As for the syntax for the secret declaration, this is also straight forward, take a look: tunnel_left_ip tunnel_right_ip : PSK "secret string".

In our case the ipsec.secrets file will be defined as:

echo '10.0.0.1 10.0.0.2 : PSK "VeryLongAndVerySecurePassword12345!@#$%"' > /etc/ipsec.d/tunnel.secrets

Start Pluto (IKE Daemon)

The Pluto daemon (service running in the background) handles the IKE protocol layer and instructs the kernel about IPsec Security Association’s. (Security Association’s - SA’s - are a collection of parameters that the two ends of a IPsec tunnel will use for establishing the tunnel)

This is all we will need to know about it at the moment but I advise you learn more about Pluto daemon in the context of Libreswan here:

ip netns exec node-a ipsec pluto --stderrlog --config /etc/ipsec.conf --nssdir /etc/ipsec.d/node-a --rundir /run/pluto-a --logfile /var/log/pluto-a.log

ip netns exec node-b ipsec pluto --stderrlog --config /etc/ipsec.conf --nssdir /etc/ipsec.d/node-b --rundir /run/pluto-b --logfile /var/log/pluto-b.log

As you can see, within each of the nodes (node-a and node-b) a ipsec pluto (sub)command is being run.

  • --stderrlog as you may imagine enables the logging of standard error (all possible error messages) to the log file for eventual troubleshooting purposes
  • --config /etc/ipsec.conf specifies the default and general (tunnel independent) IPsec configuration required for Pluto
  • Because of our multi namespace environment and resulting multi nss setup we need to explicitly tell Pluto which nss database will be used for each namespace like so: --nssdir /etc/ipsec.d/node-X
  • The above also requires for us to declare a rundir for each of the namespace individually: --rundir /run/pluto-X. The Pluto directories under /run will be created automaticaly and populated with a pluto.ctl and pluto.pid files. (pluto.pid file upon opening will tell us what is the PID - Process ID of the Pluto daemon process where as execution of the file utility on the other pluto.ctl file reveals that the file type here is “socket” which enables other software to connect and control the Pluto process programmatically - API)
  • Lastly --logfile /var/log/pluto-X.log will allow us to monitor the output (stdout and stderr) of both Pluto daemons for troubleshooting - if needed

Reloading Secrets

For good measure let’s also trigger a “secret reload” which will force Libreswan to read our freshly defined secret configuration into the running Pluto daemon (*this needs to be done after starting the Pluto daemon itself as it instructs the process to read the .secrets file(s) and without the process running these commands will fail)

ip netns exec node-a ipsec rereadsecrets --ctlsocket /run/pluto-a/pluto.ctl

ip netns exec node-b ipsec rereadsecrets --ctlsocket /run/pluto-b/pluto.ctl

Output of these commands should include the secrets file we created under /etc/ipsec.d/tunnel.secrets which signifies the commands ran successfully and Pluto is aware and able to proceed with the authentication process.

Loading the IPsec Tunnel Definition into Local Network Namespaces

With the Pluto daemons running in both namespaces, we need to inform both daemons of the specific tunnel configuration.

# Load the 'my-ipsec-tunnel' definition into the Pluto daemon for the node-a namespace
ip netns exec node-a ipsec auto --rundir /run/pluto-a --add my-ipsec-tunnel

# Load the same definition into the Pluto daemon for the node-b namespace
ip netns exec node-b ipsec auto --rundir /run/pluto-b --add my-ipsec-tunnel

Starting the IPsec tunnel

One last step. With the configuration finished there is only one more thing we need to do - actually starting our IPsec tunnel!

# Initiate the tunnel from Node A
ip netns exec node-a ipsec auto --rundir /run/pluto-a --up my-ipsec-tunnel

Instead of the --add my-ipsec-tunnel argument we used to inform daemons of the tunnel configuration, the --up my-ipsec-tunnel (executed from within node-a namespace context) actually brings up (initializes) the tunnel. In my VM the successful output looks like this:

"my-ipsec-tunnel" #6: initiating IKEv2 connection to 10.0.0.2 using UDP
"my-ipsec-tunnel" #6: sent IKE_SA_INIT request to 10.0.0.2:UDP/500
"my-ipsec-tunnel" #6: processed IKE_SA_INIT response from 10.0.0.2:UDP/500 {cipher=AES_GCM_16_256 integ=n/a prf=HMAC_SHA2_512 group=DH19}, initiating IKE_AUTH
"my-ipsec-tunnel" #6: sent IKE_AUTH request to 10.0.0.2:UDP/500 with shared-key-mac and IPV4_ADDR '10.0.0.1'; Child SA #7 {ESP <0xc86eaaea}
"my-ipsec-tunnel" #6: initiator established IKE SA; authenticated peer using authby=secret and IPV4_ADDR '10.0.0.2'
"my-ipsec-tunnel" #7: initiator established Child SA using #6; IPsec tunnel [192.168.1.0/24===192.168.2.0/24] {ESP/ESN=>0x935938aa <0xc86eaaea xfrm=AES_GCM_16_256-NONE DPD=passive}

Confirming correctly working tunnel

There are other ways of checking if our tunnel is active and functioning correctly, two of which I would like to show you right now.

ipsec whack (self described as: IPsec IKE keying daemon low-level control interface) enables us to do just that. Specifying the run directory for our Pluto daemon we can check the status of the IPsec tunnel from within node-a context:

ip netns exec node-a ipsec whack --rundir /run/pluto-a --trafficstatus

Output of which gives us some insights of the amount of traffic (bytes) that were transferred through the tunnel:

#7: "my-ipsec-tunnel", type=ESP, add_time=1771455044, inBytes=0, outBytes=0, maxBytes=2^63B, id='10.0.0.2'

Other option is to use the ip utility of which the xfrm subcommand can show us more details about our newly created tunnel:

ip netns exec node-a ip xfrm state

# Output
src 10.0.0.1 dst 10.0.0.2
	proto esp spi 0x935938aa reqid 16393 mode tunnel
	replay-window 0 flag af-unspec esn
	aead rfc4106(gcm(aes)) 0x5c131205d942d26172db6ed8d45d8ddb0b87551dd19af946338d2c43c63eaf4b3b48828d 128
	anti-replay esn context:
	 seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
	 replay_window 0, bitmap-length 0
	dir out
src 10.0.0.2 dst 10.0.0.1
	proto esp spi 0xc86eaaea reqid 16393 mode tunnel
	replay-window 0 flag af-unspec esn
	aead rfc4106(gcm(aes)) 0x11941b37d160b39c6f61ee1ce8669f803aed8c7c21dccef0a6a0376ca9a9abe4fd4f0872 128
	anti-replay esn context:
	 seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
	 replay_window 128, bitmap-length 4
	 00000000 00000000 00000000 00000000
	dir in

Source 10.0.0.1 with destination 10.0.0.2 (also src 10.0.0.2 and dst 10.0.0.1) matches with our IPsec tunnel configuration. Multiple other parameters can be observed here but we will not go into details about them in this post. This is a good opportunity for you to learn more on your own. Go for it!

Generate and analyze test traffic

With our tunnel up and running it’s time for some tests. The tunnel by itself isn’t very useful is it? It’s what we send through the tunnel that matters. And for that I propose we start with some basic pings to confirm that actual ESP packets can be routed through our virtual tunnel.

Here because we will want to have two commands running at once (one generating traffic and one monitoring the tunnel) we need some kind of software which will allow us to have two terminal windows opened at once. If you remember from the start of this post, this is where tmux comes into play.

Start by executing tmux and you will be welcomed by the same terminal but this time with a green bar at the bottom. To create a new terminal (window in tmux) start by pressing the prefix shortcut ctrl+b followed by the letter c (stands for create). This will move you to a new terminal window with its own history of commands. Test it by executing ls after which you can go back to your previous window by pressing ctrl+b and n (stands for next). Notice how on the previous window there is no ls output visible confirming it only exists in the second window. From now ctrl+b followed by n is how we are gonna switch between these two windows. Please note it is possible to have as many windows as you would like but for today two is all we will need.

In the first window (terminal session) we will start a PCAP (network packet) capture using the tcpdump utility while specifying the interface we want to capture to be our virtual ethernet cable veth-a and the packet type we want to capture being ESP packets. For good measure -v gives us slightly more verbose output and -n stops tcpdump from converting addresses (for example host addresses, port numbers) into names.

# Start packet capture looking for ESP packets
ip netns exec node-a tcpdump -v -ni veth-a -p esp

# Output
tcpdump: listening on veth-a, link-type EN10MB (Ethernet), snapshot length 262144 bytes

For now no further output will be visible as no packets are going through the tunnel but leave this command running as we will come back to it as soon as we generate some traffic.

Now in the second window (ctrl+b -> n) the ping utility can be used from within the context of node-a to generate ICMP traffic.

# Ping Node B's dummy IP using Node A's dummy IP as the source
ip netns exec node-a ping -c 5 -I 192.168.1.1 192.168.2.1

With this command 5 ICMP packets will be send from the interface / address (-I) 192.168.1.1 towards 192.168.2.1 thus from one side of the IPsec tunnel to another forcing that traffic to go through the tunnel itself (there is no other possibility for that traffic to be routed any other way).

A successful ping output will look like this:

PING 192.168.2.1 (192.168.2.1) from 192.168.1.1 : 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=30.2 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=2.14 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=1.04 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=64 time=1.25 ms
64 bytes from 192.168.2.1: icmp_seq=5 ttl=64 time=1.02 ms

--- 192.168.2.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4013ms
rtt min/avg/max/mdev = 1.016/7.119/30.159/11.527 ms

Taking a look at the first window again confirms our success:

08:30:52.916179 IP (tos 0x0, ttl 64, id 47430, offset 0, flags [DF], proto ESP (50), length 140) 10.0.0.1 > 10.0.0.2: ESP(spi=0x2ba65eb0,seq=0x10), length 120
08:30:52.917195 IP (tos 0x0, ttl 64, id 3357, offset 0, flags [none], proto ESP (50), length 140) 10.0.0.2 > 10.0.0.1: ESP(spi=0xae04ca3e,seq=0x10), length 120
08:30:53.914313 IP (tos 0x0, ttl 64, id 47869, offset 0, flags [DF], proto ESP (50), length 140) 10.0.0.1 > 10.0.0.2: ESP(spi=0x2ba65eb0,seq=0x11), length 120
08:30:53.914708 IP (tos 0x0, ttl 64, id 4281, offset 0, flags [none], proto ESP (50), length 140) 10.0.0.2 > 10.0.0.1: ESP(spi=0xae04ca3e,seq=0x11), length 120
08:30:54.915874 IP (tos 0x0, ttl 64, id 48398, offset 0, flags [DF], proto ESP (50), length 140) 10.0.0.1 > 10.0.0.2: ESP(spi=0x2ba65eb0,seq=0x12), length 120
08:30:54.916498 IP (tos 0x0, ttl 64, id 4877, offset 0, flags [none], proto ESP (50), length 140) 10.0.0.2 > 10.0.0.1: ESP(spi=0xae04ca3e,seq=0x12), length 120
08:30:55.917631 IP (tos 0x0, ttl 64, id 48421, offset 0, flags [DF], proto ESP (50), length 140) 10.0.0.1 > 10.0.0.2: ESP(spi=0x2ba65eb0,seq=0x13), length 120
08:30:55.918283 IP (tos 0x0, ttl 64, id 5432, offset 0, flags [none], proto ESP (50), length 140) 10.0.0.2 > 10.0.0.1: ESP(spi=0xae04ca3e,seq=0x13), length 120
08:30:56.919870 IP (tos 0x0, ttl 64, id 48895, offset 0, flags [DF], proto ESP (50), length 140) 10.0.0.1 > 10.0.0.2: ESP(spi=0x2ba65eb0,seq=0x14), length 120
08:30:56.920499 IP (tos 0x0, ttl 64, id 6196, offset 0, flags [none], proto ESP (50), length 140) 10.0.0.2 > 10.0.0.1: ESP(spi=0xae04ca3e,seq=0x14), length 120

10 packets captured
10 packets received by filter
0 packets dropped by kernel

ESP packets (Encapsulating Security Payload packets which are responsible for providing confidentiality, data origin authentication, and integrity for IPsec VPNs) are indeed flowing through the veth-a interface - our IPsec tunnel.

Playing around

After first achieving the virtual IPsec tunnel setup I am explaining in this post I came up with some ideas how I can test its possibilities, including:

  • Performance benchmarking
  • Simulating “bad” internet connection
  • MTU/MSS probing

So obviously this hidden knowledge will be hidden no more - let’s go through them one by one.

Benchmarking the performance of our tunnel can be done using the iperf3 utility we installed earlier exactly for this purpose. The usage of this utility is simple and comes down to launching a server on one of the nodes using iperf3 -s and connecting to it using a client by specifying the other side (IP) of the tunnel: iperf -c 192.168.2.1. Doing this in the proper context of our namespaces gives us:

# Server
ip netns exec node-b iperf3 -s

# Client
ip netns exec node-a iperf3 -c 192.168.2.1

Both commands are blocking (running until manually stopped using ctrl+c) so using tmux will be again necessary.

For my machine the output of the server with our results presents as follows:

-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.0.0.1, port 39886
[  5] local 192.168.2.1 port 5201 connected to 10.0.0.1 port 39890
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   128 MBytes  1.08 Gbits/sec
[  5]   1.00-2.00   sec   128 MBytes  1.07 Gbits/sec
[  5]   2.00-3.00   sec   136 MBytes  1.14 Gbits/sec
[  5]   3.00-4.00   sec   130 MBytes  1.09 Gbits/sec
[  5]   4.00-5.00   sec   130 MBytes  1.10 Gbits/sec
[  5]   5.00-6.00   sec   130 MBytes  1.09 Gbits/sec
[  5]   6.00-7.01   sec   128 MBytes  1.07 Gbits/sec
[  5]   7.01-8.00   sec   129 MBytes  1.08 Gbits/sec
[  5]   8.00-9.00   sec   128 MBytes  1.07 Gbits/sec
[  5]   9.00-10.00  sec   123 MBytes  1.04 Gbits/sec
[  5]  10.00-10.01  sec   256 KBytes   186 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.26 GBytes  1.08 Gbits/sec                  receiver

1 Gbits/sec which is not too bad but I bet I (and you) can do better. Maybe I should write a post about optimizing such IPsec tunnel in the future? I will put in on my to-do list meaning it will happen “eventually”...

Another utility - tc - will allow us to achieve the second test. Simulating different network conditions. tc itself is a utility used to configure Traffic Control in the Linux kernel and in our case in combination with netem classless qdisc will allow us to add arbitrary latency, jitter and packet loss onto our virtual network.

Adding latency to the veth-a virtual wire can be achieved executing:

ip netns exec node-a tc qdisc add dev veth-a root netem delay 100ms

For packet loss the command is very similar but instead of delay the loss option can be used to specify how much percent of packet loss we would like to apply:

ip netns exec node-a tc qdisc add dev veth-a root netem loss 5%

We can also combine both rules and in one command add to our network 100ms of delay, 5% of packet loss and additionally 10ms of artificial network jitter:

ip netns exec node-a tc qdisc add dev veth-a root netem delay 100ms 10ms loss 5%

Executing the above will result in the following ping output between our nodes:

# Before
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.603 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.478 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.462 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.447 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.460 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4012ms
rtt min/avg/max/mdev = 0.447/0.490/0.603/0.057 ms

# After
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=111 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=104 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=101 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=98.3 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4028ms
rtt min/avg/max/mdev = 98.281/103.638/111.475/4.937 ms

Beneficial to know will also be how to bring back our original network performance - how to clear the network rules we just applied. Here you go:

ip netns exec node-a tc qdisc del dev veth-a root

To learn more about the abilities of the Linux kernel exposed through the tc utility refer to the man page hosted on man7.org:

Last but not least a test of the Maximum Transmission Unit (MTU) can be performed to see where packets begin to fragment due to IPsec header overhead. No additional utilities are needed for this as our already familiar ping utility can help us with that.

ip netns exec node-a ping -I 192.168.1.1 -M do -s 1400 192.168.2.1

The -M do argument is important as it specifies the Path MTU Discovery strategy and using the do strategy prohibits fragmentation, even local one. Combining this strategy with the packet size of our choice -s 1400 (bytes) will give us the ability to monitor how the network responds to different packet sizes thus measuring MTU.

In my case a value of 1500 (1472 packet size specified + 8 bytes of ICMP header data + 20 bytes of IP header data) was the maximum the network was able to transmit before erroring out with Message too large giving us the MTU value of - 1500.

# ip netns exec node-a ping -M do -c 1 -s 1472 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 1472(1500) bytes of data.
1480 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.750 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.750/0.750/0.750/0.000 ms

# ip netns exec node-a ping -M do -c 1 -s 1473 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 1473(1501) bytes of data.
ping: sendmsg: Message too large

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 1ms

Conclusion

This concludes our deep dive into Linux networking and virtual IPsec tunnels. I hope you found this article either interesting or helpful. Or both. Please keep an eye on my blog as more articles are coming soon... Stay curious!