Wireguard on NanoPC-T4 with BoringTun

I was in the midst of implementing a little POC migration of a small deployment from using OpenVPN to WireGuard for a pre-production network when I ran into an issue with 3 of the network clients being NanoPC-T4 devices. I didn't initially consider that this would be an issue, but FriendlyElec in their infinite wisdom does not provide a linux-headers package for their kernels, and wireguard doesn't provide a wireguard-modules package for ARM. Other distributions such as DietPi still package the upstream kernel directly as well, so also do not provide a headers package.

After banging my head against the wall looking into custom mainline kernels for the NanoPC-T4, I considered that there might be a userspace utility for WireGuard that doesn't require compiling a kernel module or installing wireguard-dkms.

A quick search turned up both wireguard-go and boringtun*. From a cursory glance and a few user stories, the boringtun project seemed to be a slightly more mature and less error prone implementation that has seen deployments on embedded & SBC devices already.

*Editor's Note: Fuck CloudFlare. Get your FUCK CLOUDFLARE sickers today!

Installation & Configuration

I still need the wireguard-tools package in order to leverage the handy-dandy wg-quick util and related systemd integration so the VPN will connect easily on boot.

Generate client side keys for WireGuard, and set up a simple wg0.conf using the existing OpenVPN IP addresses so I don't have to reconfigure security groups.

$ cat /etc/apt/sources.list.d/unstable.list
> deb http://deb.debian.org/debian/ unstable main

$ cat /etc/apt/preferences.d/limit-unstable
> Package: *
> Pin: release a=unstable
> Pin-Priority: 90

$ apt install wireguard-tools

$ wg genkey | tee privatekey | wg pubkey > publickey

$ cat /etc/wireguard/wg0.conf

> Address = 10.8.0.2/24
> PrivateKey = <privatekey> 
> DNS = <your favourite DNS server>
>
> [Peer]
> PublicKey = <server public key> 
> PresharedKey = <server PSK>
> AllowedIPs = 10.8.0.0/24
> Endpoint = <server IP>:51820 
> PersistentKeepalive = 25

Install BoringTun & Rust

Dietpi package repo's Rust was old and decrepit, so let's just install new rust from rustup.sh. It should automagically install everything you need to build and compile your rusty applications (i.e. cargo & rustc) for whatever platform you're running. In this case: arm64/aarch64.

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Next. I had to pull down and build my own copy of BoringTun from source to fix a small issue with compilation before building and installing.

$ git clone https://github.com/cloudflare/boringtun.git
$ cd boringtun/
$ vim src/device/tun_linux.rs
# >> https://github.com/cloudflare/boringtun/issues/89/#issuecomment-508962631
cargo install --path .

The fancy new boringtun binary installs to /root/.cargo/bin, which should be on your path if you've sourced $HOME/.cargo/env since installing Rust.

Manual hax

Privilege de-escalation doesn't work properly for boringtun when logged in as root. Passing undocumented environment variables to wg-quick on startup as suggested by using WG_QUICK_USERSPACE_IMPLEMENTATION="boringtun --disable-drop-privileges" wg-quick up /etc/wireguard/wg0.conf was unsuccessful per the project's README. So hacking wg-quick in place was needed in order to accomplish a functional utility.

$ vim $(which wg-quick)
> 
> add_if() {
>         boringtun --disable-drop-privileges "$INTERFACE"
>         # jlocal ret
>         # ...
>

Note: WG_QUICK_USERSPACE_IMPLEMENTATION is purposefully undocumented and added in #b9b78f27399 to help enable userspace implementations of the WireGuard protocol.

It's Tunnel Time

Now that the hacking and slashing is behind us, we can finally bring the WireGuard VPN connection online to the server.

$ wg-quick up /etc/wireguard/wg0.conf
$ wg

interface: wg0
  public key: <redacted>
  private key: (hidden)
  listening port: 57870

peer: <server public key>
  preshared key: (hidden)
  endpoint: <server IP>:51820
  allowed ips: 10.8.0.0/24
  latest handshake: 1 minute, 16 seconds ago
  transfer: 3.79 KiB received, 119.36 KiB sent
  persistent keepalive: every 25 seconds

If you need to tunnel everything over your VPN vs. just the resources on the VPN network, you can change allowed_ips in your configuration to 0.0.0.0 and it will generate a default route and mark all packets to be destined for the WireGuard connection.

Caveats

It seems like when tearing down the connection, wg-quick doesn't actually destroy the BoringTun tunnel and creates a second process, but doesn't actually complain about it. All seems to be working fine, but after 3-4 restarts, the NanoPC's CPU is pinned at 500% usage with all the userspace agents fighting eachother.

Probably something I'll hack on later to clean up, but for now it is working as well as can be expected. Even with using a userspace vs. kernelspace agent, system load is down and overall latency has improved compared to the existing OpenVPN setup.

Notes

For those looking for a great source of information on installing and configuring WireGuard, check out the Arch Linux Wiki page on WireGuard. Top notch as always.