My machines are spread over a number of places that are all behind some kind of NAT: the home network behind an OPNsense firewall, a Mikrotik router on another site, a Raspberry Pi at the chalet, and the laptops that move around with me. A few years ago I solved the “how do I reach home” question with an SSH access through a Tor Hidden Service. It worked, but it was slow, it only gave me SSH, and only allowed me to connect to a server, not link multiple networks together.

Today all these sites dial out to a small public server which acts as a WireGuard hub and routes between them. The same server also runs an frp server so that services in my home Kubernetes cluster can be reached from the Internet without opening anything on the home connection.

Until recently, the configuration of this server was edited by hand. I have now moved it to an Ansible repository, and in this post I will explain how it is organised: the hub-and-spoke model with a single list of peers, the forwarding and iptables rules, the frp server with its systemd unit, and the generation of client configuration files on the controller.

flowchart TB
    internet(["Internet"])

    subgraph hub["Arven - public server"]
        direction LR
        frps["frps<br/>public 80 / 443<br/>control TCP 7000"]
        wg0["wg0 - 10.100.0.1/24<br/>UDP 51820"]
    end

    laptop["Laptop<br/>10.100.0.3"]
    pi["Raspberry Pi<br/>10.100.0.10"]

    subgraph site2["Other site"]
        mk["Mikrotik<br/>10.100.0.9<br/>LAN 192.168.50.0/24"]
    end

    subgraph home["Home"]
        opn["OPNsense<br/>10.100.0.8<br/>VLANs 10.20.0.0/24<br/>and 10.20.16.0/20"]
        subgraph k8s["Kubernetes cluster"]
            direction LR
            frpc["frpc pod"]
            traefik["Traefik<br/>80 / 443"]
            forgejo["Forgejo<br/>SSH 22"]
            frpc --> traefik
            frpc --> forgejo
        end
    end

    internet -- "80 / 443" --> frps

    wg0 <== WireGuard ==> laptop
    wg0 <== WireGuard ==> pi
    wg0 <== WireGuard ==> mk
    wg0 <== WireGuard ==> opn

    frpc -. "dials out to TCP 7000" .-> frps

Repository layout

The repository is small. There is a single host in the inventory and four roles applied by one playbook:

- name: Configure Arven as WireGuard hub and frp server
  hosts: arven
  become: true
  roles:
    - wireguard_hub
    - firewall
    - frps
    - client_configs

The non-secret configuration lives in group_vars/arven/wireguard.yml and group_vars/arven/frps.yml. All the secrets (the hub private key, the private keys of the clients for which I generate a configuration, and the frp token) are in an Ansible Vault file next to them. The other files only reference them with a vault_ prefix, like arven_private_key: "{{ vault_arven_private_key }}", so that the vault can be encrypted without breaking the lookups.

One list of peers

The main idea of the repository is that every tunnel ending on the hub is described in a single wireguard_peers list. Site-to-site tunnels and roadwarriors are the same thing from the point of view of the hub: a public key, a tunnel address, and optionally some networks behind it. A shortened version looks like this:

wg_interface: wg0
wg_address: 10.100.0.1/24
wg_listen_port: 51820
wg_network: 10.100.0.0/24
wg_dns: 10.100.0.1
public_endpoint: "vpn.example.org:51820"

wireguard_peers:
  - name: Laptop
    address: 10.100.0.3/32
    public_key: "<laptop-public-key>"
    keepalive: 15
    manage_config: true
    private_key: "{{ vault_laptop_private_key }}"

  - name: OPNSense
    address: 10.100.0.8/32
    public_key: "<opnsense-public-key>"
    keepalive: 15
    site_subnets:
      - 10.20.0.0/24
      - 10.20.16.0/20

  - name: Mikrotik
    address: 10.100.0.9/32
    public_key: "<mikrotik-public-key>"
    keepalive: 15
    site_subnets:
      - 192.168.50.0/24

  - name: Raspberry-Chalet
    address: 10.100.0.10/32
    public_key: "<chalet-public-key>"
    keepalive: 15

The real list has a few more entries but the structure is the same. The site_subnets key is what makes a peer a site: it lists the LANs that are reachable through it. manage_config tells Ansible that I hold the private key of that peer and that it should render a client file for it. The keepalive is set on everything because all the peers are behind NAT and the hub can never initiate the connection.

From this list, two values are derived in the same file:

wg_routed_subnets: "{{ wireguard_peers
  | selectattr('site_subnets', 'defined')
  | map(attribute='site_subnets') | flatten | unique }}"

managed_peers: "{{ wireguard_peers
  | selectattr('manage_config', 'defined')
  | selectattr('manage_config') | list }}"

The first one is the union of all the networks behind all the sites, the second one the peers for which a client configuration is generated.

The hub interface

The wireguard_hub role installs wireguard-tools, enables IPv4 forwarding in /etc/sysctl.d/99-wireguard-router.conf and renders /etc/wireguard/wg0.conf. The template loops over the peers and gives each one its tunnel address plus its site subnets as AllowedIPs:

{% for peer in wireguard_peers %}
{% if peer.public_key and not peer.public_key.startswith('REPLACE_') %}
# {{ peer.name }}
[Peer]
PublicKey = {{ peer.public_key }}
AllowedIPs = {{ ([peer.address] + (peer.site_subnets | default([])))
                | join(', ') }}
PersistentKeepalive = {{ peer.keepalive | default(25) }}

{% endif %}
{% endfor %}

With WireGuard, AllowedIPs on the hub is both the access list and the routing table: a packet for 192.168.50.10 arriving on wg0 is sent to the Mikrotik peer because that peer owns 192.168.50.0/24. There is no need for extra routes on the hub, wg-quick adds them.

The part I care most about is that a mistake in this file must not take down all the tunnels at once, since I am usually connected through one of them when I run the playbook. So there are two layers of checks. First, assertions on the controller check that the hub private key and every peer public key look like WireGuard keys (43 base64 characters followed by =). These also run with --check. Then the template task uses the validate option to load the rendered candidate into a throwaway interface that is never brought up:

wireguard_config_validate_cmd: >-
  bash -c 'ip link del dev wgvalidate 2>/dev/null || true;
  ip link add dev wgvalidate type wireguard &&
  wg setconf wgvalidate <(wg-quick strip %s);
  rc=$?;
  ip link del dev wgvalidate 2>/dev/null || true;
  exit $rc'

This is the same parsing wg-quick up does, so a malformed key or a duplicate AllowedIPs fails here. Because Ansible runs the command on the temporary file, the live wg0.conf stays untouched and the restart handler never fires.

Forwarding and iptables

Enabling net.ipv4.ip_forward is not enough, the hub also needs to accept forwarded traffic. The firewall role installs iptables-persistent and deploys /etc/iptables/rules.v4:

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

-A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Open WireGuard hub forwarding (roadwarrior <-> site tunnels).
-A FORWARD -i wg0 -o wg0 -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A FORWARD -o wg0 -j ACCEPT

COMMIT

The default policy for forwarding is to drop, and only traffic entering or leaving through wg0 is allowed. This is deliberately open between the peers: every peer can reach every advertised subnet. That is fine for me because all the peers are my own devices, but it is a trade-off. Filtering between sites is left to the OPNsense and the Mikrotik, which see the traffic arriving from the tunnel.

Client configurations on the controller

For the peers with manage_config: true, the client_configs role renders an importable .conf file. It runs with delegate_to: localhost and become: false, so the files are written on my machine into generated/clients/, a directory with mode 0700 that is ignored by git. The template is short:

[Interface]
PrivateKey = {{ peer.private_key }}
Address = {{ peer.address }}
DNS = {{ wg_dns }}

[Peer]
PublicKey = {{ arven_public_key }}
Endpoint = {{ public_endpoint }}
AllowedIPs = {{ ([wg_network] + wg_routed_subnets)
                | reject('in', peer.site_subnets | default([]))
                | list | join(', ') }}
PersistentKeepalive = {{ peer.keepalive | default(25) }}

The interesting line is AllowedIPs: each client gets the tunnel network plus every advertised site subnet, minus its own. A laptop therefore reaches the home VLANs and the Mikrotik LAN through the hub, and the Mikrotik, which is itself a managed peer, does not try to route its own LAN into the tunnel. For a laptop, the rendered file looks like this:

[Interface]
PrivateKey = <laptop-private-key>
Address = 10.100.0.3/32
DNS = 10.100.0.1

[Peer]
PublicKey = <hub-public-key>
Endpoint = vpn.example.org:51820
AllowedIPs = 10.100.0.0/24, 10.20.0.0/24, 10.20.16.0/20, 192.168.50.0/24
PersistentKeepalive = 15

It can be imported directly with nmcli, the WireGuard app on a phone, or copied to /etc/wireguard/. The client private keys stay in the vault and on the controller; the hub only ever sees public keys. Adding a new device is now a matter of generating a key pair, putting the public key in wireguard.yml, the private key in the vault, and running the playbook:

$ wg genkey | tee /tmp/priv | wg pubkey
$ ansible-playbook playbooks/site.yml --check --diff
$ ansible-playbook playbooks/site.yml

For peers like the chalet Raspberry Pi or the OPNsense, I do not manage the configuration from here. Their private key was generated on the device and never left it, so they only appear in the list with their public key.

Public ingress with frps

The second job of Arven is to be the public entry point for the services running at home. Previously this was done with HAProxy relaying TCP connections. It is now done with frp: the frps server runs on Arven, and an frpc client inside the cluster dials out to it and registers the ports it wants to expose.

The frps role downloads a pinned release from GitHub (with an optional checksum), installs the binary only if the version changed, creates a frps system user, and renders /etc/frp/frps.toml. The variables are:

frps_version: "0.69.1"
frps_bind_port: 7000
frps_vhost_http_port: false
frps_vhost_https_port: false
frps_auth_token: "{{ vault_frps_auth_token }}"
frps_dashboard_enabled: false
frps_allow_ports:
  - "single = 80"
  - "single = 443"
  - "single = 10022"

The HTTP and HTTPS virtual host features of frp are disabled because I reproduce what HAProxy did: raw TCP passthrough on ports 80 and 443, with TLS terminated in the cluster. The allowPorts list restricts the ports a client may ask for to exactly the three in use, so a leaked token would not allow somebody to open arbitrary ports on the server. Before starting a public frps, an assertion checks that the token is set and is not a placeholder, and the template task validates the new configuration with frps verify -c before it replaces the live one, the same idea as for WireGuard.

The systemd unit runs frps as the unprivileged user and only grants it the capability to bind low ports:

[Service]
Type=simple
User=frps
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/frps -c /etc/frp/frps.toml
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
LimitNOFILE=1048576

The frpc side in the cluster

On the home lab cluster, frpc is a normal Deployment in its own namespace, deployed by ArgoCD from the kubernetes-lab repository. Its configuration is a ConfigMap generated from frpc.toml:

serverAddr = "vpn.example.org"
serverPort = 7000

auth.method = "token"
auth.token = "{{ .Envs.FRP_AUTH_TOKEN }}"

[[proxies]]
name = "ingress-https"
type = "tcp"
localIP = "traefik-ingress.traefik-ingress.svc.cluster.local"
localPort = 443
remotePort = 443

[[proxies]]
name = "git-ssh"
type = "tcp"
localIP = "forgejo.forgejo.svc.cluster.local"
localPort = 22
remotePort = 10022

There is a third proxy for port 80 to the same Traefik Service. Ports 80 and 443 on Arven land on the Traefik ingress, which does the TLS and the routing to the applications, and port 10022 goes to the SSH port of Forgejo for git over SSH.

The token comes from an encrypted Secret, injected as an environment variable. Here I lost some time: frp expands these templates with Go’s html/template, which HTML-escapes the value. A base64 token with a + became + on the client and did not match the server. The fix is to use a hex token, generated with openssl rand -hex 32.

The pod runs as non-root with a read-only root filesystem and all capabilities dropped. A NetworkPolicy denies all ingress, since frpc only makes outbound connections, and limits egress to the cluster DNS, the Traefik and Forgejo namespaces, and port 7000 on the address of Arven. The Deployment uses the Recreate strategy, because frps rejects a second client registering the same proxy names, so a rolling update would leave the new pod failing until the old one is gone.

What is left

The setup has been simplified a lot by having one list to edit, but there are a few things I would like to improve. The forwarding between peers is fully open and could be restricted per peer. The return routes on the OPNsense and the Mikrotik are still configured by hand on those devices. The frps version used in the cluster image (0.61.1) and the server version are not kept in sync automatically. And Arven is a single point of failure: when it is down, the sites cannot talk to each other and the public services are unreachable.

The migration itself did not go perfectly: during one of the first playbook runs, an error in the configuration dropped the WireGuard connections of all the peers. This is where I was happy that the public traffic does not go through the tunnel. The frpc client connects to Arven over the Internet, not over wg0, so the websites and the git server stayed reachable while I was fixing the WireGuard side.

Still, compared to the Tor hidden services, I now have real routing between all the sites, the public services do not depend on the home connection accepting incoming traffic, and adding a new device is one entry in a YAML file and a playbook run.