VPN access - can't connect to internal services

I’ve been using Wireguard VPN on my OPNSense firewall/router for a while now, and have been quite happy with the setup. Recently though, I noticed I am not able to connect to some of the internal services while connected through the VPN.

All my wireguard peers are set up so that each peer has Allowed IPs in the form of 192.168.100.XXX/32, so all in the 192.168.100.0/24 network. All internal services are in the 192.168.2.0/24 network. When I couldn’t ping or connect from VPN to those internal services my first reaction was to check the routing/firewall, to see if somehow perhaps there was something wrong with the route from 192.168.100.0/24 to 192.168.2.0/24. But the automatically generated rule was there:

So it seems that the VPN was being routed correctly.

All the services I wanted to connect to are in a VM and running in docker containers. So I started to dig in that side of things, perhaps docker is somehow preventing access from VPN, from this other network.

And also, the result was that I could ping the host in the target network, but there was even no response to the ping. As if the connection only worked into one direction.

So I listed all the docker networks:

❯ docker network ls --format "{{.ID}}" | xargs -I {} sh -c 'docker network inspect -f "{{range .IPAM.Config}}{{.Subnet}}{{end}};{{.Name}};{{.Driver}}" {}' | sort | awk -F';' 'BEGIN {print "SUBNET                NETWORK NAME           DRIVER"; print "==================== ===================== ===================="}
{printf "%-20s %-21s %s\n", $1, $2, $3}'

SUBNET                NETWORK NAME           DRIVER
==================== ===================== ====================
172.17.0.0/16        bridge                bridge
172.18.0.0/16        traefik_internal      bridge
172.19.0.0/16        invidious             bridge
172.20.0.0/16        minecraft-server      bridge
172.21.0.0/16        firefly               bridge
172.22.0.0/16        authentik             bridge
172.23.0.0/16        wanderer              bridge
172.24.0.0/16        changedetection       bridge
172.25.0.0/16        netbox                bridge
172.27.0.0/16        n8n                   bridge
172.28.0.0/16        aria2                 bridge
172.29.0.0/16        dbgate                bridge
172.30.0.0/16        formio                bridge
172.31.0.0/16        immich                bridge
192.168.16.0/20      domain_watchdog       bridge
192.168.32.0/20      simplelogin           bridge
192.168.64.0/20      pinchflat             bridge
192.168.96.0/20      librum                bridge
                     host                  host

And then I noticed that with more and more networks added, docker started using new IP ranges for new subnets, the latest of which were overlapping with the IP range of the VPN pool. And that was it! After removing some of the old networks, especially the offending one 192.168.96.0/20, I could again connect to the services I needed.

Apparently, docker has a default predefined pool of network ranges it uses. This is not really documented, but is visible in the source code :

var (
    ...
    localScopeDefaultNetworks            = []*NetworkToSplit{
        {"172.17.0.0/16", 16},
        {"172.18.0.0/16", 16},
        {"172.19.0.0/16", 16},
        {"172.20.0.0/14", 16},
        {"172.24.0.0/14", 16},
        {"172.28.0.0/14", 16},
        {"192.168.0.0/16", 20}}
    globalScopeDefaultNetworks = []*NetworkToSplit{
        {"10.0.0.0/8", 24} //
    }
)

There is not that many of those ranges and by default they include a lot of IP addresses per subnet, much more than I need. So this should be customized and that is done by editing /etc/docker/daemon.json configuration file.

I ended up with the following file:

{
  "default-address-pools" : [
    {
      "base" : "172.17.0.0/12",
      "size" : 24
    },
    {
      "base" : "192.168.0.0/16",
      "size" : 24
    }
  ]
}

which allows for 254 hosts per subnet, and should go a long way before there is a conflict and overlap again. The newly created networks now started with 172.16.1.0/24.