VeilNet
Blog

Multi-cloud Kubernetes, without a VPN gateway in every cloud

Running Kubernetes across clouds usually means a VPN gateway in each, a tunnel between every pair and route tables to keep in step. Here's why a mesh VPN like Tailscale struggles with it, and a simpler way: join every node to VeilNet and run the cluster on VeilNet's interface, so nodes in every cloud share one flat, post-quantum network.

AWSnode-1ON veil0NODE IP10.66.0.1node-2ON veil0NODE IP10.66.0.2GOOGLE CLOUDnode-3ON veil0NODE IP10.66.0.3node-4ON veil0NODE IP10.66.0.4AZUREnode-5ON veil0NODE IP10.66.0.5node-6ON veil0NODE IP10.66.0.6VEILNET · POST-QUANTUM · DIRECT OR RELAYEDONE CLUSTER ACROSS THREE CLOUDS · NO VPN GATEWAYS

There are good reasons to run Kubernetes in more than one cloud: staying up when a region or a provider has a bad day, keeping data in a particular country, or simply using the best of what each cloud offers. The hard part is rarely Kubernetes. It's the network between the clouds.

This post walks through a simpler way to do it: join every node to VeilNet, then run the cluster's own networking on VeilNet's interface. Kubernetes sees one flat network, whatever cloud a node is in, and every link between clouds is post-quantum. Along the way, we'll look at why a mesh VPN like Tailscale struggles with exactly this setup.

The usual way

Joining clouds for Kubernetes usually takes:

  • A VPN gateway in every cloud.
  • A tunnel from every gateway to every other one. Three clouds need three tunnels, and four need six.
  • Route tables on both sides of every tunnel, kept in step by hand.
  • Address ranges that never overlap.

It works, but every gateway is one more thing to size, patch and pay for, and every new cloud adds a tunnel to each of the others. The tunnels also protect their keys with classical key exchange, such as Diffie-Hellman, which a large quantum computer could break.

Fig. 01 — The usual way and its chores
AWS10.1.0.0/16GWROUTES ×3, BY HANDGOOGLE CLOUD10.2.0.0/16GWROUTES ×3, BY HANDAZURE10.3.0.0/16GWROUTES ×3, BY HANDON-PREM10.4.0.0/16GWROUTES ×3, BY HANDNEW SITE: +3 TUNNELS4 GATEWAYS · 6 TUNNELS4 ROUTE TABLESCLASSICAL KEY EXCHANGE INSIDE

Every cloud needs a gateway, every pair of gateways a tunnel, and every gateway a route table kept in step by hand. Add a fourth site and the tunnels grow from three to six, each protected by classical key exchange.

Why not a mesh VPN like Tailscale?

Mesh VPNs such as Tailscale are a big step up from gateways. People join clouds with them in two ways: they run the cluster over the VPN's own interface, or they forward each cloud's networks with subnet routers. Each runs into one of Tailscale's design choices, and both are in its own documentation.

Full-size packets don't fit

Every tunnel has a largest packet it will carry, its MTU. Tailscale's is 1,280 bytes, while cloud networks and pods usually work with around 1,500 bytes (1,460 on Google Cloud). In Tailscale's own words, packets bigger than its MTU “might get dropped silently”. The fixes it suggests are lowering the MTU on the network that sends them, or clamping TCP's segment size, and its site-to-site guide has you add the clamping rule to each router by hand.

Run a cluster over the VPN, and its networking has to squeeze inside those 1,280 bytes, with its own wrapping on top. Forward networks with subnet routers, and anything that sends a full-size packet risks falling through the gap, since clamping only helps TCP. Problems like this are hard to spot, too: small requests work, and large responses quietly stall.

Routes that go the long way round

Forwarding a cloud's network reliably means two subnet routers in that cloud, and each has to accept routes to reach the other clouds. Tailscale's subnet router docs warn about exactly that: a standby router “can accept its own advertised routes from the primary router”, and then “will send all … traffic through the primary router, even though it is directly connected to that network.”

That's because, on Linux, Tailscale gives the routes it accepts priority over the machine's own. People running it have reported routing loops when a machine's own network comes back to it through Tailscale. The advice is to avoid accepting routes on those routers, which you can't do here, and the documented workaround is a hand-written routing rule for each local network on each machine, which you have to reapply at every boot.

Fig. 02 — Where a mesh VPN trips up
TOO BIG FOR THE TUNNELCLOUD NETWORKS AND PODS1,500 BTAILSCALE'S TUNNEL1,280 BPODS ON A CLUSTER RUN OVER IT≈1,230 BBIGGER PACKETS “MIGHT GET DROPPED SILENTLY”THE LONG WAY ROUNDTHE MESHrouter-1PRIMARYrouter-2STANDBY10.1.0.40A DATABASEDIRECTFIXES, ALL BY HAND: AN IPTABLES RULE ON EVERY ROUTER,AND AN IP RULE PER NETWORK ON EVERY MACHINE, AT EVERY BOOT

Tailscale's tunnel carries packets of up to 1,280 bytes, and bigger ones can be dropped without a word. A standby subnet router that accepts routes reaches its own network through the primary. And the documented fixes are all done by hand.

Neither is a bug in the usual sense. A small, fixed MTU keeps Tailscale working on awkward networks, and preferring its own routes protects a laptop on public Wi-Fi from a network that reuses the same addresses. Those choices just weren't made for a cluster that spans clouds.

Put the cluster on VeilNet's interface

Here's the simpler way. Run conflux, VeilNet's agent, on every node with conflux up. That gives each node a network interface on VeilNet, here called veil0, with an address you choose from one range, such as 10.66.0.0/16. Then tell Kubernetes to use that interface as the node's network.

From then on, Kubernetes sees what looks like one flat network. Every node can reach every other node at its 10.66.0.x address, whichever cloud it's in. The cluster's own networking works as it would in one data centre, wrapping pod traffic between nodes, and VeilNet carries that traffic from cloud to cloud: directly where it can, through a hole punched in NAT where it can't, and through a relay when nothing else works.

VeilNet also steers clear of both snags above, by design:

  • Full-size packets fit. By default, VeilNet's interface takes packets of up to 65,498 bytes on every route and carries them on a QUIC stream. A cluster that sizes its pod network from the interface has plenty of room, and nothing has to be squeezed.
  • Local routes stay local. When a node already has its own route to a network, it keeps that route and skips VeilNet's, and logs that it did. So if you also forward a cloud's own network, which we'll come to, the nodes on that network keep reaching it directly.
Fig. 03 — The cluster runs on veil0
KUBERNETES SEES ONE FLAT NETWORK · 10.66.0.0/16node-1AWSNODE IP 10.66.0.1 ON veil0PODPODPODPODS 10.42.0.0/24node-5AZURENODE IP 10.66.0.5 ON veil0PODPODPODPODS 10.42.4.0/24VEILNET · POST-QUANTUM · DIRECT OR RELAYEDVXLAN · 10.66.0.1 → 10.66.0.5THE CLUSTER WRAPS POD TRAFFIC · VEILNET CARRIES IT

Each node's Kubernetes address is its VeilNet address, on veil0. The cluster's own networking wraps pod traffic between nodes as usual, and VeilNet carries it from cloud to cloud.

Before you start

Pick one address range for the nodes, one that overlaps nothing else on your networks. With k3s, the cluster's own pod and service ranges are 10.42.0.0/16 and 10.43.0.0/16 by default, and Google Cloud's auto-mode networks use 10.128.0.0/9, so 10.66.0.0/16 is clear of all of them. Then give each node its own address from it, for example from the tool that creates your nodes. Keep them unique: VeilNet lets two machines share an address on purpose, for load balancing, so it won't stop you giving two nodes the same one.

Step 1: join every node to VeilNet

Get the install command from the get-started page and install conflux on every node. On the first node, start conflux with its address and the interface name:

the first node
$ sudo conflux up --ipv4 10.66.0.1/16 --interface veil0

The first run mints a taint, a label that names your new network, and prints it. Copy it: it's all another node needs to join. Treat it like a password and share it only with the nodes that belong on the network.

Start every other node with the taint and an address of its own:

every other node
$ sudo conflux up --taint brhk-2mq9-tzva-6pjs-k4xe-nw7d-qf --ipv4 10.66.0.2/16 --interface veil0

The /16 tells each node that the whole range is on VeilNet, so it sends traffic for any 10.66.x.x address to the node that holds it. The sudo is there because conflux creates the interface and registers itself as a service, so it comes back after a reboot. Give the nodes a minute to find each other, then check from any node:

$ sudo conflux peers

Every other node should be listed, with yes in the DATA column.

Step 2: start the cluster on veil0

Now tell Kubernetes to use that interface. With k3s, that's two flags: --node-ip for the node's address, and --flannel-iface for the interface its pod networking runs over. On the first server:

the server, 10.66.0.1
$ curl -sfL https://get.k3s.io | sh -s - --node-ip 10.66.0.1 --flannel-iface veil0

Read the join token from /var/lib/rancher/k3s/server/node-token on the server, then join each worker, in whichever cloud it is:

each worker
$ curl -sfL https://get.k3s.io | K3S_URL=https://10.66.0.1:6443 K3S_TOKEN=<token> \
    sh -s - --node-ip 10.66.0.2 --flannel-iface veil0

Other distributions take the same two settings under different names: the kubelet's node IP, and the interface your cluster's networking uses. Use networking that wraps pod traffic between nodes, such as flannel's default VXLAN mode, rather than one that needs the nodes to share a physical network.

If your nodes run a firewall, let the cluster's own traffic in on veil0: for k3s, that's TCP 6443 to the server, UDP 8472 between nodes for flannel's VXLAN, TCP 10250 for metrics, and traffic from the cluster's pod and service ranges.

Step 3: check it

List the nodes. Each one should show its VeilNet address as its internal IP:

$ kubectl get nodes -o wide

Then the fun part: from a pod in one cloud, call a pod in another at its own address. A pod in AWS can reach a pod in Azure exactly as it would reach one on the next node, and Kubernetes Services work across the whole cluster as usual.

Reaching the rest of each cloud

Your workloads will probably need things outside Kubernetes too, such as a database server or a VM on one cloud's own network. For that, forward that network from one or two nodes that sit on it, by adding --subnet with the network's range:

one or two nodes in AWS
$ sudo conflux up --taint brhk-2mq9-tzva-6pjs-k4xe-nw7d-qf --ipv4 10.66.0.2/16 --interface veil0 \
    --subnet 10.1.0.0/20

Every other node accepts the route on its own, so pods anywhere in the cluster can reach that network through those nodes. Whatever they reach there sees the traffic arrive from the forwarding node, so it needs no route back.

Run two forwarding nodes in each cloud, and they share the traffic flow by flow, and a failed one's flows move to the other once its traffic has failed for two seconds. It's the same idea behind load balancing without a load balancer. And because a node that has its own route to a network keeps it, the two forwarding nodes keep talking to their own network directly, with none of the long way round.

Compartments, and your own root

On the free network VeilNet hosts, the taint is your network: any machine started with it joins. Keep production and staging apart by giving them different taints, so they're separate networks.

When you want a network only your organisation can join, with your own root, revocation and compartments, run your own guardian. It commissions each node and can give each one a free address from your range, and a commissioned node joins without --taint, because its credential carries its compartments.

Try it

Two nodes in two clouds are enough to try this. The docs cover conflux's networking flags, forwarding local networks and how nodes find each other through NAT in full. To plan addresses and compartments for your own clusters, book a demo or write to sovereign@veilnet.net.