One address, many servers: load balancing without a load balancer
Spreading a service across several servers usually means DNS round robin or a load balancer in the middle. VeilNet does it with one shared address: each client picks a server for itself, stays on it, and moves within seconds if that server goes away. Here's how it works, and where it fits.
Sooner or later, one server isn't enough. Maybe it's too busy, or maybe you've realised that when it goes down, so does everything that depends on it. The answer is as old as the internet: run the service on several servers, and spread the work between them. That's load balancing.
The hard part has never been running a second server. It's getting clients to find the servers, spread out evenly, and stop calling the one that just died. This post is about how VeilNet does that without DNS tricks or a load balancer in the middle, and where its approach fits.
The usual ways, and their catches
There are a few well-worn ways to put several servers behind one service, and each one works. Each also comes with a catch.
- DNS round robin. You publish several addresses under one name, and clients pick one. It's cheap and simple, but answers get cached along the way. When a server dies, some clients keep calling it until their cached answer expires.
- A load balancer. A box, or a cloud service, sits in front of the servers and hands each request to one of them. It reacts quickly, but it's one more thing every request passes through, and one more thing to scale, patch, pay for and keep alive. It often decrypts the traffic to do its job, too.
- An API gateway. Much the same idea with more features, and usually with a public address that the whole internet can reach.
DNS can leave clients calling a server that has gone, and a load balancer puts one box in the path of every request. With a shared address, nothing sits in between: each client's own agent picks a server.
One address, several servers
On a VeilNet network, every machine can be given a private address of its own, and usually each one's is different. But you can give the same address to several machines on purpose, and when you do, they become one destination.
Clients keep using that one address, as they would any other. They don't know or care how many servers are behind it. There are no DNS records to publish, there's no load balancer in the middle, and the traffic stays encrypted all the way from the client to the server that answers it.
Setting it up is one flag. Start each server with the same address:
$ sudo conflux up --taint brhk-2mq9-tzva-6pjs-k4xe-nw7d-qf --ipv4 10.128.0.9/24
That's it. Every server announces the address, and every client on the network sees one destination with several servers behind it. Clients join the network as usual, and there's nothing extra to configure on them.
Every client chooses for itself
So who decides which server a client talks to? The client does. More precisely, the VeilNet agent on the client's own machine does. There's no coordinator handing out clients, and the servers take no part in the choice.
The agent picks a server once per flow, which is all the traffic from one machine to that address. Once it has picked, it keeps that flow on the same server for as long as it's busy, so a session never moves halfway through. A flow only gets a new server after a long quiet spell, or if its server goes away.
When it picks, it leans towards the servers that are quicker to reach, judged from the client's own traffic. Nearby, responsive servers get more new clients, but every server gets some. And because each client measures for itself, a client in Sydney and a client in London can each lean towards the servers that are quick for them.
Each client's own anchor picks a server for its traffic and keeps it there. New clients go more often to the servers that are quicker to reach, but every server gets some.
When a server goes away
Servers fail, and the real test of any load balancing is what happens then.
- Stopped with
sudo conflux down, a server says goodbye as it goes, and clients drop it straight away. - Crashed, powered off or cut off, it's dropped as soon as a client's traffic to it has failed for two seconds. Each client decides that for itself, without waiting for anyone else.
Either way, only that server's clients move, each to one of the servers that remain. Everyone else stays exactly where they were. When the server comes back, it doesn't snatch its old clients back; it simply starts getting new ones.
When server B stops answering, each of its clients moves to another server once its traffic to B has failed for two seconds. Nobody else moves. When B comes back, it takes no one back, and gets new clients only.
Planned work is just as calm. To replace a server, start the new one at the shared address, then stop the old one with sudo conflux down. Its clients move across, and nobody notices more than a reconnect.
The same idea, elsewhere
Shared addresses aren't a one-off feature for web servers. The same idea runs through the rest of VeilNet. If you run several subnet routers for one network, or several exit nodes, they share the load in a similar way, flow by flow, and a failed one's flows move to the others. Our guide to multi-cloud Kubernetes uses exactly that to reach each cloud's own network.
Where it fits, and where it doesn't
This approach is simple on purpose, and simple has edges. Here are the ones worth knowing:
- It balances machines, not connections. All the traffic from one client machine goes to one server, however many connections it opens. With many clients, the load evens out. With one very busy client, it won't.
- Open connections don't survive a failure. When a server goes away, the next one is a different machine, so the application has to reconnect. Most do this already.
- Servers don't share memory. A session kept on one server is unknown to the others, so keep sessions in a shared store or make them stateless, as you would behind any load balancer.
- It watches machines, not your app. VeilNet notices when a server's machine, or its VeilNet agent, goes away. It doesn't notice when your app breaks on a machine that's still running, so keep your own health checks and alerts, and stop the agent before you take a service down.
- The address is private. A shared address lives on your VeilNet network, so it serves your own machines and members, not the public internet.
Side by side
| DNS round robin | A load balancer | A shared VeilNet address | |
|---|---|---|---|
| In the middle | Nothing, but DNS answers are cached | A box every request passes through | No balancer. At most, a relay that can't read the traffic |
| A failed machine | Called until caches expire | Taken out by the balancer's health checks | Dropped at once, or after two seconds of failed traffic |
| Single point of failure | No | The balancer, unless you run several | No |
| Who can read the traffic | Depends on the app's own encryption | Often the balancer too | Only the server |
| Setup | Records for every server | A box or service to run and scale | The same address on each server |
Try it
You can try a shared address on the free network VeilNet hosts in a few minutes: install conflux from the get-started page on two servers and a laptop, give both servers the same address, and watch what happens when you stop one.
The docs go further, with how a shared address works and a step-by-step guide to running an internal API on one address. To plan it for your own network, book a demo or write to sovereign@veilnet.net.