Creation of a Geo-Redundant On-Premise Cloud

All started with three things:

I got a fixed public IPv4 at home unintentionally and for free first. This was first shocking and then I thought about what to do with it. My first guess was to connect all my locations via site-to-site VPN so I can connect to them easily. This was done quite fast and is surprisingly stable, although the other locations are hidden behind a CG-NAT. Secondly, I was in need of a website. Everybody needs a website, so I need one as well. I looked for web hoster online, but they were either expensive or got me things I did not need. Then I realised, that I need three things for hosting a website. 1.) An IP (check), 2.) a domain (check - got it for mailing purposes), and 3.) a server (check, got a NAS). So I deployed my website on the NAS first and had the first service running on that public fixed IPv4. And then I was in need for a replacement for the really old NAS, which led to my PVE setup. And then I though: why not make it redundant, why not distribute it along the other locations?

This article is supposed to document my journey through the wonders of clustering and remote networking and guiding you around pitfalls that may occur. It focuses on creating a cluster computing with limited bandwidth and latency between the members, locating them in different nets connected through a site-to-site VPN, and with limited hardware (one-disk computing nodes with no additional storage).

Setup

Visualisation of the network interaction. Three locations are connected to the internet. Location 2 and 3 are connected to location 1 via a VPN tunnel but do not have a direct connection.

The starting network setup is pictured by the image. All three locations are connected to the internet. The grey location (location 1) is supposed to be the master location as it has the fixed public IPv4 address. It has a VPN tunnel to location 2 and 3 (green links). The locations 2 and 3 do not have a VPN tunnel to each other but can connect the other location via the hop to location 1. Also there is an existing PVE node in location 1 with an existing PBS node. The plan is to get an additional PVE node to each location 2 and 3 via the following steps.

Creating a cluster and adding a cluster member in a remote network

Creating a datacenter cluster and adding a cluster member in PVE is easy and straightforward. This describes how to do this. Notice, just adding a cluster member does not activate any HA related magic. This is done in another blog entry. Read more. Adding of one member worked perfectly fine, but two members in the given network setup was highly unstable. This was due to the network setup; the two members were not able to communicate directly, due to missing network routes. Apparently, they need to communicate direclty and cannot use the primary server as proxy. Disconnecting one of the members physically and removing it from the cluster, was enough to stabilise the cluster again.

Creation of cluster available networking

TODO

Creation of VM/CT groups and Fencing

TODO

Enabling HA for VMs and CTs

TODO

Previous Blog Entry Next Blog Entry

Last update: 2025-09-15