Proxmox VE clustering represents the convergence of software-defined compute and storage within high-density network infrastructure. In the broader technical stack of cloud service providers or industrial utility management; maintaining near-zero downtime is a fundamental requirement. Single-node architecture introduces a catastrophic point of failure; if the underlying hardware undergoes thermal-shutdown or critical power loss, the entire service chain collapses. Proxmox VE clustering addresses this by utilizing a distributed cluster file system known as pmxcfs and the Corosync communication engine. By clustering nodes, administrators achieve an idempotent state where configuration changes propagate across the entire fabric simultaneously. This transition from isolated machines to a unified cluster fabric allows for automated failover and live migration of virtual machines. The goal of this manual is to detail the deployment of a highly available environment; ensuring that even in scenarios of high latency or localized hardware interference, the cluster maintains quorum and functional operational integrity.
Technical Specifications
| Requirement | Default Port/Range | Protocol/Standard | Impact Level (1-10) | Recommended Resources |
| :— | :— | :— | :— | :— |
| Corosync Cluster Traffic | 5404:5405 | UDP / Unicast | 10 | Low Latency < 2ms |
| Web Management Interface | 8006 | HTTPS | 7 | 2 vCPU / 2GB RAM |
| Cluster File System | N/A | pmxcfs (FUSE) | 9 | High-Speed SSD/NVMe |
| Resource Migration | Multiple (Dynamic) | TCP / SSH | 6 | 10GbE Bonded NICs |
| Hardware Fencing | N/A | Linux Watchdog | 10 | IPMI / iDRAC / iLO |
The Configuration Protocol
Environment Prerequisites:
Successful clustering requires Proxmox VE version 7.x or 8.x installed on all participating nodes. Networking must support IPv4 or IPv6 multicast/unicast, though unicast is the modern standard for Corosync in PVE. All nodes must have synchronized clocks via NTP or Chrony; clock drift exceeding 0.5 seconds will cause authentication ticket expiration and quorum instability. Users must possess root or Administrator permissions and have a reliable SSH link between nodes. Hardware must be homogeneous where possible to prevent thermal-inertia bottlenecks during peak load distribution.
Section A: Implementation Logic:
The logic of a Proxmox cluster relies on the Quorum algorithm. In a distributed system, quorum is the minimum number of votes required to make decisions. For a cluster to remain “inquorate” and functional, more than half of the nodes must be online and communicating. In a two-node setup, a single node failure results in a loss of quorum, halting all HA operations. Therefore, a three-node minimum is the architectural standard to ensure a majority exists during the failure of a single physical asset. The pmxcfs layer acts as a database-backed file system that mirrors the /etc/pve directory across all nodes; this ensures that an action taken on Node A is reflected on Node C with minimal overhead.
Step-By-Step Execution
1. Initialize the Primary Cluster Identity
On the designated master node, execute the command pvecm create
System Note: This action generates the /etc/pve/corosync.conf file and initializes the corosync.service. The kernel creates a new cluster membership ID and begins broadcasting its presence on the local network interface.
2. Configure Corosync Network Links
Navigate to the /etc/pve/corosync.conf file and ensure the ring0_addr is set to a dedicated, low-latency network interface.
System Note: By isolating cluster traffic to a specific VLAN or physical NIC, you prevent packet-loss caused by high-volume storage or VM traffic. This reduces signal-attenuation in virtualized networking layers.
3. Join Subsequent Nodes to the Fabric
From the web interface of the target joiner node, or via CLI using pvecm add
System Note: The pve-cluster.service on the joining node will stop, synchronize its local /etc/pve directory with the master node via an rsync-like mechanism inside pmxcfs, and then restart. This ensures an idempotent configuration across the pool.
4. Verification of Cluster Totality
Run the command pvecm status to verify all nodes are listed as “Online” and that the “Quorum” status is “OK”.
System Note: This command queries the corosync-quorumtool to check if the node has a valid vote. If the node is “Partitioned”, it will enter read-only mode for /etc/pve to prevent a split-brain scenario.
5. Define High Availability Groups
Within the Proxmox UI or via ha-manager, create an HA group that includes specific nodes. Assign priority levels to define which node should take the workload first.
System Note: The pve-ha-lrm (Local Resource Manager) and pve-ha-crm (Cluster Resource Manager) services begin monitoring the state of the VMs. If a node heart-beat is lost, the CRM initiates a recovery on another node in the group.
Section B: Dependency Fault-Lines:
Software-defined clustering is sensitive to network jitter. A common failure point is the use of a shared network for Corosync and Ceph storage; high storage throughput can flood the buffers, causing Corosync to drop packets. This results in “node flapping” where a node repeatedly joins and leaves the cluster. Another bottleneck is the disk I/O latency on the root partition; because pmxcfs writes frequently to a small SQLite database, high latency on the OS drive can cause the file system to lock, rendering the management UI unresponsive.
The Troubleshooting Matrix
Section C: Logs & Debugging:
When a node becomes “inquorate”, the first point of inspection is journalctl -xeu corosync. Look for “Token Lost” or “Retransmit List” errors. If these strings appear, it indicates that the network cannot handle the concurrency required for cluster heartbeats.
Specific Error Strings:
1. “pmxcfs: [status] notice: update failed”: This suggests the local node cannot write to its database. Check for disk space using df -h or disk health using smartctl.
2. “corosync [TOTEM] Retransmitting”: This identifies packet-loss on the cluster network. Use mtr or ping between nodes to check for signal-attenuation or physical cable faults.
3. “ha-manager: fence-node failed”: This indicates the cluster tried to reboot a failed node via the watchdog or IPMI but failed. Check the path /dev/watchdog to ensure the kernel module is loaded via lsmod | grep dog.
Log analysis should always prioritize the file /var/log/pve/tasks/index for high-level management errors and /var/log/corosync/corosync.log for low-level heartbeat issues.
Optimization & Hardening
Performance Tuning:
To increase throughput and reduce latency, set the Corosync redundancy mode to active-passive by defining multiple rings in corosync.conf. This provides a failover path if the primary NIC fails. Additionally, ensure the MTU (Maximum Transmission Unit) is consistent across the cluster. Performance is further enhanced by setting the CPU Governor to “performance” mode on all nodes to minimize the context-switching overhead that can delay heartbeat processing.
Security Hardening:
Restrict cluster communication to a private, non-routable VLAN. Enable the Proxmox Firewall at the cluster level and create a rule to allow UDP 5404:5405 only from known node IPs. Ensure all SSH communication uses RSA or ED25519 keys with limited permissions. Encapsulation of cluster traffic via a VPN is discouraged due to the extreme latency penalty, but using a dedicated physical crossover for two-node setups can harden the link against external interception.
Scaling Logic:
As the cluster grows beyond 8 nodes, the overhead of Corosync’s “full mesh” communication increases. To maintain efficiency, consider splitting large environments into multiple smaller clusters or utilizing a separate Proxmox Backup Server (PBS) to offload the I/O payload. When adding nodes, always maintain an odd number of members to simplify quorum calculations and prevent tie-break failures.
The Admin Desk
How do I fix a “Permission Denied” error when joining a node?
Ensure the root SSH keys are synchronized. Use ssh-copy-id to manually push the public key to the target node. Check that the /etc/hosts file on both nodes contains the correct IP entries for every cluster member.
Can I run a Proxmox cluster with only two nodes?
Yes; however, you should use a QDevice (Quorum Device). This is a lightweight service running on a separate third machine (even a Raspberry Pi) that provides a tie-breaker vote to prevent “split-brain” scenarios during a node failure.
Why is my cluster file system read-only?
The node has likely lost quorum. If more than half the nodes are down; the remaining nodes go read-only to prevent data corruption. Restore the network link or manually force quorum using pvecm expected 1 for emergency recovery.
What happens if the master node dies?
In Proxmox VE, all nodes are peers. There is no single master node after the cluster is formed. The pmxcfs handles the distribution, and any surviving node can manage the cluster as long as quorum is maintained across the fabric.
How do I reduce Corosync latency?
Disable any unnecessary flow control on your network switches. Ensure that the network interfaces used for clustering are not shared with virtual machine bridges or heavy storage traffic. Use dedicated physical hardware whenever possible to avoid encapsulation delays.
