core/docs/DESIGN-cluster-registry.md

Design: PG-Backed Cluster Registry & Self-Assembling Mesh

Status: Proposed — Pre-MVP / MVP wrap-up
Author: Jeff
Date: 2026-03-30
Scope: Kernel primitive — zero domain awareness

---

Problem

Armature-core is tightly coupled to PostgreSQL. Scaling horizontally requires instance coordination: peer discovery, health monitoring, ephemeral event routing, and (optionally) leader election. Traditional HA solutions (Raft, etcd, Consul) introduce a second consensus layer on top of PG — doubling operational complexity for a system that already has serializable transactions and LISTEN/NOTIFY.

Principle

PG is already the consensus system. Use it as the sole coordinator for service discovery, health, pub/sub, and cluster state. Zero new infrastructure dependencies.

---

Design

1. Node Registry (Unlogged Table)

CREATE UNLOGGED TABLE IF NOT EXISTS node_registry (
    node_id       TEXT PRIMARY KEY,
    endpoint      TEXT NOT NULL,
    seq           SERIAL,
    registered_at TIMESTAMPTZ DEFAULT now(),
    heartbeat     TIMESTAMPTZ DEFAULT now(),
    stats         JSONB DEFAULT '{}'
);

Why UNLOGGED: No WAL overhead, visible to all connections, survives session disconnect. Contents do NOT survive a PG crash — but a PG crash means all nodes are dead anyway. Clean slate on recovery is the correct behavior; all nodes re-register on restart.

Why not TEMPORARY: PG temporary tables are session-scoped and invisible to other connections. Useless for cross-instance coordination.

2. Node Lifecycle

Startup

boot → connect PG → register self → read peer list →
subscribe LISTEN/NOTIFY channels → open WS to clients → ready

-- Self-registration (atomic, idempotent)
INSERT INTO node_registry (node_id, endpoint)
VALUES ($1, $2)
ON CONFLICT (node_id) DO UPDATE
SET endpoint = EXCLUDED.endpoint,
    registered_at = now(),
    heartbeat = now(),
    stats = '{}';

node_id is generated at startup: hostname + PID or a UUID. Deterministic IDs enable restart-in-place without orphaned rows.

Heartbeat Tick (every N seconds, e.g., 10s)

Each node performs two operations per tick:

// 1. Update own heartbeat + stats
stats := collectStats() // see §5
db.Exec(`UPDATE node_registry SET heartbeat = now(), stats = $2 WHERE node_id = $1`,
    nodeID, stats)

// 2. Sweep stale entries (idempotent — safe for concurrent execution)
db.Exec(`DELETE FROM node_registry WHERE heartbeat < now() - interval '30 seconds'`)

The sweep is the distributed health check. Every node runs it. Multiple nodes deleting the same stale row is a no-op. No ring topology required — the sweep is the safety net that guarantees eventual cleanup regardless of which nodes are alive.

Self-Eviction

If a node's own heartbeat UPDATE returns 0 rows affected, it has been swept by a peer:

result := db.Exec(`UPDATE node_registry SET heartbeat = now() WHERE node_id = $1`, nodeID)
if result.RowsAffected() == 0 {
    log.Error("evicted from registry, shutting down")
    os.Exit(1)
}

Kubernetes restarts the process. On boot it re-registers as a fresh node. The restart loop IS the recovery mechanism.

3. Leaderless by Default

No role column. No primary/replica distinction. All nodes are equal peers. PG is the sole coordinator.

If a leader is ever needed (future use case), it's a small lift:

ALTER TABLE node_registry ADD COLUMN role TEXT DEFAULT 'peer';

-- Atomic leader claim: first writer wins
UPDATE node_registry SET role = 'primary'
WHERE node_id = $1
AND NOT EXISTS (SELECT 1 FROM node_registry WHERE role = 'primary');

No quorum, no voting, no term numbers. PG serialization handles it. But the current design intentionally avoids this — consistent with the existing pattern where scheduled tasks use first-to-mark-wins execution, not leader assignment.

4. Realtime Event Routing (Two Tiers)

Tier 1: Durable Events (messages, state changes)

These are PG writes. Fan-out via LISTEN/NOTIFY:

User A sends message → Instance 1 writes to PG →
pg_notify('channel:xyz', '{"type":"message","id":"..."}') →
All instances receive → push to local WS clients subscribed to channel:xyz

This maps directly to the v0.5.0 realtime.publish() Starlark primitive. Kernel implementation is pg_notify(channel, payload).

Tier 2: Ephemeral Events (typing indicators, presence, cursor position)

These NEVER hit durable storage.

Phase 1 (MVP): Use PG LISTEN/NOTIFY for ephemeral events too.

User A types → Instance 1 WS handler →
pg_notify('ephemeral:xyz', '{"type":"typing","user":"A"}') →
Instance 2 receives → pushes to User B's WS
Instance 3 receives → pushes to User C's WS

PG NOTIFY payload limit is 8KB. A typing indicator is ~60 bytes. At homelab scale (3 nodes, dozens of users) this is a non-issue. One codepath, one transport, zero new infrastructure.

Phase 2 (Post-MVP, if needed): Direct instance-to-instance mesh for ephemeral events.

• Peer endpoints are already in the registry
• Each node dials peers discovered from the registry on startup and on peer list changes
• Ephemeral events route over the mesh; durable events stay on PG NOTIFY
• The realtime.publish() Starlark API does not change — kernel swaps the underlying transport

                    ┌─────────────────────────────────┐
                    │         PG (source of truth)     │
                    │  ┌───────────┐ ┌──────────────┐  │
                    │  │  node_    │ │  LISTEN/     │  │
                    │  │  registry │ │  NOTIFY      │  │
                    │  └───────────┘ └──────────────┘  │
                    └──────┬──────────────┬────────────┘
                           │   durable    │
              ┌────────────┼──────────────┼────────────┐
              │            │              │            │
         ┌────┴────┐  ┌───┴─────┐  ┌────┴─────┐
         │ Node 1  │◄─┤ Node 2  ├─►│  Node 3  │
         │ User A  │  │ User B  │  │  User C  │
         └─────────┘  └─────────┘  └──────────┘
              ◄──── ephemeral mesh ────►
                   (Phase 2, if needed)

Channel subscription model (Phase 1): Broadcast and discard. Every node receives every event, checks local WebSocket subscriptions, drops if irrelevant. At 3 nodes this is the correct answer — trivial, no subscription state to sync, fully leaderless.

5. Stats Collection & Admin Dashboard

Each heartbeat tick publishes runtime stats in the stats JSONB column:

var m runtime.MemStats
runtime.ReadMemStats(&m)

stats := map[string]any{
    "goroutines":     runtime.NumGoroutine(),
    "heap_alloc":     m.HeapAlloc,
    "heap_sys":       m.HeapSys,
    "gc_cycles":      m.NumGC,
    "gc_pause_ns":    m.PauseNs[(m.NumGC+255)%256],
    "uptime_sec":     time.Since(startTime).Seconds(),
    "ws_clients":     wsHub.Count(),
    "extensions":     sandbox.LoadedCount(),
}

Admin dashboard API endpoint:

SELECT node_id, endpoint, registered_at, heartbeat, stats
FROM node_registry
ORDER BY seq;

ICD envelope: {"data": [...]} — consistent with list endpoint convention.

The admin dashboard extension renders one card per node. One node = one card. Three nodes = three cards. The extension has zero awareness of clustering — it reads a list and renders it. This is the extensibility proof working as designed.

Since stats is JSONB, future additions (mesh peer latency, extension execution counts, queue depths) require no schema migration. The dashboard extension renders whatever keys are present.

6. Self-Assembling Mesh Properties

The mesh builds itself from the registry with zero configuration:

Event	Behavior
Node starts	Inserts into registry → peers discover it on next heartbeat tick
Node dies	Heartbeat goes stale → swept by peers → mesh shrinks
Node restarts	Re-registers → peers discover it → mesh restores
Scale up (K8s)	New replicas register → mesh grows automatically
Scale down (K8s)	Terminated nodes go stale → swept → mesh shrinks
PG crash	All nodes lose connection → all exit → K8s restarts all → clean re-registration

No config files listing peers. No seed nodes. No service discovery sidecar. No bootstrap ceremony. The registry table IS the service mesh control plane.

---

What This Enables (Future)

These are NOT in scope for this work. Listed to validate the design supports them without rearchitecting.

• Work distribution: Primary (if elected) assigns extension execution shards via registry metadata. Replicas poll for assignments. No message queue.
• Realtime routing (v0.5.0): When a node owns a pub/sub channel, it registers ownership in PG. Other nodes route publishes to the owner. Node dies → sweep clears routes → subscribers reconnect.
• Rolling deploys: New version registers, old version's heartbeat goes stale, clean handoff with zero coordination protocol.
• Session affinity: Registry tracks which node holds a user's WS connection. Targeted event delivery instead of broadcast (optimization for scale).

---

Implementation Notes

Migration

Pre-MVP: fold CREATE UNLOGGED TABLE into existing schema initialization. No new migration file needed. DB rebuild required (acceptable — no install base).

Configuration

Setting	Default	Notes
CLUSTER_NODE_ID	hostname-PID	Override for deterministic identity across restarts
CLUSTER_HEARTBEAT_INTERVAL	10s	Tick frequency
CLUSTER_STALE_THRESHOLD	30s	3x heartbeat = stale
CLUSTER_ENDPOINT	auto-detect	Advertised address for peer mesh (Phase 2)

Single-Node Behavior

When only one node is registered, the system behaves identically to pre-cluster armature. The registry has one row. LISTEN/NOTIFY delivers events back to the same instance. No special-casing required.

Health Endpoint Integration

GET /health includes cluster status:

{
    "status": "healthy",
    "node_id": "node-abc-1234",
    "cluster": {
        "size": 3,
        "peers": ["node-def-5678", "node-ghi-9012"],
        "heartbeat_age_ms": 2340
    }
}

Satisfies the v0.6.0 MVP health monitoring requirement.

---

Rejected Alternatives

Alternative	Why rejected
etcd / Consul	Second consensus layer on top of PG. Doubles operational complexity for a system already tightly coupled to PG. Violates KISS.
Redis pub/sub for ephemeral events	Second stateful service. Two failure domains, two connection pools, two things to monitor. PG NOTIFY handles the same job at homelab scale.
Raft implementation in Go	Massive complexity for a problem PG serializable transactions already solve.
Ring topology monitoring	Each node watches the one above it. Adjacent failures create blind spots. Sweep-all is simpler and has no edge cases.
Leader election at startup	Unnecessary. Leaderless design with first-to-mark-wins task execution already works. Leader election is a future option, not a requirement.

---

Checklist

• node_registry unlogged table in schema init
• Node registration on startup
• Heartbeat tick with stats collection and stale sweep
• Self-eviction on 0-rows-affected heartbeat
• PG LISTEN/NOTIFY integration for realtime events
• Admin API endpoint: GET /api/admin/cluster
• Admin dashboard extension: cluster node cards
• Health endpoint cluster status
• Configuration via environment variables
• Single-node regression test (no behavioral change)
• Multi-node integration test (Docker Compose, 3 instances)