# Design: PG-Backed Cluster Registry & Self-Assembling Mesh

**Status:** Proposed — Pre-MVP / MVP wrap-up  
**Author:** Jeff  
**Date:** 2026-03-30  
**Scope:** Kernel primitive — zero domain awareness

---

## Problem

Switchboard-core is tightly coupled to PostgreSQL. Scaling horizontally requires instance coordination: peer discovery, health monitoring, ephemeral event routing, and (optionally) leader election. Traditional HA solutions (Raft, etcd, Consul) introduce a second consensus layer on top of PG — doubling operational complexity for a system that already has serializable transactions and LISTEN/NOTIFY.

## Principle

PG is already the consensus system. Use it as the sole coordinator for service discovery, health, pub/sub, and cluster state. Zero new infrastructure dependencies.

---

## Design

### 1. Node Registry (Unlogged Table)

```sql
CREATE UNLOGGED TABLE IF NOT EXISTS node_registry (
    node_id       TEXT PRIMARY KEY,
    endpoint      TEXT NOT NULL,
    seq           SERIAL,
    registered_at TIMESTAMPTZ DEFAULT now(),
    heartbeat     TIMESTAMPTZ DEFAULT now(),
    stats         JSONB DEFAULT '{}'
);
```

**Why UNLOGGED:** No WAL overhead, visible to all connections, survives session disconnect. Contents do NOT survive a PG crash — but a PG crash means all nodes are dead anyway. Clean slate on recovery is the correct behavior; all nodes re-register on restart.

**Why not TEMPORARY:** PG temporary tables are session-scoped and invisible to other connections. Useless for cross-instance coordination.

### 2. Node Lifecycle

#### Startup

```
boot → connect PG → register self → read peer list →
subscribe LISTEN/NOTIFY channels → open WS to clients → ready
```

```sql
-- Self-registration (atomic, idempotent)
INSERT INTO node_registry (node_id, endpoint)
VALUES ($1, $2)
ON CONFLICT (node_id) DO UPDATE
SET endpoint = EXCLUDED.endpoint,
    registered_at = now(),
    heartbeat = now(),
    stats = '{}';
```

`node_id` is generated at startup: `hostname + PID` or a UUID. Deterministic IDs enable restart-in-place without orphaned rows.

#### Heartbeat Tick (every N seconds, e.g., 10s)

Each node performs two operations per tick:

```go
// 1. Update own heartbeat + stats
stats := collectStats() // see §5
db.Exec(`UPDATE node_registry SET heartbeat = now(), stats = $2 WHERE node_id = $1`,
    nodeID, stats)

// 2. Sweep stale entries (idempotent — safe for concurrent execution)
db.Exec(`DELETE FROM node_registry WHERE heartbeat < now() - interval '30 seconds'`)
```

The sweep is the distributed health check. Every node runs it. Multiple nodes deleting the same stale row is a no-op. No ring topology required — the sweep is the safety net that guarantees eventual cleanup regardless of which nodes are alive.

#### Self-Eviction

If a node's own heartbeat UPDATE returns 0 rows affected, it has been swept by a peer:

```go
result := db.Exec(`UPDATE node_registry SET heartbeat = now() WHERE node_id = $1`, nodeID)
if result.RowsAffected() == 0 {
    log.Error("evicted from registry, shutting down")
    os.Exit(1)
}
```

Kubernetes restarts the process. On boot it re-registers as a fresh node. The restart loop IS the recovery mechanism.

### 3. Leaderless by Default

No `role` column. No primary/replica distinction. All nodes are equal peers. PG is the sole coordinator.

**If a leader is ever needed** (future use case), it's a small lift:

```sql
ALTER TABLE node_registry ADD COLUMN role TEXT DEFAULT 'peer';

-- Atomic leader claim: first writer wins
UPDATE node_registry SET role = 'primary'
WHERE node_id = $1
AND NOT EXISTS (SELECT 1 FROM node_registry WHERE role = 'primary');
```

No quorum, no voting, no term numbers. PG serialization handles it. But the current design intentionally avoids this — consistent with the existing pattern where scheduled tasks use first-to-mark-wins execution, not leader assignment.

### 4. Realtime Event Routing (Two Tiers)

#### Tier 1: Durable Events (messages, state changes)

These are PG writes. Fan-out via `LISTEN/NOTIFY`:

```
User A sends message → Instance 1 writes to PG →
pg_notify('channel:xyz', '{"type":"message","id":"..."}') →
All instances receive → push to local WS clients subscribed to channel:xyz
```

This maps directly to the v0.5.0 `realtime.publish()` Starlark primitive. Kernel implementation is `pg_notify(channel, payload)`.

#### Tier 2: Ephemeral Events (typing indicators, presence, cursor position)

These NEVER hit durable storage.

**Phase 1 (MVP):** Use PG LISTEN/NOTIFY for ephemeral events too.

```
User A types → Instance 1 WS handler →
pg_notify('ephemeral:xyz', '{"type":"typing","user":"A"}') →
Instance 2 receives → pushes to User B's WS
Instance 3 receives → pushes to User C's WS
```

PG NOTIFY payload limit is 8KB. A typing indicator is ~60 bytes. At homelab scale (3 nodes, dozens of users) this is a non-issue. One codepath, one transport, zero new infrastructure.

**Phase 2 (Post-MVP, if needed):** Direct instance-to-instance mesh for ephemeral events.

- Peer endpoints are already in the registry
- Each node dials peers discovered from the registry on startup and on peer list changes
- Ephemeral events route over the mesh; durable events stay on PG NOTIFY
- The `realtime.publish()` Starlark API does not change — kernel swaps the underlying transport

```
                    ┌─────────────────────────────────┐
                    │         PG (source of truth)     │
                    │  ┌───────────┐ ┌──────────────┐  │
                    │  │  node_    │ │  LISTEN/     │  │
                    │  │  registry │ │  NOTIFY      │  │
                    │  └───────────┘ └──────────────┘  │
                    └──────┬──────────────┬────────────┘
                           │   durable    │
              ┌────────────┼──────────────┼────────────┐
              │            │              │            │
         ┌────┴────┐  ┌───┴─────┐  ┌────┴─────┐
         │ Node 1  │◄─┤ Node 2  ├─►│  Node 3  │
         │ User A  │  │ User B  │  │  User C  │
         └─────────┘  └─────────┘  └──────────┘
              ◄──── ephemeral mesh ────►
                   (Phase 2, if needed)
```

**Channel subscription model (Phase 1):** Broadcast and discard. Every node receives every event, checks local WebSocket subscriptions, drops if irrelevant. At 3 nodes this is the correct answer — trivial, no subscription state to sync, fully leaderless.

### 5. Stats Collection & Admin Dashboard

Each heartbeat tick publishes runtime stats in the `stats` JSONB column:

```go
var m runtime.MemStats
runtime.ReadMemStats(&m)

stats := map[string]any{
    "goroutines":     runtime.NumGoroutine(),
    "heap_alloc":     m.HeapAlloc,
    "heap_sys":       m.HeapSys,
    "gc_cycles":      m.NumGC,
    "gc_pause_ns":    m.PauseNs[(m.NumGC+255)%256],
    "uptime_sec":     time.Since(startTime).Seconds(),
    "ws_clients":     wsHub.Count(),
    "extensions":     sandbox.LoadedCount(),
}
```

Admin dashboard API endpoint:

```sql
SELECT node_id, endpoint, registered_at, heartbeat, stats
FROM node_registry
ORDER BY seq;
```

**ICD envelope:** `{"data": [...]}` — consistent with list endpoint convention.

The admin dashboard extension renders one card per node. One node = one card. Three nodes = three cards. The extension has zero awareness of clustering — it reads a list and renders it. This is the extensibility proof working as designed.

Since `stats` is JSONB, future additions (mesh peer latency, extension execution counts, queue depths) require no schema migration. The dashboard extension renders whatever keys are present.

### 6. Self-Assembling Mesh Properties

The mesh builds itself from the registry with zero configuration:

| Event | Behavior |
|---|---|
| Node starts | Inserts into registry → peers discover it on next heartbeat tick |
| Node dies | Heartbeat goes stale → swept by peers → mesh shrinks |
| Node restarts | Re-registers → peers discover it → mesh restores |
| Scale up (K8s) | New replicas register → mesh grows automatically |
| Scale down (K8s) | Terminated nodes go stale → swept → mesh shrinks |
| PG crash | All nodes lose connection → all exit → K8s restarts all → clean re-registration |

No config files listing peers. No seed nodes. No service discovery sidecar. No bootstrap ceremony. The registry table IS the service mesh control plane.

---

## What This Enables (Future)

These are NOT in scope for this work. Listed to validate the design supports them without rearchitecting.

- **Work distribution:** Primary (if elected) assigns extension execution shards via registry metadata. Replicas poll for assignments. No message queue.
- **Realtime routing (v0.5.0):** When a node owns a pub/sub channel, it registers ownership in PG. Other nodes route publishes to the owner. Node dies → sweep clears routes → subscribers reconnect.
- **Rolling deploys:** New version registers, old version's heartbeat goes stale, clean handoff with zero coordination protocol.
- **Session affinity:** Registry tracks which node holds a user's WS connection. Targeted event delivery instead of broadcast (optimization for scale).

---

## Implementation Notes

### Migration

Pre-MVP: fold `CREATE UNLOGGED TABLE` into existing schema initialization. No new migration file needed. DB rebuild required (acceptable — no install base).

### Configuration

| Setting | Default | Notes |
|---|---|---|
| `CLUSTER_NODE_ID` | `hostname-PID` | Override for deterministic identity across restarts |
| `CLUSTER_HEARTBEAT_INTERVAL` | `10s` | Tick frequency |
| `CLUSTER_STALE_THRESHOLD` | `30s` | 3x heartbeat = stale |
| `CLUSTER_ENDPOINT` | auto-detect | Advertised address for peer mesh (Phase 2) |

### Single-Node Behavior

When only one node is registered, the system behaves identically to pre-cluster switchboard-core. The registry has one row. LISTEN/NOTIFY delivers events back to the same instance. No special-casing required.

### Health Endpoint Integration

`GET /health` includes cluster status:

```json
{
    "status": "healthy",
    "node_id": "node-abc-1234",
    "cluster": {
        "size": 3,
        "peers": ["node-def-5678", "node-ghi-9012"],
        "heartbeat_age_ms": 2340
    }
}
```

Satisfies the v0.6.0 MVP health monitoring requirement.

---

## Rejected Alternatives

| Alternative | Why rejected |
|---|---|
| etcd / Consul | Second consensus layer on top of PG. Doubles operational complexity for a system already tightly coupled to PG. Violates KISS. |
| Redis pub/sub for ephemeral events | Second stateful service. Two failure domains, two connection pools, two things to monitor. PG NOTIFY handles the same job at homelab scale. |
| Raft implementation in Go | Massive complexity for a problem PG serializable transactions already solve. |
| Ring topology monitoring | Each node watches the one above it. Adjacent failures create blind spots. Sweep-all is simpler and has no edge cases. |
| Leader election at startup | Unnecessary. Leaderless design with first-to-mark-wins task execution already works. Leader election is a future option, not a requirement. |

---

## Checklist

- [ ] `node_registry` unlogged table in schema init
- [ ] Node registration on startup
- [ ] Heartbeat tick with stats collection and stale sweep
- [ ] Self-eviction on 0-rows-affected heartbeat
- [ ] PG LISTEN/NOTIFY integration for realtime events
- [ ] Admin API endpoint: `GET /api/admin/cluster`
- [ ] Admin dashboard extension: cluster node cards
- [ ] Health endpoint cluster status
- [ ] Configuration via environment variables
- [ ] Single-node regression test (no behavioral change)
- [ ] Multi-node integration test (Docker Compose, 3 instances)