Jeffrey Smith
680ec3b897
All checks were successful
Co-authored-by: Jeffrey Smith <jasafpro@gmail.com> Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
291 lines
12 KiB
Markdown
291 lines
12 KiB
Markdown
# Design: PG-Backed Cluster Registry & Self-Assembling Mesh
|
|
|
|
**Status:** Proposed — Pre-MVP / MVP wrap-up
|
|
**Author:** Jeff
|
|
**Date:** 2026-03-30
|
|
**Scope:** Kernel primitive — zero domain awareness
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
Armature-core is tightly coupled to PostgreSQL. Scaling horizontally requires instance coordination: peer discovery, health monitoring, ephemeral event routing, and (optionally) leader election. Traditional HA solutions (Raft, etcd, Consul) introduce a second consensus layer on top of PG — doubling operational complexity for a system that already has serializable transactions and LISTEN/NOTIFY.
|
|
|
|
## Principle
|
|
|
|
PG is already the consensus system. Use it as the sole coordinator for service discovery, health, pub/sub, and cluster state. Zero new infrastructure dependencies.
|
|
|
|
---
|
|
|
|
## Design
|
|
|
|
### 1. Node Registry (Unlogged Table)
|
|
|
|
```sql
|
|
CREATE UNLOGGED TABLE IF NOT EXISTS node_registry (
|
|
node_id TEXT PRIMARY KEY,
|
|
endpoint TEXT NOT NULL,
|
|
seq SERIAL,
|
|
registered_at TIMESTAMPTZ DEFAULT now(),
|
|
heartbeat TIMESTAMPTZ DEFAULT now(),
|
|
stats JSONB DEFAULT '{}'
|
|
);
|
|
```
|
|
|
|
**Why UNLOGGED:** No WAL overhead, visible to all connections, survives session disconnect. Contents do NOT survive a PG crash — but a PG crash means all nodes are dead anyway. Clean slate on recovery is the correct behavior; all nodes re-register on restart.
|
|
|
|
**Why not TEMPORARY:** PG temporary tables are session-scoped and invisible to other connections. Useless for cross-instance coordination.
|
|
|
|
### 2. Node Lifecycle
|
|
|
|
#### Startup
|
|
|
|
```
|
|
boot → connect PG → register self → read peer list →
|
|
subscribe LISTEN/NOTIFY channels → open WS to clients → ready
|
|
```
|
|
|
|
```sql
|
|
-- Self-registration (atomic, idempotent)
|
|
INSERT INTO node_registry (node_id, endpoint)
|
|
VALUES ($1, $2)
|
|
ON CONFLICT (node_id) DO UPDATE
|
|
SET endpoint = EXCLUDED.endpoint,
|
|
registered_at = now(),
|
|
heartbeat = now(),
|
|
stats = '{}';
|
|
```
|
|
|
|
`node_id` is generated at startup: `hostname + PID` or a UUID. Deterministic IDs enable restart-in-place without orphaned rows.
|
|
|
|
#### Heartbeat Tick (every N seconds, e.g., 10s)
|
|
|
|
Each node performs two operations per tick:
|
|
|
|
```go
|
|
// 1. Update own heartbeat + stats
|
|
stats := collectStats() // see §5
|
|
db.Exec(`UPDATE node_registry SET heartbeat = now(), stats = $2 WHERE node_id = $1`,
|
|
nodeID, stats)
|
|
|
|
// 2. Sweep stale entries (idempotent — safe for concurrent execution)
|
|
db.Exec(`DELETE FROM node_registry WHERE heartbeat < now() - interval '30 seconds'`)
|
|
```
|
|
|
|
The sweep is the distributed health check. Every node runs it. Multiple nodes deleting the same stale row is a no-op. No ring topology required — the sweep is the safety net that guarantees eventual cleanup regardless of which nodes are alive.
|
|
|
|
#### Self-Eviction
|
|
|
|
If a node's own heartbeat UPDATE returns 0 rows affected, it has been swept by a peer:
|
|
|
|
```go
|
|
result := db.Exec(`UPDATE node_registry SET heartbeat = now() WHERE node_id = $1`, nodeID)
|
|
if result.RowsAffected() == 0 {
|
|
log.Error("evicted from registry, shutting down")
|
|
os.Exit(1)
|
|
}
|
|
```
|
|
|
|
Kubernetes restarts the process. On boot it re-registers as a fresh node. The restart loop IS the recovery mechanism.
|
|
|
|
### 3. Leaderless by Default
|
|
|
|
No `role` column. No primary/replica distinction. All nodes are equal peers. PG is the sole coordinator.
|
|
|
|
**If a leader is ever needed** (future use case), it's a small lift:
|
|
|
|
```sql
|
|
ALTER TABLE node_registry ADD COLUMN role TEXT DEFAULT 'peer';
|
|
|
|
-- Atomic leader claim: first writer wins
|
|
UPDATE node_registry SET role = 'primary'
|
|
WHERE node_id = $1
|
|
AND NOT EXISTS (SELECT 1 FROM node_registry WHERE role = 'primary');
|
|
```
|
|
|
|
No quorum, no voting, no term numbers. PG serialization handles it. But the current design intentionally avoids this — consistent with the existing pattern where scheduled tasks use first-to-mark-wins execution, not leader assignment.
|
|
|
|
### 4. Realtime Event Routing (Two Tiers)
|
|
|
|
#### Tier 1: Durable Events (messages, state changes)
|
|
|
|
These are PG writes. Fan-out via `LISTEN/NOTIFY`:
|
|
|
|
```
|
|
User A sends message → Instance 1 writes to PG →
|
|
pg_notify('channel:xyz', '{"type":"message","id":"..."}') →
|
|
All instances receive → push to local WS clients subscribed to channel:xyz
|
|
```
|
|
|
|
This maps directly to the v0.5.0 `realtime.publish()` Starlark primitive. Kernel implementation is `pg_notify(channel, payload)`.
|
|
|
|
#### Tier 2: Ephemeral Events (typing indicators, presence, cursor position)
|
|
|
|
These NEVER hit durable storage.
|
|
|
|
**Phase 1 (MVP):** Use PG LISTEN/NOTIFY for ephemeral events too.
|
|
|
|
```
|
|
User A types → Instance 1 WS handler →
|
|
pg_notify('ephemeral:xyz', '{"type":"typing","user":"A"}') →
|
|
Instance 2 receives → pushes to User B's WS
|
|
Instance 3 receives → pushes to User C's WS
|
|
```
|
|
|
|
PG NOTIFY payload limit is 8KB. A typing indicator is ~60 bytes. At homelab scale (3 nodes, dozens of users) this is a non-issue. One codepath, one transport, zero new infrastructure.
|
|
|
|
**Phase 2 (Post-MVP, if needed):** Direct instance-to-instance mesh for ephemeral events.
|
|
|
|
- Peer endpoints are already in the registry
|
|
- Each node dials peers discovered from the registry on startup and on peer list changes
|
|
- Ephemeral events route over the mesh; durable events stay on PG NOTIFY
|
|
- The `realtime.publish()` Starlark API does not change — kernel swaps the underlying transport
|
|
|
|
```
|
|
┌─────────────────────────────────┐
|
|
│ PG (source of truth) │
|
|
│ ┌───────────┐ ┌──────────────┐ │
|
|
│ │ node_ │ │ LISTEN/ │ │
|
|
│ │ registry │ │ NOTIFY │ │
|
|
│ └───────────┘ └──────────────┘ │
|
|
└──────┬──────────────┬────────────┘
|
|
│ durable │
|
|
┌────────────┼──────────────┼────────────┐
|
|
│ │ │ │
|
|
┌────┴────┐ ┌───┴─────┐ ┌────┴─────┐
|
|
│ Node 1 │◄─┤ Node 2 ├─►│ Node 3 │
|
|
│ User A │ │ User B │ │ User C │
|
|
└─────────┘ └─────────┘ └──────────┘
|
|
◄──── ephemeral mesh ────►
|
|
(Phase 2, if needed)
|
|
```
|
|
|
|
**Channel subscription model (Phase 1):** Broadcast and discard. Every node receives every event, checks local WebSocket subscriptions, drops if irrelevant. At 3 nodes this is the correct answer — trivial, no subscription state to sync, fully leaderless.
|
|
|
|
### 5. Stats Collection & Admin Dashboard
|
|
|
|
Each heartbeat tick publishes runtime stats in the `stats` JSONB column:
|
|
|
|
```go
|
|
var m runtime.MemStats
|
|
runtime.ReadMemStats(&m)
|
|
|
|
stats := map[string]any{
|
|
"goroutines": runtime.NumGoroutine(),
|
|
"heap_alloc": m.HeapAlloc,
|
|
"heap_sys": m.HeapSys,
|
|
"gc_cycles": m.NumGC,
|
|
"gc_pause_ns": m.PauseNs[(m.NumGC+255)%256],
|
|
"uptime_sec": time.Since(startTime).Seconds(),
|
|
"ws_clients": wsHub.Count(),
|
|
"extensions": sandbox.LoadedCount(),
|
|
}
|
|
```
|
|
|
|
Admin dashboard API endpoint:
|
|
|
|
```sql
|
|
SELECT node_id, endpoint, registered_at, heartbeat, stats
|
|
FROM node_registry
|
|
ORDER BY seq;
|
|
```
|
|
|
|
**ICD envelope:** `{"data": [...]}` — consistent with list endpoint convention.
|
|
|
|
The admin dashboard extension renders one card per node. One node = one card. Three nodes = three cards. The extension has zero awareness of clustering — it reads a list and renders it. This is the extensibility proof working as designed.
|
|
|
|
Since `stats` is JSONB, future additions (mesh peer latency, extension execution counts, queue depths) require no schema migration. The dashboard extension renders whatever keys are present.
|
|
|
|
### 6. Self-Assembling Mesh Properties
|
|
|
|
The mesh builds itself from the registry with zero configuration:
|
|
|
|
| Event | Behavior |
|
|
|---|---|
|
|
| Node starts | Inserts into registry → peers discover it on next heartbeat tick |
|
|
| Node dies | Heartbeat goes stale → swept by peers → mesh shrinks |
|
|
| Node restarts | Re-registers → peers discover it → mesh restores |
|
|
| Scale up (K8s) | New replicas register → mesh grows automatically |
|
|
| Scale down (K8s) | Terminated nodes go stale → swept → mesh shrinks |
|
|
| PG crash | All nodes lose connection → all exit → K8s restarts all → clean re-registration |
|
|
|
|
No config files listing peers. No seed nodes. No service discovery sidecar. No bootstrap ceremony. The registry table IS the service mesh control plane.
|
|
|
|
---
|
|
|
|
## What This Enables (Future)
|
|
|
|
These are NOT in scope for this work. Listed to validate the design supports them without rearchitecting.
|
|
|
|
- **Work distribution:** Primary (if elected) assigns extension execution shards via registry metadata. Replicas poll for assignments. No message queue.
|
|
- **Realtime routing (v0.5.0):** When a node owns a pub/sub channel, it registers ownership in PG. Other nodes route publishes to the owner. Node dies → sweep clears routes → subscribers reconnect.
|
|
- **Rolling deploys:** New version registers, old version's heartbeat goes stale, clean handoff with zero coordination protocol.
|
|
- **Session affinity:** Registry tracks which node holds a user's WS connection. Targeted event delivery instead of broadcast (optimization for scale).
|
|
|
|
---
|
|
|
|
## Implementation Notes
|
|
|
|
### Migration
|
|
|
|
Pre-MVP: fold `CREATE UNLOGGED TABLE` into existing schema initialization. No new migration file needed. DB rebuild required (acceptable — no install base).
|
|
|
|
### Configuration
|
|
|
|
| Setting | Default | Notes |
|
|
|---|---|---|
|
|
| `CLUSTER_NODE_ID` | `hostname-PID` | Override for deterministic identity across restarts |
|
|
| `CLUSTER_HEARTBEAT_INTERVAL` | `10s` | Tick frequency |
|
|
| `CLUSTER_STALE_THRESHOLD` | `30s` | 3x heartbeat = stale |
|
|
| `CLUSTER_ENDPOINT` | auto-detect | Advertised address for peer mesh (Phase 2) |
|
|
|
|
### Single-Node Behavior
|
|
|
|
When only one node is registered, the system behaves identically to pre-cluster armature. The registry has one row. LISTEN/NOTIFY delivers events back to the same instance. No special-casing required.
|
|
|
|
### Health Endpoint Integration
|
|
|
|
`GET /health` includes cluster status:
|
|
|
|
```json
|
|
{
|
|
"status": "healthy",
|
|
"node_id": "node-abc-1234",
|
|
"cluster": {
|
|
"size": 3,
|
|
"peers": ["node-def-5678", "node-ghi-9012"],
|
|
"heartbeat_age_ms": 2340
|
|
}
|
|
}
|
|
```
|
|
|
|
Satisfies the v0.6.0 MVP health monitoring requirement.
|
|
|
|
---
|
|
|
|
## Rejected Alternatives
|
|
|
|
| Alternative | Why rejected |
|
|
|---|---|
|
|
| etcd / Consul | Second consensus layer on top of PG. Doubles operational complexity for a system already tightly coupled to PG. Violates KISS. |
|
|
| Redis pub/sub for ephemeral events | Second stateful service. Two failure domains, two connection pools, two things to monitor. PG NOTIFY handles the same job at homelab scale. |
|
|
| Raft implementation in Go | Massive complexity for a problem PG serializable transactions already solve. |
|
|
| Ring topology monitoring | Each node watches the one above it. Adjacent failures create blind spots. Sweep-all is simpler and has no edge cases. |
|
|
| Leader election at startup | Unnecessary. Leaderless design with first-to-mark-wins task execution already works. Leader election is a future option, not a requirement. |
|
|
|
|
---
|
|
|
|
## Checklist
|
|
|
|
- [ ] `node_registry` unlogged table in schema init
|
|
- [ ] Node registration on startup
|
|
- [ ] Heartbeat tick with stats collection and stale sweep
|
|
- [ ] Self-eviction on 0-rows-affected heartbeat
|
|
- [ ] PG LISTEN/NOTIFY integration for realtime events
|
|
- [ ] Admin API endpoint: `GET /api/admin/cluster`
|
|
- [ ] Admin dashboard extension: cluster node cards
|
|
- [ ] Health endpoint cluster status
|
|
- [ ] Configuration via environment variables
|
|
- [ ] Single-node regression test (no behavioral change)
|
|
- [ ] Multi-node integration test (Docker Compose, 3 instances)
|