Feat v0.5.1 chat core (#31)
All checks were successful
All checks were successful
Co-authored-by: Jeffrey Smith <jasafpro@gmail.com> Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
This commit was merged in pull request #31.
This commit is contained in:
290
docs/DESIGN-cluster-registry.md
Normal file
290
docs/DESIGN-cluster-registry.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Design: PG-Backed Cluster Registry & Self-Assembling Mesh
|
||||
|
||||
**Status:** Proposed — Pre-MVP / MVP wrap-up
|
||||
**Author:** Jeff
|
||||
**Date:** 2026-03-30
|
||||
**Scope:** Kernel primitive — zero domain awareness
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Switchboard-core is tightly coupled to PostgreSQL. Scaling horizontally requires instance coordination: peer discovery, health monitoring, ephemeral event routing, and (optionally) leader election. Traditional HA solutions (Raft, etcd, Consul) introduce a second consensus layer on top of PG — doubling operational complexity for a system that already has serializable transactions and LISTEN/NOTIFY.
|
||||
|
||||
## Principle
|
||||
|
||||
PG is already the consensus system. Use it as the sole coordinator for service discovery, health, pub/sub, and cluster state. Zero new infrastructure dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Node Registry (Unlogged Table)
|
||||
|
||||
```sql
|
||||
CREATE UNLOGGED TABLE IF NOT EXISTS node_registry (
|
||||
node_id TEXT PRIMARY KEY,
|
||||
endpoint TEXT NOT NULL,
|
||||
seq SERIAL,
|
||||
registered_at TIMESTAMPTZ DEFAULT now(),
|
||||
heartbeat TIMESTAMPTZ DEFAULT now(),
|
||||
stats JSONB DEFAULT '{}'
|
||||
);
|
||||
```
|
||||
|
||||
**Why UNLOGGED:** No WAL overhead, visible to all connections, survives session disconnect. Contents do NOT survive a PG crash — but a PG crash means all nodes are dead anyway. Clean slate on recovery is the correct behavior; all nodes re-register on restart.
|
||||
|
||||
**Why not TEMPORARY:** PG temporary tables are session-scoped and invisible to other connections. Useless for cross-instance coordination.
|
||||
|
||||
### 2. Node Lifecycle
|
||||
|
||||
#### Startup
|
||||
|
||||
```
|
||||
boot → connect PG → register self → read peer list →
|
||||
subscribe LISTEN/NOTIFY channels → open WS to clients → ready
|
||||
```
|
||||
|
||||
```sql
|
||||
-- Self-registration (atomic, idempotent)
|
||||
INSERT INTO node_registry (node_id, endpoint)
|
||||
VALUES ($1, $2)
|
||||
ON CONFLICT (node_id) DO UPDATE
|
||||
SET endpoint = EXCLUDED.endpoint,
|
||||
registered_at = now(),
|
||||
heartbeat = now(),
|
||||
stats = '{}';
|
||||
```
|
||||
|
||||
`node_id` is generated at startup: `hostname + PID` or a UUID. Deterministic IDs enable restart-in-place without orphaned rows.
|
||||
|
||||
#### Heartbeat Tick (every N seconds, e.g., 10s)
|
||||
|
||||
Each node performs two operations per tick:
|
||||
|
||||
```go
|
||||
// 1. Update own heartbeat + stats
|
||||
stats := collectStats() // see §5
|
||||
db.Exec(`UPDATE node_registry SET heartbeat = now(), stats = $2 WHERE node_id = $1`,
|
||||
nodeID, stats)
|
||||
|
||||
// 2. Sweep stale entries (idempotent — safe for concurrent execution)
|
||||
db.Exec(`DELETE FROM node_registry WHERE heartbeat < now() - interval '30 seconds'`)
|
||||
```
|
||||
|
||||
The sweep is the distributed health check. Every node runs it. Multiple nodes deleting the same stale row is a no-op. No ring topology required — the sweep is the safety net that guarantees eventual cleanup regardless of which nodes are alive.
|
||||
|
||||
#### Self-Eviction
|
||||
|
||||
If a node's own heartbeat UPDATE returns 0 rows affected, it has been swept by a peer:
|
||||
|
||||
```go
|
||||
result := db.Exec(`UPDATE node_registry SET heartbeat = now() WHERE node_id = $1`, nodeID)
|
||||
if result.RowsAffected() == 0 {
|
||||
log.Error("evicted from registry, shutting down")
|
||||
os.Exit(1)
|
||||
}
|
||||
```
|
||||
|
||||
Kubernetes restarts the process. On boot it re-registers as a fresh node. The restart loop IS the recovery mechanism.
|
||||
|
||||
### 3. Leaderless by Default
|
||||
|
||||
No `role` column. No primary/replica distinction. All nodes are equal peers. PG is the sole coordinator.
|
||||
|
||||
**If a leader is ever needed** (future use case), it's a small lift:
|
||||
|
||||
```sql
|
||||
ALTER TABLE node_registry ADD COLUMN role TEXT DEFAULT 'peer';
|
||||
|
||||
-- Atomic leader claim: first writer wins
|
||||
UPDATE node_registry SET role = 'primary'
|
||||
WHERE node_id = $1
|
||||
AND NOT EXISTS (SELECT 1 FROM node_registry WHERE role = 'primary');
|
||||
```
|
||||
|
||||
No quorum, no voting, no term numbers. PG serialization handles it. But the current design intentionally avoids this — consistent with the existing pattern where scheduled tasks use first-to-mark-wins execution, not leader assignment.
|
||||
|
||||
### 4. Realtime Event Routing (Two Tiers)
|
||||
|
||||
#### Tier 1: Durable Events (messages, state changes)
|
||||
|
||||
These are PG writes. Fan-out via `LISTEN/NOTIFY`:
|
||||
|
||||
```
|
||||
User A sends message → Instance 1 writes to PG →
|
||||
pg_notify('channel:xyz', '{"type":"message","id":"..."}') →
|
||||
All instances receive → push to local WS clients subscribed to channel:xyz
|
||||
```
|
||||
|
||||
This maps directly to the v0.5.0 `realtime.publish()` Starlark primitive. Kernel implementation is `pg_notify(channel, payload)`.
|
||||
|
||||
#### Tier 2: Ephemeral Events (typing indicators, presence, cursor position)
|
||||
|
||||
These NEVER hit durable storage.
|
||||
|
||||
**Phase 1 (MVP):** Use PG LISTEN/NOTIFY for ephemeral events too.
|
||||
|
||||
```
|
||||
User A types → Instance 1 WS handler →
|
||||
pg_notify('ephemeral:xyz', '{"type":"typing","user":"A"}') →
|
||||
Instance 2 receives → pushes to User B's WS
|
||||
Instance 3 receives → pushes to User C's WS
|
||||
```
|
||||
|
||||
PG NOTIFY payload limit is 8KB. A typing indicator is ~60 bytes. At homelab scale (3 nodes, dozens of users) this is a non-issue. One codepath, one transport, zero new infrastructure.
|
||||
|
||||
**Phase 2 (Post-MVP, if needed):** Direct instance-to-instance mesh for ephemeral events.
|
||||
|
||||
- Peer endpoints are already in the registry
|
||||
- Each node dials peers discovered from the registry on startup and on peer list changes
|
||||
- Ephemeral events route over the mesh; durable events stay on PG NOTIFY
|
||||
- The `realtime.publish()` Starlark API does not change — kernel swaps the underlying transport
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ PG (source of truth) │
|
||||
│ ┌───────────┐ ┌──────────────┐ │
|
||||
│ │ node_ │ │ LISTEN/ │ │
|
||||
│ │ registry │ │ NOTIFY │ │
|
||||
│ └───────────┘ └──────────────┘ │
|
||||
└──────┬──────────────┬────────────┘
|
||||
│ durable │
|
||||
┌────────────┼──────────────┼────────────┐
|
||||
│ │ │ │
|
||||
┌────┴────┐ ┌───┴─────┐ ┌────┴─────┐
|
||||
│ Node 1 │◄─┤ Node 2 ├─►│ Node 3 │
|
||||
│ User A │ │ User B │ │ User C │
|
||||
└─────────┘ └─────────┘ └──────────┘
|
||||
◄──── ephemeral mesh ────►
|
||||
(Phase 2, if needed)
|
||||
```
|
||||
|
||||
**Channel subscription model (Phase 1):** Broadcast and discard. Every node receives every event, checks local WebSocket subscriptions, drops if irrelevant. At 3 nodes this is the correct answer — trivial, no subscription state to sync, fully leaderless.
|
||||
|
||||
### 5. Stats Collection & Admin Dashboard
|
||||
|
||||
Each heartbeat tick publishes runtime stats in the `stats` JSONB column:
|
||||
|
||||
```go
|
||||
var m runtime.MemStats
|
||||
runtime.ReadMemStats(&m)
|
||||
|
||||
stats := map[string]any{
|
||||
"goroutines": runtime.NumGoroutine(),
|
||||
"heap_alloc": m.HeapAlloc,
|
||||
"heap_sys": m.HeapSys,
|
||||
"gc_cycles": m.NumGC,
|
||||
"gc_pause_ns": m.PauseNs[(m.NumGC+255)%256],
|
||||
"uptime_sec": time.Since(startTime).Seconds(),
|
||||
"ws_clients": wsHub.Count(),
|
||||
"extensions": sandbox.LoadedCount(),
|
||||
}
|
||||
```
|
||||
|
||||
Admin dashboard API endpoint:
|
||||
|
||||
```sql
|
||||
SELECT node_id, endpoint, registered_at, heartbeat, stats
|
||||
FROM node_registry
|
||||
ORDER BY seq;
|
||||
```
|
||||
|
||||
**ICD envelope:** `{"data": [...]}` — consistent with list endpoint convention.
|
||||
|
||||
The admin dashboard extension renders one card per node. One node = one card. Three nodes = three cards. The extension has zero awareness of clustering — it reads a list and renders it. This is the extensibility proof working as designed.
|
||||
|
||||
Since `stats` is JSONB, future additions (mesh peer latency, extension execution counts, queue depths) require no schema migration. The dashboard extension renders whatever keys are present.
|
||||
|
||||
### 6. Self-Assembling Mesh Properties
|
||||
|
||||
The mesh builds itself from the registry with zero configuration:
|
||||
|
||||
| Event | Behavior |
|
||||
|---|---|
|
||||
| Node starts | Inserts into registry → peers discover it on next heartbeat tick |
|
||||
| Node dies | Heartbeat goes stale → swept by peers → mesh shrinks |
|
||||
| Node restarts | Re-registers → peers discover it → mesh restores |
|
||||
| Scale up (K8s) | New replicas register → mesh grows automatically |
|
||||
| Scale down (K8s) | Terminated nodes go stale → swept → mesh shrinks |
|
||||
| PG crash | All nodes lose connection → all exit → K8s restarts all → clean re-registration |
|
||||
|
||||
No config files listing peers. No seed nodes. No service discovery sidecar. No bootstrap ceremony. The registry table IS the service mesh control plane.
|
||||
|
||||
---
|
||||
|
||||
## What This Enables (Future)
|
||||
|
||||
These are NOT in scope for this work. Listed to validate the design supports them without rearchitecting.
|
||||
|
||||
- **Work distribution:** Primary (if elected) assigns extension execution shards via registry metadata. Replicas poll for assignments. No message queue.
|
||||
- **Realtime routing (v0.5.0):** When a node owns a pub/sub channel, it registers ownership in PG. Other nodes route publishes to the owner. Node dies → sweep clears routes → subscribers reconnect.
|
||||
- **Rolling deploys:** New version registers, old version's heartbeat goes stale, clean handoff with zero coordination protocol.
|
||||
- **Session affinity:** Registry tracks which node holds a user's WS connection. Targeted event delivery instead of broadcast (optimization for scale).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Migration
|
||||
|
||||
Pre-MVP: fold `CREATE UNLOGGED TABLE` into existing schema initialization. No new migration file needed. DB rebuild required (acceptable — no install base).
|
||||
|
||||
### Configuration
|
||||
|
||||
| Setting | Default | Notes |
|
||||
|---|---|---|
|
||||
| `CLUSTER_NODE_ID` | `hostname-PID` | Override for deterministic identity across restarts |
|
||||
| `CLUSTER_HEARTBEAT_INTERVAL` | `10s` | Tick frequency |
|
||||
| `CLUSTER_STALE_THRESHOLD` | `30s` | 3x heartbeat = stale |
|
||||
| `CLUSTER_ENDPOINT` | auto-detect | Advertised address for peer mesh (Phase 2) |
|
||||
|
||||
### Single-Node Behavior
|
||||
|
||||
When only one node is registered, the system behaves identically to pre-cluster switchboard-core. The registry has one row. LISTEN/NOTIFY delivers events back to the same instance. No special-casing required.
|
||||
|
||||
### Health Endpoint Integration
|
||||
|
||||
`GET /health` includes cluster status:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"node_id": "node-abc-1234",
|
||||
"cluster": {
|
||||
"size": 3,
|
||||
"peers": ["node-def-5678", "node-ghi-9012"],
|
||||
"heartbeat_age_ms": 2340
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Satisfies the v0.6.0 MVP health monitoring requirement.
|
||||
|
||||
---
|
||||
|
||||
## Rejected Alternatives
|
||||
|
||||
| Alternative | Why rejected |
|
||||
|---|---|
|
||||
| etcd / Consul | Second consensus layer on top of PG. Doubles operational complexity for a system already tightly coupled to PG. Violates KISS. |
|
||||
| Redis pub/sub for ephemeral events | Second stateful service. Two failure domains, two connection pools, two things to monitor. PG NOTIFY handles the same job at homelab scale. |
|
||||
| Raft implementation in Go | Massive complexity for a problem PG serializable transactions already solve. |
|
||||
| Ring topology monitoring | Each node watches the one above it. Adjacent failures create blind spots. Sweep-all is simpler and has no edge cases. |
|
||||
| Leader election at startup | Unnecessary. Leaderless design with first-to-mark-wins task execution already works. Leader election is a future option, not a requirement. |
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] `node_registry` unlogged table in schema init
|
||||
- [ ] Node registration on startup
|
||||
- [ ] Heartbeat tick with stats collection and stale sweep
|
||||
- [ ] Self-eviction on 0-rows-affected heartbeat
|
||||
- [ ] PG LISTEN/NOTIFY integration for realtime events
|
||||
- [ ] Admin API endpoint: `GET /api/admin/cluster`
|
||||
- [ ] Admin dashboard extension: cluster node cards
|
||||
- [ ] Health endpoint cluster status
|
||||
- [ ] Configuration via environment variables
|
||||
- [ ] Single-node regression test (no behavioral change)
|
||||
- [ ] Multi-node integration test (Docker Compose, 3 instances)
|
||||
Reference in New Issue
Block a user