Changeset 0.32.0 (#206)

docs/ROADMAP.md

@@ -30,7 +30,7 @@ v0.9.x–v0.28.7  Foundation through Platform Polish          ✅
         │                               │
    Extension Track                 Operations Track
         │                               │
-   v0.29.0 Starlark Sandbox  ✅  v0.32.0 Multi-Replica HA
+   v0.29.0 Starlark Sandbox  ✅  v0.32.0 Multi-Replica HA  ✅
    v0.29.1 API Extensions   ✅  v0.33.0 Observability
    v0.29.2 DB Extensions    ✅   v0.34.0 Data Portability
    v0.29.3 Workflow Forms   ✅          │
@@ -39,7 +39,7 @@ v0.9.x–v0.28.7  Foundation through Platform Polish          ✅
    v0.30.2 Workflow Packages ✅         │
    v0.31.0 Editor + SDK     ✅         │
    v0.31.1 SDK Exercise     ✅         │
-   v0.31.2 Team Workflows            │
+   v0.31.2 Team Workflows   ✅         │
         │                               │
         │                          v0.35.0 Workflow Product
         │                               │
@@ -350,43 +350,39 @@ Depends on: v0.31.1.
 Parallel to extension track. No Starlark dependency. Delivers
 production readiness for the target multi-team deployment.
 
-### v0.32.0 — Multi-Replica HA
+### v0.32.0 — Multi-Replica HA ✅
 
 Run 2–3 backend replicas across nodes for node-level availability.
 
 Depends on: v0.28.8.
 
+**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
+leader election. Every replica runs the scheduler poll loop; PG
+serializes task claims atomically. No K8s API dependency, no Redis,
+no new coordination infrastructure. See `docs/DESIGN-0.32.0.md`.
+
 **What already works multi-replica:**
 - REST API (stateless, JWT auth) ✅
 - PG + S3 + CephFS (shared infrastructure) ✅
 - `pg_broadcast` LISTEN/NOTIFY (cross-pod event bus) ✅
 
-**What needs work:**
-- [ ] WebSocket fan-out: wire `Bus.Subscribe` to `pg_broadcast` listener
-      so events from other pods reach local WebSocket connections.
-      LISTEN/NOTIFY plumbing exists — bridge inbound NOTIFY into per-pod
-      hub for local delivery.
-- [ ] Task scheduler leader election: Kubernetes `Lease`-based. Only
-      leader runs cron scheduler. Prevents duplicate execution.
-- [ ] Shared ticket store: `TicketStore` from `sync.Map` → PG table
-      with 30s TTL. Ensures WS ticket from pod-1 validates on pod-2.
-- [ ] Shared rate limiter: in-memory → PG or Redis counter. Without
-      this, effective rate limit = N × configured across N replicas.
-- [ ] Health check refinement: readiness probe fails fast on PG
-      connection loss.
-- [ ] Helm: `backend.replicaCount` > 1 tested. Pod anti-affinity
-      to spread across nodes.
+**Delivered (5 changesets):**
+- [x] Task scheduler: `FOR UPDATE SKIP LOCKED` atomic claim replaces
+      `ListDue`. `CreateRunExclusive` conditional insert for
+      belt-and-suspenders uniqueness. Startup jitter (0–15s).
+- [x] WebSocket cross-pod delivery: `PublishToUser` routes through
+      bus → `pg_broadcast` → remote pod. 18 `SendToUser` call sites
+      migrated. `tool.result.*` routing changed to `DirBoth` for
+      cross-pod `WaitFor`. `TargetUserID` field on `Event`.
+- [x] Shared ticket store: `ws_tickets` PG table with 30s TTL,
+      atomic `DELETE ... RETURNING` validation. Replaces `sync.Map`.
+- [x] Shared rate limiter: `rate_limit_counters` PG table with
+      fixed-window counters. Fail-open policy on DB errors.
+- [x] Health probes: `/healthz/ready` (PG ping, 2s timeout),
+      `/healthz/live` (process alive). Helm: `replicaCount: 2`,
+      pod anti-affinity, readiness/liveness probes.
 
-**Sizing (measured on v0.28.8):**
-
-| Metric | Value |
-|--------|-------|
-| Idle memory | 17Mi |
-| Peak (569 reqs, 4 users, SSE) | 216Mi |
-| Settled after GC | 34Mi |
-| Per-SSE stream | ~500KB–1MB |
-| Per-WebSocket | ~64KB |
-| DB pool | 25 max (shared) |
+**Schema:** `020_ha.sql` — `ws_tickets`, `rate_limit_counters`.
 
 ### v0.33.0 — Observability