Changeset 0.32.0 (#206)
This commit is contained in:
810
docs/DESIGN-0.32.0.md
Normal file
810
docs/DESIGN-0.32.0.md
Normal file
@@ -0,0 +1,810 @@
|
||||
# DESIGN-0.32.0 — Multi-Replica HA
|
||||
|
||||
Run 2–3 backend replicas across nodes for node-level availability.
|
||||
Five changesets. No sub-versions — the work is tightly coupled and
|
||||
each CS builds on the previous.
|
||||
|
||||
Depends on: v0.31.2 (current HEAD).
|
||||
|
||||
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
|
||||
leader election for the task scheduler. Every replica polls; PG
|
||||
serializes the claims. Extends to all shared mutable state — tickets,
|
||||
rate counters — keeping PG as the sole coordination layer. No Redis,
|
||||
no K8s API dependency, no new infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## What Already Works Multi-Replica
|
||||
|
||||
These require zero changes:
|
||||
|
||||
- **REST API** — stateless, JWT auth, any replica serves any request.
|
||||
- **PG + S3 + CephFS** — shared storage infrastructure.
|
||||
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
|
||||
→ `broadcastHook` → `pg_notify` → remote pod `publishLocal()` →
|
||||
local WS subscribers. Fully wired, just never tested at N>1.
|
||||
|
||||
---
|
||||
|
||||
## What Needs Work
|
||||
|
||||
Five areas of in-process mutable state that break at replica count > 1:
|
||||
|
||||
| State | Current | Problem at N>1 | Fix |
|
||||
|-------|---------|----------------|-----|
|
||||
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
|
||||
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
|
||||
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
|
||||
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
|
||||
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Schema: `020_ha.sql`
|
||||
|
||||
New migration for both PG and SQLite. No changes to existing 019
|
||||
task schema — `next_run_at` nullable already supports the claim
|
||||
mechanism.
|
||||
|
||||
### PG: `server/database/migrations/020_ha.sql`
|
||||
|
||||
```sql
|
||||
-- ==========================================
|
||||
-- Chat Switchboard — 020 Multi-Replica HA
|
||||
-- ==========================================
|
||||
-- Shared state tables for multi-replica operation.
|
||||
-- v0.32.0: ws_tickets, rate_limit_counters.
|
||||
-- ==========================================
|
||||
|
||||
-- =========================================
|
||||
-- WEBSOCKET TICKETS (replaces sync.Map)
|
||||
-- =========================================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||||
id TEXT PRIMARY KEY, -- 128-bit hex token
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||||
ON ws_tickets (expires_at);
|
||||
|
||||
-- =========================================
|
||||
-- RATE LIMIT COUNTERS (replaces in-memory)
|
||||
-- =========================================
|
||||
-- Token bucket approximation using time-windowed counters.
|
||||
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
|
||||
-- Window is truncated to the second for the configured rate.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||||
key TEXT NOT NULL,
|
||||
window TIMESTAMPTZ NOT NULL, -- truncated timestamp (window start)
|
||||
tokens REAL NOT NULL DEFAULT 0, -- tokens consumed in this window
|
||||
PRIMARY KEY (key, window)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||||
ON rate_limit_counters (window);
|
||||
```
|
||||
|
||||
### SQLite: `server/database/migrations/sqlite/020_ha.sql`
|
||||
|
||||
SQLite parity — structurally identical but with dialect adjustments.
|
||||
These tables are functional in SQLite for single-process test parity,
|
||||
though multi-replica is PG-only in production.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||||
ON ws_tickets (expires_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||||
key TEXT NOT NULL,
|
||||
window TEXT NOT NULL,
|
||||
tokens REAL NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (key, window)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||||
ON rate_limit_counters (window);
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [x] `020_ha.sql` (PG)
|
||||
- [x] `020_ha.sql` (SQLite)
|
||||
- [x] CI green on both pipelines (migration runs, tables exist)
|
||||
|
||||
---
|
||||
|
||||
## CS1 — Task Scheduler: SKIP LOCKED
|
||||
|
||||
Replace the read-then-execute scheduler with atomic claim semantics.
|
||||
Every replica runs the poll loop. PG serializes task handoff.
|
||||
|
||||
### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)
|
||||
|
||||
New store method. Single atomic statement — SELECT + UPDATE in one
|
||||
round trip. Returns at most one task per call.
|
||||
|
||||
**PG implementation:**
|
||||
|
||||
```sql
|
||||
UPDATE tasks
|
||||
SET next_run_at = NULL
|
||||
WHERE id = (
|
||||
SELECT id FROM tasks
|
||||
WHERE is_active = true
|
||||
AND next_run_at <= NOW()
|
||||
ORDER BY next_run_at ASC
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING <taskColumns>
|
||||
```
|
||||
|
||||
Mechanics:
|
||||
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
|
||||
candidate row, skip it instantly (no wait, no deadlock).
|
||||
- `SET next_run_at = NULL` — claimed task disappears from future
|
||||
polls. `advanceNextRun` restores it after execution.
|
||||
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
|
||||
- The existing partial index `idx_tasks_next_run` covers the WHERE
|
||||
clause (`is_active = true AND next_run_at IS NOT NULL` by
|
||||
implication — NULL rows won't satisfy `<= NOW()`).
|
||||
|
||||
**SQLite implementation:**
|
||||
|
||||
SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
|
||||
the current `ListDue` behavior unchanged for SQLite. The store
|
||||
interface accommodates both:
|
||||
|
||||
```go
|
||||
// TaskStore additions
|
||||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||||
```
|
||||
|
||||
SQLite `ClaimDueTask` just does:
|
||||
```sql
|
||||
SELECT <taskColumns> FROM tasks
|
||||
WHERE is_active = 1 AND next_run_at <= datetime('now')
|
||||
ORDER BY next_run_at ASC LIMIT 1
|
||||
```
|
||||
Then sets `next_run_at = NULL` in a second statement (single writer,
|
||||
no contention).
|
||||
|
||||
### 1.2 — Conditional `CreateRunExclusive`
|
||||
|
||||
Belt-and-suspenders: prevent double-execution even if two replicas
|
||||
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
|
||||
but defense in depth).
|
||||
|
||||
```sql
|
||||
INSERT INTO task_runs (task_id, status)
|
||||
SELECT $1, 'running'
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM task_runs
|
||||
WHERE task_id = $1 AND status IN ('running', 'queued')
|
||||
)
|
||||
RETURNING id, started_at
|
||||
```
|
||||
|
||||
Returns `sql.ErrNoRows` if a run already exists → skip execution.
|
||||
|
||||
New store method:
|
||||
```go
|
||||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
```
|
||||
|
||||
### 1.3 — Scheduler Loop Rewrite
|
||||
|
||||
`scheduler.go` changes:
|
||||
|
||||
```go
|
||||
func (s *Scheduler) poll() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
|
||||
cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
|
||||
if !cfg.Enabled {
|
||||
return
|
||||
}
|
||||
|
||||
// Claim tasks one at a time until none remain or budget exhausted.
|
||||
// Each replica claims independently — PG SKIP LOCKED serializes.
|
||||
claimed := 0
|
||||
for claimed < cfg.MaxConcurrent {
|
||||
task, err := s.stores.Tasks.ClaimDueTask(ctx)
|
||||
if err != nil {
|
||||
break // sql.ErrNoRows or real error — either way, done
|
||||
}
|
||||
claimed++
|
||||
go s.execute(ctx, task)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute()` changes:
|
||||
- Remove the `GetActiveRun` TOCTOU check — replaced by
|
||||
`CreateRunExclusive`.
|
||||
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
|
||||
But adoption also uses `CreateRunExclusive` semantics (transition
|
||||
only if still `queued`).
|
||||
- `advanceNextRun` still computes and sets `next_run_at`, making the
|
||||
task visible to future polls again.
|
||||
|
||||
### 1.4 — Remove `ListDue` from Interface
|
||||
|
||||
`ListDue` removed from `TaskStore` interface. Replaced by
|
||||
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
|
||||
`CreateRunExclusive` added alongside it.
|
||||
|
||||
Updated interface:
|
||||
```go
|
||||
type TaskStore interface {
|
||||
// ... existing CRUD ...
|
||||
|
||||
// Scheduler queries
|
||||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||||
SetNextRun(ctx context.Context, id string, nextRun interface{}) error
|
||||
SetLastRun(ctx context.Context, id string) error
|
||||
IncrementRunCount(ctx context.Context, id string) error
|
||||
|
||||
// Run history
|
||||
CreateRun(ctx context.Context, r *models.TaskRun) error
|
||||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
UpdateRun(ctx context.Context, id string, status string, ...) error
|
||||
TransitionRunStatus(ctx context.Context, id string, status string) error
|
||||
GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||||
ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
|
||||
}
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
|
||||
- [ ] `CreateRunExclusive` — PG + SQLite
|
||||
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
|
||||
- [ ] Remove `ListDue` from interface + both stores
|
||||
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
|
||||
disjoint task sets)
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS2 — PG Ticket Store
|
||||
|
||||
Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
|
||||
implementation. The middleware already programs to the `TicketValidator`
|
||||
interface — the swap is clean.
|
||||
|
||||
### 2.1 — Store Interface
|
||||
|
||||
New interface in `store/`:
|
||||
|
||||
```go
|
||||
// TicketStore manages short-lived single-use WebSocket auth tickets.
|
||||
type TicketStore interface {
|
||||
Issue(ctx context.Context, userID string) (string, error)
|
||||
Validate(ctx context.Context, ticketID string) (string, bool)
|
||||
Reap(ctx context.Context) (int, error)
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 — PG Implementation
|
||||
|
||||
**Issue:**
|
||||
```sql
|
||||
INSERT INTO ws_tickets (id, user_id, expires_at)
|
||||
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
|
||||
```
|
||||
|
||||
Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).
|
||||
|
||||
**Validate (atomic consume):**
|
||||
```sql
|
||||
DELETE FROM ws_tickets
|
||||
WHERE id = $1 AND expires_at > NOW()
|
||||
RETURNING user_id
|
||||
```
|
||||
|
||||
Single statement — delete + return. If expired or already consumed,
|
||||
zero rows → `("", false)`.
|
||||
|
||||
**Reap (TTL cleanup):**
|
||||
```sql
|
||||
DELETE FROM ws_tickets WHERE expires_at <= NOW()
|
||||
```
|
||||
|
||||
Called by a system task or inline during poll. No dedicated goroutine
|
||||
needed — the scheduler's 30s tick can piggyback, or register as a
|
||||
lightweight system function.
|
||||
|
||||
### 2.3 — SQLite Implementation
|
||||
|
||||
Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
|
||||
instead of `NOW() + INTERVAL`).
|
||||
|
||||
### 2.4 — Adapter for Middleware Interface
|
||||
|
||||
The middleware `TicketValidator` interface is:
|
||||
```go
|
||||
type TicketValidator interface {
|
||||
Validate(ticketID string) (string, bool)
|
||||
}
|
||||
```
|
||||
|
||||
The store's `Validate` takes a `context.Context`. Thin adapter:
|
||||
|
||||
```go
|
||||
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
|
||||
type TicketValidatorAdapter struct {
|
||||
Store store.TicketStore
|
||||
}
|
||||
|
||||
func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return a.Store.Validate(ctx, ticketID)
|
||||
}
|
||||
```
|
||||
|
||||
### 2.5 — Wiring in `main.go`
|
||||
|
||||
Replace:
|
||||
```go
|
||||
ticketStore := events.NewTicketStore()
|
||||
defer ticketStore.Stop()
|
||||
```
|
||||
|
||||
With:
|
||||
```go
|
||||
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
|
||||
```
|
||||
|
||||
The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
|
||||
directly. The WS auth middleware receives `ticketAdapter` (satisfies
|
||||
`TicketValidator`). No reaper goroutine — use system function or
|
||||
scheduler piggyback.
|
||||
|
||||
### 2.6 — Reap Strategy
|
||||
|
||||
Option A: Register `ticket_reap` as a system function
|
||||
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
|
||||
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
|
||||
infrastructure. Cons: requires task to exist.
|
||||
|
||||
Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
|
||||
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
|
||||
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.
|
||||
|
||||
**Recommendation: Option B.** At 50 users, the table will have at most
|
||||
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
|
||||
free. Revisit if ticket volume grows.
|
||||
|
||||
### 2.7 — Delete `events/tickets.go`
|
||||
|
||||
The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
|
||||
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
|
||||
lives in `events/` (or alongside the middleware — whichever avoids
|
||||
import cycles).
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `store.TicketStore` interface
|
||||
- [ ] PG implementation + SQLite implementation
|
||||
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
|
||||
`TicketValidator` interface
|
||||
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
|
||||
- [ ] Inline reap in scheduler poll
|
||||
- [ ] Delete `events/tickets.go`
|
||||
- [ ] Add `Tickets` field to `store.Stores`
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS3 — PG Rate Limiter
|
||||
|
||||
Replace the in-memory token bucket (`middleware.RateLimiter`) with a
|
||||
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
|
||||
burst 8), so transaction volume is trivially low.
|
||||
|
||||
### 3.1 — Store Interface
|
||||
|
||||
```go
|
||||
// RateLimitStore manages distributed rate limit counters.
|
||||
type RateLimitStore interface {
|
||||
// Allow checks if a request is within the rate limit.
|
||||
// Returns (allowed bool, tokensRemaining float64).
|
||||
// Atomically increments the counter if allowed.
|
||||
Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)
|
||||
|
||||
// Cleanup removes expired windows.
|
||||
Cleanup(ctx context.Context, maxAge time.Duration) error
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 — PG Implementation: Sliding-Window Token Bucket
|
||||
|
||||
The in-memory implementation uses a classic token bucket (refill based
|
||||
on elapsed time). The PG version approximates this with time-bucketed
|
||||
counters.
|
||||
|
||||
**`Allow` — atomic check-and-increment:**
|
||||
|
||||
```sql
|
||||
-- Upsert the current window's counter and check burst limit.
|
||||
-- Window = current second (truncated).
|
||||
INSERT INTO rate_limit_counters (key, window, tokens)
|
||||
VALUES ($1, date_trunc('second', NOW()), 1)
|
||||
ON CONFLICT (key, window)
|
||||
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
|
||||
RETURNING tokens
|
||||
```
|
||||
|
||||
Go logic after the RETURNING:
|
||||
1. Query returns `tokens` (count in the current 1-second window).
|
||||
2. If `tokens > burst` → denied, return `(false, nil)`.
|
||||
3. Else → allowed.
|
||||
|
||||
This is a fixed-window approximation, not a sliding window. For auth
|
||||
rate limiting at 5 req/s burst 8, the practical difference is
|
||||
negligible — worst case allows 2× burst at window boundaries, which
|
||||
is acceptable for this use case. A true sliding window would require
|
||||
reading adjacent windows and interpolating, adding complexity for
|
||||
near-zero benefit at this scale.
|
||||
|
||||
**Cleanup:**
|
||||
```sql
|
||||
DELETE FROM rate_limit_counters
|
||||
WHERE window < NOW() - $1::interval
|
||||
```
|
||||
|
||||
Called alongside ticket reap in the scheduler poll loop.
|
||||
|
||||
### 3.3 — SQLite Implementation
|
||||
|
||||
Same logic, `datetime('now')` for window truncation. Single-process,
|
||||
so the PG atomicity guarantees are naturally satisfied.
|
||||
|
||||
### 3.4 — Middleware Swap
|
||||
|
||||
Replace `middleware.RateLimiter` struct with one backed by the store:
|
||||
|
||||
```go
|
||||
type RateLimiter struct {
|
||||
store store.RateLimitStore
|
||||
rate float64
|
||||
burst int
|
||||
}
|
||||
|
||||
func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
|
||||
return &RateLimiter{store: store, rate: rate, burst: burst}
|
||||
}
|
||||
|
||||
func (rl *RateLimiter) Limit() gin.HandlerFunc {
|
||||
return func(c *gin.Context) {
|
||||
key := "auth:" + c.ClientIP()
|
||||
allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
|
||||
if err != nil {
|
||||
// DB error — fail open (don't block auth on rate limit DB failure)
|
||||
c.Next()
|
||||
return
|
||||
}
|
||||
if !allowed {
|
||||
c.Header("Retry-After", "1")
|
||||
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
|
||||
"error": "rate limit exceeded",
|
||||
})
|
||||
return
|
||||
}
|
||||
c.Next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Fail-open policy:** if PG is down, allow the request. The auth
|
||||
endpoints have their own protections (bcrypt, lockout). Blocking
|
||||
legitimate logins because the rate limit table is unreachable is worse
|
||||
than allowing a brief burst.
|
||||
|
||||
### 3.5 — Wiring in `main.go`
|
||||
|
||||
Replace:
|
||||
```go
|
||||
authLimiter := middleware.NewRateLimiter(5, 8)
|
||||
```
|
||||
|
||||
With:
|
||||
```go
|
||||
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
|
||||
```
|
||||
|
||||
Delete the in-memory cleanup goroutine (replaced by scheduler reap).
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `store.RateLimitStore` interface
|
||||
- [ ] PG implementation + SQLite implementation
|
||||
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
|
||||
- [ ] `main.go` wiring
|
||||
- [ ] Add `RateLimits` field to `store.Stores`
|
||||
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
|
||||
- [ ] Delete in-memory visitor map + cleanup goroutine
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS4 — WebSocket Cross-Pod Delivery
|
||||
|
||||
The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
|
||||
already implemented and handles room-scoped events correctly. The gap:
|
||||
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
|
||||
to local connection send channels. Users on other pods are never reached.
|
||||
|
||||
### Affected call sites (7 total)
|
||||
|
||||
| File | Context |
|
||||
|------|---------|
|
||||
| `main.go` | DM typing indicators |
|
||||
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
|
||||
| `handlers/stream_loop.go` | SSE stream events |
|
||||
| `handlers/tool_loop.go` | Browser tool call delivery |
|
||||
| `notifications/service.go` | Push notification delivery |
|
||||
|
||||
### 4.1 — Add `TargetUserID` to Event
|
||||
|
||||
```go
|
||||
type Event struct {
|
||||
Label string `json:"event"`
|
||||
Room string `json:"room,omitempty"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
Ts int64 `json:"ts"`
|
||||
|
||||
// Server-side metadata (not serialized to clients)
|
||||
SenderID string `json:"-"`
|
||||
ConnID string `json:"-"`
|
||||
TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
|
||||
}
|
||||
```
|
||||
|
||||
`TargetUserID` is included in the JSON serialization for `pg_broadcast`
|
||||
(so the remote pod knows who to deliver to) but stripped before sending
|
||||
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
|
||||
path, or use a separate internal envelope.
|
||||
|
||||
**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
|
||||
harmless if clients see it (they already ignore unknown fields), and
|
||||
avoids a second serialization path. The field is empty for room-scoped
|
||||
events and only populated for targeted delivery.
|
||||
|
||||
### 4.2 — Update WS Subscriber Filter
|
||||
|
||||
In `subscribeToBus`, add target filtering:
|
||||
|
||||
```go
|
||||
// Targeted delivery: if event has a target user, only deliver to that user
|
||||
if e.TargetUserID != "" && e.TargetUserID != c.userID {
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
This goes before the room filter. Targeted events skip room filtering
|
||||
entirely (they're user-scoped, not room-scoped).
|
||||
|
||||
### 4.3 — Rewrite `SendToUser` → `PublishToUser`
|
||||
|
||||
New method on `Hub`:
|
||||
|
||||
```go
|
||||
// PublishToUser sends an event to a specific user via the bus.
|
||||
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
|
||||
func (h *Hub) PublishToUser(userID string, event Event) {
|
||||
event.TargetUserID = userID
|
||||
h.bus.Publish(event)
|
||||
}
|
||||
```
|
||||
|
||||
This replaces all 7 `hub.SendToUser` call sites. The event flows:
|
||||
|
||||
1. `hub.PublishToUser(userID, evt)`
|
||||
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
|
||||
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
|
||||
4. `broadcastHook` → `pg_notify` → remote pod
|
||||
5. Remote pod: `publishLocal(evt)` → `subscribeToBus` filter → deliver
|
||||
|
||||
### 4.4 — Keep `SendToUser` as Local-Only Optimization
|
||||
|
||||
Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
|
||||
`PublishToUser` can optionally try local delivery first (fast path)
|
||||
and only broadcast if the user isn't connected locally. But this
|
||||
optimization is premature at 50 users — just always go through the
|
||||
bus. Revisit post-MVP if latency matters.
|
||||
|
||||
### 4.5 — Audit: `tool.call` + `WaitFor`
|
||||
|
||||
`tool.call.*` events use `DirToClient` routing and are delivered via
|
||||
`SendToUser`. After the rewrite, they go through the bus and will
|
||||
broadcast to all pods. The `WaitFor` on the originating pod still
|
||||
works — `tool.result.*` comes back from the client on the same pod
|
||||
(the WS connection is sticky to a pod).
|
||||
|
||||
Verify: if the user's browser tool is connected to pod-2, but the
|
||||
completion handler calling `WaitFor` is on pod-1, the tool call event
|
||||
needs to reach pod-2 (via pg_broadcast), and the result needs to come
|
||||
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
|
||||
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
|
||||
the broadcast hook skips it. **This needs to change to `DirBoth`** so
|
||||
tool results cross pod boundaries for `WaitFor` to work.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `TargetUserID` field on `Event`
|
||||
- [ ] `subscribeToBus` target filter
|
||||
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
|
||||
- [ ] Update `tool.result.*` routing to `DirBoth`
|
||||
- [ ] Rename `SendToUser` → `sendToUserLocal` (unexported)
|
||||
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
|
||||
- [ ] CI green
|
||||
|
||||
---
|
||||
|
||||
## CS5 — Health Probes + Helm Validation
|
||||
|
||||
### 5.1 — Readiness Probe Refinement
|
||||
|
||||
Current health endpoint: basic HTTP 200. Add a PG ping:
|
||||
|
||||
```go
|
||||
func (h *HealthHandler) Readiness(c *gin.Context) {
|
||||
ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
|
||||
defer cancel()
|
||||
if err := database.DB.PingContext(ctx); err != nil {
|
||||
c.JSON(503, gin.H{"error": "database unavailable"})
|
||||
return
|
||||
}
|
||||
c.JSON(200, gin.H{"status": "ok"})
|
||||
}
|
||||
```
|
||||
|
||||
Wire as `/healthz/ready` (separate from the existing `/healthz/live`
|
||||
liveness probe). Kubernetes pulls the pod from the service on readiness
|
||||
failure — new requests route to healthy replicas.
|
||||
|
||||
### 5.2 — Helm Changes
|
||||
|
||||
`chart/values.yaml`:
|
||||
```yaml
|
||||
backend:
|
||||
replicaCount: 2 # was 1
|
||||
```
|
||||
|
||||
`chart/templates/deployment-backend.yaml` additions:
|
||||
|
||||
**Pod anti-affinity** — spread replicas across nodes:
|
||||
```yaml
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: switchboard-backend
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
`preferredDuring` (not `requiredDuring`) — if fewer nodes than
|
||||
replicas, pods still schedule on the same node (degraded HA is better
|
||||
than no scheduling).
|
||||
|
||||
**Readiness probe:**
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz/ready
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
### 5.3 — Startup Jitter
|
||||
|
||||
Add random jitter (0–15s) to the scheduler's initial tick to stagger
|
||||
replica polling:
|
||||
|
||||
```go
|
||||
func (s *Scheduler) Run() {
|
||||
// ...
|
||||
jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
|
||||
time.Sleep(jitter)
|
||||
log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Not strictly necessary with `SKIP LOCKED` (contention is harmless),
|
||||
but reduces unnecessary lock acquisition at startup.
|
||||
|
||||
### 5.4 — Validation Checklist
|
||||
|
||||
Manual validation on the `gobha-ai-chat` cluster:
|
||||
|
||||
- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
|
||||
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
|
||||
- [ ] Create a cron task (every 1 min), confirm only one run per tick
|
||||
- [ ] WebSocket: connect on pod-1, send message that completes on
|
||||
pod-2, verify streaming events arrive
|
||||
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
|
||||
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
|
||||
burst regardless of which replica serves each request
|
||||
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
|
||||
continues serving + picks up scheduler duties immediately
|
||||
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load
|
||||
|
||||
### Deliverables
|
||||
|
||||
- [ ] `/healthz/ready` endpoint with PG ping
|
||||
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
|
||||
- [ ] Scheduler startup jitter
|
||||
- [ ] Validation checklist executed and documented
|
||||
- [ ] CI green, tag `v0.32.0`
|
||||
|
||||
---
|
||||
|
||||
## Summary: Store Interface Additions
|
||||
|
||||
```go
|
||||
type Stores struct {
|
||||
// ... existing fields ...
|
||||
Tickets TicketStore // v0.32.0: WS auth tickets (PG-backed)
|
||||
RateLimits RateLimitStore // v0.32.0: Distributed rate limiting
|
||||
}
|
||||
```
|
||||
|
||||
`TaskStore` changes:
|
||||
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
|
||||
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
|
||||
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`
|
||||
|
||||
---
|
||||
|
||||
## Changeset Sequence
|
||||
|
||||
| CS | Scope | Key Files | Gate |
|
||||
|----|-------|-----------|------|
|
||||
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
|
||||
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
|
||||
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
|
||||
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
|
||||
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
|
||||
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |
|
||||
|
||||
---
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
|
||||
(rate limiter — auth-only, fail-open).
|
||||
|
||||
**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
|
||||
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
|
||||
`--replicas=1`).
|
||||
|
||||
**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
|
||||
handlers, completion, notifications). The `tool.result` routing change
|
||||
(`DirFromClient` → `DirBoth`) needs careful testing — if tool results
|
||||
broadcast to all pods, the non-originating pod's `WaitFor` must not
|
||||
accidentally consume the result. Verify: `WaitFor` subscribes to
|
||||
`tool.result.{specific-id}` (exact label match), so only the waiting
|
||||
goroutine receives it. Safe.
|
||||
|
||||
**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
|
||||
always a safe rollback. Each CS should leave CI green at single-replica.
|
||||
@@ -30,7 +30,7 @@ v0.9.x–v0.28.7 Foundation through Platform Polish ✅
|
||||
│ │
|
||||
Extension Track Operations Track
|
||||
│ │
|
||||
v0.29.0 Starlark Sandbox ✅ v0.32.0 Multi-Replica HA
|
||||
v0.29.0 Starlark Sandbox ✅ v0.32.0 Multi-Replica HA ✅
|
||||
v0.29.1 API Extensions ✅ v0.33.0 Observability
|
||||
v0.29.2 DB Extensions ✅ v0.34.0 Data Portability
|
||||
v0.29.3 Workflow Forms ✅ │
|
||||
@@ -39,7 +39,7 @@ v0.9.x–v0.28.7 Foundation through Platform Polish ✅
|
||||
v0.30.2 Workflow Packages ✅ │
|
||||
v0.31.0 Editor + SDK ✅ │
|
||||
v0.31.1 SDK Exercise ✅ │
|
||||
v0.31.2 Team Workflows │
|
||||
v0.31.2 Team Workflows ✅ │
|
||||
│ │
|
||||
│ v0.35.0 Workflow Product
|
||||
│ │
|
||||
@@ -350,43 +350,39 @@ Depends on: v0.31.1.
|
||||
Parallel to extension track. No Starlark dependency. Delivers
|
||||
production readiness for the target multi-team deployment.
|
||||
|
||||
### v0.32.0 — Multi-Replica HA
|
||||
### v0.32.0 — Multi-Replica HA ✅
|
||||
|
||||
Run 2–3 backend replicas across nodes for node-level availability.
|
||||
|
||||
Depends on: v0.28.8.
|
||||
|
||||
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
|
||||
leader election. Every replica runs the scheduler poll loop; PG
|
||||
serializes task claims atomically. No K8s API dependency, no Redis,
|
||||
no new coordination infrastructure. See `docs/DESIGN-0.32.0.md`.
|
||||
|
||||
**What already works multi-replica:**
|
||||
- REST API (stateless, JWT auth) ✅
|
||||
- PG + S3 + CephFS (shared infrastructure) ✅
|
||||
- `pg_broadcast` LISTEN/NOTIFY (cross-pod event bus) ✅
|
||||
|
||||
**What needs work:**
|
||||
- [ ] WebSocket fan-out: wire `Bus.Subscribe` to `pg_broadcast` listener
|
||||
so events from other pods reach local WebSocket connections.
|
||||
LISTEN/NOTIFY plumbing exists — bridge inbound NOTIFY into per-pod
|
||||
hub for local delivery.
|
||||
- [ ] Task scheduler leader election: Kubernetes `Lease`-based. Only
|
||||
leader runs cron scheduler. Prevents duplicate execution.
|
||||
- [ ] Shared ticket store: `TicketStore` from `sync.Map` → PG table
|
||||
with 30s TTL. Ensures WS ticket from pod-1 validates on pod-2.
|
||||
- [ ] Shared rate limiter: in-memory → PG or Redis counter. Without
|
||||
this, effective rate limit = N × configured across N replicas.
|
||||
- [ ] Health check refinement: readiness probe fails fast on PG
|
||||
connection loss.
|
||||
- [ ] Helm: `backend.replicaCount` > 1 tested. Pod anti-affinity
|
||||
to spread across nodes.
|
||||
**Delivered (5 changesets):**
|
||||
- [x] Task scheduler: `FOR UPDATE SKIP LOCKED` atomic claim replaces
|
||||
`ListDue`. `CreateRunExclusive` conditional insert for
|
||||
belt-and-suspenders uniqueness. Startup jitter (0–15s).
|
||||
- [x] WebSocket cross-pod delivery: `PublishToUser` routes through
|
||||
bus → `pg_broadcast` → remote pod. 18 `SendToUser` call sites
|
||||
migrated. `tool.result.*` routing changed to `DirBoth` for
|
||||
cross-pod `WaitFor`. `TargetUserID` field on `Event`.
|
||||
- [x] Shared ticket store: `ws_tickets` PG table with 30s TTL,
|
||||
atomic `DELETE ... RETURNING` validation. Replaces `sync.Map`.
|
||||
- [x] Shared rate limiter: `rate_limit_counters` PG table with
|
||||
fixed-window counters. Fail-open policy on DB errors.
|
||||
- [x] Health probes: `/healthz/ready` (PG ping, 2s timeout),
|
||||
`/healthz/live` (process alive). Helm: `replicaCount: 2`,
|
||||
pod anti-affinity, readiness/liveness probes.
|
||||
|
||||
**Sizing (measured on v0.28.8):**
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Idle memory | 17Mi |
|
||||
| Peak (569 reqs, 4 users, SSE) | 216Mi |
|
||||
| Settled after GC | 34Mi |
|
||||
| Per-SSE stream | ~500KB–1MB |
|
||||
| Per-WebSocket | ~64KB |
|
||||
| DB pool | 25 max (shared) |
|
||||
**Schema:** `020_ha.sql` — `ws_tickets`, `rate_limit_counters`.
|
||||
|
||||
### v0.33.0 — Observability
|
||||
|
||||
|
||||
Reference in New Issue
Block a user