811 lines
26 KiB
Markdown
811 lines
26 KiB
Markdown
# DESIGN-0.32.0 — Multi-Replica HA
|
||
|
||
Run 2–3 backend replicas across nodes for node-level availability.
|
||
Five changesets. No sub-versions — the work is tightly coupled and
|
||
each CS builds on the previous.
|
||
|
||
Depends on: v0.31.2 (current HEAD).
|
||
|
||
**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
|
||
leader election for the task scheduler. Every replica polls; PG
|
||
serializes the claims. Extends to all shared mutable state — tickets,
|
||
rate counters — keeping PG as the sole coordination layer. No Redis,
|
||
no K8s API dependency, no new infrastructure.
|
||
|
||
---
|
||
|
||
## What Already Works Multi-Replica
|
||
|
||
These require zero changes:
|
||
|
||
- **REST API** — stateless, JWT auth, any replica serves any request.
|
||
- **PG + S3 + CephFS** — shared storage infrastructure.
|
||
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
|
||
→ `broadcastHook` → `pg_notify` → remote pod `publishLocal()` →
|
||
local WS subscribers. Fully wired, just never tested at N>1.
|
||
|
||
---
|
||
|
||
## What Needs Work
|
||
|
||
Five areas of in-process mutable state that break at replica count > 1:
|
||
|
||
| State | Current | Problem at N>1 | Fix |
|
||
|-------|---------|----------------|-----|
|
||
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
|
||
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
|
||
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
|
||
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
|
||
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |
|
||
|
||
---
|
||
|
||
## CS0 — Schema: `020_ha.sql`
|
||
|
||
New migration for both PG and SQLite. No changes to existing 019
|
||
task schema — `next_run_at` nullable already supports the claim
|
||
mechanism.
|
||
|
||
### PG: `server/database/migrations/020_ha.sql`
|
||
|
||
```sql
|
||
-- ==========================================
|
||
-- Chat Switchboard — 020 Multi-Replica HA
|
||
-- ==========================================
|
||
-- Shared state tables for multi-replica operation.
|
||
-- v0.32.0: ws_tickets, rate_limit_counters.
|
||
-- ==========================================
|
||
|
||
-- =========================================
|
||
-- WEBSOCKET TICKETS (replaces sync.Map)
|
||
-- =========================================
|
||
|
||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||
id TEXT PRIMARY KEY, -- 128-bit hex token
|
||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||
expires_at TIMESTAMPTZ NOT NULL
|
||
);
|
||
|
||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||
ON ws_tickets (expires_at);
|
||
|
||
-- =========================================
|
||
-- RATE LIMIT COUNTERS (replaces in-memory)
|
||
-- =========================================
|
||
-- Token bucket approximation using time-windowed counters.
|
||
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
|
||
-- Window is truncated to the second for the configured rate.
|
||
|
||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||
key TEXT NOT NULL,
|
||
window TIMESTAMPTZ NOT NULL, -- truncated timestamp (window start)
|
||
tokens REAL NOT NULL DEFAULT 0, -- tokens consumed in this window
|
||
PRIMARY KEY (key, window)
|
||
);
|
||
|
||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||
ON rate_limit_counters (window);
|
||
```
|
||
|
||
### SQLite: `server/database/migrations/sqlite/020_ha.sql`
|
||
|
||
SQLite parity — structurally identical but with dialect adjustments.
|
||
These tables are functional in SQLite for single-process test parity,
|
||
though multi-replica is PG-only in production.
|
||
|
||
```sql
|
||
CREATE TABLE IF NOT EXISTS ws_tickets (
|
||
id TEXT PRIMARY KEY,
|
||
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||
expires_at TEXT NOT NULL
|
||
);
|
||
|
||
CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
|
||
ON ws_tickets (expires_at);
|
||
|
||
CREATE TABLE IF NOT EXISTS rate_limit_counters (
|
||
key TEXT NOT NULL,
|
||
window TEXT NOT NULL,
|
||
tokens REAL NOT NULL DEFAULT 0,
|
||
PRIMARY KEY (key, window)
|
||
);
|
||
|
||
CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
|
||
ON rate_limit_counters (window);
|
||
```
|
||
|
||
### Deliverables
|
||
|
||
- [x] `020_ha.sql` (PG)
|
||
- [x] `020_ha.sql` (SQLite)
|
||
- [x] CI green on both pipelines (migration runs, tables exist)
|
||
|
||
---
|
||
|
||
## CS1 — Task Scheduler: SKIP LOCKED
|
||
|
||
Replace the read-then-execute scheduler with atomic claim semantics.
|
||
Every replica runs the poll loop. PG serializes task handoff.
|
||
|
||
### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)
|
||
|
||
New store method. Single atomic statement — SELECT + UPDATE in one
|
||
round trip. Returns at most one task per call.
|
||
|
||
**PG implementation:**
|
||
|
||
```sql
|
||
UPDATE tasks
|
||
SET next_run_at = NULL
|
||
WHERE id = (
|
||
SELECT id FROM tasks
|
||
WHERE is_active = true
|
||
AND next_run_at <= NOW()
|
||
ORDER BY next_run_at ASC
|
||
LIMIT 1
|
||
FOR UPDATE SKIP LOCKED
|
||
)
|
||
RETURNING <taskColumns>
|
||
```
|
||
|
||
Mechanics:
|
||
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
|
||
candidate row, skip it instantly (no wait, no deadlock).
|
||
- `SET next_run_at = NULL` — claimed task disappears from future
|
||
polls. `advanceNextRun` restores it after execution.
|
||
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
|
||
- The existing partial index `idx_tasks_next_run` covers the WHERE
|
||
clause (`is_active = true AND next_run_at IS NOT NULL` by
|
||
implication — NULL rows won't satisfy `<= NOW()`).
|
||
|
||
**SQLite implementation:**
|
||
|
||
SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
|
||
the current `ListDue` behavior unchanged for SQLite. The store
|
||
interface accommodates both:
|
||
|
||
```go
|
||
// TaskStore additions
|
||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||
```
|
||
|
||
SQLite `ClaimDueTask` just does:
|
||
```sql
|
||
SELECT <taskColumns> FROM tasks
|
||
WHERE is_active = 1 AND next_run_at <= datetime('now')
|
||
ORDER BY next_run_at ASC LIMIT 1
|
||
```
|
||
Then sets `next_run_at = NULL` in a second statement (single writer,
|
||
no contention).
|
||
|
||
### 1.2 — Conditional `CreateRunExclusive`
|
||
|
||
Belt-and-suspenders: prevent double-execution even if two replicas
|
||
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
|
||
but defense in depth).
|
||
|
||
```sql
|
||
INSERT INTO task_runs (task_id, status)
|
||
SELECT $1, 'running'
|
||
WHERE NOT EXISTS (
|
||
SELECT 1 FROM task_runs
|
||
WHERE task_id = $1 AND status IN ('running', 'queued')
|
||
)
|
||
RETURNING id, started_at
|
||
```
|
||
|
||
Returns `sql.ErrNoRows` if a run already exists → skip execution.
|
||
|
||
New store method:
|
||
```go
|
||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||
```
|
||
|
||
### 1.3 — Scheduler Loop Rewrite
|
||
|
||
`scheduler.go` changes:
|
||
|
||
```go
|
||
func (s *Scheduler) poll() {
|
||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||
defer cancel()
|
||
|
||
cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
|
||
if !cfg.Enabled {
|
||
return
|
||
}
|
||
|
||
// Claim tasks one at a time until none remain or budget exhausted.
|
||
// Each replica claims independently — PG SKIP LOCKED serializes.
|
||
claimed := 0
|
||
for claimed < cfg.MaxConcurrent {
|
||
task, err := s.stores.Tasks.ClaimDueTask(ctx)
|
||
if err != nil {
|
||
break // sql.ErrNoRows or real error — either way, done
|
||
}
|
||
claimed++
|
||
go s.execute(ctx, task)
|
||
}
|
||
}
|
||
```
|
||
|
||
`execute()` changes:
|
||
- Remove the `GetActiveRun` TOCTOU check — replaced by
|
||
`CreateRunExclusive`.
|
||
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
|
||
But adoption also uses `CreateRunExclusive` semantics (transition
|
||
only if still `queued`).
|
||
- `advanceNextRun` still computes and sets `next_run_at`, making the
|
||
task visible to future polls again.
|
||
|
||
### 1.4 — Remove `ListDue` from Interface
|
||
|
||
`ListDue` removed from `TaskStore` interface. Replaced by
|
||
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
|
||
`CreateRunExclusive` added alongside it.
|
||
|
||
Updated interface:
|
||
```go
|
||
type TaskStore interface {
|
||
// ... existing CRUD ...
|
||
|
||
// Scheduler queries
|
||
ClaimDueTask(ctx context.Context) (*models.Task, error)
|
||
SetNextRun(ctx context.Context, id string, nextRun interface{}) error
|
||
SetLastRun(ctx context.Context, id string) error
|
||
IncrementRunCount(ctx context.Context, id string) error
|
||
|
||
// Run history
|
||
CreateRun(ctx context.Context, r *models.TaskRun) error
|
||
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||
UpdateRun(ctx context.Context, id string, status string, ...) error
|
||
TransitionRunStatus(ctx context.Context, id string, status string) error
|
||
GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||
GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
|
||
ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
|
||
}
|
||
```
|
||
|
||
### Deliverables
|
||
|
||
- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
|
||
- [ ] `CreateRunExclusive` — PG + SQLite
|
||
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
|
||
- [ ] Remove `ListDue` from interface + both stores
|
||
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
|
||
disjoint task sets)
|
||
- [ ] CI green
|
||
|
||
---
|
||
|
||
## CS2 — PG Ticket Store
|
||
|
||
Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
|
||
implementation. The middleware already programs to the `TicketValidator`
|
||
interface — the swap is clean.
|
||
|
||
### 2.1 — Store Interface
|
||
|
||
New interface in `store/`:
|
||
|
||
```go
|
||
// TicketStore manages short-lived single-use WebSocket auth tickets.
|
||
type TicketStore interface {
|
||
Issue(ctx context.Context, userID string) (string, error)
|
||
Validate(ctx context.Context, ticketID string) (string, bool)
|
||
Reap(ctx context.Context) (int, error)
|
||
}
|
||
```
|
||
|
||
### 2.2 — PG Implementation
|
||
|
||
**Issue:**
|
||
```sql
|
||
INSERT INTO ws_tickets (id, user_id, expires_at)
|
||
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
|
||
```
|
||
|
||
Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).
|
||
|
||
**Validate (atomic consume):**
|
||
```sql
|
||
DELETE FROM ws_tickets
|
||
WHERE id = $1 AND expires_at > NOW()
|
||
RETURNING user_id
|
||
```
|
||
|
||
Single statement — delete + return. If expired or already consumed,
|
||
zero rows → `("", false)`.
|
||
|
||
**Reap (TTL cleanup):**
|
||
```sql
|
||
DELETE FROM ws_tickets WHERE expires_at <= NOW()
|
||
```
|
||
|
||
Called by a system task or inline during poll. No dedicated goroutine
|
||
needed — the scheduler's 30s tick can piggyback, or register as a
|
||
lightweight system function.
|
||
|
||
### 2.3 — SQLite Implementation
|
||
|
||
Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
|
||
instead of `NOW() + INTERVAL`).
|
||
|
||
### 2.4 — Adapter for Middleware Interface
|
||
|
||
The middleware `TicketValidator` interface is:
|
||
```go
|
||
type TicketValidator interface {
|
||
Validate(ticketID string) (string, bool)
|
||
}
|
||
```
|
||
|
||
The store's `Validate` takes a `context.Context`. Thin adapter:
|
||
|
||
```go
|
||
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
|
||
type TicketValidatorAdapter struct {
|
||
Store store.TicketStore
|
||
}
|
||
|
||
func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
|
||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||
defer cancel()
|
||
return a.Store.Validate(ctx, ticketID)
|
||
}
|
||
```
|
||
|
||
### 2.5 — Wiring in `main.go`
|
||
|
||
Replace:
|
||
```go
|
||
ticketStore := events.NewTicketStore()
|
||
defer ticketStore.Stop()
|
||
```
|
||
|
||
With:
|
||
```go
|
||
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
|
||
```
|
||
|
||
The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
|
||
directly. The WS auth middleware receives `ticketAdapter` (satisfies
|
||
`TicketValidator`). No reaper goroutine — use system function or
|
||
scheduler piggyback.
|
||
|
||
### 2.6 — Reap Strategy
|
||
|
||
Option A: Register `ticket_reap` as a system function
|
||
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
|
||
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
|
||
infrastructure. Cons: requires task to exist.
|
||
|
||
Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
|
||
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
|
||
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.
|
||
|
||
**Recommendation: Option B.** At 50 users, the table will have at most
|
||
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
|
||
free. Revisit if ticket volume grows.
|
||
|
||
### 2.7 — Delete `events/tickets.go`
|
||
|
||
The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
|
||
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
|
||
lives in `events/` (or alongside the middleware — whichever avoids
|
||
import cycles).
|
||
|
||
### Deliverables
|
||
|
||
- [ ] `store.TicketStore` interface
|
||
- [ ] PG implementation + SQLite implementation
|
||
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
|
||
`TicketValidator` interface
|
||
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
|
||
- [ ] Inline reap in scheduler poll
|
||
- [ ] Delete `events/tickets.go`
|
||
- [ ] Add `Tickets` field to `store.Stores`
|
||
- [ ] CI green
|
||
|
||
---
|
||
|
||
## CS3 — PG Rate Limiter
|
||
|
||
Replace the in-memory token bucket (`middleware.RateLimiter`) with a
|
||
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
|
||
burst 8), so transaction volume is trivially low.
|
||
|
||
### 3.1 — Store Interface
|
||
|
||
```go
|
||
// RateLimitStore manages distributed rate limit counters.
|
||
type RateLimitStore interface {
|
||
// Allow checks if a request is within the rate limit.
|
||
// Returns (allowed bool, tokensRemaining float64).
|
||
// Atomically increments the counter if allowed.
|
||
Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)
|
||
|
||
// Cleanup removes expired windows.
|
||
Cleanup(ctx context.Context, maxAge time.Duration) error
|
||
}
|
||
```
|
||
|
||
### 3.2 — PG Implementation: Sliding-Window Token Bucket
|
||
|
||
The in-memory implementation uses a classic token bucket (refill based
|
||
on elapsed time). The PG version approximates this with time-bucketed
|
||
counters.
|
||
|
||
**`Allow` — atomic check-and-increment:**
|
||
|
||
```sql
|
||
-- Upsert the current window's counter and check burst limit.
|
||
-- Window = current second (truncated).
|
||
INSERT INTO rate_limit_counters (key, window, tokens)
|
||
VALUES ($1, date_trunc('second', NOW()), 1)
|
||
ON CONFLICT (key, window)
|
||
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
|
||
RETURNING tokens
|
||
```
|
||
|
||
Go logic after the RETURNING:
|
||
1. Query returns `tokens` (count in the current 1-second window).
|
||
2. If `tokens > burst` → denied, return `(false, nil)`.
|
||
3. Else → allowed.
|
||
|
||
This is a fixed-window approximation, not a sliding window. For auth
|
||
rate limiting at 5 req/s burst 8, the practical difference is
|
||
negligible — worst case allows 2× burst at window boundaries, which
|
||
is acceptable for this use case. A true sliding window would require
|
||
reading adjacent windows and interpolating, adding complexity for
|
||
near-zero benefit at this scale.
|
||
|
||
**Cleanup:**
|
||
```sql
|
||
DELETE FROM rate_limit_counters
|
||
WHERE window < NOW() - $1::interval
|
||
```
|
||
|
||
Called alongside ticket reap in the scheduler poll loop.
|
||
|
||
### 3.3 — SQLite Implementation
|
||
|
||
Same logic, `datetime('now')` for window truncation. Single-process,
|
||
so the PG atomicity guarantees are naturally satisfied.
|
||
|
||
### 3.4 — Middleware Swap
|
||
|
||
Replace `middleware.RateLimiter` struct with one backed by the store:
|
||
|
||
```go
|
||
type RateLimiter struct {
|
||
store store.RateLimitStore
|
||
rate float64
|
||
burst int
|
||
}
|
||
|
||
func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
|
||
return &RateLimiter{store: store, rate: rate, burst: burst}
|
||
}
|
||
|
||
func (rl *RateLimiter) Limit() gin.HandlerFunc {
|
||
return func(c *gin.Context) {
|
||
key := "auth:" + c.ClientIP()
|
||
allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
|
||
if err != nil {
|
||
// DB error — fail open (don't block auth on rate limit DB failure)
|
||
c.Next()
|
||
return
|
||
}
|
||
if !allowed {
|
||
c.Header("Retry-After", "1")
|
||
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
|
||
"error": "rate limit exceeded",
|
||
})
|
||
return
|
||
}
|
||
c.Next()
|
||
}
|
||
}
|
||
```
|
||
|
||
**Fail-open policy:** if PG is down, allow the request. The auth
|
||
endpoints have their own protections (bcrypt, lockout). Blocking
|
||
legitimate logins because the rate limit table is unreachable is worse
|
||
than allowing a brief burst.
|
||
|
||
### 3.5 — Wiring in `main.go`
|
||
|
||
Replace:
|
||
```go
|
||
authLimiter := middleware.NewRateLimiter(5, 8)
|
||
```
|
||
|
||
With:
|
||
```go
|
||
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
|
||
```
|
||
|
||
Delete the in-memory cleanup goroutine (replaced by scheduler reap).
|
||
|
||
### Deliverables
|
||
|
||
- [ ] `store.RateLimitStore` interface
|
||
- [ ] PG implementation + SQLite implementation
|
||
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
|
||
- [ ] `main.go` wiring
|
||
- [ ] Add `RateLimits` field to `store.Stores`
|
||
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
|
||
- [ ] Delete in-memory visitor map + cleanup goroutine
|
||
- [ ] CI green
|
||
|
||
---
|
||
|
||
## CS4 — WebSocket Cross-Pod Delivery
|
||
|
||
The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
|
||
already implemented and handles room-scoped events correctly. The gap:
|
||
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
|
||
to local connection send channels. Users on other pods are never reached.
|
||
|
||
### Affected call sites (7 total)
|
||
|
||
| File | Context |
|
||
|------|---------|
|
||
| `main.go` | DM typing indicators |
|
||
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
|
||
| `handlers/stream_loop.go` | SSE stream events |
|
||
| `handlers/tool_loop.go` | Browser tool call delivery |
|
||
| `notifications/service.go` | Push notification delivery |
|
||
|
||
### 4.1 — Add `TargetUserID` to Event
|
||
|
||
```go
|
||
type Event struct {
|
||
Label string `json:"event"`
|
||
Room string `json:"room,omitempty"`
|
||
Payload json.RawMessage `json:"payload"`
|
||
Ts int64 `json:"ts"`
|
||
|
||
// Server-side metadata (not serialized to clients)
|
||
SenderID string `json:"-"`
|
||
ConnID string `json:"-"`
|
||
TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
|
||
}
|
||
```
|
||
|
||
`TargetUserID` is included in the JSON serialization for `pg_broadcast`
|
||
(so the remote pod knows who to deliver to) but stripped before sending
|
||
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
|
||
path, or use a separate internal envelope.
|
||
|
||
**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
|
||
harmless if clients see it (they already ignore unknown fields), and
|
||
avoids a second serialization path. The field is empty for room-scoped
|
||
events and only populated for targeted delivery.
|
||
|
||
### 4.2 — Update WS Subscriber Filter
|
||
|
||
In `subscribeToBus`, add target filtering:
|
||
|
||
```go
|
||
// Targeted delivery: if event has a target user, only deliver to that user
|
||
if e.TargetUserID != "" && e.TargetUserID != c.userID {
|
||
return
|
||
}
|
||
```
|
||
|
||
This goes before the room filter. Targeted events skip room filtering
|
||
entirely (they're user-scoped, not room-scoped).
|
||
|
||
### 4.3 — Rewrite `SendToUser` → `PublishToUser`
|
||
|
||
New method on `Hub`:
|
||
|
||
```go
|
||
// PublishToUser sends an event to a specific user via the bus.
|
||
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
|
||
func (h *Hub) PublishToUser(userID string, event Event) {
|
||
event.TargetUserID = userID
|
||
h.bus.Publish(event)
|
||
}
|
||
```
|
||
|
||
This replaces all 7 `hub.SendToUser` call sites. The event flows:
|
||
|
||
1. `hub.PublishToUser(userID, evt)`
|
||
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
|
||
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
|
||
4. `broadcastHook` → `pg_notify` → remote pod
|
||
5. Remote pod: `publishLocal(evt)` → `subscribeToBus` filter → deliver
|
||
|
||
### 4.4 — Keep `SendToUser` as Local-Only Optimization
|
||
|
||
Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
|
||
`PublishToUser` can optionally try local delivery first (fast path)
|
||
and only broadcast if the user isn't connected locally. But this
|
||
optimization is premature at 50 users — just always go through the
|
||
bus. Revisit post-MVP if latency matters.
|
||
|
||
### 4.5 — Audit: `tool.call` + `WaitFor`
|
||
|
||
`tool.call.*` events use `DirToClient` routing and are delivered via
|
||
`SendToUser`. After the rewrite, they go through the bus and will
|
||
broadcast to all pods. The `WaitFor` on the originating pod still
|
||
works — `tool.result.*` comes back from the client on the same pod
|
||
(the WS connection is sticky to a pod).
|
||
|
||
Verify: if the user's browser tool is connected to pod-2, but the
|
||
completion handler calling `WaitFor` is on pod-1, the tool call event
|
||
needs to reach pod-2 (via pg_broadcast), and the result needs to come
|
||
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
|
||
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
|
||
the broadcast hook skips it. **This needs to change to `DirBoth`** so
|
||
tool results cross pod boundaries for `WaitFor` to work.
|
||
|
||
### Deliverables
|
||
|
||
- [ ] `TargetUserID` field on `Event`
|
||
- [ ] `subscribeToBus` target filter
|
||
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
|
||
- [ ] Update `tool.result.*` routing to `DirBoth`
|
||
- [ ] Rename `SendToUser` → `sendToUserLocal` (unexported)
|
||
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
|
||
- [ ] CI green
|
||
|
||
---
|
||
|
||
## CS5 — Health Probes + Helm Validation
|
||
|
||
### 5.1 — Readiness Probe Refinement
|
||
|
||
Current health endpoint: basic HTTP 200. Add a PG ping:
|
||
|
||
```go
|
||
func (h *HealthHandler) Readiness(c *gin.Context) {
|
||
ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
|
||
defer cancel()
|
||
if err := database.DB.PingContext(ctx); err != nil {
|
||
c.JSON(503, gin.H{"error": "database unavailable"})
|
||
return
|
||
}
|
||
c.JSON(200, gin.H{"status": "ok"})
|
||
}
|
||
```
|
||
|
||
Wire as `/healthz/ready` (separate from the existing `/healthz/live`
|
||
liveness probe). Kubernetes pulls the pod from the service on readiness
|
||
failure — new requests route to healthy replicas.
|
||
|
||
### 5.2 — Helm Changes
|
||
|
||
`chart/values.yaml`:
|
||
```yaml
|
||
backend:
|
||
replicaCount: 2 # was 1
|
||
```
|
||
|
||
`chart/templates/deployment-backend.yaml` additions:
|
||
|
||
**Pod anti-affinity** — spread replicas across nodes:
|
||
```yaml
|
||
affinity:
|
||
podAntiAffinity:
|
||
preferredDuringSchedulingIgnoredDuringExecution:
|
||
- weight: 100
|
||
podAffinityTerm:
|
||
labelSelector:
|
||
matchLabels:
|
||
app: switchboard-backend
|
||
topologyKey: kubernetes.io/hostname
|
||
```
|
||
|
||
`preferredDuring` (not `requiredDuring`) — if fewer nodes than
|
||
replicas, pods still schedule on the same node (degraded HA is better
|
||
than no scheduling).
|
||
|
||
**Readiness probe:**
|
||
```yaml
|
||
readinessProbe:
|
||
httpGet:
|
||
path: /healthz/ready
|
||
port: 8080
|
||
initialDelaySeconds: 5
|
||
periodSeconds: 10
|
||
failureThreshold: 3
|
||
```
|
||
|
||
### 5.3 — Startup Jitter
|
||
|
||
Add random jitter (0–15s) to the scheduler's initial tick to stagger
|
||
replica polling:
|
||
|
||
```go
|
||
func (s *Scheduler) Run() {
|
||
// ...
|
||
jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
|
||
time.Sleep(jitter)
|
||
log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
|
||
// ...
|
||
}
|
||
```
|
||
|
||
Not strictly necessary with `SKIP LOCKED` (contention is harmless),
|
||
but reduces unnecessary lock acquisition at startup.
|
||
|
||
### 5.4 — Validation Checklist
|
||
|
||
Manual validation on the `gobha-ai-chat` cluster:
|
||
|
||
- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
|
||
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
|
||
- [ ] Create a cron task (every 1 min), confirm only one run per tick
|
||
- [ ] WebSocket: connect on pod-1, send message that completes on
|
||
pod-2, verify streaming events arrive
|
||
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
|
||
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
|
||
burst regardless of which replica serves each request
|
||
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
|
||
continues serving + picks up scheduler duties immediately
|
||
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load
|
||
|
||
### Deliverables
|
||
|
||
- [ ] `/healthz/ready` endpoint with PG ping
|
||
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
|
||
- [ ] Scheduler startup jitter
|
||
- [ ] Validation checklist executed and documented
|
||
- [ ] CI green, tag `v0.32.0`
|
||
|
||
---
|
||
|
||
## Summary: Store Interface Additions
|
||
|
||
```go
|
||
type Stores struct {
|
||
// ... existing fields ...
|
||
Tickets TicketStore // v0.32.0: WS auth tickets (PG-backed)
|
||
RateLimits RateLimitStore // v0.32.0: Distributed rate limiting
|
||
}
|
||
```
|
||
|
||
`TaskStore` changes:
|
||
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
|
||
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
|
||
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`
|
||
|
||
---
|
||
|
||
## Changeset Sequence
|
||
|
||
| CS | Scope | Key Files | Gate |
|
||
|----|-------|-----------|------|
|
||
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
|
||
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
|
||
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
|
||
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
|
||
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
|
||
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |
|
||
|
||
---
|
||
|
||
## Risk Assessment
|
||
|
||
**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
|
||
(rate limiter — auth-only, fail-open).
|
||
|
||
**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
|
||
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
|
||
`--replicas=1`).
|
||
|
||
**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
|
||
handlers, completion, notifications). The `tool.result` routing change
|
||
(`DirFromClient` → `DirBoth`) needs careful testing — if tool results
|
||
broadcast to all pods, the non-originating pod's `WaitFor` must not
|
||
accidentally consume the result. Verify: `WaitFor` subscribes to
|
||
`tool.result.{specific-id}` (exact label match), so only the waiting
|
||
goroutine receives it. Safe.
|
||
|
||
**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
|
||
always a safe rollback. Each CS should leave CI green at single-replica.
|