# DESIGN-0.32.0 — Multi-Replica HA

Run 2–3 backend replicas across nodes for node-level availability.
Five changesets. No sub-versions — the work is tightly coupled and
each CS builds on the previous.

Depends on: v0.31.2 (current HEAD).

**Design decision:** PG `SKIP LOCKED` replaces Kubernetes Lease-based
leader election for the task scheduler. Every replica polls; PG
serializes the claims. Extends to all shared mutable state — tickets,
rate counters — keeping PG as the sole coordination layer. No Redis,
no K8s API dependency, no new infrastructure.

---

## What Already Works Multi-Replica

These require zero changes:

- **REST API** — stateless, JWT auth, any replica serves any request.
- **PG + S3 + CephFS** — shared storage infrastructure.
- **`pg_broadcast` LISTEN/NOTIFY** — cross-pod event bus. `Publish()`
  → `broadcastHook` → `pg_notify` → remote pod `publishLocal()` →
  local WS subscribers. Fully wired, just never tested at N>1.

---

## What Needs Work

Five areas of in-process mutable state that break at replica count > 1:

| State | Current | Problem at N>1 | Fix |
|-------|---------|----------------|-----|
| Task scheduler | Single goroutine polls `ListDue` | All replicas fire same tasks | `SKIP LOCKED` atomic claim |
| Task run guard | `GetActiveRun` check → `CreateRun` | TOCTOU race window | Conditional `INSERT ... WHERE NOT EXISTS` |
| WS ticket store | `sync.Map` per-pod | Ticket from pod-1 invalid on pod-2 | PG table with TTL reaper |
| Rate limiter | In-memory token bucket per-pod | Effective limit = N × configured | PG counter with time bucket |
| `SendToUser` | Local hub lookup only | User on pod-2 never reached | Route through bus → `pg_broadcast` |

---

## CS0 — Schema: `020_ha.sql`

New migration for both PG and SQLite. No changes to existing 019
task schema — `next_run_at` nullable already supports the claim
mechanism.

### PG: `server/database/migrations/020_ha.sql`

```sql
-- ==========================================
-- Chat Switchboard — 020 Multi-Replica HA
-- ==========================================
-- Shared state tables for multi-replica operation.
-- v0.32.0: ws_tickets, rate_limit_counters.
-- ==========================================

-- =========================================
-- WEBSOCKET TICKETS (replaces sync.Map)
-- =========================================

CREATE TABLE IF NOT EXISTS ws_tickets (
    id         TEXT PRIMARY KEY,           -- 128-bit hex token
    user_id    UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    expires_at TIMESTAMPTZ NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
    ON ws_tickets (expires_at);

-- =========================================
-- RATE LIMIT COUNTERS (replaces in-memory)
-- =========================================
-- Token bucket approximation using time-windowed counters.
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
-- Window is truncated to the second for the configured rate.

CREATE TABLE IF NOT EXISTS rate_limit_counters (
    key        TEXT NOT NULL,
    window     TIMESTAMPTZ NOT NULL,       -- truncated timestamp (window start)
    tokens     REAL NOT NULL DEFAULT 0,    -- tokens consumed in this window
    PRIMARY KEY (key, window)
);

CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
    ON rate_limit_counters (window);
```

### SQLite: `server/database/migrations/sqlite/020_ha.sql`

SQLite parity — structurally identical but with dialect adjustments.
These tables are functional in SQLite for single-process test parity,
though multi-replica is PG-only in production.

```sql
CREATE TABLE IF NOT EXISTS ws_tickets (
    id         TEXT PRIMARY KEY,
    user_id    TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    expires_at TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
    ON ws_tickets (expires_at);

CREATE TABLE IF NOT EXISTS rate_limit_counters (
    key        TEXT NOT NULL,
    window     TEXT NOT NULL,
    tokens     REAL NOT NULL DEFAULT 0,
    PRIMARY KEY (key, window)
);

CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
    ON rate_limit_counters (window);
```

### Deliverables

- [x] `020_ha.sql` (PG)
- [x] `020_ha.sql` (SQLite)
- [x] CI green on both pipelines (migration runs, tables exist)

---

## CS1 — Task Scheduler: SKIP LOCKED

Replace the read-then-execute scheduler with atomic claim semantics.
Every replica runs the poll loop. PG serializes task handoff.

### 1.1 — Atomic `ClaimDueTask` (replaces `ListDue`)

New store method. Single atomic statement — SELECT + UPDATE in one
round trip. Returns at most one task per call.

**PG implementation:**

```sql
UPDATE tasks
SET next_run_at = NULL
WHERE id = (
    SELECT id FROM tasks
    WHERE is_active = true
      AND next_run_at <= NOW()
    ORDER BY next_run_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING <taskColumns>
```

Mechanics:
- `FOR UPDATE SKIP LOCKED` — if another replica holds a lock on a
  candidate row, skip it instantly (no wait, no deadlock).
- `SET next_run_at = NULL` — claimed task disappears from future
  polls. `advanceNextRun` restores it after execution.
- Returns zero rows if nothing is due → `sql.ErrNoRows` → no-op.
- The existing partial index `idx_tasks_next_run` covers the WHERE
  clause (`is_active = true AND next_run_at IS NOT NULL` by
  implication — NULL rows won't satisfy `<= NOW()`).

**SQLite implementation:**

SQLite has no `SKIP LOCKED`. Single-process, so not needed. Keep
the current `ListDue` behavior unchanged for SQLite. The store
interface accommodates both:

```go
// TaskStore additions
ClaimDueTask(ctx context.Context) (*models.Task, error)
```

SQLite `ClaimDueTask` just does:
```sql
SELECT <taskColumns> FROM tasks
WHERE is_active = 1 AND next_run_at <= datetime('now')
ORDER BY next_run_at ASC LIMIT 1
```
Then sets `next_run_at = NULL` in a second statement (single writer,
no contention).

### 1.2 — Conditional `CreateRunExclusive`

Belt-and-suspenders: prevent double-execution even if two replicas
somehow both claim the same task (shouldn't happen with SKIP LOCKED,
but defense in depth).

```sql
INSERT INTO task_runs (task_id, status)
SELECT $1, 'running'
WHERE NOT EXISTS (
    SELECT 1 FROM task_runs
    WHERE task_id = $1 AND status IN ('running', 'queued')
)
RETURNING id, started_at
```

Returns `sql.ErrNoRows` if a run already exists → skip execution.

New store method:
```go
CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
```

### 1.3 — Scheduler Loop Rewrite

`scheduler.go` changes:

```go
func (s *Scheduler) poll() {
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
    if !cfg.Enabled {
        return
    }

    // Claim tasks one at a time until none remain or budget exhausted.
    // Each replica claims independently — PG SKIP LOCKED serializes.
    claimed := 0
    for claimed < cfg.MaxConcurrent {
        task, err := s.stores.Tasks.ClaimDueTask(ctx)
        if err != nil {
            break // sql.ErrNoRows or real error — either way, done
        }
        claimed++
        go s.execute(ctx, task)
    }
}
```

`execute()` changes:
- Remove the `GetActiveRun` TOCTOU check — replaced by
  `CreateRunExclusive`.
- `GetQueuedRun` stays — webhook-triggered runs still need adoption.
  But adoption also uses `CreateRunExclusive` semantics (transition
  only if still `queued`).
- `advanceNextRun` still computes and sets `next_run_at`, making the
  task visible to future polls again.

### 1.4 — Remove `ListDue` from Interface

`ListDue` removed from `TaskStore` interface. Replaced by
`ClaimDueTask`. `CreateRun` remains for webhook trigger path;
`CreateRunExclusive` added alongside it.

Updated interface:
```go
type TaskStore interface {
    // ... existing CRUD ...

    // Scheduler queries
    ClaimDueTask(ctx context.Context) (*models.Task, error)
    SetNextRun(ctx context.Context, id string, nextRun interface{}) error
    SetLastRun(ctx context.Context, id string) error
    IncrementRunCount(ctx context.Context, id string) error

    // Run history
    CreateRun(ctx context.Context, r *models.TaskRun) error
    CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
    UpdateRun(ctx context.Context, id string, status string, ...) error
    TransitionRunStatus(ctx context.Context, id string, status string) error
    GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
    GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
    ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
}
```

### Deliverables

- [ ] `ClaimDueTask` — PG (`FOR UPDATE SKIP LOCKED`) + SQLite (simple select)
- [ ] `CreateRunExclusive` — PG + SQLite
- [ ] Scheduler loop rewrite — claim-per-iteration, no TOCTOU
- [ ] Remove `ListDue` from interface + both stores
- [ ] Unit tests: concurrent claim (PG only — two goroutines, verify
      disjoint task sets)
- [ ] CI green

---

## CS2 — PG Ticket Store

Replace `events.TicketStore` (in-memory `sync.Map`) with a PG-backed
implementation. The middleware already programs to the `TicketValidator`
interface — the swap is clean.

### 2.1 — Store Interface

New interface in `store/`:

```go
// TicketStore manages short-lived single-use WebSocket auth tickets.
type TicketStore interface {
    Issue(ctx context.Context, userID string) (string, error)
    Validate(ctx context.Context, ticketID string) (string, bool)
    Reap(ctx context.Context) (int, error)
}
```

### 2.2 — PG Implementation

**Issue:**
```sql
INSERT INTO ws_tickets (id, user_id, expires_at)
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')
```

Token generation stays in Go (`crypto/rand`, 16 bytes, hex-encoded).

**Validate (atomic consume):**
```sql
DELETE FROM ws_tickets
WHERE id = $1 AND expires_at > NOW()
RETURNING user_id
```

Single statement — delete + return. If expired or already consumed,
zero rows → `("", false)`.

**Reap (TTL cleanup):**
```sql
DELETE FROM ws_tickets WHERE expires_at <= NOW()
```

Called by a system task or inline during poll. No dedicated goroutine
needed — the scheduler's 30s tick can piggyback, or register as a
lightweight system function.

### 2.3 — SQLite Implementation

Identical SQL with dialect adjustments (`datetime('now', '+30 seconds')`
instead of `NOW() + INTERVAL`).

### 2.4 — Adapter for Middleware Interface

The middleware `TicketValidator` interface is:
```go
type TicketValidator interface {
    Validate(ticketID string) (string, bool)
}
```

The store's `Validate` takes a `context.Context`. Thin adapter:

```go
// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
type TicketValidatorAdapter struct {
    Store store.TicketStore
}

func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    return a.Store.Validate(ctx, ticketID)
}
```

### 2.5 — Wiring in `main.go`

Replace:
```go
ticketStore := events.NewTicketStore()
defer ticketStore.Stop()
```

With:
```go
ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}
```

The `POST /ws/ticket` endpoint calls `stores.Tickets.Issue(ctx, userID)`
directly. The WS auth middleware receives `ticketAdapter` (satisfies
`TicketValidator`). No reaper goroutine — use system function or
scheduler piggyback.

### 2.6 — Reap Strategy

Option A: Register `ticket_reap` as a system function
(`taskutil.SystemRegistry`), add a system task with `schedule: "@hourly"`
or `"*/5 * * * *"`. Pros: visible in admin UI, uses existing
infrastructure. Cons: requires task to exist.

Option B: Inline reap — call `stores.Tickets.Reap(ctx)` at the top
of the scheduler `poll()` loop (every 30s, cheap no-op when table is
empty). Pros: zero config. Cons: slightly couples scheduler to tickets.

**Recommendation: Option B.** At 50 users, the table will have at most
a few dozen rows. A `DELETE WHERE expires_at <= NOW()` every 30s is
free. Revisit if ticket volume grows.

### 2.7 — Delete `events/tickets.go`

The in-memory `TicketStore` struct, `NewTicketStore()`, and reaper
goroutine are fully replaced. Remove the file. The `TicketValidatorAdapter`
lives in `events/` (or alongside the middleware — whichever avoids
import cycles).

### Deliverables

- [ ] `store.TicketStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `TicketValidatorAdapter` — bridges `context`-aware store to
      `TicketValidator` interface
- [ ] `main.go` wiring — swap in PG store, remove `events.NewTicketStore()`
- [ ] Inline reap in scheduler poll
- [ ] Delete `events/tickets.go`
- [ ] Add `Tickets` field to `store.Stores`
- [ ] CI green

---

## CS3 — PG Rate Limiter

Replace the in-memory token bucket (`middleware.RateLimiter`) with a
PG-backed counter. Currently only applied to `/auth` routes (5 req/s,
burst 8), so transaction volume is trivially low.

### 3.1 — Store Interface

```go
// RateLimitStore manages distributed rate limit counters.
type RateLimitStore interface {
    // Allow checks if a request is within the rate limit.
    // Returns (allowed bool, tokensRemaining float64).
    // Atomically increments the counter if allowed.
    Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)

    // Cleanup removes expired windows.
    Cleanup(ctx context.Context, maxAge time.Duration) error
}
```

### 3.2 — PG Implementation: Sliding-Window Token Bucket

The in-memory implementation uses a classic token bucket (refill based
on elapsed time). The PG version approximates this with time-bucketed
counters.

**`Allow` — atomic check-and-increment:**

```sql
-- Upsert the current window's counter and check burst limit.
-- Window = current second (truncated).
INSERT INTO rate_limit_counters (key, window, tokens)
VALUES ($1, date_trunc('second', NOW()), 1)
ON CONFLICT (key, window)
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
RETURNING tokens
```

Go logic after the RETURNING:
1. Query returns `tokens` (count in the current 1-second window).
2. If `tokens > burst` → denied, return `(false, nil)`.
3. Else → allowed.

This is a fixed-window approximation, not a sliding window. For auth
rate limiting at 5 req/s burst 8, the practical difference is
negligible — worst case allows 2× burst at window boundaries, which
is acceptable for this use case. A true sliding window would require
reading adjacent windows and interpolating, adding complexity for
near-zero benefit at this scale.

**Cleanup:**
```sql
DELETE FROM rate_limit_counters
WHERE window < NOW() - $1::interval
```

Called alongside ticket reap in the scheduler poll loop.

### 3.3 — SQLite Implementation

Same logic, `datetime('now')` for window truncation. Single-process,
so the PG atomicity guarantees are naturally satisfied.

### 3.4 — Middleware Swap

Replace `middleware.RateLimiter` struct with one backed by the store:

```go
type RateLimiter struct {
    store store.RateLimitStore
    rate  float64
    burst int
}

func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
    return &RateLimiter{store: store, rate: rate, burst: burst}
}

func (rl *RateLimiter) Limit() gin.HandlerFunc {
    return func(c *gin.Context) {
        key := "auth:" + c.ClientIP()
        allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
        if err != nil {
            // DB error — fail open (don't block auth on rate limit DB failure)
            c.Next()
            return
        }
        if !allowed {
            c.Header("Retry-After", "1")
            c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
                "error": "rate limit exceeded",
            })
            return
        }
        c.Next()
    }
}
```

**Fail-open policy:** if PG is down, allow the request. The auth
endpoints have their own protections (bcrypt, lockout). Blocking
legitimate logins because the rate limit table is unreachable is worse
than allowing a brief burst.

### 3.5 — Wiring in `main.go`

Replace:
```go
authLimiter := middleware.NewRateLimiter(5, 8)
```

With:
```go
authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)
```

Delete the in-memory cleanup goroutine (replaced by scheduler reap).

### Deliverables

- [ ] `store.RateLimitStore` interface
- [ ] PG implementation + SQLite implementation
- [ ] `middleware.RateLimiter` rewrite — backed by store, fail-open
- [ ] `main.go` wiring
- [ ] Add `RateLimits` field to `store.Stores`
- [ ] Scheduler poll cleanup (rate_limit_counters + ws_tickets)
- [ ] Delete in-memory visitor map + cleanup goroutine
- [ ] CI green

---

## CS4 — WebSocket Cross-Pod Delivery

The event bus → `pg_broadcast` → remote `publishLocal` pipeline is
already implemented and handles room-scoped events correctly. The gap:
**`hub.SendToUser()` bypasses the bus entirely** — it writes directly
to local connection send channels. Users on other pods are never reached.

### Affected call sites (7 total)

| File | Context |
|------|---------|
| `main.go` | DM typing indicators |
| `handlers/completion.go` (×3) | Streaming chunks, tool results, mentions |
| `handlers/stream_loop.go` | SSE stream events |
| `handlers/tool_loop.go` | Browser tool call delivery |
| `notifications/service.go` | Push notification delivery |

### 4.1 — Add `TargetUserID` to Event

```go
type Event struct {
    Label    string          `json:"event"`
    Room     string          `json:"room,omitempty"`
    Payload  json.RawMessage `json:"payload"`
    Ts       int64           `json:"ts"`

    // Server-side metadata (not serialized to clients)
    SenderID     string `json:"-"`
    ConnID       string `json:"-"`
    TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
}
```

`TargetUserID` is included in the JSON serialization for `pg_broadcast`
(so the remote pod knows who to deliver to) but stripped before sending
to WebSocket clients. Add a custom `MarshalJSON` for the broadcast
path, or use a separate internal envelope.

**Simpler approach:** Use `json:"target_user_id,omitempty"` — it's
harmless if clients see it (they already ignore unknown fields), and
avoids a second serialization path. The field is empty for room-scoped
events and only populated for targeted delivery.

### 4.2 — Update WS Subscriber Filter

In `subscribeToBus`, add target filtering:

```go
// Targeted delivery: if event has a target user, only deliver to that user
if e.TargetUserID != "" && e.TargetUserID != c.userID {
    return
}
```

This goes before the room filter. Targeted events skip room filtering
entirely (they're user-scoped, not room-scoped).

### 4.3 — Rewrite `SendToUser` → `PublishToUser`

New method on `Hub`:

```go
// PublishToUser sends an event to a specific user via the bus.
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
func (h *Hub) PublishToUser(userID string, event Event) {
    event.TargetUserID = userID
    h.bus.Publish(event)
}
```

This replaces all 7 `hub.SendToUser` call sites. The event flows:

1. `hub.PublishToUser(userID, evt)`
2. `bus.Publish(evt)` → local subscribers + `broadcastHook`
3. Local pod: `subscribeToBus` filter matches `TargetUserID` → deliver
4. `broadcastHook` → `pg_notify` → remote pod
5. Remote pod: `publishLocal(evt)` → `subscribeToBus` filter → deliver

### 4.4 — Keep `SendToUser` as Local-Only Optimization

Don't delete `SendToUser` — rename to `sendToUserLocal` (unexported).
`PublishToUser` can optionally try local delivery first (fast path)
and only broadcast if the user isn't connected locally. But this
optimization is premature at 50 users — just always go through the
bus. Revisit post-MVP if latency matters.

### 4.5 — Audit: `tool.call` + `WaitFor`

`tool.call.*` events use `DirToClient` routing and are delivered via
`SendToUser`. After the rewrite, they go through the bus and will
broadcast to all pods. The `WaitFor` on the originating pod still
works — `tool.result.*` comes back from the client on the same pod
(the WS connection is sticky to a pod).

Verify: if the user's browser tool is connected to pod-2, but the
completion handler calling `WaitFor` is on pod-1, the tool call event
needs to reach pod-2 (via pg_broadcast), and the result needs to come
back to pod-1 (via pg_broadcast of `tool.result.*`). Check
`RouteFor("tool.result.*")` — currently `DirFromClient`, which means
the broadcast hook skips it. **This needs to change to `DirBoth`** so
tool results cross pod boundaries for `WaitFor` to work.

### Deliverables

- [ ] `TargetUserID` field on `Event`
- [ ] `subscribeToBus` target filter
- [ ] `Hub.PublishToUser` — replaces all `SendToUser` call sites
- [ ] Update `tool.result.*` routing to `DirBoth`
- [ ] Rename `SendToUser` → `sendToUserLocal` (unexported)
- [ ] Verify `WaitFor` works cross-pod (tool bridge scenario)
- [ ] CI green

---

## CS5 — Health Probes + Helm Validation

### 5.1 — Readiness Probe Refinement

Current health endpoint: basic HTTP 200. Add a PG ping:

```go
func (h *HealthHandler) Readiness(c *gin.Context) {
    ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
    defer cancel()
    if err := database.DB.PingContext(ctx); err != nil {
        c.JSON(503, gin.H{"error": "database unavailable"})
        return
    }
    c.JSON(200, gin.H{"status": "ok"})
}
```

Wire as `/healthz/ready` (separate from the existing `/healthz/live`
liveness probe). Kubernetes pulls the pod from the service on readiness
failure — new requests route to healthy replicas.

### 5.2 — Helm Changes

`chart/values.yaml`:
```yaml
backend:
  replicaCount: 2     # was 1
```

`chart/templates/deployment-backend.yaml` additions:

**Pod anti-affinity** — spread replicas across nodes:
```yaml
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: switchboard-backend
          topologyKey: kubernetes.io/hostname
```

`preferredDuring` (not `requiredDuring`) — if fewer nodes than
replicas, pods still schedule on the same node (degraded HA is better
than no scheduling).

**Readiness probe:**
```yaml
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3
```

### 5.3 — Startup Jitter

Add random jitter (0–15s) to the scheduler's initial tick to stagger
replica polling:

```go
func (s *Scheduler) Run() {
    // ...
    jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
    time.Sleep(jitter)
    log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
    // ...
}
```

Not strictly necessary with `SKIP LOCKED` (contention is harmless),
but reduces unnecessary lock acquisition at startup.

### 5.4 — Validation Checklist

Manual validation on the `gobha-ai-chat` cluster:

- [ ] Scale to 2 replicas: `kubectl scale deploy switchboard-backend --replicas=2`
- [ ] Verify pods land on different nodes (`kubectl get pods -o wide`)
- [ ] Create a cron task (every 1 min), confirm only one run per tick
- [ ] WebSocket: connect on pod-1, send message that completes on
      pod-2, verify streaming events arrive
- [ ] Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
- [ ] Rate limit: hit `/auth/login` 10× rapidly — confirm 429 after
      burst regardless of which replica serves each request
- [ ] Kill one pod (`kubectl delete pod ...`), confirm the other
      continues serving + picks up scheduler duties immediately
- [ ] `kubectl top pod` — memory baseline at 2 replicas under light load

### Deliverables

- [ ] `/healthz/ready` endpoint with PG ping
- [ ] Helm: `replicaCount: 2`, pod anti-affinity, readiness probe
- [ ] Scheduler startup jitter
- [ ] Validation checklist executed and documented
- [ ] CI green, tag `v0.32.0`

---

## Summary: Store Interface Additions

```go
type Stores struct {
    // ... existing fields ...
    Tickets    TicketStore     // v0.32.0: WS auth tickets (PG-backed)
    RateLimits RateLimitStore  // v0.32.0: Distributed rate limiting
}
```

`TaskStore` changes:
- **Add:** `ClaimDueTask(ctx) (*Task, error)`
- **Add:** `CreateRunExclusive(ctx, taskID) (*TaskRun, error)`
- **Remove:** `ListDue(ctx, limit) ([]Task, error)`

---

## Changeset Sequence

| CS | Scope | Key Files | Gate |
|----|-------|-----------|------|
| CS0 | Schema | `020_ha.sql` (PG + SQLite) | Migrations run, tables exist |
| CS1 | Task scheduler | `store/task_iface.go`, `store/postgres/tasks.go`, `store/sqlite/tasks.go`, `scheduler/scheduler.go` | Concurrent claim test passes |
| CS2 | Ticket store | `store/ticket_iface.go`, `store/postgres/tickets.go`, `store/sqlite/tickets.go`, `events/tickets.go` (delete), `main.go` | WS auth works with PG tickets |
| CS3 | Rate limiter | `store/ratelimit_iface.go`, `store/postgres/ratelimit.go`, `store/sqlite/ratelimit.go`, `middleware/ratelimit.go`, `main.go` | Rate limit shared across replicas |
| CS4 | WS fan-out | `events/types.go`, `events/ws.go`, all `SendToUser` call sites | Cross-pod event delivery |
| CS5 | Health + Helm | `handlers/health.go`, `chart/`, `scheduler/scheduler.go` | 2-replica cluster validated |

---

## Risk Assessment

**Low risk:** CS0 (schema), CS2 (ticket store — simple CRUD), CS3
(rate limiter — auth-only, fail-open).

**Medium risk:** CS1 (scheduler rewrite — core behavior change, but
well-scoped and testable), CS5 (Helm — infra changes, but rollback is
`--replicas=1`).

**Higher risk:** CS4 (WS fan-out — touches 7 call sites across
handlers, completion, notifications). The `tool.result` routing change
(`DirFromClient` → `DirBoth`) needs careful testing — if tool results
broadcast to all pods, the non-originating pod's `WaitFor` must not
accidentally consume the result. Verify: `WaitFor` subscribes to
`tool.result.{specific-id}` (exact label match), so only the waiting
goroutine receives it. Safe.

**Fallback:** If any CS breaks multi-replica, `replicaCount: 1` is
always a safe rollback. Each CS should leave CI green at single-replica.