core/docs/DESIGN-0.32.0.md

DESIGN-0.32.0 — Multi-Replica HA

Run 2–3 backend replicas across nodes for node-level availability. Five changesets. No sub-versions — the work is tightly coupled and each CS builds on the previous.

Depends on: v0.31.2 (current HEAD).

Design decision: PG SKIP LOCKED replaces Kubernetes Lease-based leader election for the task scheduler. Every replica polls; PG serializes the claims. Extends to all shared mutable state — tickets, rate counters — keeping PG as the sole coordination layer. No Redis, no K8s API dependency, no new infrastructure.

---

What Already Works Multi-Replica

These require zero changes:

• REST API — stateless, JWT auth, any replica serves any request.
• PG + S3 + CephFS — shared storage infrastructure.
• pg_broadcast LISTEN/NOTIFY — cross-pod event bus. Publish() → broadcastHook → pg_notify → remote pod publishLocal() → local WS subscribers. Fully wired, just never tested at N>1.

---

What Needs Work

Five areas of in-process mutable state that break at replica count > 1:

State	Current	Problem at N>1	Fix
Task scheduler	Single goroutine polls ListDue	All replicas fire same tasks	SKIP LOCKED atomic claim
Task run guard	GetActiveRun check → CreateRun	TOCTOU race window	Conditional INSERT ... WHERE NOT EXISTS
WS ticket store	sync.Map per-pod	Ticket from pod-1 invalid on pod-2	PG table with TTL reaper
Rate limiter	In-memory token bucket per-pod	Effective limit = N × configured	PG counter with time bucket
SendToUser	Local hub lookup only	User on pod-2 never reached	Route through bus → pg_broadcast

---

CS0 — Schema: 020_ha.sql

New migration for both PG and SQLite. No changes to existing 019 task schema — next_run_at nullable already supports the claim mechanism.

PG: server/database/migrations/020_ha.sql

-- ==========================================
-- Chat Switchboard — 020 Multi-Replica HA
-- ==========================================
-- Shared state tables for multi-replica operation.
-- v0.32.0: ws_tickets, rate_limit_counters.
-- ==========================================

-- =========================================
-- WEBSOCKET TICKETS (replaces sync.Map)
-- =========================================

CREATE TABLE IF NOT EXISTS ws_tickets (
    id         TEXT PRIMARY KEY,           -- 128-bit hex token
    user_id    UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    expires_at TIMESTAMPTZ NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
    ON ws_tickets (expires_at);

-- =========================================
-- RATE LIMIT COUNTERS (replaces in-memory)
-- =========================================
-- Token bucket approximation using time-windowed counters.
-- Key format: "{scope}:{identifier}" e.g. "auth:192.168.1.1"
-- Window is truncated to the second for the configured rate.

CREATE TABLE IF NOT EXISTS rate_limit_counters (
    key        TEXT NOT NULL,
    window     TIMESTAMPTZ NOT NULL,       -- truncated timestamp (window start)
    tokens     REAL NOT NULL DEFAULT 0,    -- tokens consumed in this window
    PRIMARY KEY (key, window)
);

CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
    ON rate_limit_counters (window);

SQLite: server/database/migrations/sqlite/020_ha.sql

SQLite parity — structurally identical but with dialect adjustments. These tables are functional in SQLite for single-process test parity, though multi-replica is PG-only in production.

CREATE TABLE IF NOT EXISTS ws_tickets (
    id         TEXT PRIMARY KEY,
    user_id    TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    expires_at TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_ws_tickets_expires
    ON ws_tickets (expires_at);

CREATE TABLE IF NOT EXISTS rate_limit_counters (
    key        TEXT NOT NULL,
    window     TEXT NOT NULL,
    tokens     REAL NOT NULL DEFAULT 0,
    PRIMARY KEY (key, window)
);

CREATE INDEX IF NOT EXISTS idx_rate_limit_counters_window
    ON rate_limit_counters (window);

Deliverables

• 020_ha.sql (PG)
• 020_ha.sql (SQLite)
• CI green on both pipelines (migration runs, tables exist)

---

CS1 — Task Scheduler: SKIP LOCKED

Replace the read-then-execute scheduler with atomic claim semantics. Every replica runs the poll loop. PG serializes task handoff.

1.1 — Atomic ClaimDueTask (replaces ListDue)

New store method. Single atomic statement — SELECT + UPDATE in one round trip. Returns at most one task per call.

PG implementation:

UPDATE tasks
SET next_run_at = NULL
WHERE id = (
    SELECT id FROM tasks
    WHERE is_active = true
      AND next_run_at <= NOW()
    ORDER BY next_run_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING <taskColumns>

Mechanics:

• FOR UPDATE SKIP LOCKED — if another replica holds a lock on a candidate row, skip it instantly (no wait, no deadlock).
• SET next_run_at = NULL — claimed task disappears from future polls. advanceNextRun restores it after execution.
• Returns zero rows if nothing is due → sql.ErrNoRows → no-op.
• The existing partial index idx_tasks_next_run covers the WHERE clause (is_active = true AND next_run_at IS NOT NULL by implication — NULL rows won't satisfy <= NOW()).

SQLite implementation:

SQLite has no SKIP LOCKED. Single-process, so not needed. Keep the current ListDue behavior unchanged for SQLite. The store interface accommodates both:

// TaskStore additions
ClaimDueTask(ctx context.Context) (*models.Task, error)

SQLite ClaimDueTask just does:

SELECT <taskColumns> FROM tasks
WHERE is_active = 1 AND next_run_at <= datetime('now')
ORDER BY next_run_at ASC LIMIT 1

Then sets next_run_at = NULL in a second statement (single writer, no contention).

1.2 — Conditional CreateRunExclusive

Belt-and-suspenders: prevent double-execution even if two replicas somehow both claim the same task (shouldn't happen with SKIP LOCKED, but defense in depth).

INSERT INTO task_runs (task_id, status)
SELECT $1, 'running'
WHERE NOT EXISTS (
    SELECT 1 FROM task_runs
    WHERE task_id = $1 AND status IN ('running', 'queued')
)
RETURNING id, started_at

Returns sql.ErrNoRows if a run already exists → skip execution.

New store method:

CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)

1.3 — Scheduler Loop Rewrite

scheduler.go changes:

func (s *Scheduler) poll() {
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    cfg := taskutil.LoadTaskConfig(ctx, s.stores.GlobalConfig)
    if !cfg.Enabled {
        return
    }

    // Claim tasks one at a time until none remain or budget exhausted.
    // Each replica claims independently — PG SKIP LOCKED serializes.
    claimed := 0
    for claimed < cfg.MaxConcurrent {
        task, err := s.stores.Tasks.ClaimDueTask(ctx)
        if err != nil {
            break // sql.ErrNoRows or real error — either way, done
        }
        claimed++
        go s.execute(ctx, task)
    }
}

execute() changes:

• Remove the GetActiveRun TOCTOU check — replaced by CreateRunExclusive.
• GetQueuedRun stays — webhook-triggered runs still need adoption. But adoption also uses CreateRunExclusive semantics (transition only if still queued).
• advanceNextRun still computes and sets next_run_at, making the task visible to future polls again.

1.4 — Remove ListDue from Interface

ListDue removed from TaskStore interface. Replaced by ClaimDueTask. CreateRun remains for webhook trigger path; CreateRunExclusive added alongside it.

Updated interface:

type TaskStore interface {
    // ... existing CRUD ...

    // Scheduler queries
    ClaimDueTask(ctx context.Context) (*models.Task, error)
    SetNextRun(ctx context.Context, id string, nextRun interface{}) error
    SetLastRun(ctx context.Context, id string) error
    IncrementRunCount(ctx context.Context, id string) error

    // Run history
    CreateRun(ctx context.Context, r *models.TaskRun) error
    CreateRunExclusive(ctx context.Context, taskID string) (*models.TaskRun, error)
    UpdateRun(ctx context.Context, id string, status string, ...) error
    TransitionRunStatus(ctx context.Context, id string, status string) error
    GetActiveRun(ctx context.Context, taskID string) (*models.TaskRun, error)
    GetQueuedRun(ctx context.Context, taskID string) (*models.TaskRun, error)
    ListRuns(ctx context.Context, taskID string, limit int) ([]models.TaskRun, error)
}

Deliverables

• ClaimDueTask — PG (FOR UPDATE SKIP LOCKED) + SQLite (simple select)
• CreateRunExclusive — PG + SQLite
• Scheduler loop rewrite — claim-per-iteration, no TOCTOU
• Remove ListDue from interface + both stores
• Unit tests: concurrent claim (PG only — two goroutines, verify disjoint task sets)
• CI green

---

CS2 — PG Ticket Store

Replace events.TicketStore (in-memory sync.Map) with a PG-backed implementation. The middleware already programs to the TicketValidator interface — the swap is clean.

2.1 — Store Interface

New interface in store/:

// TicketStore manages short-lived single-use WebSocket auth tickets.
type TicketStore interface {
    Issue(ctx context.Context, userID string) (string, error)
    Validate(ctx context.Context, ticketID string) (string, bool)
    Reap(ctx context.Context) (int, error)
}

2.2 — PG Implementation

Issue:

INSERT INTO ws_tickets (id, user_id, expires_at)
VALUES ($1, $2, NOW() + INTERVAL '30 seconds')

Token generation stays in Go (crypto/rand, 16 bytes, hex-encoded).

Validate (atomic consume):

DELETE FROM ws_tickets
WHERE id = $1 AND expires_at > NOW()
RETURNING user_id

Single statement — delete + return. If expired or already consumed, zero rows → ("", false).

Reap (TTL cleanup):

DELETE FROM ws_tickets WHERE expires_at <= NOW()

Called by a system task or inline during poll. No dedicated goroutine needed — the scheduler's 30s tick can piggyback, or register as a lightweight system function.

2.3 — SQLite Implementation

Identical SQL with dialect adjustments (datetime('now', '+30 seconds') instead of NOW() + INTERVAL).

2.4 — Adapter for Middleware Interface

The middleware TicketValidator interface is:

type TicketValidator interface {
    Validate(ticketID string) (string, bool)
}

The store's Validate takes a context.Context. Thin adapter:

// TicketValidatorAdapter bridges store.TicketStore → middleware.TicketValidator.
type TicketValidatorAdapter struct {
    Store store.TicketStore
}

func (a *TicketValidatorAdapter) Validate(ticketID string) (string, bool) {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    return a.Store.Validate(ctx, ticketID)
}

2.5 — Wiring in main.go

Replace:

ticketStore := events.NewTicketStore()
defer ticketStore.Stop()

With:

ticketAdapter := &events.TicketValidatorAdapter{Store: stores.Tickets}

The POST /ws/ticket endpoint calls stores.Tickets.Issue(ctx, userID) directly. The WS auth middleware receives ticketAdapter (satisfies TicketValidator). No reaper goroutine — use system function or scheduler piggyback.

2.6 — Reap Strategy

Option A: Register ticket_reap as a system function (taskutil.SystemRegistry), add a system task with schedule: "@hourly" or "*/5 * * * *". Pros: visible in admin UI, uses existing infrastructure. Cons: requires task to exist.

Option B: Inline reap — call stores.Tickets.Reap(ctx) at the top of the scheduler poll() loop (every 30s, cheap no-op when table is empty). Pros: zero config. Cons: slightly couples scheduler to tickets.

Recommendation: Option B. At 50 users, the table will have at most a few dozen rows. A DELETE WHERE expires_at <= NOW() every 30s is free. Revisit if ticket volume grows.

2.7 — Delete events/tickets.go

The in-memory TicketStore struct, NewTicketStore(), and reaper goroutine are fully replaced. Remove the file. The TicketValidatorAdapter lives in events/ (or alongside the middleware — whichever avoids import cycles).

Deliverables

• store.TicketStore interface
• PG implementation + SQLite implementation
• TicketValidatorAdapter — bridges context-aware store to TicketValidator interface
• main.go wiring — swap in PG store, remove events.NewTicketStore()
• Inline reap in scheduler poll
• Delete events/tickets.go
• Add Tickets field to store.Stores
• CI green

---

CS3 — PG Rate Limiter

Replace the in-memory token bucket (middleware.RateLimiter) with a PG-backed counter. Currently only applied to /auth routes (5 req/s, burst 8), so transaction volume is trivially low.

3.1 — Store Interface

// RateLimitStore manages distributed rate limit counters.
type RateLimitStore interface {
    // Allow checks if a request is within the rate limit.
    // Returns (allowed bool, tokensRemaining float64).
    // Atomically increments the counter if allowed.
    Allow(ctx context.Context, key string, rate float64, burst int) (bool, error)

    // Cleanup removes expired windows.
    Cleanup(ctx context.Context, maxAge time.Duration) error
}

3.2 — PG Implementation: Sliding-Window Token Bucket

The in-memory implementation uses a classic token bucket (refill based on elapsed time). The PG version approximates this with time-bucketed counters.

Allow — atomic check-and-increment:

-- Upsert the current window's counter and check burst limit.
-- Window = current second (truncated).
INSERT INTO rate_limit_counters (key, window, tokens)
VALUES ($1, date_trunc('second', NOW()), 1)
ON CONFLICT (key, window)
DO UPDATE SET tokens = rate_limit_counters.tokens + 1
RETURNING tokens

Go logic after the RETURNING:

1. Query returns tokens (count in the current 1-second window).
2. If tokens > burst → denied, return (false, nil).
3. Else → allowed.

This is a fixed-window approximation, not a sliding window. For auth rate limiting at 5 req/s burst 8, the practical difference is negligible — worst case allows 2× burst at window boundaries, which is acceptable for this use case. A true sliding window would require reading adjacent windows and interpolating, adding complexity for near-zero benefit at this scale.

Cleanup:

DELETE FROM rate_limit_counters
WHERE window < NOW() - $1::interval

Called alongside ticket reap in the scheduler poll loop.

3.3 — SQLite Implementation

Same logic, datetime('now') for window truncation. Single-process, so the PG atomicity guarantees are naturally satisfied.

3.4 — Middleware Swap

Replace middleware.RateLimiter struct with one backed by the store:

type RateLimiter struct {
    store store.RateLimitStore
    rate  float64
    burst int
}

func NewRateLimiter(store store.RateLimitStore, rate float64, burst int) *RateLimiter {
    return &RateLimiter{store: store, rate: rate, burst: burst}
}

func (rl *RateLimiter) Limit() gin.HandlerFunc {
    return func(c *gin.Context) {
        key := "auth:" + c.ClientIP()
        allowed, err := rl.store.Allow(c.Request.Context(), key, rl.rate, rl.burst)
        if err != nil {
            // DB error — fail open (don't block auth on rate limit DB failure)
            c.Next()
            return
        }
        if !allowed {
            c.Header("Retry-After", "1")
            c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
                "error": "rate limit exceeded",
            })
            return
        }
        c.Next()
    }
}

Fail-open policy: if PG is down, allow the request. The auth endpoints have their own protections (bcrypt, lockout). Blocking legitimate logins because the rate limit table is unreachable is worse than allowing a brief burst.

3.5 — Wiring in main.go

Replace:

authLimiter := middleware.NewRateLimiter(5, 8)

With:

authLimiter := middleware.NewRateLimiter(stores.RateLimits, 5, 8)

Delete the in-memory cleanup goroutine (replaced by scheduler reap).

Deliverables

• store.RateLimitStore interface
• PG implementation + SQLite implementation
• middleware.RateLimiter rewrite — backed by store, fail-open
• main.go wiring
• Add RateLimits field to store.Stores
• Scheduler poll cleanup (rate_limit_counters + ws_tickets)
• Delete in-memory visitor map + cleanup goroutine
• CI green

---

CS4 — WebSocket Cross-Pod Delivery

The event bus → pg_broadcast → remote publishLocal pipeline is already implemented and handles room-scoped events correctly. The gap: hub.SendToUser() bypasses the bus entirely — it writes directly to local connection send channels. Users on other pods are never reached.

Affected call sites (7 total)

File	Context
main.go	DM typing indicators
handlers/completion.go (×3)	Streaming chunks, tool results, mentions
handlers/stream_loop.go	SSE stream events
handlers/tool_loop.go	Browser tool call delivery
notifications/service.go	Push notification delivery

4.1 — Add TargetUserID to Event

type Event struct {
    Label    string          `json:"event"`
    Room     string          `json:"room,omitempty"`
    Payload  json.RawMessage `json:"payload"`
    Ts       int64           `json:"ts"`

    // Server-side metadata (not serialized to clients)
    SenderID     string `json:"-"`
    ConnID       string `json:"-"`
    TargetUserID string `json:"-"` // v0.32.0: cross-pod targeted delivery
}

TargetUserID is included in the JSON serialization for pg_broadcast (so the remote pod knows who to deliver to) but stripped before sending to WebSocket clients. Add a custom MarshalJSON for the broadcast path, or use a separate internal envelope.

Simpler approach: Use json:"target_user_id,omitempty" — it's harmless if clients see it (they already ignore unknown fields), and avoids a second serialization path. The field is empty for room-scoped events and only populated for targeted delivery.

4.2 — Update WS Subscriber Filter

In subscribeToBus, add target filtering:

// Targeted delivery: if event has a target user, only deliver to that user
if e.TargetUserID != "" && e.TargetUserID != c.userID {
    return
}

This goes before the room filter. Targeted events skip room filtering entirely (they're user-scoped, not room-scoped).

4.3 — Rewrite SendToUser → PublishToUser

New method on Hub:

// PublishToUser sends an event to a specific user via the bus.
// Cross-pod safe: the bus broadcast hook fans out via pg_notify.
func (h *Hub) PublishToUser(userID string, event Event) {
    event.TargetUserID = userID
    h.bus.Publish(event)
}

This replaces all 7 hub.SendToUser call sites. The event flows:

1. hub.PublishToUser(userID, evt)
2. bus.Publish(evt) → local subscribers + broadcastHook
3. Local pod: subscribeToBus filter matches TargetUserID → deliver
4. broadcastHook → pg_notify → remote pod
5. Remote pod: publishLocal(evt) → subscribeToBus filter → deliver

4.4 — Keep SendToUser as Local-Only Optimization

Don't delete SendToUser — rename to sendToUserLocal (unexported). PublishToUser can optionally try local delivery first (fast path) and only broadcast if the user isn't connected locally. But this optimization is premature at 50 users — just always go through the bus. Revisit post-MVP if latency matters.

4.5 — Audit: tool.call + WaitFor

tool.call.* events use DirToClient routing and are delivered via SendToUser. After the rewrite, they go through the bus and will broadcast to all pods. The WaitFor on the originating pod still works — tool.result.* comes back from the client on the same pod (the WS connection is sticky to a pod).

Verify: if the user's browser tool is connected to pod-2, but the completion handler calling WaitFor is on pod-1, the tool call event needs to reach pod-2 (via pg_broadcast), and the result needs to come back to pod-1 (via pg_broadcast of tool.result.*). Check RouteFor("tool.result.*") — currently DirFromClient, which means the broadcast hook skips it. This needs to change to DirBoth so tool results cross pod boundaries for WaitFor to work.

Deliverables

• TargetUserID field on Event
• subscribeToBus target filter
• Hub.PublishToUser — replaces all SendToUser call sites
• Update tool.result.* routing to DirBoth
• Rename SendToUser → sendToUserLocal (unexported)
• Verify WaitFor works cross-pod (tool bridge scenario)
• CI green

---

CS5 — Health Probes + Helm Validation

5.1 — Readiness Probe Refinement

Current health endpoint: basic HTTP 200. Add a PG ping:

func (h *HealthHandler) Readiness(c *gin.Context) {
    ctx, cancel := context.WithTimeout(c.Request.Context(), 2*time.Second)
    defer cancel()
    if err := database.DB.PingContext(ctx); err != nil {
        c.JSON(503, gin.H{"error": "database unavailable"})
        return
    }
    c.JSON(200, gin.H{"status": "ok"})
}

Wire as /healthz/ready (separate from the existing /healthz/live liveness probe). Kubernetes pulls the pod from the service on readiness failure — new requests route to healthy replicas.

5.2 — Helm Changes

chart/values.yaml:

backend:
  replicaCount: 2     # was 1

chart/templates/deployment-backend.yaml additions:

Pod anti-affinity — spread replicas across nodes:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: switchboard-backend
          topologyKey: kubernetes.io/hostname

preferredDuring (not requiredDuring) — if fewer nodes than replicas, pods still schedule on the same node (degraded HA is better than no scheduling).

Readiness probe:

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

5.3 — Startup Jitter

Add random jitter (0–15s) to the scheduler's initial tick to stagger replica polling:

func (s *Scheduler) Run() {
    // ...
    jitter := time.Duration(rand.Intn(15000)) * time.Millisecond
    time.Sleep(jitter)
    log.Printf("[scheduler] Started (jitter=%s, interval=%s)", jitter, s.interval)
    // ...
}

Not strictly necessary with SKIP LOCKED (contention is harmless), but reduces unnecessary lock acquisition at startup.

5.4 — Validation Checklist

Manual validation on the gobha-ai-chat cluster:

• Scale to 2 replicas: kubectl scale deploy switchboard-backend --replicas=2
• Verify pods land on different nodes (kubectl get pods -o wide)
• Create a cron task (every 1 min), confirm only one run per tick
• WebSocket: connect on pod-1, send message that completes on pod-2, verify streaming events arrive
• Issue WS ticket on pod-1, connect WS on pod-2 — ticket validates
• Rate limit: hit /auth/login 10× rapidly — confirm 429 after burst regardless of which replica serves each request
• Kill one pod (kubectl delete pod ...), confirm the other continues serving + picks up scheduler duties immediately
• kubectl top pod — memory baseline at 2 replicas under light load

Deliverables

• /healthz/ready endpoint with PG ping
• Helm: replicaCount: 2, pod anti-affinity, readiness probe
• Scheduler startup jitter
• Validation checklist executed and documented
• CI green, tag v0.32.0

---

Summary: Store Interface Additions

type Stores struct {
    // ... existing fields ...
    Tickets    TicketStore     // v0.32.0: WS auth tickets (PG-backed)
    RateLimits RateLimitStore  // v0.32.0: Distributed rate limiting
}

TaskStore changes:

• Add: ClaimDueTask(ctx) (*Task, error)
• Add: CreateRunExclusive(ctx, taskID) (*TaskRun, error)
• Remove: ListDue(ctx, limit) ([]Task, error)

---

Changeset Sequence

CS	Scope	Key Files	Gate
CS0	Schema	020_ha.sql (PG + SQLite)	Migrations run, tables exist
CS1	Task scheduler	store/task_iface.go, store/postgres/tasks.go, store/sqlite/tasks.go, scheduler/scheduler.go	Concurrent claim test passes
CS2	Ticket store	store/ticket_iface.go, store/postgres/tickets.go, store/sqlite/tickets.go, events/tickets.go (delete), main.go	WS auth works with PG tickets
CS3	Rate limiter	store/ratelimit_iface.go, store/postgres/ratelimit.go, store/sqlite/ratelimit.go, middleware/ratelimit.go, main.go	Rate limit shared across replicas
CS4	WS fan-out	events/types.go, events/ws.go, all SendToUser call sites	Cross-pod event delivery
CS5	Health + Helm	handlers/health.go, chart/, scheduler/scheduler.go	2-replica cluster validated

---

Risk Assessment

Low risk: CS0 (schema), CS2 (ticket store — simple CRUD), CS3 (rate limiter — auth-only, fail-open).

Medium risk: CS1 (scheduler rewrite — core behavior change, but well-scoped and testable), CS5 (Helm — infra changes, but rollback is --replicas=1).

Higher risk: CS4 (WS fan-out — touches 7 call sites across handlers, completion, notifications). The tool.result routing change (DirFromClient → DirBoth) needs careful testing — if tool results broadcast to all pods, the non-originating pod's WaitFor must not accidentally consume the result. Verify: WaitFor subscribes to tool.result.{specific-id} (exact label match), so only the waiting goroutine receives it. Safe.

Fallback: If any CS breaks multi-replica, replicaCount: 1 is always a safe rollback. Each CS should leave CI green at single-replica.