core/docs/DESIGN-0.15.0.md

DESIGN-0.15.0 — Compaction

Overview

Automatic conversation compaction. A background service monitors channels for context pressure and triggers summarization via the utility model role, replacing the manual "Summarize & Continue" button as the primary compaction path. Manual summarization remains available as an explicit user action.

Depends on: utility model role (v0.10.0), summarize handler (v0.10.2).

Design principle: don't reinvent the wheel. The existing SummarizeHandler already has the full pipeline — path loading, summary boundary detection, prompt construction, role resolution, tree insertion, cursor update, usage logging. This design extracts that core logic into a shared compaction package and adds a background scanner on top.

---

1. Extract: compaction Package

Move the reusable summarization pipeline out of handlers/summarize.go into server/compaction/compaction.go. The HTTP handler becomes a thin wrapper.

Core type

package compaction

// Service provides conversation compaction (summarization).
// Used by both the HTTP handler (manual) and the background scanner (auto).
type Service struct {
    stores   store.Stores
    resolver *roles.Resolver
}

func NewService(stores store.Stores, resolver *roles.Resolver) *Service {
    return &Service{stores: stores, resolver: resolver}
}

Extracted method: Compact

The current SummarizeHandler.Summarize body becomes Service.Compact:

type CompactRequest struct {
    ChannelID string
    UserID    string     // channel owner
    TeamID    *string    // for role resolution
    Trigger   string     // "manual" | "auto"
}

type CompactResult struct {
    SummaryID       string
    SummarizedCount int
    Model           string
    UsedFallback    bool
    Content         string
    InputTokens     int
    OutputTokens    int
}

func (s *Service) Compact(ctx context.Context, req CompactRequest) (*CompactResult, error)

The method performs the same steps as today's handler:

1. Load active path via getActivePath (already exported within handlers)
2. Find summary boundary, collect messages to summarize
3. Guard: minimum 4 messages since last summary
4. Build summarization prompt
5. Call resolver.Complete(ctx, RoleUtility, ...)
6. Insert summary tree node with metadata
7. Update cursor
8. Log usage

The only new metadata field is "trigger" ("manual" or "auto") so the frontend can distinguish how the summary was created.

Refactored handler

handlers/summarize.go shrinks to:

func (h *SummarizeHandler) Summarize(c *gin.Context) {
    // ownership check, rate limit check (unchanged)
    // ...
    result, err := h.compaction.Compact(c.Request.Context(), compaction.CompactRequest{
        ChannelID: channelID,
        UserID:    userID,
        TeamID:    teamID,
        Trigger:   "manual",
    })
    // return JSON response (unchanged)
}

Tree helpers

getActivePath, isSummaryMessage, nextSiblingIndex, updateCursor are currently unexported in handlers/tree.go. Two options:

Option A — Export them from handlers and import in compaction. Clean but creates a dependency from compaction → handlers.

Option B — Move tree helpers into a treepath (or similar) package imported by both handlers and compaction.

Recommend Option B to keep the dependency graph clean: handlers → compaction → treepath ← handlers. The treepath package is pure data + SQL queries with no handler logic.

server/
  treepath/          ← NEW: extracted from handlers/tree.go
    path.go          (getActivePath, getPathToLeaf, getActiveLeaf)
    summary.go       (isSummaryMessage, summary metadata helpers)
    siblings.go      (getSiblingCount, getSiblings, findLeafFromMessage)
    cursor.go        (updateCursor, nextSiblingIndex)
  compaction/        ← NEW
    compaction.go    (Service, Compact)
    scanner.go       (Scanner, background loop)
    estimator.go     (token estimation)
  handlers/
    tree.go          → thin wrappers or deleted, imports treepath
    summarize.go     → thin HTTP wrapper, delegates to compaction.Service
    completion.go    → imports treepath for path building

---

2. Server-Side Token Estimation

The background scanner needs to evaluate context pressure without making an LLM call. The frontend already does this with a chars / 4 heuristic (tokens.js). Replicate server-side:

// compaction/estimator.go

// EstimateTokens returns a rough token count using the ~4 chars/token
// heuristic. Matches the frontend Tokens.estimate() for consistency.
func EstimateTokens(text string) int {
    return (len(text) + 3) / 4
}

// EstimatePath returns total estimated tokens for a message path,
// including system prompt overhead.
func EstimatePath(path []treepath.PathMessage, systemPromptLen int) int {
    total := 0
    if systemPromptLen > 0 {
        total += EstimateTokens(string(make([]byte, systemPromptLen))) + 4
    }
    for _, m := range path {
        total += EstimateTokens(m.Content) + 4 // +4 per-message overhead
    }
    return total
}

Not exact, but it's the same bar the user already sees in the UI. If we need better accuracy later, swap in a tokenizer library without changing the interface.

Context budget resolution

The scanner needs max_context for whatever model the channel is using. Resolution chain:

1. Channel has a model field → look up model_catalog for capabilities
2. Channel has a provider_config_id → use that provider's catalog entry
3. Fallback: use the utility role's model context window as a proxy (it's the model that would be doing the summarization anyway)
4. Hard fallback: 128K tokens (conservative default)

func (s *Scanner) getContextBudget(ctx context.Context, ch *models.Channel) int {
    if ch.Model != "" {
        if caps, err := s.stores.Catalog.GetCapabilities(ctx, ch.Model); err == nil {
            if caps.MaxContext > 0 {
                return caps.MaxContext
            }
        }
    }
    return DefaultContextBudget // 128000
}

---

3. Background Scanner

Follows the Ingester pattern from knowledge/ingest.go: goroutine pool with semaphore, WaitGroup for graceful shutdown.

Scanner type

// compaction/scanner.go

type Scanner struct {
    service    *Service
    stores     store.Stores
    sem        chan struct{}
    wg         sync.WaitGroup
    stopCh     chan struct{}
    ticker     *time.Ticker
}

type ScannerConfig struct {
    Enabled     bool          // global kill switch
    Interval    time.Duration // scan frequency (default: 5 minutes)
    Concurrency int           // max parallel compactions (default: 2)
}

func NewScanner(svc *Service, stores store.Stores, cfg ScannerConfig) *Scanner
func (sc *Scanner) Start()
func (sc *Scanner) Stop()  // signals stop + drains in-flight

Scan loop

every <interval>:
    if !globalEnabled(): continue

    candidates = findCandidates()
    for each candidate:
        sem.acquire()
        go compact(candidate)

Candidate query

A single SQL query finds channels that need compaction:

SELECT c.id, c.user_id, c.model, c.settings,
       COUNT(m.id) as msg_count,
       SUM(LENGTH(m.content)) as total_chars
FROM channels c
JOIN messages m ON m.channel_id = c.id
    AND m.deleted_at IS NULL
WHERE c.deleted_at IS NULL
  AND c.type = 'direct'              -- only user conversations (not groups yet)
  AND c.is_archived = false
  -- Skip channels with very recent activity (let the user finish typing)
  AND c.updated_at < NOW() - INTERVAL '2 minutes'
  -- Only channels active in the last 7 days (don't compact stale channels)
  AND c.updated_at > NOW() - INTERVAL '7 days'
GROUP BY c.id
HAVING COUNT(m.id) >= 10             -- minimum message count
   AND SUM(LENGTH(m.content)) > 20000 -- rough: 20K chars ≈ 5K tokens minimum
ORDER BY c.updated_at DESC
LIMIT 50                              -- batch size cap

This is a coarse filter. Each candidate then gets a precise check:

func (sc *Scanner) shouldCompact(ctx context.Context, ch *models.Channel) bool {
    // 1. Check channel-level opt-out
    if ch.Settings != nil {
        if v, ok := ch.Settings["auto_compaction"]; ok && v == false {
            return false
        }
    }

    // 2. Load active path for the channel owner
    path, err := treepath.GetActivePath(ch.ID, ch.UserID)
    if err != nil || len(path) < 10 {
        return false
    }

    // 3. Find post-summary messages only
    startIdx := 0
    for i, m := range path {
        if treepath.IsSummaryMessage(&m) {
            startIdx = i + 1
        }
    }
    postSummary := path[startIdx:]
    if len(postSummary) < 8 {
        return false // not enough new messages since last summary
    }

    // 4. Estimate tokens and check threshold
    tokens := EstimatePath(postSummary, 0)
    budget := sc.getContextBudget(ctx, ch)
    threshold := sc.getThreshold(ch) // default 0.70

    return float64(tokens) / float64(budget) >= threshold
}

Threshold

Default: 0.70 (70% of context window). This is intentionally lower than the frontend's 75% warning bar — auto-compaction should fire before the user sees a warning, not after.

Configurable per-channel via settings.compaction_threshold (float) and globally via global_settings key auto_compaction_threshold.

Rate limiting

Auto-compaction calls use the utility role and are org-funded. They share the existing utility_rate_limit budget but get a separate counter prefix so admins can see auto vs manual usage:

entry := &models.UsageEntry{
    // ...
    Role:     &role,          // "utility"
    Metadata: models.JSONMap{
        "trigger": "auto",   // distinguishes from manual
    },
}

If the global utility rate limit is hit, auto-compaction silently skips the channel and retries next scan cycle. It never blocks or errors.

---

4. Per-Channel Configuration

Uses the existing Channel.Settings JSONB column. No migration needed.

Key	Type	Default	Description
auto_compaction	bool	true (from global)	Enable/disable for this channel
compaction_threshold	float	0.70 (from global)	Context usage ratio to trigger

Frontend: channel settings

Add a "Compaction" section to the channel settings panel (the gear icon in the chat header). Two controls:

┌─ Compaction ──────────────────────────────────┐
│ Auto-compact   [✓]                            │
│ Threshold      [70]%  of context window       │
└───────────────────────────────────────────────┘

The manual "Summarize & Continue" button remains in the context warning bar and works regardless of auto-compaction settings.

---

5. Admin Controls

New global_settings keys:

Key	Type	Default	Description
auto_compaction_enabled	bool	false	Global kill switch. Ships off — admins opt in.
auto_compaction_threshold	float	0.70	Default threshold for channels without override
auto_compaction_interval_minutes	int	5	Scanner tick interval
auto_compaction_concurrency	int	2	Max parallel auto-compactions
auto_compaction_cooldown_minutes	int	30	Min time between auto-compactions of the same channel

Admin UI

Add an "Auto-Compaction" card to the existing admin AI settings panel (alongside the utility rate limit and model roles):

┌─ Auto-Compaction ─────────────────────────────┐
│ Enabled        [✓]                            │
│ Threshold      [70]%                          │
│ Scan interval  [5] minutes                    │
│ Concurrency    [2] parallel                   │
│ Cooldown       [30] minutes per channel       │
└───────────────────────────────────────────────┘

---

6. Cooldown + Dedup

Prevent rapid re-compaction of the same channel:

1. Cooldown — after compacting channel X, don't re-evaluate it for cooldown_minutes. Track in-memory:

   lastCompacted map[string]time.Time  // channelID → timestamp
   

2. In-flight dedup — if channel X is currently being compacted (sem acquired, goroutine running), skip it in the current scan. Track via:

   inFlight sync.Map  // channelID → struct{}
   

3. Stacking — the existing summary boundary logic already handles multiple summaries correctly. Auto-compaction produces the same tree nodes as manual, so stacking "just works."

---

7. Lifecycle + Wiring

main.go changes

// After role resolver and stores setup:
compactionSvc := compaction.NewService(stores, roleResolver)
compactionScanner := compaction.NewScanner(compactionSvc, stores, compaction.ScannerConfig{
    Enabled:     getGlobalBool(stores, "auto_compaction_enabled", false),
    Interval:    time.Duration(getGlobalInt(stores, "auto_compaction_interval_minutes", 5)) * time.Minute,
    Concurrency: getGlobalInt(stores, "auto_compaction_concurrency", 2),
})
compactionScanner.Start()
defer compactionScanner.Stop()  // drain in-flight on shutdown

// Summarize handler now wraps compaction service:
summarize := handlers.NewSummarizeHandler(stores, roleResolver, compactionSvc)

Follows the same pattern as kbIngester with defer kbIngester.Wait().

Config reload

The scanner re-reads global settings at the start of each scan cycle. No restart required to change thresholds, intervals, or the kill switch. If auto_compaction_enabled flips to false, the scanner skips all work on the next tick.

---

8. Observability

Logging

🔍 compaction: scanning 50 candidates
🔍 compaction: channel abc123 qualifies (78% of 128K context, 42 messages post-summary)
📝 compaction: compacting channel abc123 for user xyz (trigger=auto)
✅ compaction: channel abc123 done (42 messages → 847 chars, model=claude-sonnet-4-5-20250929)
⏭ compaction: channel def456 skipped (cooldown, 12m remaining)
⏭ compaction: channel ghi789 skipped (rate limit)

Audit log

Auto-compaction events are logged to audit_log:

stores.Audit.Log(ctx, &models.AuditEntry{
    Actor:    "system:compaction",
    Action:   "compaction.auto",
    Resource: "channel:" + channelID,
    Details: models.JSONMap{
        "summarized_count": result.SummarizedCount,
        "model":            result.Model,
        "trigger":          "auto",
    },
})

Usage tracking

All auto-compaction calls flow through the existing usage logging path (same as manual summarization). The metadata.trigger = "auto" field lets the admin usage dashboard distinguish auto from manual.

---

9. Frontend Changes

Minimal — the backend does the heavy lifting.

Auto-compaction indicator

When loading a conversation that was auto-compacted, the frontend already handles summary boundaries correctly (the collapsed history toggle, the "X earlier messages summarized" bar). The only new UI is a subtle indicator on the summary node:

▸ 42 earlier messages summarized (auto)

The (auto) label comes from metadata.trigger on the summary message.

Channel settings

Add compaction controls to the channel settings panel (Section 4 above). Saves to PATCH /channels/:id with settings field — no new endpoint.

Admin panel

Add the auto-compaction card to the AI settings section (Section 5). Saves via existing PUT /admin/settings/:key endpoints.

---

10. Migration

No new migration required.

• Channel compaction settings → existing Channel.Settings JSONB
• Admin settings → existing global_settings table
• Summary messages → existing messages table with metadata
• Usage tracking → existing usage_log table
• Audit logging → existing audit_log table

---

11. File Manifest

server/
  treepath/                    ← NEW package
    path.go                    (extracted from handlers/tree.go)
    summary.go                 (isSummaryMessage + helpers)
    siblings.go                (sibling queries)
    cursor.go                  (cursor + sibling index)
  compaction/                  ← NEW package
    compaction.go              (Service, Compact, CompactRequest/Result)
    scanner.go                 (Scanner, scan loop, candidate query)
    estimator.go               (EstimateTokens, EstimatePath)
  handlers/
    tree.go                    (updated: delegates to treepath)
    summarize.go               (updated: delegates to compaction.Service)
    completion.go              (updated: imports treepath)
  main.go                      (updated: wire compaction scanner)
src/js/
  ui-core.js                   (updated: show "(auto)" on auto-compacted summaries)
  ui-settings.js               (updated: channel compaction controls)
  ui-admin.js                  (updated: auto-compaction admin card)

---

12. Implementation Order

Phase 1: Extract — Move tree helpers to treepath, core summarization to compaction.Service. Refactor SummarizeHandler to delegate. All existing tests must still pass. No new behavior.

Phase 2: Estimator + Scanner — Add estimator.go, scanner.go, candidate query, shouldCompact logic. Wire into main.go with Start()/Stop(). Global settings for admin controls. Ship with auto_compaction_enabled defaulting to false.

Phase 3: Frontend — Channel compaction settings, admin panel card, auto-compaction indicator on summary nodes.

Phase 4: Soak + tune — Enable on a test instance, watch logs, tune default threshold and cooldown. Adjust candidate query filters if scan is too expensive.

---

13. What This Intentionally Doesn't Do

• Token-accurate estimation — chars/4 is good enough. A real tokenizer adds a dependency for marginal accuracy gain. Revisit if users report summaries firing too early or too late.

• Per-model prompt tuning — the summarization prompt is one-size-fits -all. Works fine across Claude/GPT/Llama. Revisit if quality varies significantly across providers.

• Group channel compaction — auto-compaction targets direct channels only (single-owner). Group/channel types need multi-cursor awareness (which user's path to compact?). Deferred to v0.19.0 multi-participant.

• Compaction snapshots for KB — the roadmap mentions this. The summary nodes we create here are already stored as messages with full metadata. KB integration can query them later without changes to this design.

• Event-driven triggers — a post-completion hook that checks immediately after each message would reduce latency vs. the ticker. Worth adding later but the ticker is simpler to ship and debug. The 2-minute activity gap in the candidate query achieves a similar effect (don't compact mid-conversation).