585 lines
19 KiB
Markdown
585 lines
19 KiB
Markdown
# DESIGN-0.15.0 — Compaction
|
|
|
|
## Overview
|
|
|
|
Automatic conversation compaction. A background service monitors channels
|
|
for context pressure and triggers summarization via the utility model role,
|
|
replacing the manual "Summarize & Continue" button as the primary
|
|
compaction path. Manual summarization remains available as an explicit
|
|
user action.
|
|
|
|
Depends on: utility model role (v0.10.0), summarize handler (v0.10.2).
|
|
|
|
**Design principle: don't reinvent the wheel.** The existing
|
|
`SummarizeHandler` already has the full pipeline — path loading, summary
|
|
boundary detection, prompt construction, role resolution, tree insertion,
|
|
cursor update, usage logging. This design extracts that core logic into
|
|
a shared `compaction` package and adds a background scanner on top.
|
|
|
|
---
|
|
|
|
## 1. Extract: `compaction` Package
|
|
|
|
Move the reusable summarization pipeline out of `handlers/summarize.go`
|
|
into `server/compaction/compaction.go`. The HTTP handler becomes a thin
|
|
wrapper.
|
|
|
|
### Core type
|
|
|
|
```go
|
|
package compaction
|
|
|
|
// Service provides conversation compaction (summarization).
|
|
// Used by both the HTTP handler (manual) and the background scanner (auto).
|
|
type Service struct {
|
|
stores store.Stores
|
|
resolver *roles.Resolver
|
|
}
|
|
|
|
func NewService(stores store.Stores, resolver *roles.Resolver) *Service {
|
|
return &Service{stores: stores, resolver: resolver}
|
|
}
|
|
```
|
|
|
|
### Extracted method: `Compact`
|
|
|
|
The current `SummarizeHandler.Summarize` body becomes `Service.Compact`:
|
|
|
|
```go
|
|
type CompactRequest struct {
|
|
ChannelID string
|
|
UserID string // channel owner
|
|
TeamID *string // for role resolution
|
|
Trigger string // "manual" | "auto"
|
|
}
|
|
|
|
type CompactResult struct {
|
|
SummaryID string
|
|
SummarizedCount int
|
|
Model string
|
|
UsedFallback bool
|
|
Content string
|
|
InputTokens int
|
|
OutputTokens int
|
|
}
|
|
|
|
func (s *Service) Compact(ctx context.Context, req CompactRequest) (*CompactResult, error)
|
|
```
|
|
|
|
The method performs the same steps as today's handler:
|
|
|
|
1. Load active path via `getActivePath` (already exported within `handlers`)
|
|
2. Find summary boundary, collect messages to summarize
|
|
3. Guard: minimum 4 messages since last summary
|
|
4. Build summarization prompt
|
|
5. Call `resolver.Complete(ctx, RoleUtility, ...)`
|
|
6. Insert summary tree node with metadata
|
|
7. Update cursor
|
|
8. Log usage
|
|
|
|
The only new metadata field is `"trigger"` (`"manual"` or `"auto"`) so
|
|
the frontend can distinguish how the summary was created.
|
|
|
|
### Refactored handler
|
|
|
|
`handlers/summarize.go` shrinks to:
|
|
|
|
```go
|
|
func (h *SummarizeHandler) Summarize(c *gin.Context) {
|
|
// ownership check, rate limit check (unchanged)
|
|
// ...
|
|
result, err := h.compaction.Compact(c.Request.Context(), compaction.CompactRequest{
|
|
ChannelID: channelID,
|
|
UserID: userID,
|
|
TeamID: teamID,
|
|
Trigger: "manual",
|
|
})
|
|
// return JSON response (unchanged)
|
|
}
|
|
```
|
|
|
|
### Tree helpers
|
|
|
|
`getActivePath`, `isSummaryMessage`, `nextSiblingIndex`, `updateCursor`
|
|
are currently unexported in `handlers/tree.go`. Two options:
|
|
|
|
**Option A** — Export them from `handlers` and import in `compaction`.
|
|
Clean but creates a dependency from `compaction` → `handlers`.
|
|
|
|
**Option B** — Move tree helpers into a `treepath` (or similar) package
|
|
imported by both `handlers` and `compaction`.
|
|
|
|
Recommend **Option B** to keep the dependency graph clean:
|
|
`handlers` → `compaction` → `treepath` ← `handlers`. The `treepath`
|
|
package is pure data + SQL queries with no handler logic.
|
|
|
|
```
|
|
server/
|
|
treepath/ ← NEW: extracted from handlers/tree.go
|
|
path.go (getActivePath, getPathToLeaf, getActiveLeaf)
|
|
summary.go (isSummaryMessage, summary metadata helpers)
|
|
siblings.go (getSiblingCount, getSiblings, findLeafFromMessage)
|
|
cursor.go (updateCursor, nextSiblingIndex)
|
|
compaction/ ← NEW
|
|
compaction.go (Service, Compact)
|
|
scanner.go (Scanner, background loop)
|
|
estimator.go (token estimation)
|
|
handlers/
|
|
tree.go → thin wrappers or deleted, imports treepath
|
|
summarize.go → thin HTTP wrapper, delegates to compaction.Service
|
|
completion.go → imports treepath for path building
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Server-Side Token Estimation
|
|
|
|
The background scanner needs to evaluate context pressure without making
|
|
an LLM call. The frontend already does this with a `chars / 4` heuristic
|
|
(`tokens.js`). Replicate server-side:
|
|
|
|
```go
|
|
// compaction/estimator.go
|
|
|
|
// EstimateTokens returns a rough token count using the ~4 chars/token
|
|
// heuristic. Matches the frontend Tokens.estimate() for consistency.
|
|
func EstimateTokens(text string) int {
|
|
return (len(text) + 3) / 4
|
|
}
|
|
|
|
// EstimatePath returns total estimated tokens for a message path,
|
|
// including system prompt overhead.
|
|
func EstimatePath(path []treepath.PathMessage, systemPromptLen int) int {
|
|
total := 0
|
|
if systemPromptLen > 0 {
|
|
total += EstimateTokens(string(make([]byte, systemPromptLen))) + 4
|
|
}
|
|
for _, m := range path {
|
|
total += EstimateTokens(m.Content) + 4 // +4 per-message overhead
|
|
}
|
|
return total
|
|
}
|
|
```
|
|
|
|
Not exact, but it's the same bar the user already sees in the UI. If we
|
|
need better accuracy later, swap in a tokenizer library without changing
|
|
the interface.
|
|
|
|
### Context budget resolution
|
|
|
|
The scanner needs `max_context` for whatever model the channel is using.
|
|
Resolution chain:
|
|
|
|
1. Channel has a `model` field → look up `model_catalog` for capabilities
|
|
2. Channel has a `provider_config_id` → use that provider's catalog entry
|
|
3. Fallback: use the utility role's model context window as a proxy (it's
|
|
the model that would be doing the summarization anyway)
|
|
4. Hard fallback: 128K tokens (conservative default)
|
|
|
|
```go
|
|
func (s *Scanner) getContextBudget(ctx context.Context, ch *models.Channel) int {
|
|
if ch.Model != "" {
|
|
if caps, err := s.stores.Catalog.GetCapabilities(ctx, ch.Model); err == nil {
|
|
if caps.MaxContext > 0 {
|
|
return caps.MaxContext
|
|
}
|
|
}
|
|
}
|
|
return DefaultContextBudget // 128000
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Background Scanner
|
|
|
|
Follows the Ingester pattern from `knowledge/ingest.go`: goroutine pool
|
|
with semaphore, WaitGroup for graceful shutdown.
|
|
|
|
### Scanner type
|
|
|
|
```go
|
|
// compaction/scanner.go
|
|
|
|
type Scanner struct {
|
|
service *Service
|
|
stores store.Stores
|
|
sem chan struct{}
|
|
wg sync.WaitGroup
|
|
stopCh chan struct{}
|
|
ticker *time.Ticker
|
|
}
|
|
|
|
type ScannerConfig struct {
|
|
Enabled bool // global kill switch
|
|
Interval time.Duration // scan frequency (default: 5 minutes)
|
|
Concurrency int // max parallel compactions (default: 2)
|
|
}
|
|
|
|
func NewScanner(svc *Service, stores store.Stores, cfg ScannerConfig) *Scanner
|
|
func (sc *Scanner) Start()
|
|
func (sc *Scanner) Stop() // signals stop + drains in-flight
|
|
```
|
|
|
|
### Scan loop
|
|
|
|
```
|
|
every <interval>:
|
|
if !globalEnabled(): continue
|
|
|
|
candidates = findCandidates()
|
|
for each candidate:
|
|
sem.acquire()
|
|
go compact(candidate)
|
|
```
|
|
|
|
### Candidate query
|
|
|
|
A single SQL query finds channels that need compaction:
|
|
|
|
```sql
|
|
SELECT c.id, c.user_id, c.model, c.settings,
|
|
COUNT(m.id) as msg_count,
|
|
SUM(LENGTH(m.content)) as total_chars
|
|
FROM channels c
|
|
JOIN messages m ON m.channel_id = c.id
|
|
AND m.deleted_at IS NULL
|
|
WHERE c.deleted_at IS NULL
|
|
AND c.type = 'direct' -- only user conversations (not groups yet)
|
|
AND c.is_archived = false
|
|
-- Skip channels with very recent activity (let the user finish typing)
|
|
AND c.updated_at < NOW() - INTERVAL '2 minutes'
|
|
-- Only channels active in the last 7 days (don't compact stale channels)
|
|
AND c.updated_at > NOW() - INTERVAL '7 days'
|
|
GROUP BY c.id
|
|
HAVING COUNT(m.id) >= 10 -- minimum message count
|
|
AND SUM(LENGTH(m.content)) > 20000 -- rough: 20K chars ≈ 5K tokens minimum
|
|
ORDER BY c.updated_at DESC
|
|
LIMIT 50 -- batch size cap
|
|
```
|
|
|
|
This is a coarse filter. Each candidate then gets a precise check:
|
|
|
|
```go
|
|
func (sc *Scanner) shouldCompact(ctx context.Context, ch *models.Channel) bool {
|
|
// 1. Check channel-level opt-out
|
|
if ch.Settings != nil {
|
|
if v, ok := ch.Settings["auto_compaction"]; ok && v == false {
|
|
return false
|
|
}
|
|
}
|
|
|
|
// 2. Load active path for the channel owner
|
|
path, err := treepath.GetActivePath(ch.ID, ch.UserID)
|
|
if err != nil || len(path) < 10 {
|
|
return false
|
|
}
|
|
|
|
// 3. Find post-summary messages only
|
|
startIdx := 0
|
|
for i, m := range path {
|
|
if treepath.IsSummaryMessage(&m) {
|
|
startIdx = i + 1
|
|
}
|
|
}
|
|
postSummary := path[startIdx:]
|
|
if len(postSummary) < 8 {
|
|
return false // not enough new messages since last summary
|
|
}
|
|
|
|
// 4. Estimate tokens and check threshold
|
|
tokens := EstimatePath(postSummary, 0)
|
|
budget := sc.getContextBudget(ctx, ch)
|
|
threshold := sc.getThreshold(ch) // default 0.70
|
|
|
|
return float64(tokens) / float64(budget) >= threshold
|
|
}
|
|
```
|
|
|
|
### Threshold
|
|
|
|
Default: **0.70** (70% of context window). This is intentionally lower
|
|
than the frontend's 75% warning bar — auto-compaction should fire before
|
|
the user sees a warning, not after.
|
|
|
|
Configurable per-channel via `settings.compaction_threshold` (float) and
|
|
globally via `global_settings` key `auto_compaction_threshold`.
|
|
|
|
### Rate limiting
|
|
|
|
Auto-compaction calls use the utility role and are org-funded. They share
|
|
the existing `utility_rate_limit` budget but get a separate counter
|
|
prefix so admins can see auto vs manual usage:
|
|
|
|
```go
|
|
entry := &models.UsageEntry{
|
|
// ...
|
|
Role: &role, // "utility"
|
|
Metadata: models.JSONMap{
|
|
"trigger": "auto", // distinguishes from manual
|
|
},
|
|
}
|
|
```
|
|
|
|
If the global utility rate limit is hit, auto-compaction silently skips
|
|
the channel and retries next scan cycle. It never blocks or errors.
|
|
|
|
---
|
|
|
|
## 4. Per-Channel Configuration
|
|
|
|
Uses the existing `Channel.Settings` JSONB column. No migration needed.
|
|
|
|
| Key | Type | Default | Description |
|
|
|-----|------|---------|-------------|
|
|
| `auto_compaction` | `bool` | `true` (from global) | Enable/disable for this channel |
|
|
| `compaction_threshold` | `float` | `0.70` (from global) | Context usage ratio to trigger |
|
|
|
|
### Frontend: channel settings
|
|
|
|
Add a "Compaction" section to the channel settings panel (the gear icon
|
|
in the chat header). Two controls:
|
|
|
|
```
|
|
┌─ Compaction ──────────────────────────────────┐
|
|
│ Auto-compact [✓] │
|
|
│ Threshold [70]% of context window │
|
|
└───────────────────────────────────────────────┘
|
|
```
|
|
|
|
The manual "Summarize & Continue" button remains in the context warning
|
|
bar and works regardless of auto-compaction settings.
|
|
|
|
---
|
|
|
|
## 5. Admin Controls
|
|
|
|
New `global_settings` keys:
|
|
|
|
| Key | Type | Default | Description |
|
|
|-----|------|---------|-------------|
|
|
| `auto_compaction_enabled` | `bool` | `false` | Global kill switch. Ships **off** — admins opt in. |
|
|
| `auto_compaction_threshold` | `float` | `0.70` | Default threshold for channels without override |
|
|
| `auto_compaction_interval_minutes` | `int` | `5` | Scanner tick interval |
|
|
| `auto_compaction_concurrency` | `int` | `2` | Max parallel auto-compactions |
|
|
| `auto_compaction_cooldown_minutes` | `int` | `30` | Min time between auto-compactions of the same channel |
|
|
|
|
### Admin UI
|
|
|
|
Add an "Auto-Compaction" card to the existing admin AI settings panel
|
|
(alongside the utility rate limit and model roles):
|
|
|
|
```
|
|
┌─ Auto-Compaction ─────────────────────────────┐
|
|
│ Enabled [✓] │
|
|
│ Threshold [70]% │
|
|
│ Scan interval [5] minutes │
|
|
│ Concurrency [2] parallel │
|
|
│ Cooldown [30] minutes per channel │
|
|
└───────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Cooldown + Dedup
|
|
|
|
Prevent rapid re-compaction of the same channel:
|
|
|
|
1. **Cooldown** — after compacting channel X, don't re-evaluate it for
|
|
`cooldown_minutes`. Track in-memory:
|
|
```go
|
|
lastCompacted map[string]time.Time // channelID → timestamp
|
|
```
|
|
|
|
2. **In-flight dedup** — if channel X is currently being compacted (sem
|
|
acquired, goroutine running), skip it in the current scan. Track via:
|
|
```go
|
|
inFlight sync.Map // channelID → struct{}
|
|
```
|
|
|
|
3. **Stacking** — the existing summary boundary logic already handles
|
|
multiple summaries correctly. Auto-compaction produces the same tree
|
|
nodes as manual, so stacking "just works."
|
|
|
|
---
|
|
|
|
## 7. Lifecycle + Wiring
|
|
|
|
### main.go changes
|
|
|
|
```go
|
|
// After role resolver and stores setup:
|
|
compactionSvc := compaction.NewService(stores, roleResolver)
|
|
compactionScanner := compaction.NewScanner(compactionSvc, stores, compaction.ScannerConfig{
|
|
Enabled: getGlobalBool(stores, "auto_compaction_enabled", false),
|
|
Interval: time.Duration(getGlobalInt(stores, "auto_compaction_interval_minutes", 5)) * time.Minute,
|
|
Concurrency: getGlobalInt(stores, "auto_compaction_concurrency", 2),
|
|
})
|
|
compactionScanner.Start()
|
|
defer compactionScanner.Stop() // drain in-flight on shutdown
|
|
|
|
// Summarize handler now wraps compaction service:
|
|
summarize := handlers.NewSummarizeHandler(stores, roleResolver, compactionSvc)
|
|
```
|
|
|
|
Follows the same pattern as `kbIngester` with `defer kbIngester.Wait()`.
|
|
|
|
### Config reload
|
|
|
|
The scanner re-reads global settings at the start of each scan cycle.
|
|
No restart required to change thresholds, intervals, or the kill switch.
|
|
If `auto_compaction_enabled` flips to `false`, the scanner skips all
|
|
work on the next tick.
|
|
|
|
---
|
|
|
|
## 8. Observability
|
|
|
|
### Logging
|
|
|
|
```
|
|
🔍 compaction: scanning 50 candidates
|
|
🔍 compaction: channel abc123 qualifies (78% of 128K context, 42 messages post-summary)
|
|
📝 compaction: compacting channel abc123 for user xyz (trigger=auto)
|
|
✅ compaction: channel abc123 done (42 messages → 847 chars, model=claude-sonnet-4-5-20250929)
|
|
⏭ compaction: channel def456 skipped (cooldown, 12m remaining)
|
|
⏭ compaction: channel ghi789 skipped (rate limit)
|
|
```
|
|
|
|
### Audit log
|
|
|
|
Auto-compaction events are logged to `audit_log`:
|
|
|
|
```go
|
|
stores.Audit.Log(ctx, &models.AuditEntry{
|
|
Actor: "system:compaction",
|
|
Action: "compaction.auto",
|
|
Resource: "channel:" + channelID,
|
|
Details: models.JSONMap{
|
|
"summarized_count": result.SummarizedCount,
|
|
"model": result.Model,
|
|
"trigger": "auto",
|
|
},
|
|
})
|
|
```
|
|
|
|
### Usage tracking
|
|
|
|
All auto-compaction calls flow through the existing usage logging path
|
|
(same as manual summarization). The `metadata.trigger = "auto"` field
|
|
lets the admin usage dashboard distinguish auto from manual.
|
|
|
|
---
|
|
|
|
## 9. Frontend Changes
|
|
|
|
Minimal — the backend does the heavy lifting.
|
|
|
|
### Auto-compaction indicator
|
|
|
|
When loading a conversation that was auto-compacted, the frontend
|
|
already handles summary boundaries correctly (the collapsed history
|
|
toggle, the "X earlier messages summarized" bar). The only new UI is
|
|
a subtle indicator on the summary node:
|
|
|
|
```
|
|
▸ 42 earlier messages summarized (auto)
|
|
```
|
|
|
|
The `(auto)` label comes from `metadata.trigger` on the summary message.
|
|
|
|
### Channel settings
|
|
|
|
Add compaction controls to the channel settings panel (Section 4 above).
|
|
Saves to `PATCH /channels/:id` with `settings` field — no new endpoint.
|
|
|
|
### Admin panel
|
|
|
|
Add the auto-compaction card to the AI settings section (Section 5).
|
|
Saves via existing `PUT /admin/settings/:key` endpoints.
|
|
|
|
---
|
|
|
|
## 10. Migration
|
|
|
|
**No new migration required.**
|
|
|
|
- Channel compaction settings → existing `Channel.Settings` JSONB
|
|
- Admin settings → existing `global_settings` table
|
|
- Summary messages → existing `messages` table with metadata
|
|
- Usage tracking → existing `usage_log` table
|
|
- Audit logging → existing `audit_log` table
|
|
|
|
---
|
|
|
|
## 11. File Manifest
|
|
|
|
```
|
|
server/
|
|
treepath/ ← NEW package
|
|
path.go (extracted from handlers/tree.go)
|
|
summary.go (isSummaryMessage + helpers)
|
|
siblings.go (sibling queries)
|
|
cursor.go (cursor + sibling index)
|
|
compaction/ ← NEW package
|
|
compaction.go (Service, Compact, CompactRequest/Result)
|
|
scanner.go (Scanner, scan loop, candidate query)
|
|
estimator.go (EstimateTokens, EstimatePath)
|
|
handlers/
|
|
tree.go (updated: delegates to treepath)
|
|
summarize.go (updated: delegates to compaction.Service)
|
|
completion.go (updated: imports treepath)
|
|
main.go (updated: wire compaction scanner)
|
|
src/js/
|
|
ui-core.js (updated: show "(auto)" on auto-compacted summaries)
|
|
ui-settings.js (updated: channel compaction controls)
|
|
ui-admin.js (updated: auto-compaction admin card)
|
|
```
|
|
|
|
---
|
|
|
|
## 12. Implementation Order
|
|
|
|
**Phase 1: Extract** — Move tree helpers to `treepath`, core
|
|
summarization to `compaction.Service`. Refactor `SummarizeHandler` to
|
|
delegate. All existing tests must still pass. No new behavior.
|
|
|
|
**Phase 2: Estimator + Scanner** — Add `estimator.go`, `scanner.go`,
|
|
candidate query, `shouldCompact` logic. Wire into `main.go` with
|
|
`Start()`/`Stop()`. Global settings for admin controls. Ship with
|
|
`auto_compaction_enabled` defaulting to `false`.
|
|
|
|
**Phase 3: Frontend** — Channel compaction settings, admin panel card,
|
|
auto-compaction indicator on summary nodes.
|
|
|
|
**Phase 4: Soak + tune** — Enable on a test instance, watch logs, tune
|
|
default threshold and cooldown. Adjust candidate query filters if scan
|
|
is too expensive.
|
|
|
|
---
|
|
|
|
## 13. What This Intentionally Doesn't Do
|
|
|
|
- **Token-accurate estimation** — chars/4 is good enough. A real
|
|
tokenizer adds a dependency for marginal accuracy gain. Revisit if
|
|
users report summaries firing too early or too late.
|
|
|
|
- **Per-model prompt tuning** — the summarization prompt is one-size-fits
|
|
-all. Works fine across Claude/GPT/Llama. Revisit if quality varies
|
|
significantly across providers.
|
|
|
|
- **Group channel compaction** — auto-compaction targets `direct` channels
|
|
only (single-owner). Group/channel types need multi-cursor awareness
|
|
(which user's path to compact?). Deferred to v0.19.0 multi-participant.
|
|
|
|
- **Compaction snapshots for KB** — the roadmap mentions this. The
|
|
summary nodes we create here are already stored as messages with full
|
|
metadata. KB integration can query them later without changes to this
|
|
design.
|
|
|
|
- **Event-driven triggers** — a post-completion hook that checks
|
|
immediately after each message would reduce latency vs. the ticker.
|
|
Worth adding later but the ticker is simpler to ship and debug. The
|
|
2-minute activity gap in the candidate query achieves a similar effect
|
|
(don't compact mid-conversation).
|