Changeset 0.15.1 (#73)
This commit is contained in:
332
docs/DESIGN-0.15.1.md
Normal file
332
docs/DESIGN-0.15.1.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# DESIGN-0.15.1 — Context Recall Tools
|
||||
|
||||
## Overview
|
||||
|
||||
After compaction (v0.15.0) or in long conversations, the model loses access
|
||||
to earlier material — attachment content injected once at send time, and
|
||||
message details compressed into summaries. These tools let the model
|
||||
re-read source material on demand.
|
||||
|
||||
Depends on: file handling (v0.12.0), tool framework (v0.11.0), compaction (v0.15.0).
|
||||
|
||||
**Four work streams:**
|
||||
|
||||
1. `attachment_recall` — re-read attachment content from storage
|
||||
2. `conversation_search` — keyword search over channel messages
|
||||
3. Token estimator improvements — attachment-aware context counting
|
||||
4. KB auto-injection — automatic knowledge base context in system prompt
|
||||
|
||||
---
|
||||
|
||||
## 1. `attachment_recall` Tool
|
||||
|
||||
### Purpose
|
||||
|
||||
Attachments are injected into context once (at the user message that
|
||||
references them). On subsequent turns the model relies on its own response
|
||||
as a natural summary. After compaction, that summary is gone. This tool
|
||||
lets the model re-read any attachment in the current channel.
|
||||
|
||||
### Two-action design
|
||||
|
||||
Single tool, `action` parameter selects behavior:
|
||||
|
||||
- **`list`** — returns metadata for all attachments in the channel
|
||||
(id, filename, content_type, size, created_at). The model calls this
|
||||
first to discover what's available.
|
||||
- **`read`** — returns content for a specific attachment_id.
|
||||
Documents → `extracted_text` from DB. Images → base64 data URI.
|
||||
|
||||
### Security
|
||||
|
||||
Channel-scoped: `attachment_id` must belong to `ExecutionContext.ChannelID`.
|
||||
The store query is `GetByChannel(channelID)` so cross-channel access is
|
||||
impossible by construction.
|
||||
|
||||
### Dependencies
|
||||
|
||||
Late registration (same pattern as `kb_search`):
|
||||
|
||||
```go
|
||||
func RegisterAttachmentRecall(stores store.Stores, objStore storage.ObjectStore)
|
||||
```
|
||||
|
||||
Called from `main.go` after storage init. If `objStore` is nil (storage
|
||||
not configured), the tool is **not registered** — it simply won't appear
|
||||
in the tool list.
|
||||
|
||||
### Implementation: `server/tools/attachment_recall.go`
|
||||
|
||||
```go
|
||||
type attachmentRecallTool struct {
|
||||
stores store.Stores
|
||||
objStore storage.ObjectStore
|
||||
}
|
||||
|
||||
func (t *attachmentRecallTool) Definition() ToolDef {
|
||||
return ToolDef{
|
||||
Name: "attachment_recall",
|
||||
DisplayName: "Attachments",
|
||||
Category: "context",
|
||||
Description: "Re-read file attachments from this conversation. " +
|
||||
"Use action='list' to see available files, then action='read' " +
|
||||
"with an attachment_id to get the content.",
|
||||
Parameters: JSONSchema(map[string]interface{}{
|
||||
"action": PropEnum("Action to perform", "list", "read"),
|
||||
"attachment_id": Prop("string", "Attachment ID (required for read)"),
|
||||
}, []string{"action"}),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`list` action returns:**
|
||||
```json
|
||||
{
|
||||
"attachments": [
|
||||
{
|
||||
"id": "abc-123",
|
||||
"filename": "report.pdf",
|
||||
"content_type": "application/pdf",
|
||||
"size_bytes": 45200,
|
||||
"has_text": true,
|
||||
"created_at": "2026-02-26T10:00:00Z"
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
**`read` action returns:**
|
||||
```json
|
||||
{
|
||||
"id": "abc-123",
|
||||
"filename": "report.pdf",
|
||||
"content_type": "application/pdf",
|
||||
"content": "... extracted text ...",
|
||||
"content_type_returned": "text"
|
||||
}
|
||||
```
|
||||
|
||||
For images:
|
||||
```json
|
||||
{
|
||||
"id": "def-456",
|
||||
"filename": "diagram.png",
|
||||
"content_type": "image/png",
|
||||
"content": "data:image/png;base64,...",
|
||||
"content_type_returned": "base64_image"
|
||||
}
|
||||
```
|
||||
|
||||
Image base64 is capped at 10MB to prevent context blow-up. Larger images
|
||||
return an error suggesting the user re-upload.
|
||||
|
||||
---
|
||||
|
||||
## 2. `conversation_search` Tool
|
||||
|
||||
### Purpose
|
||||
|
||||
After compaction, message details are compressed into summaries. If the
|
||||
user asks about something the summary omitted, the model needs to search
|
||||
the original messages. Also useful in long pre-compaction conversations
|
||||
where the model's attention has drifted.
|
||||
|
||||
### Design
|
||||
|
||||
Keyword search (full-text) over messages in the current channel using
|
||||
PostgreSQL's `plainto_tsquery`. No migration needed — `to_tsvector` is
|
||||
computed on the fly. This is acceptable because:
|
||||
|
||||
- Search is scoped to a single channel (bounded message count)
|
||||
- Called infrequently (tool invocation, not every request)
|
||||
- Channel messages rarely exceed ~10K rows
|
||||
|
||||
If performance becomes an issue, a `search_vector tsvector` column with
|
||||
GIN index can be added in a future migration.
|
||||
|
||||
### Parameters
|
||||
|
||||
```go
|
||||
Parameters: JSONSchema(map[string]interface{}{
|
||||
"query": Prop("string", "Search keywords"),
|
||||
"max_results": map[string]interface{}{
|
||||
"type": "integer",
|
||||
"description": "Maximum results (1-20, default 5)",
|
||||
},
|
||||
"role_filter": PropEnum("Filter by message role", "all", "user", "assistant"),
|
||||
}, []string{"query"})
|
||||
```
|
||||
|
||||
### Returns
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"message_id": "...",
|
||||
"role": "user",
|
||||
"excerpt": "...highlighted excerpt...",
|
||||
"timestamp": "2026-02-26T10:00:00Z",
|
||||
"rank": 0.95
|
||||
}
|
||||
],
|
||||
"query": "original query",
|
||||
"total": 3,
|
||||
"channel_id": "..."
|
||||
}
|
||||
```
|
||||
|
||||
Uses `ts_headline` for highlighted excerpts (same pattern as `note_search`).
|
||||
|
||||
### Security
|
||||
|
||||
Channel-scoped via `ExecutionContext.ChannelID`. Only searches messages
|
||||
in the current channel. Soft-deleted messages (`deleted_at IS NOT NULL`)
|
||||
are excluded.
|
||||
|
||||
### Dependencies
|
||||
|
||||
Late registration:
|
||||
|
||||
```go
|
||||
func RegisterConversationSearch(stores store.Stores)
|
||||
```
|
||||
|
||||
No external dependencies beyond stores (uses `database.DB` directly for
|
||||
the full-text query, same pattern as `note_search`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Token Estimator Improvements
|
||||
|
||||
### Frontend (`tokens.js`)
|
||||
|
||||
Current `estimateConversation` counts message text only. Enhancements:
|
||||
|
||||
- **Staged attachments**: Before send, show estimated tokens for queued
|
||||
attachments. Documents: `extracted_text.length / 4`. Images: flat
|
||||
~765 tokens per image (OpenAI's base tile estimate, reasonable cross-provider).
|
||||
- **Sent attachments**: After send, the augmented content is already in
|
||||
the message text (documents are inlined). Images were ephemeral
|
||||
ContentParts — add their estimate to the message's token count.
|
||||
|
||||
### Implementation
|
||||
|
||||
Add to `Tokens`:
|
||||
|
||||
```javascript
|
||||
estimateAttachments(attachments) {
|
||||
let total = 0;
|
||||
for (const att of attachments) {
|
||||
if (att.content_type?.startsWith('image/')) {
|
||||
total += 765; // base tile estimate
|
||||
} else if (att.extracted_text) {
|
||||
total += this.estimate(att.extracted_text);
|
||||
} else {
|
||||
total += Math.ceil(att.size_bytes / 4); // rough fallback
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
```
|
||||
|
||||
Wire into `updateInputTokens()` to include staged attachment estimates.
|
||||
|
||||
---
|
||||
|
||||
## 4. KB Auto-Injection
|
||||
|
||||
### Purpose
|
||||
|
||||
Currently the model must explicitly call `kb_search` to access knowledge
|
||||
base content. For common patterns (channel linked to a KB, user asks a
|
||||
question), automatic injection into the system prompt is more natural.
|
||||
|
||||
### Design
|
||||
|
||||
In the completion handler, after loading conversation history and before
|
||||
building the provider request:
|
||||
|
||||
1. Check if channel has linked KBs (or user has personal KBs)
|
||||
2. Extract the user's latest message as the query
|
||||
3. Run similarity search (top-K, K=3 default)
|
||||
4. If results found AND context budget allows, prepend to system prompt
|
||||
|
||||
### Context budget check
|
||||
|
||||
```go
|
||||
contextUsed := estimateTokens(systemPrompt + allMessages)
|
||||
kbTokens := estimateTokens(kbContent)
|
||||
if contextUsed + kbTokens > maxContext * 0.85 {
|
||||
// Skip injection — context too full
|
||||
log.Printf("⏭ KB auto-inject skipped: context %.0f%% full", pct*100)
|
||||
}
|
||||
```
|
||||
|
||||
### Channel settings
|
||||
|
||||
```json
|
||||
{
|
||||
"kb_injection": "auto" | "tool_only" | "both" | "off"
|
||||
}
|
||||
```
|
||||
|
||||
Default: `"auto"` — inject when KBs are linked. `"tool_only"` disables
|
||||
auto-injection but keeps the `kb_search` tool available. `"both"` does
|
||||
both. `"off"` disables all KB context.
|
||||
|
||||
### System prompt format
|
||||
|
||||
```
|
||||
[Knowledge Base Context]
|
||||
The following information was retrieved from the user's knowledge bases
|
||||
and may be relevant to this conversation:
|
||||
|
||||
---
|
||||
Source: Technical Docs (chunk 3 of report.pdf, similarity: 0.87)
|
||||
Content: ...
|
||||
|
||||
---
|
||||
Source: FAQ (chunk 1 of faq.md, similarity: 0.82)
|
||||
Content: ...
|
||||
[End Knowledge Base Context]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Inventory
|
||||
|
||||
### New files
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `server/tools/attachment_recall.go` | attachment_recall tool |
|
||||
| `server/tools/conversation_search.go` | conversation_search tool |
|
||||
| `docs/DESIGN-0.15.1.md` | This design doc |
|
||||
|
||||
### Modified files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `server/main.go` | Register new tools |
|
||||
| `server/handlers/completion.go` | KB auto-injection in completion flow |
|
||||
| `src/js/tokens.js` | Attachment-aware token estimation |
|
||||
| `docs/ROADMAP.md` | Mark v0.15.1 items |
|
||||
|
||||
### No migration required
|
||||
|
||||
- `conversation_search` uses on-the-fly `to_tsvector` (no new column)
|
||||
- `attachment_recall` reads existing `extracted_text` column
|
||||
- KB auto-injection uses existing `kb_search` infrastructure
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Phase 1**: `attachment_recall` tool + registration ✅
|
||||
2. **Phase 2**: `conversation_search` tool + registration ✅
|
||||
3. **Phase 3**: Token estimator improvements (frontend) ✅
|
||||
4. **Phase 4**: KB auto-injection (completion handler) — separate delivery,
|
||||
requires adding embedder dependency to `CompletionHandler` and careful
|
||||
latency budgeting (embedding query on every completion call)
|
||||
@@ -786,28 +786,30 @@ summary. After compaction (v0.15.0) or in long conversations, the model
|
||||
may need to re-read source material. These tools bridge that gap.
|
||||
|
||||
**`attachment_recall` tool**
|
||||
- [ ] `attachment_recall(attachment_id)` — re-reads attachment content from storage
|
||||
- [ ] Images: re-inject as base64 content part (vision gating still applies)
|
||||
- [ ] Documents: return extracted text
|
||||
- [ ] Security: scoped to current channel's attachments only
|
||||
- [ ] Extend `ExecutionContext` with optional storage backend reference
|
||||
- [ ] Tool definition includes attachment list hint so model knows what's available
|
||||
- [x] Two-action design: `list` (channel metadata) + `read` (content by ID)
|
||||
- [x] Images: base64 data URI from object storage (10MB cap)
|
||||
- [x] Documents: return `extracted_text` from DB
|
||||
- [x] Security: channel-scoped — attachment must belong to `ExecutionContext.ChannelID`
|
||||
- [x] Late registration: only registered when `objStore` is non-nil
|
||||
- [x] Category: `context` (groups with conversation_search in toggle UI)
|
||||
|
||||
**`conversation_search` tool**
|
||||
- [ ] Keyword search over compacted/archived messages in current channel
|
||||
- [ ] Returns matching message excerpts with timestamps
|
||||
- [ ] Useful post-compaction when summary dropped details the user asks about
|
||||
- [x] Full-text keyword search via `plainto_tsquery` over channel messages
|
||||
- [x] `ts_headline` for highlighted excerpts with `**` markers
|
||||
- [x] Role filter: `all`, `user`, `assistant`
|
||||
- [x] Scoped to current channel, excludes soft-deleted messages
|
||||
- [x] On-the-fly `to_tsvector` (no migration needed — channel-bounded)
|
||||
|
||||
**Token estimator improvements**
|
||||
- [ ] Include attachment token estimates in context counter
|
||||
- [ ] Images: use provider-specific sizing rules (e.g. tile-based for OpenAI)
|
||||
- [ ] Documents: count extracted text tokens
|
||||
- [ ] Staged attachments reflected in input token counter before send
|
||||
- [x] `Tokens.estimateAttachments()` — images ~765 tokens, docs by text length
|
||||
- [x] Staged attachment tokens reflected in input counter before send
|
||||
- [x] Context percentage includes attachment estimates
|
||||
|
||||
**KB auto-injection** (depends on: knowledge bases v0.14.0, compaction v0.15.0)
|
||||
- [ ] Pre-pend top-K KB results to system prompt (automatic, no tool call needed)
|
||||
- [ ] Context budget aware: skip injection if context already near limit
|
||||
- [ ] Per-channel toggle: tool-only vs auto-inject vs both
|
||||
- _(deferred — requires adding embedder to CompletionHandler, latency budgeting)_
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user