docs: ROADMAP full restore + #98-143 tech debt links + archive historical DESIGN/CHANGES (#144)
This commit is contained in:
191
docs/archive/CHANGES-0.18.0-phase2.md
Normal file
191
docs/archive/CHANGES-0.18.0-phase2.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# v0.18.0 Phase 2 — Changes Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 2 adds: automatic memory extraction via background scanner,
|
||||
embedding-powered semantic recall, memory review pipeline API, and
|
||||
persona memory configuration.
|
||||
|
||||
## New Files (drop in place)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `server/database/migrations/005_v0180_memory_phase2.sql` | Postgres: persona memory columns, HNSW index, extraction log table |
|
||||
| `server/database/migrations/sqlite/004_v0180_memory_phase2.sql` | SQLite equivalent |
|
||||
| `server/memory/extractor.go` | Extraction service — calls utility model to extract facts from conversations |
|
||||
| `server/memory/scanner.go` | Background scanner — finds conversations to extract, same pattern as compaction.Scanner |
|
||||
| `server/store/postgres/memory_hybrid.go` | RecallHybrid — pgvector cosine distance + keyword merge |
|
||||
| `server/store/sqlite/memory_hybrid.go` | RecallHybrid — app-level cosine similarity + keyword merge |
|
||||
| `server/handlers/memory.go` | REST API: list, edit, delete, approve/reject memories |
|
||||
| `server/handlers/memory_inject.go` | **REPLACES Phase 1 version** — now supports hybrid recall with embeddings |
|
||||
|
||||
## Updated Files (replace Phase 1 versions)
|
||||
|
||||
| File | What Changed |
|
||||
|------|-------------|
|
||||
| `server/tools/memory.go` | **REPLACES Phase 1 version** — RegisterMemoryTools now takes embedder param, memory_save embeds on save |
|
||||
|
||||
## Existing File Modifications
|
||||
|
||||
### 1. `server/models/models.go` — Persona struct
|
||||
|
||||
Add after the `KBIDs` field:
|
||||
|
||||
```go
|
||||
MemoryEnabled bool `json:\"memory_enabled\" db:\"memory_enabled\"`
|
||||
MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\" db:\"memory_extraction_prompt\"`
|
||||
```
|
||||
|
||||
Add to `PersonaPatch`:
|
||||
|
||||
```go
|
||||
MemoryEnabled *bool `json:\"memory_enabled,omitempty\"`
|
||||
MemoryExtractionPrompt *string `json:\"memory_extraction_prompt,omitempty\"`
|
||||
```
|
||||
|
||||
### 2. `server/store/store_memory.go` — MemoryStore interface
|
||||
|
||||
Add to the interface:
|
||||
|
||||
```go
|
||||
// RecallHybrid returns active memories using vector similarity + keyword search.
|
||||
// If queryVec is nil/empty, falls back to keyword-only (same as Recall).
|
||||
RecallHybrid(ctx context.Context, userID string, personaID *string, query string, queryVec []float64, limit int) ([]models.Memory, error)
|
||||
```
|
||||
|
||||
### 3. `server/main.go` — Tool registration + scanner startup
|
||||
|
||||
**Update** the existing RegisterMemoryTools call (~line 170) to pass the embedder:
|
||||
|
||||
```go
|
||||
// Memory tools (v0.18.0) — late registration, needs stores + embedder
|
||||
tools.RegisterMemoryTools(stores, kbEmbedder)
|
||||
```
|
||||
|
||||
**Add** memory extraction scanner startup (after compaction scanner, or near the end
|
||||
of the startup block — see compaction scanner as pattern reference):
|
||||
|
||||
```go
|
||||
// Memory extraction scanner (v0.18.0 Phase 2)
|
||||
// Opt-in: requires global_settings.memory_extraction_enabled = true
|
||||
memExtractor := memory.NewExtractor(stores, roleResolver, kbEmbedder)
|
||||
memScanner := memory.NewScanner(memExtractor, stores, memory.ScannerConfig{})
|
||||
memScanner.Start()
|
||||
defer memScanner.Stop()
|
||||
```
|
||||
|
||||
Import: `\"git.gobha.me/xcaliber/chat-switchboard/memory\"`
|
||||
|
||||
### 4. `server/main.go` — Memory API routes
|
||||
|
||||
Add under the protected routes (~after presets/personas routes):
|
||||
|
||||
```go
|
||||
// Memory management (v0.18.0)
|
||||
memH := handlers.NewMemoryHandler(stores)
|
||||
protected.GET(\"/memories\", memH.ListMyMemories)
|
||||
protected.PUT(\"/memories/:id\", memH.UpdateMemory)
|
||||
protected.DELETE(\"/memories/:id\", memH.DeleteMemory)
|
||||
protected.POST(\"/memories/:id/approve\", memH.ApproveMemory)
|
||||
protected.POST(\"/memories/:id/reject\", memH.RejectMemory)
|
||||
protected.GET(\"/memories/count\", memH.MemoryCount)
|
||||
```
|
||||
|
||||
Add under admin routes:
|
||||
|
||||
```go
|
||||
// Admin memory review (v0.18.0)
|
||||
adminMemH := handlers.NewMemoryHandler(stores)
|
||||
admin.GET(\"/memories/pending\", adminMemH.ListPendingReview)
|
||||
admin.POST(\"/memories/bulk-approve\", adminMemH.BulkApprove)
|
||||
```
|
||||
|
||||
### 5. `server/handlers/completion.go` — BuildMemoryHint signature change
|
||||
|
||||
The `BuildMemoryHint` call in `loadConversation()` changes signature.
|
||||
Update from Phase 1:
|
||||
|
||||
```go
|
||||
// OLD (Phase 1):
|
||||
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != \"\" {
|
||||
|
||||
// NEW (Phase 2) — pass embedder and last user message for semantic recall:
|
||||
lastUserMsg := \"\"
|
||||
for i := len(messages) - 1; i >= 0; i-- {
|
||||
if messages[i].Role == \"user\" {
|
||||
lastUserMsg = messages[i].Content
|
||||
break
|
||||
}
|
||||
}
|
||||
if memHint := BuildMemoryHint(context.Background(), h.stores, h.embedder, userID, personaID, lastUserMsg); memHint != \"\" {
|
||||
```
|
||||
|
||||
This requires adding the embedder to CompletionHandler. In the struct:
|
||||
|
||||
```go
|
||||
type CompletionHandler struct {
|
||||
vault *crypto.KeyResolver
|
||||
stores store.Stores
|
||||
hub *events.Hub
|
||||
objStore storage.ObjectStore
|
||||
embedder *knowledge.Embedder // NEW: for memory semantic recall
|
||||
}
|
||||
```
|
||||
|
||||
And in `NewCompletionHandler`:
|
||||
|
||||
```go
|
||||
func NewCompletionHandler(vault *crypto.KeyResolver, stores store.Stores, hub *events.Hub, objStore storage.ObjectStore, embedder *knowledge.Embedder) *CompletionHandler {
|
||||
return &CompletionHandler{vault: vault, stores: stores, hub: hub, objStore: objStore, embedder: embedder}
|
||||
}
|
||||
```
|
||||
|
||||
Update the call site in main.go:
|
||||
|
||||
```go
|
||||
comp := handlers.NewCompletionHandler(keyResolver, stores, hub, objStore, kbEmbedder)
|
||||
```
|
||||
|
||||
### 6. `server/store/postgres/persona.go` — Scan memory fields
|
||||
|
||||
Any Persona scan queries need to include the new columns. Add
|
||||
`memory_enabled` and `memory_extraction_prompt` to SELECT lists and Scan calls
|
||||
in `GetByID`, `ListForUser`, etc.
|
||||
|
||||
### 7. `server/store/sqlite/persona.go` — Same as above for SQLite.
|
||||
|
||||
---
|
||||
|
||||
## Admin Setup
|
||||
|
||||
The extraction scanner is **opt-in**. To enable:
|
||||
|
||||
1. Set global config: `memory_extraction_enabled` = `true`
|
||||
2. The scanner runs every 10 minutes (configurable)
|
||||
3. Extracted memories start as `pending_review`
|
||||
4. Admin reviews via `GET /api/v1/admin/memories/pending`
|
||||
5. Approve: `POST /api/v1/memories/:id/approve`
|
||||
6. Reject: `POST /api/v1/memories/:id/reject`
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
1. **Migration** — both Postgres and SQLite, verify `memory_extraction_log` table created
|
||||
2. **Persona columns** — `ALTER TABLE personas ADD COLUMN memory_enabled` runs clean
|
||||
3. **HNSW index** — Postgres only, verify with `\\di+ idx_memories_embedding`
|
||||
4. **Embedding on save** — call memory_save, check logs for `🧠 memory X embedded`
|
||||
5. **Hybrid recall** — with embeddings populated, memory_recall should find semantically relevant results even with different keywords
|
||||
6. **Extraction scanner** — set `memory_extraction_enabled=true` in global config, wait for scan cycle, verify `🧠 memory extraction:` logs
|
||||
7. **Extracted status** — auto-extracted memories should have `status=pending_review`
|
||||
8. **Review API** — `GET /memories?status=pending_review` returns pending items
|
||||
9. **Approve/reject** — POST approve changes status to active, reject archives
|
||||
10. **BuildMemoryHint** — with embedder, verify semantic recall contextualizes based on user's message
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Extraction scanner** follows compaction.Scanner pattern: ticker loop, semaphore concurrency, in-flight dedup, graceful shutdown via Stop()
|
||||
- **Extraction log** (`memory_extraction_log`) prevents re-processing: tracks last_message_id per channel+user
|
||||
- **RecallHybrid** merges vector + keyword results with dedup by ID; semantic results rank first
|
||||
- **SQLite hybrid** loads all embedded memories into Go and computes cosine similarity in-process (acceptable for single-user deployments)
|
||||
- **Persona memory toggle** (`memory_enabled`) gates both tool-based and extraction-based memory for that persona
|
||||
- **Extraction prompt** is customizable per persona — a helpdesk persona extracts FAQ patterns, a tutoring persona extracts learning progress
|
||||
- **Phase 3** will add the frontend UI: Settings → Memory panel, admin review queue, per-persona toggle in persona editor
|
||||
Reference in New Issue
Block a user