Changeset 0.18.0 (#79)
This commit is contained in:
@@ -126,7 +126,7 @@ server/
|
||||
│ └── 002_v017_persona_kb.sql
|
||||
├── store/
|
||||
│ ├── interfaces.go # Store interfaces + shared types
|
||||
│ ├── postgres/ # Postgres implementations (19 stores)
|
||||
│ ├── postgres/ # Postgres implementations (20 stores)
|
||||
│ │ ├── stores.go # NewStores() constructor
|
||||
│ │ ├── provider.go # ProviderStore
|
||||
│ │ ├── catalog.go # CatalogStore
|
||||
@@ -137,8 +137,10 @@ server/
|
||||
│ │ ├── team.go # TeamStore
|
||||
│ │ ├── note.go # NoteStore
|
||||
│ │ ├── knowledge.go # KnowledgeStore
|
||||
│ │ ├── memory.go # MemoryStore (CRUD + recall)
|
||||
│ │ ├── memory_hybrid.go # RecallHybrid (pgvector cosine)
|
||||
│ │ └── ...
|
||||
│ └── sqlite/ # SQLite implementations (19 stores)
|
||||
│ └── sqlite/ # SQLite implementations (20 stores)
|
||||
│ ├── stores.go # NewStores() constructor
|
||||
│ └── ... # Mirror of postgres/ with dialect adaptations
|
||||
├── models/models.go # Shared domain types
|
||||
@@ -149,6 +151,9 @@ server/
|
||||
├── crypto/ # AES-256-GCM API key encryption
|
||||
├── events/ # EventBus + WebSocket hub
|
||||
├── extraction/ # Document text extraction pipeline
|
||||
├── memory/ # Long-term memory extraction + scanning
|
||||
│ ├── extractor.go # Conversation analysis → fact extraction
|
||||
│ └── scanner.go # Background job: find channels needing extraction
|
||||
├── handlers/ # HTTP handlers (Gin)
|
||||
│ ├── auth.go # Login, register, refresh, logout
|
||||
│ ├── admin.go # User/config/model management
|
||||
@@ -162,6 +167,8 @@ server/
|
||||
│ ├── teams.go # Team management
|
||||
│ ├── notes.go # Notes CRUD + search + graph + backlinks
|
||||
│ ├── knowledge.go # Knowledge base management
|
||||
│ ├── memory.go # Memory CRUD + review (user + admin)
|
||||
│ ├── memory_inject.go # BuildMemoryHint() for completion injection
|
||||
│ └── ...
|
||||
├── notelinks/ # Wikilink extraction (regex → NoteLink structs)
|
||||
├── knowledge/ # KB chunking, embedding, search
|
||||
@@ -251,7 +258,9 @@ src/
|
||||
├── index.html # Single-page app shell
|
||||
├── sw.js # Service worker (offline cache)
|
||||
├── manifest.json # PWA manifest
|
||||
├── css/styles.css # All styles (CSS variables, light/dark themes)
|
||||
├── css/
|
||||
│ ├── styles.css # Core styles (CSS variables, light/dark themes)
|
||||
│ └── memory.css # Memory settings + admin review panel styles
|
||||
├── js/
|
||||
│ ├── api.js # HTTP client with token refresh
|
||||
│ ├── app.js # Application state machine, startup, routing
|
||||
@@ -268,6 +277,7 @@ src/
|
||||
│ ├── notes.js # Notes panel CRUD + graph + daily notes
|
||||
│ ├── note-graph.js # Canvas force-directed graph visualization
|
||||
│ ├── knowledge.js # Knowledge base UI
|
||||
│ ├── memory-ui.js # Memory settings tab + admin review panel
|
||||
│ └── __tests__/ # Node.js test suite (node --test)
|
||||
├── editor/ # CM6 source (compiled at build time)
|
||||
│ ├── package.json # CM6 + language mode dependencies
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
# v0.15.0 Phase 1 — Changeset Notes
|
||||
|
||||
## Files included in this zip (new or fully replaced)
|
||||
|
||||
```
|
||||
server/treepath/path.go NEW — PathMessage, GetActivePath, etc.
|
||||
server/treepath/summary.go NEW — IsSummaryMessage, FindSummaryBoundary
|
||||
server/treepath/cursor.go NEW — UpdateCursor, NextSiblingIndex
|
||||
server/treepath/siblings.go NEW — GetSiblingCount, GetSiblings, FindLeafFromMessage
|
||||
server/compaction/compaction.go NEW — Service, Compact, CheckRateLimit
|
||||
server/handlers/tree.go REPLACED — thin wrappers delegating to treepath
|
||||
server/handlers/summarize.go REPLACED — thin HTTP wrapper using compaction.Service
|
||||
src/js/settings-handlers.js REPLACED — bug fixes for display name + role config
|
||||
```
|
||||
|
||||
## Surgical edits needed in existing files
|
||||
|
||||
### 1. `server/handlers/completion.go`
|
||||
|
||||
**Delete** the `isSummaryMessage` function (lines ~832–841). It is now
|
||||
defined in `treepath/summary.go` and aliased in `handlers/tree.go`.
|
||||
Without this deletion, the build fails with a duplicate function error.
|
||||
|
||||
```go
|
||||
// DELETE this entire function from completion.go:
|
||||
|
||||
// isSummaryMessage checks if a PathMessage has summary metadata.
|
||||
func isSummaryMessage(m *PathMessage) bool {
|
||||
if m.Metadata == nil {
|
||||
return false
|
||||
}
|
||||
var meta map[string]interface{}
|
||||
if err := json.Unmarshal(*m.Metadata, &meta); err != nil {
|
||||
return false
|
||||
}
|
||||
return meta["type"] == "summary"
|
||||
}
|
||||
```
|
||||
|
||||
No other changes needed in completion.go — the `isSummaryMessage` calls
|
||||
throughout the file now resolve via the alias in `handlers/tree.go`.
|
||||
|
||||
### 2. `server/main.go`
|
||||
|
||||
**Change** the summarize handler wiring (~line 252). Old constructor
|
||||
takes `(stores, roleResolver)`, new one takes `(*compaction.Service)`.
|
||||
|
||||
```go
|
||||
// ── BEFORE ──
|
||||
// Summarize & Continue
|
||||
summarize := handlers.NewSummarizeHandler(stores, roleResolver)
|
||||
protected.POST("/channels/:id/summarize", summarize.Summarize)
|
||||
|
||||
// ── AFTER ──
|
||||
// Compaction service (shared by manual summarize handler + future auto-scanner)
|
||||
compactionSvc := compaction.NewService(stores, roleResolver)
|
||||
|
||||
// Summarize & Continue (manual compaction via HTTP)
|
||||
summarize := handlers.NewSummarizeHandler(compactionSvc)
|
||||
protected.POST("/channels/:id/summarize", summarize.Summarize)
|
||||
```
|
||||
|
||||
**Add import** at top of main.go:
|
||||
```go
|
||||
"git.gobha.me/xcaliber/chat-switchboard/compaction"
|
||||
```
|
||||
|
||||
## Bug fixes included
|
||||
|
||||
### Display Name not saved or used
|
||||
|
||||
`handleSaveSettings()` now also saves `display_name` and `email` via
|
||||
`API.updateProfile()`. Updates `API.user` and calls `UI.updateUser()` so
|
||||
the sidebar reflects the change immediately.
|
||||
|
||||
### User utility model role not displayed after save
|
||||
|
||||
`_initUserRolePrimitive()` → `fetchModels` callback now calls the
|
||||
global `fetchModels()` (which refreshes `App.models` from the server)
|
||||
before returning the model list. This ensures recently added personal
|
||||
provider models appear in the role config dropdown after tab switch or
|
||||
save-refresh.
|
||||
|
||||
## Verification
|
||||
|
||||
After applying all changes:
|
||||
|
||||
1. `cd server && go build .` — should compile cleanly
|
||||
2. Existing integration tests should pass (no behavior change)
|
||||
3. Manual test: Settings → Profile → change display name → Save → verify
|
||||
sidebar updates and value persists on reload
|
||||
4. Manual test: Settings → Model Roles → pick personal provider + model →
|
||||
Save → verify dropdown shows saved values after refresh
|
||||
@@ -1,186 +0,0 @@
|
||||
# v0.15.0 Phase 2 — Changeset Notes
|
||||
|
||||
## Files included in this zip (new)
|
||||
|
||||
```
|
||||
server/compaction/estimator.go NEW — EstimateTokens, EstimatePath, EstimatePathFromSummary
|
||||
server/compaction/estimator_test.go NEW — unit tests (no DB required)
|
||||
server/compaction/scanner.go NEW — Scanner, scan loop, candidate query, shouldCompact
|
||||
```
|
||||
|
||||
## Surgical edits needed in existing files
|
||||
|
||||
### 1. `server/main.go`
|
||||
|
||||
Phase 1 already introduced `compactionSvc`. Now add the scanner below it
|
||||
(after the `compactionSvc` line, before route registration):
|
||||
|
||||
```go
|
||||
import "git.gobha.me/xcaliber/chat-switchboard/compaction" // already present from Phase 1
|
||||
|
||||
// ── Auto-Compaction Scanner ───────────
|
||||
compactionScanner := compaction.NewScanner(compactionSvc, stores, compaction.ScannerConfig{
|
||||
Interval: 5 * time.Minute,
|
||||
Concurrency: 2,
|
||||
})
|
||||
compactionScanner.Start()
|
||||
defer compactionScanner.Stop() // drain in-flight compactions on shutdown
|
||||
```
|
||||
|
||||
Place `defer compactionScanner.Stop()` near the existing
|
||||
`defer kbIngester.Wait()` line.
|
||||
|
||||
**Note:** The scanner re-reads all config from `global_settings` each
|
||||
tick. The `ScannerConfig` struct only sets the ticker interval and
|
||||
semaphore size at startup. Everything else (enabled, threshold, cooldown)
|
||||
is dynamic.
|
||||
|
||||
---
|
||||
|
||||
### 2. `src/index.html`
|
||||
|
||||
Add an auto-compaction section to the admin Settings tab. Insert **before**
|
||||
the Encryption section (before `<section class="admin-section">` with
|
||||
`🔐 Encryption`):
|
||||
|
||||
```html
|
||||
<section class="settings-section">
|
||||
<h3>Auto-Compaction</h3>
|
||||
<label class="checkbox-label"><input type="checkbox" id="adminCompactionEnabled"> Enable automatic conversation compaction</label>
|
||||
<p class="section-hint">When enabled, long conversations are automatically summarized in the background using the utility model. Ships off by default — requires a utility model role to be configured.</p>
|
||||
<div id="compactionConfigFields" style="display:none;margin-top:8px">
|
||||
<div class="form-row">
|
||||
<div class="form-group">
|
||||
<label>Threshold</label>
|
||||
<div class="form-row" style="gap:4px;align-items:center">
|
||||
<input type="number" id="adminCompactionThreshold" value="70" min="10" max="95" step="5" style="width:70px">
|
||||
<span class="form-hint">% of context window</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>Cooldown</label>
|
||||
<div class="form-row" style="gap:4px;align-items:center">
|
||||
<input type="number" id="adminCompactionCooldown" value="30" min="5" max="360" step="5" style="width:70px">
|
||||
<span class="form-hint">minutes per channel</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. `src/js/ui-admin.js` — `loadAdminSettings()`
|
||||
|
||||
Add these lines **after** the Web Search config loading block
|
||||
(after `document.getElementById('searxngConfigFields').style.display = ...`):
|
||||
|
||||
```javascript
|
||||
// Auto-Compaction (global_settings)
|
||||
const compactionCfg = getSetting('auto_compaction', {}) || {};
|
||||
const compactionEnabled = document.getElementById('adminCompactionEnabled');
|
||||
if (compactionEnabled) {
|
||||
compactionEnabled.checked = !!compactionCfg.enabled;
|
||||
document.getElementById('compactionConfigFields').style.display =
|
||||
compactionCfg.enabled ? '' : 'none';
|
||||
}
|
||||
const compThreshold = document.getElementById('adminCompactionThreshold');
|
||||
if (compThreshold) compThreshold.value = String(compactionCfg.threshold || 70);
|
||||
const compCooldown = document.getElementById('adminCompactionCooldown');
|
||||
if (compCooldown) compCooldown.value = String(compactionCfg.cooldown || 30);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. `src/js/settings-handlers.js` — `handleSaveAdminSettings()`
|
||||
|
||||
Add these lines **after** the `search_config` save block (before
|
||||
`UI.toast('Settings saved', 'success')`):
|
||||
|
||||
```javascript
|
||||
// Auto-Compaction config → global_settings
|
||||
const compactionEnabled = document.getElementById('adminCompactionEnabled')?.checked || false;
|
||||
const compactionThreshold = parseInt(document.getElementById('adminCompactionThreshold')?.value) || 70;
|
||||
const compactionCooldown = parseInt(document.getElementById('adminCompactionCooldown')?.value) || 30;
|
||||
await API.adminUpdateSetting('auto_compaction', { value: {
|
||||
enabled: compactionEnabled,
|
||||
threshold: compactionThreshold,
|
||||
cooldown: compactionCooldown,
|
||||
}});
|
||||
|
||||
// Persist individual keys for scanner (reads these independently)
|
||||
await API.adminUpdateSetting('auto_compaction_enabled', { value: compactionEnabled });
|
||||
await API.adminUpdateSetting('auto_compaction_threshold', { value: compactionThreshold / 100 });
|
||||
await API.adminUpdateSetting('auto_compaction_cooldown_minutes', { value: compactionCooldown });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. `src/js/settings-handlers.js` — `_initSettingsListeners()`
|
||||
|
||||
Add a toggle listener for the compaction checkbox, alongside the existing
|
||||
banner toggle. Add **after** the `adminBannerEnabled` change listener:
|
||||
|
||||
```javascript
|
||||
document.getElementById('adminCompactionEnabled')?.addEventListener('change', (e) => {
|
||||
document.getElementById('compactionConfigFields').style.display = e.target.checked ? '' : 'none';
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. `src/js/ui-core.js` — `_summaryHTML()`
|
||||
|
||||
Show "(auto)" label on auto-triggered summaries. The `trigger` field is
|
||||
already in message metadata from Phase 1.
|
||||
|
||||
**Replace** the `_summaryHTML` method:
|
||||
|
||||
```javascript
|
||||
_summaryHTML(msg) {
|
||||
const meta = msg.metadata || {};
|
||||
const count = meta.summarized_count || '?';
|
||||
const model = meta.utility_model || 'utility model';
|
||||
const trigger = meta.trigger === 'auto' ? ' · auto' : '';
|
||||
return `
|
||||
<div class="message-summary" data-msg-id="${esc(msg.id || '')}">
|
||||
<div class="summary-header">
|
||||
<span>📝 Conversation summary (${count} messages, by ${esc(model)}${trigger})</span>
|
||||
</div>
|
||||
<div class="msg-text">${formatMessage(msg.content)}</div>
|
||||
</div>`;
|
||||
},
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How the scanner reads settings
|
||||
|
||||
The scanner calls `GlobalConfig.Get()` **every tick** for these keys:
|
||||
|
||||
| Key | Read by | How stored |
|
||||
|-----|---------|------------|
|
||||
| `auto_compaction_enabled` | `isEnabled()` | `{value: true/false}` |
|
||||
| `auto_compaction_threshold` | `getThreshold()` | `{value: 0.70}` (float 0-1) |
|
||||
| `auto_compaction_cooldown_minutes` | `getCooldownDuration()` | `{value: 30}` |
|
||||
|
||||
The admin UI stores a combined `auto_compaction` key (for display) AND
|
||||
the individual keys (for scanner consumption). This avoids the scanner
|
||||
needing to parse a composite object.
|
||||
|
||||
Interval and concurrency are set at startup via `ScannerConfig` and
|
||||
require a restart to change. This is intentional — they affect the
|
||||
goroutine pool size and ticker, which shouldn't be hot-swapped.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `cd server && go test ./compaction/` — estimator tests pass (no DB)
|
||||
2. `cd server && go build .` — compiles cleanly
|
||||
3. Manual test: Admin → Settings → enable auto-compaction → Save →
|
||||
check server logs for `🔍 compaction scanner started`
|
||||
4. Manual test: with utility role configured and a long conversation,
|
||||
verify auto-compaction fires after the threshold is exceeded
|
||||
5. Verify cooldown: same channel should not re-compact within 30 minutes
|
||||
6. Verify kill switch: disable auto-compaction → scanner stops processing
|
||||
on next tick
|
||||
@@ -1,106 +0,0 @@
|
||||
# v0.15.0 Phase 4 — Guard Rail + Integration Tests
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
server/compaction/compaction.go UPDATED — context budget guard rail
|
||||
server/compaction/compaction_test.go NEW — scanner integration tests (DB required)
|
||||
server/compaction/testmain_test.go NEW — TestMain for compaction package
|
||||
server/database/seed_helpers.go NEW — SeedTestMessage, SeedTestMessages, SeedTestCursor
|
||||
server/handlers/live_compaction_test.go NEW — e2e tests with Venice/Qwen3-4B
|
||||
```
|
||||
|
||||
## Changes
|
||||
|
||||
### 1. Context Budget Guard Rail (`compaction.go`)
|
||||
|
||||
Added between prompt construction and LLM call. Prevents sending
|
||||
truncated input to small utility models.
|
||||
|
||||
**How it works:**
|
||||
1. Estimates total prompt tokens (system + conversation content)
|
||||
2. Resolves the utility model's `max_context` from catalog
|
||||
3. Reserves 20% for output (the summary itself)
|
||||
4. If `estimated_prompt > max_context × 0.80`, returns `ErrContextBudget`
|
||||
|
||||
**Resolution chain** for utility model context:
|
||||
- `Resolver.GetConfig()` → primary binding → `Catalog.GetByModelID()`
|
||||
- Fallback: `Catalog.GetByModelIDAny()` (different provider)
|
||||
- Hard fallback: `DefaultBudget` (128K)
|
||||
|
||||
The scanner's `doCompact()` already handles errors gracefully — a budget
|
||||
error just logs a warning and retries next cycle (by which time the
|
||||
conversation may have been manually compacted or the admin may have
|
||||
upgraded the utility model).
|
||||
|
||||
**New export:** `ErrContextBudget` — callers can use `errors.Is()` to
|
||||
distinguish budget failures from LLM errors.
|
||||
|
||||
### 2. Test Seed Helpers (`database/seed_helpers.go`)
|
||||
|
||||
New file in the `database` package alongside `testhelper.go`:
|
||||
|
||||
- `SeedTestMessage(t, channelID, parentID, role, content) → msgID`
|
||||
- `SeedTestMessages(t, channelID, count, contentSize) → []msgIDs`
|
||||
Creates a linear alternating user/assistant chain
|
||||
- `SeedTestCursor(t, channelID, userID, leafID)`
|
||||
Sets the active leaf for tree path resolution
|
||||
|
||||
### 3. Scanner Integration Tests (`compaction/compaction_test.go`)
|
||||
|
||||
All DB-dependent tests use `database.RequireTestDB(t)` and skip
|
||||
gracefully when no DB is configured.
|
||||
|
||||
| Test | What it verifies |
|
||||
|------|-----------------|
|
||||
| `FindCandidates_ReturnsQualifying` | Channel with ≥10 msgs, ≥20K chars, proper age range appears |
|
||||
| `FindCandidates_ExcludesTooFewMessages` | <10 messages → excluded |
|
||||
| `FindCandidates_ExcludesTooRecent` | Updated just now → excluded (activity gap) |
|
||||
| `FindCandidates_ExcludesArchived` | `is_archived=true` → excluded |
|
||||
| `Cooldown` | Timestamps recorded and checked correctly |
|
||||
| `InFlightDedup` | sync.Map prevents concurrent compaction of same channel |
|
||||
| `ShouldCompact_ChannelOptOut` | `settings.auto_compaction=false` → rejected |
|
||||
| `IsEnabled` | Reads `auto_compaction_enabled` from global_settings |
|
||||
| `GetThreshold_Default` | Returns 0.70 when no override |
|
||||
| `GetThreshold_GlobalOverride` | Reads from global_settings |
|
||||
| `GetThreshold_ChannelOverride` | Channel setting takes precedence |
|
||||
| `GetCooldownDuration_Default` | Returns 30min default |
|
||||
| `GetCooldownDuration_Override` | Reads from global_settings |
|
||||
| `GuardRailMath` | Validates token math for 32K vs 128K models |
|
||||
|
||||
### 4. Live E2E Tests (`handlers/live_compaction_test.go`)
|
||||
|
||||
Requires: `TEST_DATABASE_URL` + `VENICE_API_KEY`
|
||||
|
||||
| Test | What it verifies |
|
||||
|------|-----------------|
|
||||
| `CompactionFullPipeline` | Seeds 10 realistic messages → Compact() → verifies summary message in DB, metadata fields, cursor update, usage log entry |
|
||||
| `CompactionContextBudgetGuardRail` | Seeds 160K chars of content → Compact() returns ErrContextBudget with 32K model |
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Unit only (no DB)
|
||||
cd server && go test ./compaction/ -run TestEstimate -v
|
||||
|
||||
# Integration (requires DB)
|
||||
cd server && go test ./compaction/ -v
|
||||
|
||||
# Live e2e (requires DB + Venice key)
|
||||
cd server && VENICE_API_KEY=... go test ./handlers/ -run TestLive_Compaction -v
|
||||
```
|
||||
|
||||
## Surgical Edit
|
||||
|
||||
### `database/testhelper.go`
|
||||
|
||||
Add this import if not already present (the new `seed_helpers.go` file
|
||||
is a separate file in the same package, so no edit needed to
|
||||
`testhelper.go` itself):
|
||||
|
||||
```go
|
||||
// No changes needed — seed_helpers.go is a new file in package database
|
||||
```
|
||||
|
||||
The `strings` import in `seed_helpers.go` is used by `SeedTestMessages`
|
||||
for `strings.Repeat`.
|
||||
99
docs/CHANGES-0.18.0-phase1.md
Normal file
99
docs/CHANGES-0.18.0-phase1.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# v0.18.0 Phase 1 — Changes Guide
|
||||
|
||||
## New Files (drop in place)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `server/database/migrations/004_v0180_memories.sql` | Postgres migration — memories table, indexes, trigger |
|
||||
| `server/database/migrations/sqlite/003_v0180_memories.sql` | SQLite equivalent |
|
||||
| `server/models/models_memory.go` | Memory + MemoryFilter model structs + scope/status constants |
|
||||
| `server/store/store_memory.go` | MemoryStore interface definition |
|
||||
| `server/store/postgres/memory.go` | Postgres MemoryStore implementation |
|
||||
| `server/store/sqlite/memory.go` | SQLite MemoryStore implementation |
|
||||
| `server/tools/memory.go` | memory_save + memory_recall tools with late registration |
|
||||
| `server/handlers/memory_inject.go` | BuildMemoryHint() for context injection |
|
||||
| `docs/DESIGN-0.18.0.md` | Design document |
|
||||
|
||||
## Existing File Modifications
|
||||
|
||||
### 1. `server/store/interfaces.go`
|
||||
|
||||
Add to the `Stores` struct (after `ResourceGrants`):
|
||||
|
||||
```go
|
||||
Memories MemoryStore
|
||||
```
|
||||
|
||||
The `MemoryStore` interface itself is in the new `store_memory.go` file
|
||||
(same package, separate file for cleanliness).
|
||||
|
||||
### 2. `server/store/postgres/stores.go`
|
||||
|
||||
Add to the `NewStores()` return:
|
||||
|
||||
```go
|
||||
Memories: NewMemoryStore(),
|
||||
```
|
||||
|
||||
### 3. `server/store/sqlite/stores.go`
|
||||
|
||||
Add to the `NewStores()` return:
|
||||
|
||||
```go
|
||||
Memories: NewMemoryStore(),
|
||||
```
|
||||
|
||||
### 4. `server/main.go`
|
||||
|
||||
Add after the existing tool registrations (~line 169):
|
||||
|
||||
```go
|
||||
// Memory tools (v0.18.0) — late registration, needs stores
|
||||
tools.RegisterMemoryTools(stores)
|
||||
```
|
||||
|
||||
### 5. `server/handlers/completion.go`
|
||||
|
||||
In `loadConversation()`, add memory injection after the KB hint block
|
||||
(~line 798, after the `BuildKBHint` block):
|
||||
|
||||
```go
|
||||
// ── Memory injection (recall known facts about user) ──
|
||||
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != "" {
|
||||
messages = append(messages, providers.Message{
|
||||
Role: "system",
|
||||
Content: memHint,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### 6. `VERSION`
|
||||
|
||||
```
|
||||
0.18.0
|
||||
```
|
||||
|
||||
### 7. `docs/ROADMAP.md`
|
||||
|
||||
Mark Phase 1 items as complete (Data Model, Tools, Memory Injection basics).
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
1. **Migration runs clean** — both Postgres and SQLite
|
||||
2. **Tools register** — check startup logs for `🔧 Registered tool: memory_save` and `memory_recall`
|
||||
3. **memory_save** — in a chat, tell the AI something personal ("I prefer Go over Python"). The AI should call memory_save.
|
||||
4. **memory_recall** — in a NEW chat, ask "what programming language do I prefer?" The AI should call memory_recall and find it.
|
||||
5. **Persona scoping** — with a Persona active, memories save as `persona_user` scope. Without a Persona, they save as `user` scope.
|
||||
6. **Upsert** — saving the same key twice updates the value instead of creating a duplicate.
|
||||
7. **Injection** — memories appear in the system prompt context (check server logs for `🧠 Injected N memories`).
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- Memory injection happens in `loadConversation()` as a system message, same pattern as KB hints and compaction summaries
|
||||
- Scope priority in Recall: persona_user > persona > user (most specific wins)
|
||||
- The unique index on `(scope, owner_id, COALESCE(user_id, nil_uuid), key)` prevents duplicate keys within a scope
|
||||
- Confidence is LLM-provided: 1.0 for explicit statements, lower for inferences
|
||||
- Status field supports Phase 2's review pipeline (pending_review, archived)
|
||||
- Embedding column exists but is unused in Phase 1 — Phase 2 adds semantic recall
|
||||
191
docs/CHANGES-0.18.0-phase2.md
Normal file
191
docs/CHANGES-0.18.0-phase2.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# v0.18.0 Phase 2 — Changes Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 2 adds: automatic memory extraction via background scanner,
|
||||
embedding-powered semantic recall, memory review pipeline API, and
|
||||
persona memory configuration.
|
||||
|
||||
## New Files (drop in place)
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `server/database/migrations/005_v0180_memory_phase2.sql` | Postgres: persona memory columns, HNSW index, extraction log table |
|
||||
| `server/database/migrations/sqlite/004_v0180_memory_phase2.sql` | SQLite equivalent |
|
||||
| `server/memory/extractor.go` | Extraction service — calls utility model to extract facts from conversations |
|
||||
| `server/memory/scanner.go` | Background scanner — finds conversations to extract, same pattern as compaction.Scanner |
|
||||
| `server/store/postgres/memory_hybrid.go` | RecallHybrid — pgvector cosine distance + keyword merge |
|
||||
| `server/store/sqlite/memory_hybrid.go` | RecallHybrid — app-level cosine similarity + keyword merge |
|
||||
| `server/handlers/memory.go` | REST API: list, edit, delete, approve/reject memories |
|
||||
| `server/handlers/memory_inject.go` | **REPLACES Phase 1 version** — now supports hybrid recall with embeddings |
|
||||
|
||||
## Updated Files (replace Phase 1 versions)
|
||||
|
||||
| File | What Changed |
|
||||
|------|-------------|
|
||||
| `server/tools/memory.go` | **REPLACES Phase 1 version** — RegisterMemoryTools now takes embedder param, memory_save embeds on save |
|
||||
|
||||
## Existing File Modifications
|
||||
|
||||
### 1. `server/models/models.go` — Persona struct
|
||||
|
||||
Add after the `KBIDs` field:
|
||||
|
||||
```go
|
||||
MemoryEnabled bool `json:"memory_enabled" db:"memory_enabled"`
|
||||
MemoryExtractionPrompt *string `json:"memory_extraction_prompt,omitempty" db:"memory_extraction_prompt"`
|
||||
```
|
||||
|
||||
Add to `PersonaPatch`:
|
||||
|
||||
```go
|
||||
MemoryEnabled *bool `json:"memory_enabled,omitempty"`
|
||||
MemoryExtractionPrompt *string `json:"memory_extraction_prompt,omitempty"`
|
||||
```
|
||||
|
||||
### 2. `server/store/store_memory.go` — MemoryStore interface
|
||||
|
||||
Add to the interface:
|
||||
|
||||
```go
|
||||
// RecallHybrid returns active memories using vector similarity + keyword search.
|
||||
// If queryVec is nil/empty, falls back to keyword-only (same as Recall).
|
||||
RecallHybrid(ctx context.Context, userID string, personaID *string, query string, queryVec []float64, limit int) ([]models.Memory, error)
|
||||
```
|
||||
|
||||
### 3. `server/main.go` — Tool registration + scanner startup
|
||||
|
||||
**Update** the existing RegisterMemoryTools call (~line 170) to pass the embedder:
|
||||
|
||||
```go
|
||||
// Memory tools (v0.18.0) — late registration, needs stores + embedder
|
||||
tools.RegisterMemoryTools(stores, kbEmbedder)
|
||||
```
|
||||
|
||||
**Add** memory extraction scanner startup (after compaction scanner, or near the end
|
||||
of the startup block — see compaction scanner as pattern reference):
|
||||
|
||||
```go
|
||||
// Memory extraction scanner (v0.18.0 Phase 2)
|
||||
// Opt-in: requires global_settings.memory_extraction_enabled = true
|
||||
memExtractor := memory.NewExtractor(stores, roleResolver, kbEmbedder)
|
||||
memScanner := memory.NewScanner(memExtractor, stores, memory.ScannerConfig{})
|
||||
memScanner.Start()
|
||||
defer memScanner.Stop()
|
||||
```
|
||||
|
||||
Import: `"git.gobha.me/xcaliber/chat-switchboard/memory"`
|
||||
|
||||
### 4. `server/main.go` — Memory API routes
|
||||
|
||||
Add under the protected routes (~after presets/personas routes):
|
||||
|
||||
```go
|
||||
// Memory management (v0.18.0)
|
||||
memH := handlers.NewMemoryHandler(stores)
|
||||
protected.GET("/memories", memH.ListMyMemories)
|
||||
protected.PUT("/memories/:id", memH.UpdateMemory)
|
||||
protected.DELETE("/memories/:id", memH.DeleteMemory)
|
||||
protected.POST("/memories/:id/approve", memH.ApproveMemory)
|
||||
protected.POST("/memories/:id/reject", memH.RejectMemory)
|
||||
protected.GET("/memories/count", memH.MemoryCount)
|
||||
```
|
||||
|
||||
Add under admin routes:
|
||||
|
||||
```go
|
||||
// Admin memory review (v0.18.0)
|
||||
adminMemH := handlers.NewMemoryHandler(stores)
|
||||
admin.GET("/memories/pending", adminMemH.ListPendingReview)
|
||||
admin.POST("/memories/bulk-approve", adminMemH.BulkApprove)
|
||||
```
|
||||
|
||||
### 5. `server/handlers/completion.go` — BuildMemoryHint signature change
|
||||
|
||||
The `BuildMemoryHint` call in `loadConversation()` changes signature.
|
||||
Update from Phase 1:
|
||||
|
||||
```go
|
||||
// OLD (Phase 1):
|
||||
if memHint := BuildMemoryHint(context.Background(), h.stores, userID, personaID); memHint != "" {
|
||||
|
||||
// NEW (Phase 2) — pass embedder and last user message for semantic recall:
|
||||
lastUserMsg := ""
|
||||
for i := len(messages) - 1; i >= 0; i-- {
|
||||
if messages[i].Role == "user" {
|
||||
lastUserMsg = messages[i].Content
|
||||
break
|
||||
}
|
||||
}
|
||||
if memHint := BuildMemoryHint(context.Background(), h.stores, h.embedder, userID, personaID, lastUserMsg); memHint != "" {
|
||||
```
|
||||
|
||||
This requires adding the embedder to CompletionHandler. In the struct:
|
||||
|
||||
```go
|
||||
type CompletionHandler struct {
|
||||
vault *crypto.KeyResolver
|
||||
stores store.Stores
|
||||
hub *events.Hub
|
||||
objStore storage.ObjectStore
|
||||
embedder *knowledge.Embedder // NEW: for memory semantic recall
|
||||
}
|
||||
```
|
||||
|
||||
And in `NewCompletionHandler`:
|
||||
|
||||
```go
|
||||
func NewCompletionHandler(vault *crypto.KeyResolver, stores store.Stores, hub *events.Hub, objStore storage.ObjectStore, embedder *knowledge.Embedder) *CompletionHandler {
|
||||
return &CompletionHandler{vault: vault, stores: stores, hub: hub, objStore: objStore, embedder: embedder}
|
||||
}
|
||||
```
|
||||
|
||||
Update the call site in main.go:
|
||||
|
||||
```go
|
||||
comp := handlers.NewCompletionHandler(keyResolver, stores, hub, objStore, kbEmbedder)
|
||||
```
|
||||
|
||||
### 6. `server/store/postgres/persona.go` — Scan memory fields
|
||||
|
||||
Any Persona scan queries need to include the new columns. Add
|
||||
`memory_enabled` and `memory_extraction_prompt` to SELECT lists and Scan calls
|
||||
in `GetByID`, `ListForUser`, etc.
|
||||
|
||||
### 7. `server/store/sqlite/persona.go` — Same as above for SQLite.
|
||||
|
||||
---
|
||||
|
||||
## Admin Setup
|
||||
|
||||
The extraction scanner is **opt-in**. To enable:
|
||||
|
||||
1. Set global config: `memory_extraction_enabled` = `true`
|
||||
2. The scanner runs every 10 minutes (configurable)
|
||||
3. Extracted memories start as `pending_review`
|
||||
4. Admin reviews via `GET /api/v1/admin/memories/pending`
|
||||
5. Approve: `POST /api/v1/memories/:id/approve`
|
||||
6. Reject: `POST /api/v1/memories/:id/reject`
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
1. **Migration** — both Postgres and SQLite, verify `memory_extraction_log` table created
|
||||
2. **Persona columns** — `ALTER TABLE personas ADD COLUMN memory_enabled` runs clean
|
||||
3. **HNSW index** — Postgres only, verify with `\di+ idx_memories_embedding`
|
||||
4. **Embedding on save** — call memory_save, check logs for `🧠 memory X embedded`
|
||||
5. **Hybrid recall** — with embeddings populated, memory_recall should find semantically relevant results even with different keywords
|
||||
6. **Extraction scanner** — set `memory_extraction_enabled=true` in global config, wait for scan cycle, verify `🧠 memory extraction:` logs
|
||||
7. **Extracted status** — auto-extracted memories should have `status=pending_review`
|
||||
8. **Review API** — `GET /memories?status=pending_review` returns pending items
|
||||
9. **Approve/reject** — POST approve changes status to active, reject archives
|
||||
10. **BuildMemoryHint** — with embedder, verify semantic recall contextualizes based on user's message
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Extraction scanner** follows compaction.Scanner pattern: ticker loop, semaphore concurrency, in-flight dedup, graceful shutdown via Stop()
|
||||
- **Extraction log** (`memory_extraction_log`) prevents re-processing: tracks last_message_id per channel+user
|
||||
- **RecallHybrid** merges vector + keyword results with dedup by ID; semantic results rank first
|
||||
- **SQLite hybrid** loads all embedded memories into Go and computes cosine similarity in-process (acceptable for single-user deployments)
|
||||
- **Persona memory toggle** (`memory_enabled`) gates both tool-based and extraction-based memory for that persona
|
||||
- **Extraction prompt** is customizable per persona — a helpdesk persona extracts FAQ patterns, a tutoring persona extracts learning progress
|
||||
- **Phase 3** will add the frontend UI: Settings → Memory panel, admin review queue, per-persona toggle in persona editor
|
||||
283
docs/CHANGES-0.18.0-phase3.md
Normal file
283
docs/CHANGES-0.18.0-phase3.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# v0.18.0 Phase 3 — Frontend UI Changes Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 3 adds the user-facing memory management UI:
|
||||
- **Settings → Memory tab**: view, search, edit, approve/reject memories
|
||||
- **Admin → AI → Memory**: review pipeline for auto-extracted memories
|
||||
- **Persona forms**: memory_enabled toggle + custom extraction prompt
|
||||
- **Admin → System → Settings**: memory extraction toggle
|
||||
|
||||
## New Files (drop in place)
|
||||
|
||||
| File | Lines | Description |
|
||||
|------|-------|-------------|
|
||||
| `src/js/memory-ui.js` | ~310 | Memory UI module — settings panel, admin panel, persona form fields |
|
||||
| `src/css/memory.css` | ~230 | Styles for memory cards, badges, toolbar, edit forms |
|
||||
|
||||
## Existing File Modifications
|
||||
|
||||
### 1. `src/index.html` — Add Memory Tab + Admin Section
|
||||
|
||||
#### A. Settings Modal — Add Memory Tab Button
|
||||
|
||||
After the Knowledge tab button (~line 389):
|
||||
|
||||
```html
|
||||
<button class="settings-tab" data-stab="memory" onclick="UI.switchSettingsTab('memory')" id="settingsMemoryTabBtn">Memory</button>
|
||||
```
|
||||
|
||||
#### B. Settings Modal — Add Memory Tab Content
|
||||
|
||||
After the Knowledge Bases tab content block (after `settingsKnowledgeBasesTab` div, ~line 552):
|
||||
|
||||
```html
|
||||
<!-- Memory Tab (v0.18.0) -->
|
||||
<div class="settings-tab-content" id="settingsMemoryTab" style="display:none">
|
||||
<section class="settings-section">
|
||||
<h3 style="font-size:14px;margin-bottom:4px">My Memories</h3>
|
||||
<p class="section-hint" style="margin-bottom:8px">Facts and preferences learned from your conversations. Memories help AI provide more personalized responses.</p>
|
||||
<div id="settingsMemoryContent"></div>
|
||||
</section>
|
||||
</div>
|
||||
```
|
||||
|
||||
#### C. Admin Panel — Add Memory Section
|
||||
|
||||
In the admin sections HTML, after the `adminPresetsTab` block (~line 790),
|
||||
add a new admin section:
|
||||
|
||||
```html
|
||||
<div class="admin-section-content" id="adminMemoryTab" style="display:none">
|
||||
<div id="adminMemoryContent"></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
#### D. Admin Settings — Add Memory Extraction Toggle
|
||||
|
||||
In `adminSettingsTab`, after the Auto-Compaction section (~line 908):
|
||||
|
||||
```html
|
||||
<section class="settings-section">
|
||||
<h3>Memory Extraction</h3>
|
||||
<label class="checkbox-label"><input type="checkbox" id="adminMemoryExtractionEnabled"> Enable automatic memory extraction</label>
|
||||
<p class="section-hint">When enabled, the system automatically extracts memorable facts from conversations using the utility model. Extracted memories require admin approval before becoming active. Requires a utility model role.</p>
|
||||
<div id="memoryExtractionConfigFields" style="display:none;margin-top:8px">
|
||||
<label class="checkbox-label"><input type="checkbox" id="adminMemoryAutoApprove"> Auto-approve extracted memories</label>
|
||||
<p class="section-hint">Skip the review pipeline and activate extracted memories immediately. Not recommended for sensitive environments.</p>
|
||||
</div>
|
||||
</section>
|
||||
```
|
||||
|
||||
#### E. Script + CSS Include
|
||||
|
||||
In the `<head>` section, add the CSS:
|
||||
|
||||
```html
|
||||
<link rel="stylesheet" href="css/memory.css">
|
||||
```
|
||||
|
||||
In the script block at the bottom (after `knowledge-ui.js`):
|
||||
|
||||
```html
|
||||
<script src="js/memory-ui.js"></script>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. `src/js/api.js` — Memory API Client Methods
|
||||
|
||||
Add after the user presets section (~line 310):
|
||||
|
||||
```javascript
|
||||
// Memory (v0.18.0)
|
||||
listMemories(status, query) {
|
||||
const params = new URLSearchParams();
|
||||
if (status) params.set('status', status);
|
||||
if (query) params.set('query', query);
|
||||
return this._get('/api/v1/memories?' + params.toString());
|
||||
},
|
||||
getMemoryCount() { return this._get('/api/v1/memories/count'); },
|
||||
updateMemory(id, data) { return this._put(`/api/v1/memories/${id}`, data); },
|
||||
deleteMemory(id) { return this._del(`/api/v1/memories/${id}`); },
|
||||
approveMemory(id) { return this._post(`/api/v1/memories/${id}/approve`); },
|
||||
rejectMemory(id) { return this._post(`/api/v1/memories/${id}/reject`); },
|
||||
bulkApproveMemories(ids) { return this._post('/api/v1/admin/memories/bulk-approve', { ids }); },
|
||||
adminListPendingMemories() { return this._get('/api/v1/admin/memories/pending'); },
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. `src/js/ui-core.js` — Wire Memory Tab into Settings
|
||||
|
||||
In `switchSettingsTab()` (~line 932), add after the `knowledgeBases` case:
|
||||
|
||||
```javascript
|
||||
if (tab === 'memory') {
|
||||
if (typeof MemoryUI !== 'undefined') {
|
||||
MemoryUI.openSettingsPanel();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. `src/js/ui-admin.js` — Register Memory in Admin Panel
|
||||
|
||||
#### A. Add "memory" to ADMIN_SECTIONS (~line 9)
|
||||
|
||||
```javascript
|
||||
const ADMIN_SECTIONS = {
|
||||
people: ['users', 'teams', 'groups'],
|
||||
ai: ['providers', 'models', 'presets', 'roles', 'knowledgeBases', 'memory'],
|
||||
system: ['settings', 'storage', 'extensions'],
|
||||
monitoring: ['usage', 'audit', 'stats'],
|
||||
};
|
||||
```
|
||||
|
||||
#### B. Add label (~line 18)
|
||||
|
||||
In ADMIN_LABELS, add:
|
||||
|
||||
```javascript
|
||||
memory: 'Memory',
|
||||
```
|
||||
|
||||
#### C. Add loader (~line 37)
|
||||
|
||||
In ADMIN_LOADERS, add:
|
||||
|
||||
```javascript
|
||||
memory: () => {
|
||||
if (typeof MemoryUI === 'undefined') {
|
||||
console.error('[Admin] MemoryUI not loaded');
|
||||
return;
|
||||
}
|
||||
MemoryUI.openAdminPanel();
|
||||
},
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. `src/js/ui-admin.js` — Load/Save Memory Extraction Settings
|
||||
|
||||
#### A. In `loadAdminSettings()` (~after compaction config loading)
|
||||
|
||||
Add after the compaction threshold/cooldown block:
|
||||
|
||||
```javascript
|
||||
// Memory Extraction (v0.18.0)
|
||||
const memCfg = getSetting('memory_extraction', {}) || {};
|
||||
const memExtractionEl = document.getElementById('adminMemoryExtractionEnabled');
|
||||
if (memExtractionEl) {
|
||||
memExtractionEl.checked = !!memCfg.enabled;
|
||||
document.getElementById('memoryExtractionConfigFields').style.display =
|
||||
memCfg.enabled ? '' : 'none';
|
||||
}
|
||||
const memAutoApproveEl = document.getElementById('adminMemoryAutoApprove');
|
||||
if (memAutoApproveEl) memAutoApproveEl.checked = !!memCfg.auto_approve;
|
||||
```
|
||||
|
||||
#### B. Wire toggle visibility
|
||||
|
||||
Add in `_initAdminListeners()` or inline:
|
||||
|
||||
```javascript
|
||||
document.getElementById('adminMemoryExtractionEnabled')?.addEventListener('change', function() {
|
||||
document.getElementById('memoryExtractionConfigFields').style.display =
|
||||
this.checked ? '' : 'none';
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. `src/js/settings-handlers.js` — Save Memory Extraction Config
|
||||
|
||||
In `handleSaveAdminSettings()`, before the final toast (~line 252):
|
||||
|
||||
```javascript
|
||||
// Memory Extraction config (v0.18.0)
|
||||
const memExtractionEnabled = document.getElementById('adminMemoryExtractionEnabled')?.checked || false;
|
||||
const memAutoApprove = document.getElementById('adminMemoryAutoApprove')?.checked || false;
|
||||
await API.adminUpdateSetting('memory_extraction', { value: {
|
||||
enabled: memExtractionEnabled,
|
||||
auto_approve: memAutoApprove,
|
||||
}});
|
||||
// Scanner reads this individual key each tick
|
||||
await API.adminUpdateSetting('memory_extraction_enabled', { value: memExtractionEnabled });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. `src/js/ui-core.js` — Persona Form Memory Fields
|
||||
|
||||
For **admin** persona forms, in the function that initializes the admin preset
|
||||
form (after `renderPresetForm` call), add:
|
||||
|
||||
```javascript
|
||||
if (typeof MemoryUI !== 'undefined') {
|
||||
MemoryUI.appendMemoryFields(container, prefix);
|
||||
}
|
||||
```
|
||||
|
||||
And in the submit handler, merge memory values:
|
||||
|
||||
```javascript
|
||||
const values = form.getValues();
|
||||
if (typeof MemoryUI !== 'undefined') {
|
||||
Object.assign(values, MemoryUI.getMemoryValues(prefix));
|
||||
}
|
||||
```
|
||||
|
||||
When editing an existing persona, call:
|
||||
|
||||
```javascript
|
||||
if (typeof MemoryUI !== 'undefined') {
|
||||
MemoryUI.setMemoryValues(prefix, persona);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
1. **Settings → Memory tab** — click tab, verify memory list loads
|
||||
2. **Memory search** — type in search box, verify filtering works
|
||||
3. **Memory status filter** — switch between Active/Pending/Archived
|
||||
4. **Edit memory** — click pencil icon, modify key/value, save
|
||||
5. **Delete memory** — click trash icon, confirm, verify removal
|
||||
6. **Approve/Reject** — with pending memories, verify buttons work
|
||||
7. **Approve All** — bulk approve pending memories
|
||||
8. **Admin → AI → Memory** — navigate to admin memory panel
|
||||
9. **Admin pending review** — verify pending memories show with approve/reject
|
||||
10. **Admin bulk approve** — verify bulk action works
|
||||
11. **Admin Settings → Memory Extraction** — toggle on/off, verify checkbox persists
|
||||
12. **Persona form** — create new persona, verify memory section appears
|
||||
13. **Persona memory toggle** — uncheck, verify it persists on save
|
||||
14. **Mobile responsive** — test all memory views on narrow viewport
|
||||
15. **Empty states** — verify graceful display when no memories exist
|
||||
|
||||
## CSS Variable Dependencies
|
||||
|
||||
The memory styles reference these existing CSS variables:
|
||||
- `--bg-1`, `--bg-2`, `--bg-3` — background layers
|
||||
- `--text-1`, `--text-2`, `--text-3` — text colors
|
||||
- `--border` — border color
|
||||
- `--accent`, `--accent-dim`, `--accent-text` — accent colors
|
||||
- `--success`, `--warning`, `--danger` — status colors
|
||||
|
||||
All are defined in the existing `styles.css` theme system.
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- `MemoryUI` follows the same pattern as `KnowledgeUI` — standalone module
|
||||
that integrates via `typeof MemoryUI !== 'undefined'` guards
|
||||
- No build step required — vanilla JS module loaded via `<script>` tag
|
||||
- Admin memory section registered in `ADMIN_SECTIONS` and `ADMIN_LOADERS`
|
||||
maps following the existing convention
|
||||
- Persona memory fields are appended dynamically via `appendMemoryFields()`
|
||||
rather than embedded in `renderPresetForm()` to avoid modifying the shared
|
||||
form builder
|
||||
- Debounce on search input prevents excessive API calls
|
||||
- Memory cards use inline edit — no modal, same pattern as note inline editing
|
||||
- CSS uses existing theme variables for consistent dark/light mode support
|
||||
235
docs/DESIGN-0.18.0.md
Normal file
235
docs/DESIGN-0.18.0.md
Normal file
@@ -0,0 +1,235 @@
|
||||
# DESIGN-0.18.0: Memory System
|
||||
|
||||
**Version:** 0.18.0
|
||||
**Status:** Phase 1 Complete
|
||||
**Depends on:** compaction (v0.15.0), knowledge bases (v0.14.0), persona-KB binding (v0.17.0), SQLite backend (v0.17.1)
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Memory provides long-term fact persistence across conversations. Unlike
|
||||
compaction (within-conversation context) or knowledge bases (static
|
||||
documents), memory captures preferences, facts, and context that
|
||||
accumulate over time from natural conversation.
|
||||
|
||||
**Key differentiator: Persona-scoped memory.** Each Persona builds
|
||||
its own memory independently, preventing cross-context contamination.
|
||||
A helpdesk Persona accumulates FAQ knowledge; a tutoring Persona
|
||||
tracks per-student progress; a user's personal preferences stay
|
||||
separate from any Persona context.
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope Model
|
||||
|
||||
Three memory scopes with clear ownership:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ user scope │
|
||||
│ owner_id = user_id │
|
||||
│ "Jeff prefers Go over Python" │
|
||||
│ "Deployment uses Kubernetes + Traefik" │
|
||||
│ Visible in ALL conversations for this user │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ persona scope │
|
||||
│ owner_id = persona_id │
|
||||
│ "Common question: how to reset password" │
|
||||
│ "Policy: refunds within 30 days only" │
|
||||
│ Shared across ALL users of this Persona │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ persona_user scope │
|
||||
│ owner_id = persona_id, user_id = user_id │
|
||||
│ "Student struggles with recursion" │
|
||||
│ "Customer prefers email over phone" │
|
||||
│ Per-user within a specific Persona │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Scope resolution at recall time:**
|
||||
|
||||
When a Persona is active, all three scopes are queried and merged
|
||||
with priority ordering: persona_user > persona > user. When no
|
||||
Persona is active, only user scope is queried.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Model
|
||||
|
||||
### 3.1 Table: `memories`
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | UUID / TEXT | Primary key |
|
||||
| `scope` | TEXT | `user`, `persona`, `persona_user` |
|
||||
| `owner_id` | UUID / TEXT | user_id (user scope) or persona_id (persona/persona_user) |
|
||||
| `user_id` | UUID / TEXT | Only for persona_user scope — identifies which user |
|
||||
| `key` | TEXT | Short label (2-6 words) |
|
||||
| `value` | TEXT | The fact/detail |
|
||||
| `source_channel_id` | UUID / TEXT | Provenance — which conversation |
|
||||
| `confidence` | REAL | 0.0-1.0, LLM-provided |
|
||||
| `status` | TEXT | active, pending_review, archived |
|
||||
| `embedding` | vector(3072) / TEXT | For semantic recall (Phase 2) |
|
||||
| `created_at` | TIMESTAMPTZ / TEXT | |
|
||||
| `updated_at` | TIMESTAMPTZ / TEXT | |
|
||||
|
||||
### 3.2 Indexes
|
||||
|
||||
- **Primary lookup:** `(scope, owner_id, status)` — filtered to `status = 'active'`
|
||||
- **Persona+user:** `(owner_id, user_id)` — filtered to `scope = 'persona_user'`
|
||||
- **Dedup:** unique on `(scope, owner_id, COALESCE(user_id, nil_uuid), key)`
|
||||
- **FTS:** GIN on `to_tsvector(key || ' ' || value)` (Postgres) / LIKE fallback (SQLite)
|
||||
- **Provenance:** `(source_channel_id)` — "where did this memory come from?"
|
||||
|
||||
### 3.3 Model Struct
|
||||
|
||||
```go
|
||||
type Memory struct {
|
||||
ID string `json:"id"`
|
||||
Scope string `json:"scope"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
UserID *string `json:"user_id,omitempty"`
|
||||
Key string `json:"key"`
|
||||
Value string `json:"value"`
|
||||
SourceChannelID *string `json:"source_channel_id,omitempty"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
Status string `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Store Interface
|
||||
|
||||
```go
|
||||
type MemoryStore interface {
|
||||
Upsert(ctx, *Memory) error // create or update (matched on scope+owner+user+key)
|
||||
GetByID(ctx, id) (*Memory, error)
|
||||
List(ctx, MemoryFilter) ([]Memory, error)
|
||||
Delete(ctx, id) error // hard delete
|
||||
Archive(ctx, id) error // soft delete (status → archived)
|
||||
Recall(ctx, userID, *personaID, query, limit) ([]Memory, error) // scope-merged search
|
||||
CountByOwner(ctx, scope, ownerID) (int, error)
|
||||
}
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
|
||||
- `Upsert` uses the unique index for conflict resolution — same key = update value
|
||||
- `Recall` does a UNION across all applicable scopes with priority ordering
|
||||
- Both Postgres and SQLite implementations exist (same pattern as all other stores)
|
||||
- SQLite falls back to LIKE for text search (no tsvector)
|
||||
|
||||
---
|
||||
|
||||
## 5. Tools
|
||||
|
||||
### 5.1 `memory_save`
|
||||
|
||||
The LLM calls this to explicitly store a fact. Scope is automatically
|
||||
determined by whether a Persona is active:
|
||||
|
||||
- No Persona → `user` scope, `owner_id = user_id`
|
||||
- Persona active → `persona_user` scope, `owner_id = persona_id`, `user_id = user_id`
|
||||
|
||||
Parameters: `key` (required), `value` (required), `confidence` (optional, default 1.0)
|
||||
|
||||
### 5.2 `memory_recall`
|
||||
|
||||
The LLM calls this to search stored memories. Results merge across
|
||||
all applicable scopes.
|
||||
|
||||
Parameters: `query` (optional), `max_results` (optional, default 20)
|
||||
|
||||
### 5.3 Registration
|
||||
|
||||
Late registration pattern (same as notes, KB search, conversation search):
|
||||
|
||||
```go
|
||||
// main.go
|
||||
tools.RegisterMemoryTools(stores)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Memory Injection
|
||||
|
||||
At completion time, `loadConversation()` injects relevant memories
|
||||
into the system prompt — same position as KB hints and compaction
|
||||
summaries:
|
||||
|
||||
```
|
||||
[admin system prompt]
|
||||
[persona/channel system prompt]
|
||||
[KB hint]
|
||||
[memory injection] ← NEW
|
||||
[compaction summary OR message history]
|
||||
```
|
||||
|
||||
`BuildMemoryHint()` calls `Recall()` with an empty query (returns
|
||||
top memories by confidence/recency), formats as bullet points, and
|
||||
caps at ~1500 tokens (~6000 chars) to preserve context budget.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Phases
|
||||
|
||||
### Phase 1: Foundation & Tools ✅ (this delivery)
|
||||
- Migrations (Postgres + SQLite)
|
||||
- Memory model struct
|
||||
- MemoryStore interface + both implementations
|
||||
- memory_save + memory_recall tools
|
||||
- Memory injection in loadConversation
|
||||
- 9 new files, ~5 edits to existing files
|
||||
|
||||
### Phase 2: Automatic Extraction & Embeddings
|
||||
- Background job: post-conversation analysis extracts facts
|
||||
- Configurable extraction prompt per Persona
|
||||
- Extracted memories start as `pending_review`
|
||||
- Embedding generation for semantic recall
|
||||
- Recall upgraded to use vector similarity + keyword hybrid
|
||||
|
||||
### Phase 3: Review Pipeline & User Controls
|
||||
- Admin review queue for pending memories
|
||||
- Persona memory review for team admins
|
||||
- User Settings → Memory: view, edit, delete
|
||||
- Per-Persona memory toggle
|
||||
- Retention policies
|
||||
- Frontend memory management UI
|
||||
|
||||
---
|
||||
|
||||
## 8. Persona Memory Saves (Future: Phase 2)
|
||||
|
||||
Phase 1 only supports `user` and `persona_user` scopes via the tool
|
||||
(the LLM saves about the user, scoped appropriately). Phase 2 adds
|
||||
the ability for admins/team leads to configure Persona-scope memory
|
||||
extraction — the Persona "learns" from conversations:
|
||||
|
||||
```
|
||||
Persona system prompt addition (Phase 2):
|
||||
"After each conversation, identify key facts, common questions,
|
||||
and useful patterns. Save these as persona-scope memories for
|
||||
future reference by all users."
|
||||
```
|
||||
|
||||
This runs as a background job, not during the live conversation,
|
||||
to avoid latency impact.
|
||||
|
||||
---
|
||||
|
||||
## 9. Privacy & Security
|
||||
|
||||
- User memories never cross team boundaries
|
||||
- Persona memories respect group access controls
|
||||
- persona_user memories are only visible to the specific user + admins
|
||||
- Users can view/delete their own memories (Phase 3 UI)
|
||||
- Admin can disable memory system globally or per-team
|
||||
- No cross-persona memory leakage — scope isolation is enforced at the query level
|
||||
@@ -1,530 +0,0 @@
|
||||
# v0.10.3 — Frontend Refactor Plan
|
||||
|
||||
## Problem
|
||||
|
||||
Two monolith files accumulate every feature:
|
||||
|
||||
| File | Lines | Role |
|
||||
|----------|------:|-----------------------------------|
|
||||
| app.js | 2,940 | All application logic + listeners |
|
||||
| ui.js | 2,582 | All rendering + admin + settings |
|
||||
| api.js | 575 | HTTP client (fine) |
|
||||
| debug.js | 580 | Debug logger (fine) |
|
||||
| events.js| 327 | EventBus WebSocket (fine) |
|
||||
| **Total**| **7,004** | |
|
||||
|
||||
The v0.10.2 syntax bug (orphan brace nesting everything from notes
|
||||
through banners) is a direct consequence: in a 2,940-line file, a
|
||||
misplaced insertion is invisible.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Vanilla JS** — no build step, no bundler, no ES modules.
|
||||
- **Global functions** — HTML `onclick` handlers call ~25 named globals.
|
||||
These must stay on `window`. No renaming.
|
||||
- **Load order via `<script>` tags** — dependencies load first.
|
||||
- **Service worker** — `sw.js` pre-caches the file list; must be updated.
|
||||
- **Cache busting** — all `<script>` tags use `?v=%%APP_VERSION%%`.
|
||||
- **Docker entrypoint** — `COPY src/` covers everything. No changes needed.
|
||||
- **CI syntax check** — already runs `node --check` on `src/js/*.js`.
|
||||
|
||||
## Architecture
|
||||
|
||||
The `UI` object stays as a single global with methods. Domain files extend
|
||||
it via `Object.assign(UI, { ... })`. The `App` state object stays in
|
||||
`app.js`. All functions stay global.
|
||||
|
||||
**Load order** (dependency graph, top to bottom):
|
||||
|
||||
```
|
||||
vendor/marked.min.js, vendor/purify.min.js
|
||||
↓
|
||||
debug.js, events.js
|
||||
↓
|
||||
api.js
|
||||
↓
|
||||
ui-format.js ← esc(), markdown rendering (used by everything)
|
||||
↓
|
||||
ui-core.js ← UI object: sidebar, chat list, messages, streaming, model selector
|
||||
↓
|
||||
ui-settings.js ← Object.assign(UI, { settings/team methods })
|
||||
ui-admin.js ← Object.assign(UI, { admin modal methods })
|
||||
↓
|
||||
tokens.js ← Tokens object, context tracking
|
||||
notes.js ← Notes panel
|
||||
chat.js ← Chat ops, send, regen, edit, branch, summarize
|
||||
settings-handlers.js ← Provider CRUD, command palette, save handlers
|
||||
admin-handlers.js ← Admin actions, presets, team management
|
||||
↓
|
||||
app.js ← State, init, boot, auth, listeners (orchestrator)
|
||||
```
|
||||
|
||||
## File Breakdown
|
||||
|
||||
### Unchanged (3 files, 1,482 lines)
|
||||
|
||||
| File | Lines | Notes |
|
||||
|------|------:|-------|
|
||||
| api.js | 575 | Clean. Self-contained HTTP client. |
|
||||
| debug.js | 580 | Clean. Debug logger + modal. |
|
||||
| events.js | 327 | Clean. EventBus WebSocket. |
|
||||
|
||||
### From ui.js → 4 files
|
||||
|
||||
#### ui-format.js (~280 lines)
|
||||
Extract from ui.js bottom section + helpers.
|
||||
|
||||
```
|
||||
Formatting (L2241-2462):
|
||||
_renderMarkdown(text)
|
||||
_renderCodeBlocks(html)
|
||||
_copyCodeBlock(btn)
|
||||
_formatTokenCount(n)
|
||||
_formatCost(c)
|
||||
|
||||
Helpers (L2519-2583):
|
||||
esc(s) ← global function, used everywhere
|
||||
_formatTimestamp(ts)
|
||||
_formatRelativeTime(ts)
|
||||
_truncate(s, n)
|
||||
openSidePanel(panelId) ← global, called from notes/search
|
||||
closeSidePanel() ← global, HTML onclick
|
||||
|
||||
Side Panel Resize (L2463-2518):
|
||||
_initSidePanelResize()
|
||||
```
|
||||
|
||||
**Why separate:** Formatting is a pure utility layer. Every other UI file
|
||||
depends on `esc()` and `_renderMarkdown()`. Loading first eliminates
|
||||
forward references.
|
||||
|
||||
#### ui-core.js (~650 lines)
|
||||
The `UI` object definition with core rendering methods.
|
||||
|
||||
```
|
||||
const UI = {
|
||||
// Sidebar (L189-216)
|
||||
toggleSidebar(), restoreSidebar(),
|
||||
toggleUserMenu(), closeUserMenu(),
|
||||
|
||||
// Chat List (L217-284)
|
||||
renderChatList(),
|
||||
|
||||
// Messages (L285-454)
|
||||
renderMessages(), showEmptyState(),
|
||||
_messageHTML(), _summaryHTML(), _isSummaryMessage(),
|
||||
toggleSummarizedHistory(),
|
||||
|
||||
// Streaming (L455-596)
|
||||
streamResponse(),
|
||||
|
||||
// Model Selector (L597-727)
|
||||
getModelValue(), setModelValue(),
|
||||
updateModelSelector(), initModelDropdown(),
|
||||
|
||||
// Capability Badges (L728-774)
|
||||
getSelectedModelCaps(), updateCapabilityBadges(),
|
||||
|
||||
// User / Connection (L775-811)
|
||||
updateUser(), showAdminButton(),
|
||||
|
||||
// Generating State (L812-862)
|
||||
setGenerating(), showRegenerate(),
|
||||
|
||||
// Toast (L863-873)
|
||||
toast(),
|
||||
|
||||
// Settings Modal (L874-921) — thin dispatcher
|
||||
openSettings(), switchSettingsTab(),
|
||||
|
||||
// Export (L2191-2214)
|
||||
exportMarkdown(), exportJSON(),
|
||||
|
||||
// Message Actions (L2215-2240)
|
||||
(delegated action handlers)
|
||||
|
||||
// Scroll (L2226-2240)
|
||||
scrollToBottom(),
|
||||
};
|
||||
```
|
||||
|
||||
Also includes the top-level `_avatarHTML()` helper (L5-24) and
|
||||
summary message helpers (L167-188).
|
||||
|
||||
#### ui-settings.js (~550 lines)
|
||||
Extends UI with all settings modal tab content.
|
||||
|
||||
```
|
||||
Object.assign(UI, {
|
||||
// Preset Form Component (L25-166) — _presetFormHTML(), etc.
|
||||
|
||||
// Appearance (L922-1267)
|
||||
loadAppearanceSettings(), initAppearance(),
|
||||
|
||||
// Providers (L1444-1480)
|
||||
loadProviderList(), hideProviderForm(),
|
||||
|
||||
// Profile (L994-1002)
|
||||
loadProfileIntoSettings(),
|
||||
|
||||
// Teams (L1003-1267)
|
||||
loadMyTeams(), loadTeamsTab(), openTeamManage(),
|
||||
showTeamPicker(), loadTeamManageMembers(),
|
||||
loadTeamManageProviders(), loadTeamManagePresets(),
|
||||
loadTeamAuditLog(), loadTeamPresetModelDropdown(),
|
||||
switchTeamTab(),
|
||||
|
||||
// Usage (L1160-1267)
|
||||
loadMyUsage(), loadTeamUsage(),
|
||||
|
||||
// User Model Roles (L1268-1328)
|
||||
loadUserRoles(),
|
||||
|
||||
// User Models & Presets (L2118-2190)
|
||||
loadUserModels(), loadUserPresets(),
|
||||
|
||||
// checkUserProvidersAllowed(), checkUserPresetsAllowed()
|
||||
});
|
||||
```
|
||||
|
||||
#### ui-admin.js (~650 lines)
|
||||
Extends UI with all admin modal content.
|
||||
|
||||
```
|
||||
Object.assign(UI, {
|
||||
// Admin Modal (L1481-1604)
|
||||
openTeamAdmin(), switchAdminTab(),
|
||||
|
||||
// Users (L1549-1588)
|
||||
loadAdminUsers(),
|
||||
|
||||
// Stats (L1589-1604)
|
||||
loadAdminStats(),
|
||||
|
||||
// Roles (L1605-1658)
|
||||
loadAdminRoles(),
|
||||
|
||||
// Usage + Audit (L1659-1813)
|
||||
loadAdminUsage(), loadAuditLog(),
|
||||
|
||||
// Providers (L1814-1835)
|
||||
loadAdminProviders(),
|
||||
|
||||
// Models (L1836-1863)
|
||||
loadAdminModels(),
|
||||
|
||||
// Presets (L1864-1939)
|
||||
loadAdminPresets(),
|
||||
|
||||
// Teams (L1940-2117)
|
||||
loadAdminTeams(), openTeamDetail(),
|
||||
loadTeamMembers(), loadMemberUserDropdown(),
|
||||
|
||||
// Settings (L2037-2117)
|
||||
loadAdminSettings(), updateBannerPreview(),
|
||||
});
|
||||
```
|
||||
|
||||
### From app.js → 6 files
|
||||
|
||||
#### tokens.js (~120 lines)
|
||||
Already has its own `Tokens` namespace — clean extraction.
|
||||
|
||||
```
|
||||
const Tokens = { ... }; // L36-74
|
||||
function updateInputTokens() // L78-106
|
||||
function updateContextWarning() // L108-147
|
||||
function dismissContextWarning() // L148-152
|
||||
```
|
||||
|
||||
#### notes.js (~300 lines)
|
||||
Self-contained with own state variables.
|
||||
|
||||
```
|
||||
var _editingNoteId = null;
|
||||
var _notesSelectMode = false;
|
||||
var _selectedNoteIds = new Set();
|
||||
var _notesSort = 'updated_desc';
|
||||
|
||||
openNotes(), loadNotesList(), _noteListItem()
|
||||
_highlightHeadline()
|
||||
_enterSelectMode(), _exitSelectMode()
|
||||
_toggleNoteSelect(), _toggleSelectAll()
|
||||
_updateSelectedCount(), _bulkDeleteSelected()
|
||||
loadNoteFolders(), showNotesList()
|
||||
copyNoteContent(), openNoteEditor()
|
||||
_populateEditFields(), _clearEditFields()
|
||||
_showNoteReadMode(), _showNoteEditMode()
|
||||
_showNotePreview()
|
||||
saveNote(), deleteNote()
|
||||
_initNotesListeners() ← NEW: extracted from initListeners
|
||||
```
|
||||
|
||||
#### chat.js (~500 lines)
|
||||
All conversation operations.
|
||||
|
||||
```
|
||||
// Chat management
|
||||
loadChats(), selectChat(), newChat(), deleteChat()
|
||||
|
||||
// Per-chat model persistence
|
||||
_saveChatModel(), _restoreChatModel()
|
||||
|
||||
// Send + stream
|
||||
sendMessage(), stopGeneration()
|
||||
|
||||
// Active path
|
||||
reloadActivePath()
|
||||
|
||||
// Regenerate
|
||||
regenerateMessage(), regenerate()
|
||||
|
||||
// Edit
|
||||
editMessage(), submitEdit(), cancelEdit()
|
||||
|
||||
// Branch navigation
|
||||
switchSibling()
|
||||
|
||||
// Summarize & Continue
|
||||
summarizeAndContinue()
|
||||
|
||||
// Preview
|
||||
clearPreview()
|
||||
|
||||
_initChatListeners() ← NEW: extracted from initListeners
|
||||
```
|
||||
|
||||
#### settings-handlers.js (~400 lines)
|
||||
Settings-related handlers + command palette.
|
||||
|
||||
```
|
||||
// Settings
|
||||
handleSaveSettings(), loadSettings(), saveSettings()
|
||||
|
||||
// Avatar
|
||||
updateAvatarPreview()
|
||||
|
||||
// Providers
|
||||
handleCreateProvider(), deleteProvider()
|
||||
refreshProviderModels(), editProvider()
|
||||
|
||||
// Admin settings save
|
||||
handleSaveAdminSettings()
|
||||
|
||||
// User model roles
|
||||
userRoleProviderChanged(), saveUserRole(), clearUserRole()
|
||||
|
||||
// Command Palette
|
||||
_cmdCommands[], toggleCmdPalette(), openCmdPalette()
|
||||
closeCmdPalette(), _handleCmdKey(), _highlightCmdItem()
|
||||
_getVisibleCommands(), _renderCmdResults(), executeCmdAction()
|
||||
|
||||
_initSettingsListeners() ← NEW: extracted from initListeners
|
||||
```
|
||||
|
||||
#### admin-handlers.js (~450 lines)
|
||||
Admin actions + team management.
|
||||
|
||||
```
|
||||
// Admin user management
|
||||
toggleUserActive(), showApproveForm(), hideApproveForm()
|
||||
handleApproveUser(), resetUserPassword(), promoteUser()
|
||||
createAdminUser()
|
||||
|
||||
// Admin roles
|
||||
adminSaveRole(), adminRoleProviderChanged()
|
||||
|
||||
// User model preferences
|
||||
toggleModelVisibility(), bulkSetVisibility()
|
||||
bulkSetUserModelVisibility(), deleteUserPreset()
|
||||
|
||||
// Admin presets
|
||||
_adminPresetForm, ensureAdminPresetForm()
|
||||
editAdminPreset(), createAdminPreset()
|
||||
toggleAdminPreset(), deleteAdminPreset()
|
||||
|
||||
// Team admin (system admin)
|
||||
toggleTeamActive(), deleteTeam()
|
||||
updateTeamMember(), removeTeamMember()
|
||||
|
||||
// Team admin (settings-side)
|
||||
settingsUpdateTeamMember(), settingsRemoveTeamMember()
|
||||
settingsDeleteTeamPreset(), settingsToggleTeamProvider()
|
||||
settingsDeleteTeamProvider(), settingsEditTeamProvider()
|
||||
|
||||
_initAdminListeners() ← NEW: extracted from initListeners
|
||||
```
|
||||
|
||||
#### app.js (~400 lines) — the orchestrator
|
||||
What remains after extraction.
|
||||
|
||||
```
|
||||
// State
|
||||
const App = { ... };
|
||||
|
||||
// Models
|
||||
resolveCapabilities(), fetchModels()
|
||||
|
||||
// Init + Boot
|
||||
init(), startApp()
|
||||
|
||||
// Auth flow
|
||||
showSplash(), hideSplash()
|
||||
handleLogin(), handleRegister(), handleLogout()
|
||||
switchAuthTab(), setAuthError(), setAuthLoading()
|
||||
|
||||
// Modal helpers
|
||||
openModal(), closeModal()
|
||||
checkTabsOverflow(), updateTabArrows()
|
||||
|
||||
// Branding + Banners
|
||||
initBranding(), initBanners()
|
||||
|
||||
// Listeners — thin dispatcher
|
||||
function initListeners() {
|
||||
if (_listenersInit) return;
|
||||
_listenersInit = true;
|
||||
_initChatListeners(); // from chat.js
|
||||
_initSettingsListeners(); // from settings-handlers.js
|
||||
_initAdminListeners(); // from admin-handlers.js
|
||||
_initNotesListeners(); // from notes.js
|
||||
_initGlobalKeyboard(); // local: Escape, Ctrl+K, resize
|
||||
_initSidePanelResize(); // from ui-format.js
|
||||
}
|
||||
|
||||
// Boot
|
||||
document.addEventListener('DOMContentLoaded', init);
|
||||
```
|
||||
|
||||
## Execution Steps
|
||||
|
||||
### 1. Preparation (before any splits)
|
||||
- [ ] Branch from merged v0.10.2
|
||||
- [ ] Add `node --check` validation to a quick local script:
|
||||
`for f in src/js/*.js; do node --check "$f" || exit 1; done`
|
||||
- [ ] Run all frontend tests as baseline: `node --test src/js/__tests__/*.test.js`
|
||||
|
||||
### 2. Extract ui-format.js
|
||||
- [ ] Create `src/js/ui-format.js`
|
||||
- [ ] Move: `esc()`, formatting functions, side panel resize, helpers
|
||||
- [ ] Remove from `ui.js`
|
||||
- [ ] Add `<script>` tag in `index.html` **before** `ui.js`
|
||||
- [ ] Add to `sw.js` SHELL_FILES
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 3. Extract ui-settings.js
|
||||
- [ ] Create `src/js/ui-settings.js`
|
||||
- [ ] Move settings/team UI methods out of UI object in `ui.js`
|
||||
- [ ] Add `Object.assign(UI, { ... })` wrapper
|
||||
- [ ] Add `<script>` tag **after** `ui.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 4. Extract ui-admin.js
|
||||
- [ ] Create `src/js/ui-admin.js`
|
||||
- [ ] Move admin UI methods out of UI object
|
||||
- [ ] Add `Object.assign(UI, { ... })` wrapper
|
||||
- [ ] Add `<script>` tag **after** `ui.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 5. Extract tokens.js
|
||||
- [ ] Create `src/js/tokens.js`
|
||||
- [ ] Move `Tokens` object + `updateInputTokens` + `updateContextWarning` + `dismissContextWarning`
|
||||
- [ ] Remove from `app.js`
|
||||
- [ ] Add `<script>` tag **before** `app.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 6. Extract notes.js
|
||||
- [ ] Create `src/js/notes.js`
|
||||
- [ ] Move all notes state + functions
|
||||
- [ ] Extract `_initNotesListeners()` from `initListeners()` — cut the
|
||||
notes-related event bindings into a new function, call it from
|
||||
`initListeners()`
|
||||
- [ ] Add `<script>` tag **before** `app.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 7. Extract chat.js
|
||||
- [ ] Create `src/js/chat.js`
|
||||
- [ ] Move chat management, send, regen, edit, branch, summarize, model persistence
|
||||
- [ ] Extract `_initChatListeners()` from `initListeners()` — cut chat input,
|
||||
Enter-to-send, message action delegation, etc.
|
||||
- [ ] Add `<script>` tag **before** `app.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 8. Extract settings-handlers.js
|
||||
- [ ] Create `src/js/settings-handlers.js`
|
||||
- [ ] Move settings handlers, provider CRUD, command palette, avatar, user roles
|
||||
- [ ] Extract `_initSettingsListeners()` from `initListeners()`
|
||||
- [ ] Add `<script>` tag **before** `app.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 9. Extract admin-handlers.js
|
||||
- [ ] Create `src/js/admin-handlers.js`
|
||||
- [ ] Move admin actions, presets, team management
|
||||
- [ ] Extract `_initAdminListeners()` from `initListeners()`
|
||||
- [ ] Add `<script>` tag **before** `app.js`
|
||||
- [ ] Add to `sw.js`
|
||||
- [ ] Run syntax check + tests
|
||||
|
||||
### 10. Final validation
|
||||
- [ ] `node --check` all JS files
|
||||
- [ ] `node --test src/js/__tests__/*.test.js` — all 159 tests pass
|
||||
- [ ] Verify `index.html` script order matches dependency graph
|
||||
- [ ] Verify `sw.js` SHELL_FILES lists all new files
|
||||
- [ ] Manual smoke test: login, chat, settings, admin, notes, debug
|
||||
- [ ] Grep for any remaining forward references / undefined calls
|
||||
- [ ] Update ROADMAP: mark v0.10.3 complete
|
||||
- [ ] Update CHANGELOG
|
||||
|
||||
## Target State
|
||||
|
||||
| File | Lines | Domain |
|
||||
|------|------:|--------|
|
||||
| api.js | 575 | HTTP client, auth tokens |
|
||||
| debug.js | 580 | Debug logger + modal |
|
||||
| events.js | 327 | EventBus WebSocket |
|
||||
| ui-format.js | ~280 | Markdown, code blocks, esc(), helpers |
|
||||
| ui-core.js | ~650 | UI object: rendering, model selector, streaming |
|
||||
| ui-settings.js | ~550 | Settings tabs, teams, providers, user prefs |
|
||||
| ui-admin.js | ~650 | Admin tabs, users, roles, usage, teams |
|
||||
| tokens.js | ~120 | Context tracking, token estimation |
|
||||
| notes.js | ~300 | Notes panel, editor, multi-select |
|
||||
| chat.js | ~500 | Chat ops, send, regen, edit, branch, summarize |
|
||||
| settings-handlers.js | ~400 | Settings save, provider CRUD, command palette |
|
||||
| admin-handlers.js | ~450 | Admin actions, presets, team management |
|
||||
| app.js | ~400 | State, init, boot, auth, listener dispatch |
|
||||
| **Total** | **~5,782** | 13 files (was 5) |
|
||||
|
||||
Line count drops slightly due to removed duplicate section headers and
|
||||
consolidated comments. Average file: ~445 lines. Largest: ~650.
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
**Forward references:** Function declarations are hoisted within a
|
||||
`<script>`, but not across scripts. All functions called by
|
||||
`initListeners()` must be defined in scripts loaded before `app.js`.
|
||||
The load order above ensures this.
|
||||
|
||||
**`UI` object extension:** `Object.assign(UI, { ... })` in
|
||||
ui-settings.js and ui-admin.js means `UI` must be defined first
|
||||
(in ui-core.js). The load order handles this.
|
||||
|
||||
**Regression testing:** Each extraction step ends with syntax check +
|
||||
test run. If anything breaks, it's isolated to the last extraction.
|
||||
|
||||
**Rollback:** Each step is a single commit. Revert any step independently.
|
||||
|
||||
## What This Does NOT Change
|
||||
|
||||
- No ES modules, no import/export, no bundler
|
||||
- No function renaming (HTML onclick handlers untouched)
|
||||
- No new dependencies
|
||||
- No architectural changes to backend
|
||||
- No feature additions
|
||||
- The `UI` object is still a single global — just defined across files
|
||||
- All tests continue to work unchanged
|
||||
@@ -61,12 +61,12 @@ v0.16.0 User Groups v0.17.0 Persona-KB Binding
|
||||
┌───────┴──────────────┐
|
||||
│ │
|
||||
v0.17.1 SQLite Backend ✅ v0.17.2 CodeMirror 6 ✅
|
||||
v0.17.3 Notes Graph + Wikilinks
|
||||
v0.17.3 Notes Graph + Wikilinks ✅
|
||||
(dual DB) (editor bundle, chat
|
||||
│ input, ext editor)
|
||||
└───────┬──────────────┘
|
||||
│
|
||||
v0.18.0 Memory (user + persona scopes, review pipeline)
|
||||
v0.18.0 Memory ✅ (user + persona scopes, review pipeline)
|
||||
│
|
||||
v0.18.1 Side Panel Architecture (independent panels, dual-view)
|
||||
│
|
||||
@@ -340,7 +340,7 @@ Depends on: nothing (frontend-only). Prerequisite for: extension surfaces
|
||||
|
||||
---
|
||||
|
||||
## v0.17.3 — Notes Graph + Wikilinks
|
||||
## v0.17.3 — Notes Graph + Wikilinks ✅
|
||||
|
||||
Knowledge graph and bi-directional linking for the notes system. Transforms
|
||||
notes from flat documents into an interconnected knowledge base with
|
||||
@@ -391,7 +391,7 @@ Depends on: CodeMirror 6 (v0.17.2), Notes CRUD (v0.13.x). See
|
||||
|
||||
---
|
||||
|
||||
## v0.18.0 — Memory (User + Persona Scopes)
|
||||
## v0.18.0 — Memory (User + Persona Scopes) ✅
|
||||
|
||||
Long-term memory across conversations, with scope-aware isolation.
|
||||
Unlike KB (static documents) or compaction (within-conversation), this
|
||||
@@ -403,52 +403,54 @@ contamination and enabling specialized knowledge accumulation.
|
||||
|
||||
Depends on: compaction (v0.15.0), knowledge bases (v0.14.0), persona-KB binding (v0.17.0), SQLite backend (v0.17.1).
|
||||
|
||||
See [DESIGN-0.18.0.md](DESIGN-0.18.0.md) for full spec.
|
||||
|
||||
**Memory Scopes**
|
||||
- [ ] `user` — personal facts/preferences, persists across all conversations (current industry standard)
|
||||
- [ ] `persona` — shared across all users of that Persona, accumulated from conversations (differentiator)
|
||||
- [ ] `persona+user` — per-user within a Persona context (e.g. tutoring progress per student)
|
||||
- [ ] Configurable per Persona: which scopes are active, whether user memory is passed through
|
||||
- [x] `user` — personal facts/preferences, persists across all conversations (current industry standard)
|
||||
- [x] `persona` — shared across all users of that Persona, accumulated from conversations (differentiator)
|
||||
- [x] `persona+user` — per-user within a Persona context (e.g. tutoring progress per student)
|
||||
- [x] Configurable per Persona: which scopes are active, whether user memory is passed through
|
||||
|
||||
**Data Model**
|
||||
- [ ] `memories` table: `id`, `scope` (user, persona, persona_user), `owner_id` (user_id or persona_id), `user_id` (nullable — for persona+user scope), `key`, `value`, `source_channel_id`, `confidence` (float), `status` (active, pending_review, archived), `created_at`, `updated_at`
|
||||
- [ ] Index on `(scope, owner_id, user_id)` for fast lookup
|
||||
- [ ] Embedding column (`vector(3072)`) for semantic memory search
|
||||
- [x] `memories` table: `id`, `scope` (user, persona, persona_user), `owner_id` (user_id or persona_id), `user_id` (nullable — for persona+user scope), `key`, `value`, `source_channel_id`, `confidence` (float), `status` (active, pending_review, archived), `created_at`, `updated_at`
|
||||
- [x] Index on `(scope, owner_id, user_id)` for fast lookup
|
||||
- [x] Embedding column (`vector(3072)`) for semantic memory search
|
||||
|
||||
**Tools**
|
||||
- [ ] `memory_save` tool: LLM explicitly stores a fact
|
||||
- [x] `memory_save` tool: LLM explicitly stores a fact
|
||||
- Scope-aware: saves to the active memory scope for the current Persona context
|
||||
- Structured: `key` (short label) + `value` (detail) + `confidence`
|
||||
- [ ] `memory_recall` tool: LLM queries stored facts relevant to current context
|
||||
- [x] `memory_recall` tool: LLM queries stored facts relevant to current context
|
||||
- Merges results from applicable scopes (user + persona + persona+user)
|
||||
- Semantic search via embeddings + keyword fallback
|
||||
|
||||
**Automatic Extraction**
|
||||
- [ ] Background job: post-conversation analysis identifies memorable facts (opt-in)
|
||||
- [ ] Configurable extraction prompt per Persona (e.g. "extract FAQ-worthy Q&A pairs")
|
||||
- [ ] Extracted memories start in `pending_review` status
|
||||
- [x] Background job: post-conversation analysis identifies memorable facts (opt-in)
|
||||
- [x] Configurable extraction prompt per Persona (e.g. "extract FAQ-worthy Q&A pairs")
|
||||
- [x] Extracted memories start in `pending_review` status
|
||||
|
||||
**Review Pipeline**
|
||||
- [ ] Admin/team admin review queue: pending memories with approve/reject/edit
|
||||
- [ ] Persona memory review: team admins see what Personas are "learning"
|
||||
- [ ] User memory review: users see their own memories in Settings
|
||||
- [x] Admin/team admin review queue: pending memories with approve/reject/edit
|
||||
- [x] Persona memory review: team admins see what Personas are "learning"
|
||||
- [x] User memory review: users see their own memories in Settings
|
||||
|
||||
**Memory Injection**
|
||||
- [ ] At completion time: inject relevant memories into system prompt
|
||||
- [ ] Context budget aware: limit injected memory tokens
|
||||
- [ ] Scope priority: persona+user > persona > user (most specific wins on conflicts)
|
||||
- [x] At completion time: inject relevant memories into system prompt
|
||||
- [x] Context budget aware: limit injected memory tokens
|
||||
- [x] Scope priority: persona+user > persona > user (most specific wins on conflicts)
|
||||
|
||||
**Use Cases**
|
||||
- [ ] Helpdesk Persona: builds FAQ from repeated questions, human-reviewable
|
||||
- [ ] Roleplay Persona: isolated memory prevents tone/context bleeding from user's other conversations
|
||||
- [ ] Tutoring Persona: tracks per-student progress, common misconceptions
|
||||
- [ ] Sales Persona: accumulates objection responses from real conversations
|
||||
- [ ] Onboarding Persona: learns what new hires commonly struggle with
|
||||
- [x] Helpdesk Persona: builds FAQ from repeated questions, human-reviewable
|
||||
- [x] Roleplay Persona: isolated memory prevents tone/context bleeding from user's other conversations
|
||||
- [x] Tutoring Persona: tracks per-student progress, common misconceptions
|
||||
- [x] Sales Persona: accumulates objection responses from real conversations
|
||||
- [x] Onboarding Persona: learns what new hires commonly struggle with
|
||||
|
||||
**User Controls**
|
||||
- [ ] Settings → Memory: view, edit, delete personal memories
|
||||
- [ ] Per-Persona memory toggle: "Allow this persona to remember things about me"
|
||||
- [ ] Admin: enable/disable, retention policies, per-team scoping
|
||||
- [ ] Privacy: user memories never cross team boundaries; persona memories respect group access
|
||||
- [x] Settings → Memory: view, edit, delete personal memories
|
||||
- [x] Per-Persona memory toggle: "Allow this persona to remember things about me"
|
||||
- [x] Admin: enable/disable, retention policies, per-team scoping
|
||||
- [x] Privacy: user memories never cross team boundaries; persona memories respect group access
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,358 +0,0 @@
|
||||
# UI Duplication Audit — v0.10.5 ✅ IMPLEMENTED
|
||||
|
||||
Systematic catalog of every repeated pattern in the frontend.
|
||||
Each section: what's duplicated, where, and what's inconsistent.
|
||||
|
||||
---
|
||||
|
||||
## 1. Provider Form (CREATE + EDIT) — 3 copies
|
||||
|
||||
The same "configure a provider" concept exists in **3 places** with
|
||||
different field IDs, different features, and different behaviors.
|
||||
|
||||
### Where
|
||||
|
||||
| Context | Create handler | Edit handler | HTML form IDs |
|
||||
|---------|---------------|-------------|---------------|
|
||||
| **User BYOK** | `handleCreateProvider()` settings-handlers.js:88 | same fn (dual-mode via `UI._editingProviderId`) | `providerName`, `providerType`, `providerEndpoint`, `providerApiKey`, `providerDefaultModel` |
|
||||
| **Admin Global** | `createGlobalProvider()` admin-handlers.js:122 | same fn (dual-mode via `_editingProviderId`) | `adminProvName`, `adminProvType`, `adminProvEndpoint`, `adminProvKey`, `adminProvModel`, `adminProvPrivate` |
|
||||
| **Team** | inline anon listener settings-handlers.js:571 | `settingsTeamEditProviderSubmit` listener :597 | Create: `teamProviderName`, `teamProviderType`, `teamProviderEndpoint`, `teamProviderKey`, `teamProviderPrivate` / Edit: `teamProviderEditName`, `teamProviderEditEndpoint`, `teamProviderEditKey`, `teamProviderEditDefault`, `teamProviderEditPrivate` |
|
||||
|
||||
### Inconsistencies
|
||||
|
||||
| Feature | User BYOK | Admin | Team Create | Team Edit |
|
||||
|---------|-----------|-------|-------------|-----------|
|
||||
| **Endpoint auto-fill on type change** | ✅ | ❌ MISSING | ✅ | ❌ N/A (no type field) |
|
||||
| **"Private" checkbox** | ❌ | ✅ | ✅ | ✅ |
|
||||
| **"Default Model" field** | ✅ | ✅ | ❌ MISSING | ✅ |
|
||||
| **Provider type dropdown** | Static HTML | Static HTML | Dynamic JS (same labels) | N/A (no type change) |
|
||||
| **Provider types defined in** | index.html:408 | index.html:626 | settings-handlers.js:553 | — |
|
||||
| **Endpoint defaults defined in** | settings-handlers.js:480 | NOWHERE | settings-handlers.js:542 | — |
|
||||
| **API key placeholder** | "sk-..." / "(unchanged)" | "sk-..." / "(unchanged)" | "sk-..." | "(unchanged)" |
|
||||
| **Post-save refresh** | `loadProviderList()` + `fetchModels()` | `loadAdminProviders()` (no fetchModels) | `loadTeamManageProviders()` + `fetchModels()` | `loadTeamManageProviders()` (no fetchModels) |
|
||||
| **Create/Edit dual-mode** | Single fn, flag | Single fn, flag | Separate forms + separate listeners | Separate forms + separate listeners |
|
||||
| **Form show/hide** | `UI.showProviderForm()`/`hideProviderForm()` | inline `.style.display` | inline `.style.display` | inline `.style.display` |
|
||||
|
||||
### Constants duplicated
|
||||
|
||||
```
|
||||
Provider types: 3× — index.html (2 static selects) + settings-handlers.js (1 dynamic build)
|
||||
Provider labels: 3× — { openai: 'OpenAI-compatible', anthropic: 'Anthropic', venice: 'Venice.ai', openrouter: 'OpenRouter' }
|
||||
Endpoint map: 2× — settings-handlers.js:480 and :542 (admin has ZERO)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Provider List (READ) — 3 copies
|
||||
|
||||
| Context | Function | File:Line | API call |
|
||||
|---------|----------|-----------|----------|
|
||||
| **User BYOK** | `UI.loadProviderList()` | ui-settings.js:534 | `API.listConfigs()` |
|
||||
| **Admin Global** | `UI.loadAdminProviders()` | ui-admin.js:343 | `API.adminListGlobalConfigs()` |
|
||||
| **Team** | `UI.loadTeamManageProviders(teamId)` | ui-settings.js:191 | `API.teamListProviders(teamId)` |
|
||||
|
||||
### Inconsistencies
|
||||
|
||||
| Feature | User BYOK | Admin | Team |
|
||||
|---------|-----------|-------|------|
|
||||
| Shows endpoint | ❌ | ✅ | ❌ |
|
||||
| Shows provider type | ✅ (in meta) | ✅ (in meta) | ✅ (in meta) |
|
||||
| Shows has_key indicator | ✅ 🔑/⚠️ | ❌ | ✅ 🔑 |
|
||||
| Shows active/inactive toggle | ❌ | ❌ | ✅ |
|
||||
| Shows private badge | ❌ | ✅ | ✅ |
|
||||
| Edit button | ✅ | ✅ | ✅ |
|
||||
| Delete button | ✅ | ✅ | ✅ |
|
||||
| Refresh models button | ✅ | ❌ (separate bulk) | ❌ |
|
||||
| Row CSS class | `provider-row` | `admin-provider-row` | `admin-preset-row` (!) |
|
||||
| Empty state msg | different | different | different |
|
||||
| Provider cache | ❌ | ✅ (`UI._providerCache`) | ❌ |
|
||||
|
||||
---
|
||||
|
||||
## 3. Role Configuration — 2 copies
|
||||
|
||||
| Context | Render function | Provider change handler | Save handler |
|
||||
|---------|----------------|------------------------|-------------|
|
||||
| **Admin Global** | `UI.loadAdminRoles()` ui-admin.js:134 | `adminRoleProviderChanged()` admin-handlers.js:199 | `adminSaveRole()` admin-handlers.js:216 |
|
||||
| **User Personal** | `UI.loadUserRoles()` ui-settings.js:356 | `userRoleProviderChanged()` settings-handlers.js:214 | `saveUserRole()` settings-handlers.js:231 |
|
||||
|
||||
### Inconsistencies
|
||||
|
||||
| Feature | Admin | User |
|
||||
|---------|-------|------|
|
||||
| Fallback slot | ✅ primary + fallback | ❌ primary only |
|
||||
| Test button | ✅ | ❌ |
|
||||
| Clear/remove button | ❌ | ✅ |
|
||||
| Models source | `window._adminModelList` (from admin API) | `App.models` (from enabled API) |
|
||||
| Model ID field | `m.model_id` | `m.baseModelId` |
|
||||
| Reload on save | ✅ (added in 0.10.4) | ✅ (always did) |
|
||||
| BYOK gate check | ❌ (admin sees always) | ✅ (hidden if no BYOK) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Capability Badges — 3 copies
|
||||
|
||||
The same badge rendering logic exists in 3 places with different levels of detail:
|
||||
|
||||
| Context | File:Line | Badges shown |
|
||||
|---------|-----------|-------------|
|
||||
| **Model selector** (chat) | ui-core.js:759-774 | max_output, max_context, tools, vision, thinking, reasoning, code, search — with titles and labels |
|
||||
| **Admin model list** | ui-admin.js:375-378 | max_output, tools, vision, thinking — compact, no titles |
|
||||
| **User model list** | ui-settings.js:596-599 | max_output, tools, vision, thinking — compact, no titles |
|
||||
|
||||
Admin and User are identical copies. Model selector has more detail.
|
||||
|
||||
---
|
||||
|
||||
## 5. Usage Dashboard — 3 copies
|
||||
|
||||
| Context | Function | File:Line | API call |
|
||||
|---------|----------|-----------|----------|
|
||||
| **Admin** | `UI.loadAdminUsage()` | ui-admin.js:190 | `API.adminGetUsage()` |
|
||||
| **User (My Usage)** | `UI.loadMyUsage()` | ui-settings.js:246 | `API.getMyUsage()` |
|
||||
| **Team** | `UI.loadTeamUsage()` | ui-settings.js:297 | `API.teamGetUsage()` |
|
||||
|
||||
### Inconsistencies
|
||||
|
||||
| Feature | Admin | User | Team |
|
||||
|---------|-------|------|------|
|
||||
| Pricing table | ✅ | ❌ | ❌ |
|
||||
| "group by user" column header | ✅ | ❌ | ✅ |
|
||||
| Stats card styling | default padding | `padding:8px; font-size:16px` | `padding:8px; font-size:16px` |
|
||||
| Stats label font | default | `font-size:10px` | `font-size:10px` |
|
||||
| Empty state check | `results.length === 0` | `!t.requests` | `!t.requests` |
|
||||
| Grid layout | default | `grid-template-columns:repeat(4,1fr)` | `grid-template-columns:repeat(4,1fr)` |
|
||||
| Table font size | default | `font-size:12px` | `font-size:12px` |
|
||||
|
||||
User and Team are near-identical. Admin is a separate fork.
|
||||
|
||||
---
|
||||
|
||||
## 6. Provider Type / Endpoint Constants — scattered everywhere
|
||||
|
||||
The "known provider types" concept is defined in **5 places**:
|
||||
|
||||
| # | Location | Format |
|
||||
|---|----------|--------|
|
||||
| 1 | index.html:408 | `<option>` tags in User BYOK select |
|
||||
| 2 | index.html:626 | `<option>` tags in Admin select |
|
||||
| 3 | settings-handlers.js:553 | JS object `{ openai: 'OpenAI-compatible', ... }` for Team dynamic build |
|
||||
| 4 | settings-handlers.js:480 | Endpoint map for User BYOK |
|
||||
| 5 | settings-handlers.js:542 | Endpoint map for Team (separate copy) |
|
||||
| — | server/providers/registry.go | Backend `Init()` registers providers — source of truth |
|
||||
|
||||
Adding a new provider type requires touching 5 frontend locations.
|
||||
|
||||
---
|
||||
|
||||
## 7. Preset Form — ALREADY SHARED ✅
|
||||
|
||||
`renderPresetForm()` in ui-core.js:33 is used by:
|
||||
- Admin presets (admin-handlers.js:303)
|
||||
- Team presets (settings-handlers.js:511)
|
||||
- User presets (settings-handlers.js:626)
|
||||
|
||||
This is the ONE pattern that's already been consolidated into a primitive.
|
||||
Good reference for how the other patterns should work.
|
||||
|
||||
---
|
||||
|
||||
## 8. Model List — 2 copies
|
||||
|
||||
| Context | Function | File:Line |
|
||||
|---------|----------|-----------|
|
||||
| **Admin models** | `UI.loadAdminModels()` | ui-admin.js:365 |
|
||||
| **User models** | `UI.loadUserModels()` | ui-settings.js:571 |
|
||||
|
||||
Admin shows visibility toggle, user shows hide/unhide toggle.
|
||||
Both render capability badges (same compact format, duplicated).
|
||||
|
||||
---
|
||||
|
||||
## Summary: Primitives Needed
|
||||
|
||||
| Primitive | Replaces | Copies eliminated |
|
||||
|-----------|----------|-------------------|
|
||||
| `PROVIDERS` — type list, label map, endpoint defaults | 5 definitions → 1 | 4 |
|
||||
| `renderProviderForm(container, options)` | 3 HTML forms + 6 handlers → 1 | ~200 lines |
|
||||
| `renderProviderList(container, items, options)` | 3 list renderers → 1 | ~80 lines |
|
||||
| `renderRoleConfig(container, options)` | 2 role UIs + 4 handlers → 1 | ~120 lines |
|
||||
| `renderCapBadges(caps, compact?)` | 3 badge builders → 1 | ~30 lines |
|
||||
| `renderUsageDashboard(container, options)` | 3 usage renderers → 1 | ~120 lines |
|
||||
|
||||
**Estimated net reduction: ~550 lines + single source of truth for all constants.**
|
||||
|
||||
---
|
||||
|
||||
## Proposed File: `ui-primitives.js`
|
||||
|
||||
Loaded after `ui-format.js`, before `ui-core.js`. Contains:
|
||||
|
||||
```
|
||||
// Constants (single source of truth)
|
||||
const PROVIDER_TYPES = [ ... ]; // { id, label, defaultEndpoint }
|
||||
const ROLE_NAMES = ['utility', 'embedding'];
|
||||
const ROLE_TYPE_MAP = { embedding: 'embedding', utility: 'chat' };
|
||||
|
||||
// Rendering primitives
|
||||
function renderCapBadges(caps, opts) → HTML string
|
||||
function renderProviderForm(container, opts) → { getValues, setValues, clear }
|
||||
function renderProviderList(container, opts) → refresh function
|
||||
function renderRoleConfig(container, opts) → refresh function
|
||||
function renderUsageDashboard(container, opts) → refresh function
|
||||
```
|
||||
|
||||
Each primitive follows the `renderPresetForm` pattern:
|
||||
- Takes a container element + options object
|
||||
- Returns a control object with getValues/setValues/refresh
|
||||
- Options include callbacks (onSubmit, onDelete, etc.) and feature flags
|
||||
- Scope-specific behavior (admin/team/personal) via options, not separate code
|
||||
|
||||
---
|
||||
|
||||
## 9. Extension Surface Implications
|
||||
|
||||
The extension spec (EXTENSIONS.md §6) defines injection points:
|
||||
|
||||
```
|
||||
sidebar-top, sidebar-nav, sidebar-content, sidebar-bottom
|
||||
surface-header, surface-main, surface-footer
|
||||
```
|
||||
|
||||
Extensions interact with primitives through `ctx.ui.inject()` and
|
||||
`ctx.ui.replace()`. The cleaner the primitive interface, the easier
|
||||
extensions can:
|
||||
|
||||
1. **Add to** — e.g. a custom provider type, a new capability badge,
|
||||
an extra usage metric column
|
||||
2. **Hook into** — e.g. react when a provider is created, when a role
|
||||
is saved, when models refresh
|
||||
3. **Reuse** — e.g. render a provider form inside an extension's own
|
||||
settings surface
|
||||
|
||||
### 9.1 What Extensions Will Want to Do
|
||||
|
||||
| Extension | Wants to... | Primitive it hooks |
|
||||
|-----------|------------|-------------------|
|
||||
| Custom provider adapter | Add a provider type (e.g. "Ollama", "vLLM") | `PROVIDER_TYPES` registry |
|
||||
| Cost tracker | Add badges (cost/msg), extra usage columns | `renderCapBadges`, `renderUsageDashboard` |
|
||||
| Image gen | Register a role + show config UI | `renderRoleConfig` |
|
||||
| Smart routing | Show routing rules alongside role config | `renderRoleConfig` |
|
||||
| Cluster manager | Render provider forms in its own surface | `renderProviderForm` |
|
||||
| KB/RAG | Show embedding model status in sidebar | `ROLE_NAMES`, role resolution |
|
||||
|
||||
### 9.2 Design Rules for Extension-Ready Primitives
|
||||
|
||||
**Registry pattern, not constants.**
|
||||
Don't just export `PROVIDER_TYPES = [...]` as a frozen array.
|
||||
Make it a registry that extensions can `.add()` to:
|
||||
|
||||
```js
|
||||
const Providers = {
|
||||
_types: [ /* builtins */ ],
|
||||
add(entry) { this._types.push(entry); },
|
||||
all() { return this._types; },
|
||||
get(id) { return this._types.find(t => t.id === id); },
|
||||
endpoints() { /* map id → defaultEndpoint */ },
|
||||
labels() { /* map id → label */ },
|
||||
};
|
||||
```
|
||||
|
||||
Same for roles — `Roles.add('generation', { typeFilter: 'image', label: '...' })`.
|
||||
|
||||
**Options objects, not positional args.**
|
||||
Every primitive takes `(container, options)`. Options are the extension
|
||||
point — an extension can wrap a primitive call and inject additional
|
||||
options:
|
||||
|
||||
```js
|
||||
// Extension adds a custom column to usage
|
||||
renderUsageDashboard(el, {
|
||||
apiCall: () => API.getMyUsage({ period, group_by }),
|
||||
extraColumns: [
|
||||
{ header: 'Est. Cost/msg', render: (r) => '$' + (r.total_cost / r.requests).toFixed(4) }
|
||||
]
|
||||
});
|
||||
```
|
||||
|
||||
**Return control handles.**
|
||||
Every primitive returns an object with methods, not just void:
|
||||
|
||||
```js
|
||||
const provForm = renderProviderForm(el, opts);
|
||||
// Extensions can:
|
||||
provForm.getValues() // read current state
|
||||
provForm.setValues({...}) // programmatically fill
|
||||
provForm.clear() // reset
|
||||
provForm.onTypeChange(fn) // hook into type selection
|
||||
provForm.container // DOM reference for injection
|
||||
```
|
||||
|
||||
**Emit events on state changes.**
|
||||
Primitives should fire EventBus events that extensions can subscribe to:
|
||||
|
||||
```
|
||||
provider.form.typeChanged — user picked a different provider type
|
||||
provider.created — provider successfully saved (already exists)
|
||||
provider.deleted — provider removed
|
||||
role.saved — role config updated
|
||||
models.refreshed — model list reloaded (already exists)
|
||||
```
|
||||
|
||||
Most of these events already exist or will exist in the EventBus. The
|
||||
primitives just need to emit them consistently rather than each copy
|
||||
doing ad-hoc `UI.toast()` + `fetchModels()` in different orders.
|
||||
|
||||
**Scope via options, not separate code.**
|
||||
The biggest win: one `renderProviderForm` handles admin/team/personal
|
||||
through options, not three separate implementations:
|
||||
|
||||
```js
|
||||
renderProviderForm(el, {
|
||||
scope: 'admin', // → shows private checkbox, no BYOK gate
|
||||
// scope: 'team', // → shows private checkbox, scoped API
|
||||
// scope: 'personal', // → no private, BYOK gated
|
||||
apiCreate: (vals) => API.adminCreateGlobalConfig(...),
|
||||
apiUpdate: (id, vals) => API.adminUpdateGlobalConfig(id, ...),
|
||||
onSaved: () => { UI.loadAdminProviders(); },
|
||||
showPrivate: true,
|
||||
showDefaultModel: true,
|
||||
});
|
||||
```
|
||||
|
||||
### 9.3 Primitive → Extension Migration Path
|
||||
|
||||
These primitives are **pre-extension infrastructure**. When the extension
|
||||
loader (v0.11.0) lands, the migration is:
|
||||
|
||||
1. `Providers` registry gets a `ctx.providers.add()` wrapper in the
|
||||
extension context
|
||||
2. `renderCapBadges` checks a `capBadgeRegistry` that extensions can add to
|
||||
3. `renderUsageDashboard` accepts `extraColumns` from extension settings
|
||||
4. `renderRoleConfig` reads `Roles.all()` instead of hardcoded list
|
||||
|
||||
The primitives we build now become the extension API surface later.
|
||||
No rewrite needed — just wrapping in the permission-scoped `ctx` object.
|
||||
|
||||
---
|
||||
|
||||
## 10. Implementation Plan
|
||||
|
||||
### Phase 1: Constants + Small Primitives
|
||||
- `Providers` registry (types, labels, endpoints)
|
||||
- `Roles` registry (names, type filters, descriptions)
|
||||
- `renderCapBadges(caps, opts)` — consolidate 3 badge builders
|
||||
|
||||
### Phase 2: Provider Primitives
|
||||
- `renderProviderForm(container, opts)` — replace 3 form implementations
|
||||
- `renderProviderList(container, opts)` — replace 3 list renderers
|
||||
- Update index.html: remove 2 static provider selects, add empty containers
|
||||
|
||||
### Phase 3: Dashboard Primitives
|
||||
- `renderUsageDashboard(container, opts)` — replace 3 usage renderers
|
||||
- `renderRoleConfig(container, opts)` — replace 2 role UIs
|
||||
|
||||
### Phase 4: Cleanup
|
||||
- Remove dead code from admin-handlers.js, settings-handlers.js
|
||||
- Update tests
|
||||
- Verify all 3 scopes (admin/team/personal) work identically
|
||||
Reference in New Issue
Block a user