Changeset 0.34.0 (#208)
Co-authored-by: Jeffrey Smith <jasafpro@gmail.com> Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
This commit is contained in:
217
docs/DESIGN-0.34.0.md
Normal file
217
docs/DESIGN-0.34.0.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# DESIGN-0.34.0 — Data Portability
|
||||
|
||||
Export your data, import it elsewhere, delete your account. Six
|
||||
changesets covering user export, user import, GDPR delete, admin
|
||||
team export/import, ChatGPT import, and K8s scheduled backups.
|
||||
|
||||
Depends on: v0.33.0 (Observability, current HEAD).
|
||||
|
||||
**Design decision:** One new migration (021) — indexes only, no new
|
||||
tables. The export archive is a ZIP file (`.switchboard` extension)
|
||||
containing a JSON manifest and per-entity JSON files. Sensitive fields
|
||||
(password hashes, encrypted keys, storage keys) are stripped at export
|
||||
time via entity sanitization.
|
||||
|
||||
**Archive format:** `.switchboard` ZIP with `format_version=1`:
|
||||
```
|
||||
manifest.json
|
||||
data/users.json
|
||||
data/channels.json
|
||||
data/messages.json
|
||||
data/channel_participants.json
|
||||
data/channel_models.json
|
||||
data/channel_cursors.json
|
||||
data/notes.json
|
||||
data/note_links.json
|
||||
data/memories.json
|
||||
data/projects.json
|
||||
data/project_channels.json
|
||||
data/project_knowledge_bases.json
|
||||
data/project_notes.json
|
||||
data/workspaces.json
|
||||
data/workspace_files.json
|
||||
data/folders.json
|
||||
data/user_settings.json
|
||||
data/notification_preferences.json
|
||||
data/usage_entries.json
|
||||
data/persona_groups.json
|
||||
data/persona_group_members.json
|
||||
data/files.json
|
||||
files/{file_id}/{filename}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Already Existed
|
||||
|
||||
- **Package export** — `.pkg` ZIP format, `package_export.go`
|
||||
- **Markdown export** — pandoc-based, `export.go`
|
||||
- **Workspace archive** — ZIP/tar.gz with size limits, `workspace/archive.go`
|
||||
- **Soft deletes** — `deleted_at` on messages
|
||||
- **Audit trail** — `audit_logs` table
|
||||
- **39 store interfaces** covering all entity types
|
||||
|
||||
---
|
||||
|
||||
## CS0 — Export Format + User Export
|
||||
|
||||
Foundation changeset. Defines the archive format and ExportStore interface.
|
||||
|
||||
### `server/export/format.go` (new)
|
||||
|
||||
Archive types: `Manifest`, `ManifestScope`, `ArchiveWriter`, `ArchiveReader`.
|
||||
Size limits: 500 MB archive, 100 MB per file, 10K files max.
|
||||
Key methods: `WriteManifest`, `WriteEntityJSON`, `WriteFile`,
|
||||
`ReadManifest`, `ReadEntityJSON`, `FileEntries`.
|
||||
|
||||
### `server/export/entities.go` (new)
|
||||
|
||||
Per-entity sanitization functions. Export types strip sensitive fields:
|
||||
- `ExportChannel` — removes `provider_config_id`
|
||||
- `ExportMessage` — no changes needed
|
||||
- `ExportWorkspace` — removes `root_path`, git credentials
|
||||
- `ExportUsageEntry` — removes `provider_config_id`
|
||||
- Users — removes `password_hash`, `encrypted_uek`, `uek_salt`
|
||||
|
||||
### `server/store/interfaces.go` (modified)
|
||||
|
||||
Added `Export ExportStore` to `Stores` struct. The `ExportStore`
|
||||
interface provides 21 user-scoped read methods, 20 import methods
|
||||
(returning `(imported, skipped, error)` counts), 14 team-scoped
|
||||
read methods, and 4 GDPR methods.
|
||||
|
||||
### `server/store/{postgres,sqlite}/export.go` (new)
|
||||
|
||||
Full implementations for both dialects (~1400-1500 lines each).
|
||||
Import uses `INSERT ON CONFLICT DO NOTHING` (PG) / `INSERT OR IGNORE`
|
||||
(SQLite) for UUID-based dedup.
|
||||
|
||||
### `server/handlers/export_data.go` (new)
|
||||
|
||||
`DataExportHandler.ExportMyData` — `GET /api/v1/export/me`.
|
||||
Streams ZIP directly to response using `zip.NewWriter(c.Writer)`.
|
||||
Includes file blobs from object store.
|
||||
|
||||
### Migration 021
|
||||
|
||||
Indexes only — `idx_messages_channel_active`, `idx_channels_user_id`,
|
||||
`idx_files_user_id`. PG version uses partial index
|
||||
(`WHERE deleted_at IS NULL`).
|
||||
|
||||
---
|
||||
|
||||
## CS1 — User Import
|
||||
|
||||
### `server/handlers/import_data.go` (new)
|
||||
|
||||
`DataImportHandler.ImportMyData` — `POST /api/v1/import/me`.
|
||||
|
||||
Opens archive, validates manifest `format_version`, reads entities
|
||||
in FK-dependency order, remaps `user_id` for cross-instance import.
|
||||
Returns `{imported: {channels: N, ...}, skipped: {...}, errors: [...]}`.
|
||||
|
||||
Import order: Folders → Projects → Workspaces → Channels →
|
||||
Participants/Models/Cursors → Messages → Notes → Note Links →
|
||||
Memories → Project links → Workspace files → Files + blobs →
|
||||
Settings → Persona groups.
|
||||
|
||||
---
|
||||
|
||||
## CS2 — GDPR Account Delete
|
||||
|
||||
### `server/handlers/gdpr.go` (new)
|
||||
|
||||
`GDPRHandler.DeleteMyAccount` — `DELETE /api/v1/me`.
|
||||
|
||||
Requires `{"confirm": "DELETE", "password": "..."}`. Verifies
|
||||
password, prevents last-admin deletion, then:
|
||||
|
||||
1. Soft-deletes messages, archives channels
|
||||
2. Hard-deletes: notes, memories, workspaces, files, projects, settings
|
||||
3. Revokes tokens (refresh + WS tickets)
|
||||
4. Anonymizes user: `deleted-user-{sha256(id)[:12]}`
|
||||
5. Writes audit log entry
|
||||
|
||||
---
|
||||
|
||||
## CS3 — Admin Team Export/Import
|
||||
|
||||
`DataExportHandler.ExportTeam` — `GET /api/v1/admin/teams/:id/export`
|
||||
`DataImportHandler.ImportTeam` — `POST /api/v1/admin/teams/:id/import`
|
||||
|
||||
Same archive format, scoped to team entities (channels, personas,
|
||||
knowledge bases, workflows, groups, projects, resource grants).
|
||||
Team import skips members whose `user_id` doesn't exist on target.
|
||||
|
||||
---
|
||||
|
||||
## CS4 — ChatGPT Import
|
||||
|
||||
### `server/export/chatgpt.go` (new)
|
||||
|
||||
Parses ChatGPT `conversations.json` format. Each conversation maps
|
||||
to one channel (type=direct). The `mapping` DAG is walked depth-first
|
||||
to produce ordered messages with `parent_id` and `sibling_index`.
|
||||
|
||||
Role mapping: `user`→user, `assistant`→assistant, `system`→system,
|
||||
`tool`→tool. Content: `parts[]` joined with `\n`, non-string parts
|
||||
skipped. Timestamps: `float64` epoch → `time.Time`.
|
||||
|
||||
`DataImportHandler.ImportChatGPT` — `POST /api/v1/import/chatgpt`.
|
||||
Accepts multipart upload of `conversations.json` or ZIP containing it.
|
||||
|
||||
---
|
||||
|
||||
## CS5 — K8s Backup + ICD Tests
|
||||
|
||||
### `chart/templates/cronjob-backup.yaml` (new)
|
||||
|
||||
CronJob running `pg_dump | gzip` on schedule (default daily 02:00 UTC).
|
||||
Optional S3 upload via AWS CLI. Local retention pruning. Gated on
|
||||
`backup.enabled=true` and `database.driver=postgres`.
|
||||
|
||||
### `chart/templates/pvc-backup.yaml` (new)
|
||||
|
||||
Conditional PVC for local backup storage (when `backup.persistence.enabled`).
|
||||
|
||||
### `chart/values.yaml` (modified)
|
||||
|
||||
New `backup:` section with schedule, retention, S3 config, persistence,
|
||||
and resource limits.
|
||||
|
||||
### ICD Tests
|
||||
|
||||
~30 new tests in `crud/portability.js` covering:
|
||||
- Export download + archive validation (ZIP structure, manifest, entities)
|
||||
- Sensitive field exclusion verification
|
||||
- Import re-import (dedup validation)
|
||||
- Invalid archive rejection
|
||||
- ChatGPT import (valid, invalid JSON, empty array)
|
||||
- ChatGPT channel/message verification
|
||||
- GDPR delete flow (register → create data → delete → verify lockout)
|
||||
- GDPR validation (missing confirm, wrong password)
|
||||
- Admin team export/import
|
||||
|
||||
---
|
||||
|
||||
## New Routes
|
||||
|
||||
| Method | Path | Auth | Handler |
|
||||
|--------|------|------|---------|
|
||||
| GET | `/api/v1/export/me` | JWT | `DataExportHandler.ExportMyData` |
|
||||
| POST | `/api/v1/import/me` | JWT | `DataImportHandler.ImportMyData` |
|
||||
| POST | `/api/v1/import/chatgpt` | JWT | `DataImportHandler.ImportChatGPT` |
|
||||
| DELETE | `/api/v1/me` | JWT | `GDPRHandler.DeleteMyAccount` |
|
||||
| GET | `/api/v1/admin/teams/:id/export` | Admin | `DataExportHandler.ExportTeam` |
|
||||
| POST | `/api/v1/admin/teams/:id/import` | Admin | `DataImportHandler.ImportTeam` |
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. `cd server && go build ./...` — all changesets compile clean
|
||||
2. `helm template ./chart --set backup.enabled=true` — CronJob renders
|
||||
3. ICD test suite: ~615 tests (585 existing + ~30 portability)
|
||||
4. Manual smoke: export → inspect ZIP → re-import → verify dedup
|
||||
5. GDPR: delete → verify login fails → inspect anonymized DB record
|
||||
6. Sensitive fields: no `password_hash`, `encrypted_uek`, `storage_key` in export
|
||||
Reference in New Issue
Block a user