core/docs/DESIGN-0.34.0.md

DESIGN-0.34.0 — Data Portability

Export your data, import it elsewhere, delete your account. Six changesets covering user export, user import, GDPR delete, admin team export/import, ChatGPT import, and K8s scheduled backups.

Depends on: v0.33.0 (Observability, current HEAD).

Design decision: One new migration (021) — indexes only, no new tables. The export archive is a ZIP file (.switchboard extension) containing a JSON manifest and per-entity JSON files. Sensitive fields (password hashes, encrypted keys, storage keys) are stripped at export time via entity sanitization.

Archive format: .switchboard ZIP with format_version=1:

manifest.json
data/users.json
data/channels.json
data/messages.json
data/channel_participants.json
data/channel_models.json
data/channel_cursors.json
data/notes.json
data/note_links.json
data/memories.json
data/projects.json
data/project_channels.json
data/project_knowledge_bases.json
data/project_notes.json
data/workspaces.json
data/workspace_files.json
data/folders.json
data/user_settings.json
data/notification_preferences.json
data/usage_entries.json
data/persona_groups.json
data/persona_group_members.json
data/files.json
files/{file_id}/{filename}

---

What Already Existed

• Package export — .pkg ZIP format, package_export.go
• Markdown export — pandoc-based, export.go
• Workspace archive — ZIP/tar.gz with size limits, workspace/archive.go
• Soft deletes — deleted_at on messages
• Audit trail — audit_logs table
• 39 store interfaces covering all entity types

---

CS0 — Export Format + User Export

Foundation changeset. Defines the archive format and ExportStore interface.

server/export/format.go (new)

Archive types: Manifest, ManifestScope, ArchiveWriter, ArchiveReader. Size limits: 500 MB archive, 100 MB per file, 10K files max. Key methods: WriteManifest, WriteEntityJSON, WriteFile, ReadManifest, ReadEntityJSON, FileEntries.

server/export/entities.go (new)

Per-entity sanitization functions. Export types strip sensitive fields:

• ExportChannel — removes provider_config_id
• ExportMessage — no changes needed
• ExportWorkspace — removes root_path, git credentials
• ExportUsageEntry — removes provider_config_id
• Users — removes password_hash, encrypted_uek, uek_salt

server/store/interfaces.go (modified)

Added Export ExportStore to Stores struct. The ExportStore interface provides 21 user-scoped read methods, 20 import methods (returning (imported, skipped, error) counts), 14 team-scoped read methods, and 4 GDPR methods.

server/store/{postgres,sqlite}/export.go (new)

Full implementations for both dialects (~1400-1500 lines each). Import uses INSERT ON CONFLICT DO NOTHING (PG) / INSERT OR IGNORE (SQLite) for UUID-based dedup.

server/handlers/export_data.go (new)

DataExportHandler.ExportMyData — GET /api/v1/export/me. Streams ZIP directly to response using zip.NewWriter(c.Writer). Includes file blobs from object store.

Migration 021

Indexes only — idx_messages_channel_active, idx_channels_user_id, idx_files_user_id. PG version uses partial index (WHERE deleted_at IS NULL).

---

CS1 — User Import

server/handlers/import_data.go (new)

DataImportHandler.ImportMyData — POST /api/v1/import/me.

Opens archive, validates manifest format_version, reads entities in FK-dependency order, remaps user_id for cross-instance import. Returns {imported: {channels: N, ...}, skipped: {...}, errors: [...]}.

Import order: Folders → Projects → Workspaces → Channels → Participants/Models/Cursors → Messages → Notes → Note Links → Memories → Project links → Workspace files → Files + blobs → Settings → Persona groups.

---

CS2 — GDPR Account Delete

server/handlers/gdpr.go (new)

GDPRHandler.DeleteMyAccount — DELETE /api/v1/me.

Requires {"confirm": "DELETE", "password": "..."}. Verifies password, prevents last-admin deletion, then:

1. Soft-deletes messages, archives channels
2. Hard-deletes: notes, memories, workspaces, files, projects, settings
3. Revokes tokens (refresh + WS tickets)
4. Anonymizes user: deleted-user-{sha256(id)[:12]}
5. Writes audit log entry

---

CS3 — Admin Team Export/Import

DataExportHandler.ExportTeam — GET /api/v1/admin/teams/:id/export DataImportHandler.ImportTeam — POST /api/v1/admin/teams/:id/import

Same archive format, scoped to team entities (channels, personas, knowledge bases, workflows, groups, projects, resource grants). Team import skips members whose user_id doesn't exist on target.

---

CS4 — ChatGPT Import

server/export/chatgpt.go (new)

Parses ChatGPT conversations.json format. Each conversation maps to one channel (type=direct). The mapping DAG is walked depth-first to produce ordered messages with parent_id and sibling_index.

Role mapping: user→user, assistant→assistant, system→system, tool→tool. Content: parts[] joined with \n, non-string parts skipped. Timestamps: float64 epoch → time.Time.

DataImportHandler.ImportChatGPT — POST /api/v1/import/chatgpt. Accepts multipart upload of conversations.json or ZIP containing it.

---

CS5 — K8s Backup + ICD Tests

chart/templates/cronjob-backup.yaml (new)

CronJob running pg_dump | gzip on schedule (default daily 02:00 UTC). Optional S3 upload via AWS CLI. Local retention pruning. Gated on backup.enabled=true and database.driver=postgres.

chart/templates/pvc-backup.yaml (new)

Conditional PVC for local backup storage (when backup.persistence.enabled).

chart/values.yaml (modified)

New backup: section with schedule, retention, S3 config, persistence, and resource limits.

ICD Tests

~30 new tests in crud/portability.js covering:

• Export download + archive validation (ZIP structure, manifest, entities)
• Sensitive field exclusion verification
• Import re-import (dedup validation)
• Invalid archive rejection
• ChatGPT import (valid, invalid JSON, empty array)
• ChatGPT channel/message verification
• GDPR delete flow (register → create data → delete → verify lockout)
• GDPR validation (missing confirm, wrong password)
• Admin team export/import

---

New Routes

Method	Path	Auth	Handler
GET	/api/v1/export/me	JWT	DataExportHandler.ExportMyData
POST	/api/v1/import/me	JWT	DataImportHandler.ImportMyData
POST	/api/v1/import/chatgpt	JWT	DataImportHandler.ImportChatGPT
DELETE	/api/v1/me	JWT	GDPRHandler.DeleteMyAccount
GET	/api/v1/admin/teams/:id/export	Admin	DataExportHandler.ExportTeam
POST	/api/v1/admin/teams/:id/import	Admin	DataImportHandler.ImportTeam

---

Verification

1. cd server && go build ./... — all changesets compile clean
2. helm template ./chart --set backup.enabled=true — CronJob renders
3. ICD test suite: ~615 tests (585 existing + ~30 portability)
4. Manual smoke: export → inspect ZIP → re-import → verify dedup
5. GDPR: delete → verify login fails → inspect anonymized DB record
6. Sensitive fields: no password_hash, encrypted_uek, storage_key in export