Changeset 0.37.16 (#228)

This commit is contained in:
2026-03-24 11:46:13 +00:00
parent d005e8a30f
commit e0687d2ea6
19 changed files with 2900 additions and 13 deletions

View File

@@ -0,0 +1,456 @@
# DESIGN — Multi-File Starlark Packages
Promotes Starlark scripts from a single inline blob to a proper
file tree with `load()` support. Prerequisite for library packages.
---
## Current State
Three problems:
1. **Scripts live in the DB.** The installer injects `script.star`
contents into `manifest["_starlark_script"]`. The runner reads
the script from the manifest JSONB column at runtime. This means
the entire script — no matter how large — is serialized into a
JSON string inside a JSON object inside a database row.
2. **`load()` is disabled.** The sandbox callback hard-returns an
error. There's no way to split code across files.
3. **Only static assets are extracted.** `extractableRelPath()`
allows `js/`, `css/`, `assets/` — nothing else. Starlark files
land in the DB, not on disk.
The hotfix in v0.37.14 (reading `script.star` from the archive and
injecting it as `_starlark_script`) made single-file clean but
didn't fix the underlying architecture.
---
## Design
### Archive Structure
```
gitea-client/
├── manifest.json ← metadata only, no _starlark_script
├── script.star ← entry point (required for starlark tier)
├── star/ ← optional: additional modules
│ ├── repos.star
│ ├── issues.star
│ ├── auth.star
│ └── ci.star
├── js/ ← optional: surface assets
│ └── main.js
└── css/
└── main.css
```
Convention: `script.star` is always the entry point. Submodules live
in `star/`. This parallels `js/main.js` as the JS entry point with
supporting files alongside it.
### Manifest Change
The manifest gains an optional `entry_point` field for packages that
want a different entry script name. Default: `script.star`.
```json
{
"id": "gitea-client",
"type": "library",
"tier": "starlark",
"entry_point": "script.star"
}
```
`_starlark_script` is no longer written to the manifest. The field
is still read for backward compatibility (existing packages that have
it in their DB row continue to work until reinstalled).
### Extraction
**`extractableRelPath()`** adds `star/` and bare `*.star` files:
```go
var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}
func extractableRelPath(name string) string {
// Existing prefix matching for directories...
// Also extract bare .star files at archive root
base := filepath.Base(name)
if strings.HasSuffix(base, ".star") {
// Strip leading directory (package-id/) if present
if idx := strings.Index(name, "/"); idx >= 0 {
rest := name[idx+1:]
if rest == base || strings.HasPrefix(rest, "star/") {
return rest
}
}
if name == base {
return name
}
}
return ""
}
```
After install, the packages directory looks like:
```
/data/packages/gitea-client/
├── script.star
├── star/
│ ├── repos.star
│ ├── issues.star
│ ├── auth.star
│ └── ci.star
├── js/
│ └── main.js
└── css/
└── main.css
```
### Installer Changes
**`packages.go` `InstallPackage()`:**
1. Stop injecting `_starlark_script` into the manifest. Remove the
hotfix code that reads `script.star` from the archive and stuffs
it into the manifest JSON.
2. Let `extractableRelPath()` handle `.star` files naturally — they
get extracted to disk alongside `js/` and `css/`.
3. Validate: if `tier == "starlark"`, confirm the archive contains
`script.star` (or the manifest's `entry_point` value). Return
400 if missing.
### Runner Changes
**`runner.go` `ExecPackage()`:**
Replace manifest-based script loading with file-based:
```go
func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
if pkg.Status != models.PackageStatusActive {
return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
}
if pkg.Tier != models.ExtTierStarlark {
return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
}
// ── Load script from disk (primary) or manifest (legacy) ──
script, err := r.loadScript(pkg)
if err != nil {
return nil, err
}
modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
if err != nil {
return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
}
// Build package-scoped load callback
loader := r.packageLoader(pkg.ID, modules)
log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))
return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
}
func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
// Primary: read from disk
if r.packagesDir != "" {
entryPoint := "script.star"
if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
entryPoint = ep
}
path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
data, err := os.ReadFile(path)
if err == nil && len(data) > 0 {
return string(data), nil
}
// Fall through to legacy
}
// Legacy: inline in manifest
script, ok := pkg.Manifest["_starlark_script"].(string)
if ok && script != "" {
return script, nil
}
return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
}
```
### Sandbox Changes
**`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but
accepts a `load` callback:
```go
// LoadFunc resolves load("path") calls to Starlark source.
// Returns the module's globals. The sandbox caches results per
// thread (Starlark handles this via thread.Load dedup).
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)
func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {
// ... same as Exec() but thread.Load = loader instead of error ...
}
```
The existing `Exec()` becomes a thin wrapper that passes a
nil/error loader for backward compatibility.
### Package-Scoped Loader
**`runner.go`** builds the load callback scoped to the package's
directory:
```go
func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
if r.packagesDir == "" {
return nil // no disk = no load support
}
pkgDir := filepath.Join(r.packagesDir, pkgID)
cache := make(map[string]*loadEntry) // dedup + cycle detection
return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
// ── Security: reject traversal ──
if strings.Contains(module, "..") || filepath.IsAbs(module) {
return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
}
// ── Resolve to package directory ──
resolved := filepath.Join(pkgDir, module)
if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
return nil, fmt.Errorf("load: path escapes package directory: %q", module)
}
// ── Must be a .star file ──
if !strings.HasSuffix(resolved, ".star") {
return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
}
// ── Cache / cycle detection ──
if entry, ok := cache[module]; ok {
if entry.loading {
return nil, fmt.Errorf("load: circular dependency: %q", module)
}
return entry.globals, nil
}
entry := &loadEntry{loading: true}
cache[module] = entry
// ── Read and execute ──
data, err := os.ReadFile(resolved)
if err != nil {
return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
}
globals, err := starlark.ExecFile(thread, module, string(data),
r.predeclaredForLoad(modules))
if err != nil {
return nil, fmt.Errorf("load: error in %q: %w", module, err)
}
entry.globals = globals
entry.loading = false
return globals, nil
}
}
type loadEntry struct {
loading bool
globals starlark.StringDict
}
```
Key constraints:
- **Package-scoped.** Can only load files from within the package's
own directory. Path traversal (`..`) and absolute paths are rejected.
One package cannot load another package's files.
- **`.star` files only.** Cannot load `.js`, `.json`, or anything
else. This is a Starlark sandbox, not a file reader.
- **Circular dependency detection.** If A loads B loads A → error.
- **Cached per invocation.** If `script.star` and `repos.star` both
load `auth.star`, it executes once. Starlark's thread-level load
dedup handles this, plus our explicit cache.
- **Same predeclared modules.** Loaded files get the same `db`,
`http`, `json`, `settings`, `connections` modules as the entry
point. They run in the same permission context.
### Starlark Usage
```python
# script.star (entry point)
load("star/auth.star", "auth_headers", "auth_headers_json")
load("star/repos.star", "get_repos", "sync_repos")
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
load("star/ci.star", "get_ci_status")
def on_request(req):
# ... dispatch to imported functions ...
def on_tool_call(tool_name, params):
# ... dispatch to imported functions ...
```
```python
# star/auth.star
def auth_headers(conn):
return {"Authorization": "token " + conn.get("api_token", "")}
def auth_headers_json(conn):
h = auth_headers(conn)
h["Content-Type"] = "application/json"
return h
```
```python
# star/repos.star
load("star/auth.star", "auth_headers")
def get_repos(conn):
url = conn["base_url"] + "/api/v1/repos/search?limit=50"
resp = http.get(url=url, headers=auth_headers(conn))
if int(resp["status"]) >= 400:
return None
return json.decode(resp["body"])
def sync_repos(conn):
repos = get_repos(conn)
# ... db.insert into private tables ...
```
Submodules can load other submodules. The dependency graph is a DAG
(cycles are rejected).
### Build Script
**`packages/build.sh`** already includes `script.star`. Add `star/`:
```bash
local dirs=""
[ -d "$dir/js" ] && dirs="$dirs js/"
[ -d "$dir/css" ] && dirs="$dirs css/"
[ -d "$dir/assets" ] && dirs="$dirs assets/"
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
[ -d "$dir/star" ] && dirs="$dirs star/"
[ -d "$dir/migrations" ] && dirs="$dirs migrations/"
```
### Runner Constructor
**`runner.go`** gains `packagesDir`:
```go
type Runner struct {
sandbox *Sandbox
stores store.Stores
packagesDir string // NEW
notifier NotificationSender
resolver ProviderResolver
db *sql.DB
dbPostgres bool
}
func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
return &Runner{sandbox: sb, stores: stores}
}
func (r *Runner) SetPackagesDir(dir string) {
r.packagesDir = dir
}
```
**`main.go`** wires it:
```go
starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")
```
---
## Backward Compatibility
| Scenario | Behavior |
|----------|----------|
| Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. |
| Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. |
| New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. |
| New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. |
| Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. |
| Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". |
No migration needed. Old packages work as-is. New packages benefit
from the file-based path. Reinstalling an old package extracts its
`script.star` to disk (via the updated `extractableRelPath`).
---
## Security
| Threat | Mitigation |
|--------|------------|
| Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. |
| Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. |
| Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. |
| Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. |
| Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. |
| Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. |
| Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. |
---
## Changeset Plan
| CS | Scope | Files |
|----|-------|-------|
| CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. |
| CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. |
| CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. |
| CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. |
| CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. |
CS1-CS4 can land in a single session. CS5 validates. All five are
backend-only — no FE changes.
---
## What This Enables
- **Library packages** — a `gitea-client` with `script.star` as
entry point, `star/repos.star`, `star/issues.star`, `star/ci.star`
as submodules. Clean separation of concerns.
- **Large extensions** — a workflow automation package with 20+
tool handlers doesn't have to be one 2000-line file.
- **Shared utilities** — within a single package, common helpers
(auth, validation, formatting) live in their own file and are
loaded by multiple entry points.
- **Testable modules** — each `.star` file can be loaded and
tested independently in a dev sandbox.
This does NOT enable cross-package loading. Package A cannot load
Package B's files. That's what `lib.load()` is for (see
DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is
intra-package. `lib.load()` is inter-package. Different mechanisms,
different trust boundaries.

View File

@@ -183,10 +183,11 @@ Each surface rebuilt as Preact component tree. Old JS deleted per surface.
| v0.37.12 | Scorched Earth II ✅ | 23 files deleted (~2,600 lines): 9 JS, 6 CSS, 8 admin templates; sw.js + base.html cleaned |
| v0.37.13 | Scorched Earth III ✅ | 5 JS deleted (2,269 lines): ui-primitives, code-editor, file-tree, user-menu, app-state; SDK slimmed, Theme→Preact |
| v0.37.14 | SE IV + Pane audit ✅ | 4 JS deleted (~1,672 lines); double-unwrap cleanup (93 sites); API envelope normalization (18 Go handlers); thinking tags persistence; deleteMessage endpoint; extension surface user menu fix; 7 bug fixes |
| v0.37.15 | Workflow surfaces | Form renderer, review views (design-first) |
| v0.37.16 | Projects surface | Project views, note/channel associations (design-first) |
| v0.37.17 | Debug / model surface | Debug tooling, model configuration UI; debug modal Preact rebuild (CR P2-5) |
| v0.37.18 | Tag | Light mode CSS audit, dead code hunt, all features verified; `sw.can()` RBAC gates across surfaces (CR P2-1), `__USER__`/`__PAGE_DATA__` removal (CR P2-4), `_getScale``sw.shell.getScale()` (CR P3-3) |
| v0.37.15 | Workflow surfaces | Workflow ownership/lifecycle, stage CRUD, team-admin tabs, assignments queue |
| v0.37.16 | Projects surface ✅ | Card grid + detail, inline ChatPane, context menu, sidebar pickers, deep-link |
| v0.37.17 | Workspaces | File storage layer, archive upload/download (zip/tar), project file integration |
| v0.37.18 | Debug / model surface | Debug tooling, model configuration UI; debug modal Preact rebuild (CR P2-5) |
| v0.37.19 | Tag | Light mode CSS audit, dead code hunt, all features verified; `sw.can()` RBAC gates across surfaces (CR P2-1), `__USER__`/`__PAGE_DATA__` removal (CR P2-4), `_getScale``sw.shell.getScale()` (CR P3-3) |
**v0.37.14 — Scorched Earth IV + Pane Audit ✅** (10 sessions, v0.37.14.0.22):
@@ -203,15 +204,32 @@ Team-admin workflow management as first-class path. See
Instance cancel/unclaim/reassign, stage CRUD FE wiring, team-admin
workflow queue + stage editor + instance monitor surfaces.
**v0.37.16 — Projects Surface:**
**v0.37.16 — Projects Surface:**
Project views, note/channel associations, project-scoped navigation.
Card-grid list + two-column detail (inline ChatPane, context menu, sidebar
pickers, star toggle, deep-linking). UserMenu in header. Backend was already
complete (16+2 endpoints, 18 SDK methods). File upload UI wired but backend
storage deferred to v0.37.17.
**v0.37.17 — Debug + Model Surface:**
**v0.37.17 — Workspaces (File Storage):**
Workspace storage layer for project files. Archive (zip, tar, etc.)
upload/download is **required for MVP** — projects need document attachment.
Schema and model FK already exist (`workspaces` table, `Project.WorkspaceID`).
Git integration (clone, pull, branch tracking) is optional/post-MVP.
- [ ] Workspace CRUD endpoints wiring (create-on-demand for projects)
- [ ] File upload backend: disk storage with workspace root path
- [ ] Archive upload: zip/tar extraction + individual file indexing
- [ ] File download: single file + archive download (zip bundle)
- [ ] File browser UI in project sidebar (list, preview, delete)
- [ ] Storage quota enforcement (`max_bytes` on workspace)
**v0.37.18 — Debug + Model Surface:**
Debug modal Preact rebuild, model configuration UI, provider diagnostics.
**v0.37.18 — Tag ("UI Complete"):**
**v0.37.19 — Tag ("UI Complete"):**
`sw.can()` RBAC gates, `__USER__`/`__PAGE_DATA__` removal, `_getScale` SDK,
light mode CSS audit, dead code hunt. Every capability has a UI.
@@ -258,7 +276,7 @@ reading Go source code. Team admins build workflows visually.
package management, backup/restore
- [ ] Mobile-responsive layouts (proper mobile navigation,
touch-optimized chat, not just CSS responsive)
- [ ] Project creation dialog (replace `prompt()` with modal)
- [x] Project creation dialog (v0.37.16, sw.prompt-based)
- [ ] Admin-level project management (cross-instance visibility)
---
@@ -297,10 +315,10 @@ No ordering constraints between items.
- Memory analytics dashboard
**Projects**
- `/p/:id` shared project view
- Project-specific file uploads (own endpoint, storage path)
- `/p/:id` shared project view (public/team-scoped read-only link)
- Project templates (predefined configurations)
- Sub-projects / nested hierarchy
- Git integration (clone, pull, branch tracking — optional workspace feature)
**UX**
- "Group" scope badge on model selector / KB list