# DESIGN — Multi-File Starlark Packages Promotes Starlark scripts from a single inline blob to a proper file tree with `load()` support. Prerequisite for library packages. --- ## Current State Three problems: 1. **Scripts live in the DB.** The installer injects `script.star` contents into `manifest["_starlark_script"]`. The runner reads the script from the manifest JSONB column at runtime. This means the entire script — no matter how large — is serialized into a JSON string inside a JSON object inside a database row. 2. **`load()` is disabled.** The sandbox callback hard-returns an error. There's no way to split code across files. 3. **Only static assets are extracted.** `extractableRelPath()` allows `js/`, `css/`, `assets/` — nothing else. Starlark files land in the DB, not on disk. The hotfix in v0.37.14 (reading `script.star` from the archive and injecting it as `_starlark_script`) made single-file clean but didn't fix the underlying architecture. --- ## Design ### Archive Structure ``` gitea-client/ ├── manifest.json ← metadata only, no _starlark_script ├── script.star ← entry point (required for starlark tier) ├── star/ ← optional: additional modules │ ├── repos.star │ ├── issues.star │ ├── auth.star │ └── ci.star ├── js/ ← optional: surface assets │ └── main.js └── css/ └── main.css ``` Convention: `script.star` is always the entry point. Submodules live in `star/`. This parallels `js/main.js` as the JS entry point with supporting files alongside it. ### Manifest Change The manifest gains an optional `entry_point` field for packages that want a different entry script name. Default: `script.star`. ```json { "id": "gitea-client", "type": "library", "tier": "starlark", "entry_point": "script.star" } ``` `_starlark_script` is no longer written to the manifest. The field is still read for backward compatibility (existing packages that have it in their DB row continue to work until reinstalled). ### Extraction **`extractableRelPath()`** adds `star/` and bare `*.star` files: ```go var staticPrefixes = []string{"js/", "css/", "assets/", "star/"} func extractableRelPath(name string) string { // Existing prefix matching for directories... // Also extract bare .star files at archive root base := filepath.Base(name) if strings.HasSuffix(base, ".star") { // Strip leading directory (package-id/) if present if idx := strings.Index(name, "/"); idx >= 0 { rest := name[idx+1:] if rest == base || strings.HasPrefix(rest, "star/") { return rest } } if name == base { return name } } return "" } ``` After install, the packages directory looks like: ``` /data/packages/gitea-client/ ├── script.star ├── star/ │ ├── repos.star │ ├── issues.star │ ├── auth.star │ └── ci.star ├── js/ │ └── main.js └── css/ └── main.css ``` ### Installer Changes **`packages.go` `InstallPackage()`:** 1. Stop injecting `_starlark_script` into the manifest. Remove the hotfix code that reads `script.star` from the archive and stuffs it into the manifest JSON. 2. Let `extractableRelPath()` handle `.star` files naturally — they get extracted to disk alongside `js/` and `css/`. 3. Validate: if `tier == "starlark"`, confirm the archive contains `script.star` (or the manifest's `entry_point` value). Return 400 if missing. ### Runner Changes **`runner.go` `ExecPackage()`:** Replace manifest-based script loading with file-based: ```go func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) { if pkg.Status != models.PackageStatusActive { return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status) } if pkg.Tier != models.ExtTierStarlark { return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier) } // ── Load script from disk (primary) or manifest (legacy) ── script, err := r.loadScript(pkg) if err != nil { return nil, err } modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc) if err != nil { return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err) } // Build package-scoped load callback loader := r.packageLoader(pkg.ID, modules) log.Printf(" 🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules)) return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader) } func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) { // Primary: read from disk if r.packagesDir != "" { entryPoint := "script.star" if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" { entryPoint = ep } path := filepath.Join(r.packagesDir, pkg.ID, entryPoint) data, err := os.ReadFile(path) if err == nil && len(data) > 0 { return string(data), nil } // Fall through to legacy } // Legacy: inline in manifest script, ok := pkg.Manifest["_starlark_script"].(string) if ok && script != "" { return script, nil } return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID) } ``` ### Sandbox Changes **`sandbox.go`** gains `ExecWithLoader()` — same as `Exec()` but accepts a `load` callback: ```go // LoadFunc resolves load("path") calls to Starlark source. // Returns the module's globals. The sandbox caches results per // thread (Starlark handles this via thread.Load dedup). type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error) func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string, modules map[string]starlark.Value, loader LoadFunc) (*Result, error) { // ... same as Exec() but thread.Load = loader instead of error ... } ``` The existing `Exec()` becomes a thin wrapper that passes a nil/error loader for backward compatibility. ### Package-Scoped Loader **`runner.go`** builds the load callback scoped to the package's directory: ```go func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc { if r.packagesDir == "" { return nil // no disk = no load support } pkgDir := filepath.Join(r.packagesDir, pkgID) cache := make(map[string]*loadEntry) // dedup + cycle detection return func(thread *starlark.Thread, module string) (starlark.StringDict, error) { // ── Security: reject traversal ── if strings.Contains(module, "..") || filepath.IsAbs(module) { return nil, fmt.Errorf("load: path traversal not allowed: %q", module) } // ── Resolve to package directory ── resolved := filepath.Join(pkgDir, module) if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) { return nil, fmt.Errorf("load: path escapes package directory: %q", module) } // ── Must be a .star file ── if !strings.HasSuffix(resolved, ".star") { return nil, fmt.Errorf("load: only .star files can be loaded: %q", module) } // ── Cache / cycle detection ── if entry, ok := cache[module]; ok { if entry.loading { return nil, fmt.Errorf("load: circular dependency: %q", module) } return entry.globals, nil } entry := &loadEntry{loading: true} cache[module] = entry // ── Read and execute ── data, err := os.ReadFile(resolved) if err != nil { return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID) } globals, err := starlark.ExecFile(thread, module, string(data), r.predeclaredForLoad(modules)) if err != nil { return nil, fmt.Errorf("load: error in %q: %w", module, err) } entry.globals = globals entry.loading = false return globals, nil } } type loadEntry struct { loading bool globals starlark.StringDict } ``` Key constraints: - **Package-scoped.** Can only load files from within the package's own directory. Path traversal (`..`) and absolute paths are rejected. One package cannot load another package's files. - **`.star` files only.** Cannot load `.js`, `.json`, or anything else. This is a Starlark sandbox, not a file reader. - **Circular dependency detection.** If A loads B loads A → error. - **Cached per invocation.** If `script.star` and `repos.star` both load `auth.star`, it executes once. Starlark's thread-level load dedup handles this, plus our explicit cache. - **Same predeclared modules.** Loaded files get the same `db`, `http`, `json`, `settings`, `connections` modules as the entry point. They run in the same permission context. ### Starlark Usage ```python # script.star (entry point) load("star/auth.star", "auth_headers", "auth_headers_json") load("star/repos.star", "get_repos", "sync_repos") load("star/issues.star", "get_issues", "create_issue", "cache_issue") load("star/ci.star", "get_ci_status") def on_request(req): # ... dispatch to imported functions ... def on_tool_call(tool_name, params): # ... dispatch to imported functions ... ``` ```python # star/auth.star def auth_headers(conn): return {"Authorization": "token " + conn.get("api_token", "")} def auth_headers_json(conn): h = auth_headers(conn) h["Content-Type"] = "application/json" return h ``` ```python # star/repos.star load("star/auth.star", "auth_headers") def get_repos(conn): url = conn["base_url"] + "/api/v1/repos/search?limit=50" resp = http.get(url=url, headers=auth_headers(conn)) if int(resp["status"]) >= 400: return None return json.decode(resp["body"]) def sync_repos(conn): repos = get_repos(conn) # ... db.insert into private tables ... ``` Submodules can load other submodules. The dependency graph is a DAG (cycles are rejected). ### Build Script **`packages/build.sh`** already includes `script.star`. Add `star/`: ```bash local dirs="" [ -d "$dir/js" ] && dirs="$dirs js/" [ -d "$dir/css" ] && dirs="$dirs css/" [ -d "$dir/assets" ] && dirs="$dirs assets/" [ -f "$dir/script.star" ] && dirs="$dirs script.star" [ -d "$dir/star" ] && dirs="$dirs star/" [ -d "$dir/migrations" ] && dirs="$dirs migrations/" ``` ### Runner Constructor **`runner.go`** gains `packagesDir`: ```go type Runner struct { sandbox *Sandbox stores store.Stores packagesDir string // NEW notifier NotificationSender resolver ProviderResolver db *sql.DB dbPostgres bool } func NewRunner(sb *Sandbox, stores store.Stores) *Runner { return &Runner{sandbox: sb, stores: stores} } func (r *Runner) SetPackagesDir(dir string) { r.packagesDir = dir } ``` **`main.go`** wires it: ```go starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores) starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages") ``` --- ## Backward Compatibility | Scenario | Behavior | |----------|----------| | Existing package with `_starlark_script` in manifest, no files on disk | Works. `loadScript()` falls through to legacy path. | | Package installed with v0.37.14 hotfix (`script.star` injected into manifest) | Works. Same legacy path. | | New package with `script.star` on disk, no `_starlark_script` in manifest | Works. Primary path. | | New package with `script.star` + `star/` submodules | Works. Loader resolves `load()` calls. | | Package with both `_starlark_script` AND `script.star` on disk | Disk wins. Manifest is legacy fallback only. | | Package that calls `load()` but has no files on disk | Error: "load() not available (no package directory)". | No migration needed. Old packages work as-is. New packages benefit from the file-based path. Reinstalling an old package extracts its `script.star` to disk (via the updated `extractableRelPath`). --- ## Security | Threat | Mitigation | |--------|------------| | Path traversal (`load("../../etc/passwd")`) | `..` rejected. `filepath.Clean` + prefix check against package dir. | | Absolute paths (`load("/etc/shadow")`) | `filepath.IsAbs` check → rejected. | | Loading non-Starlark files (`load("js/main.js")`) | `.star` suffix required. | | Cross-package load (`load("../other-pkg/script.star")`) | `..` rejected. Resolution is always relative to the package's own directory. | | Circular loads (`A→B→A`) | `loadEntry.loading` flag → error on re-entry. | | Symlink escape | `filepath.Clean` resolves symlinks. Production: packages dir should be on a non-symlink-following mount. | | Infinite load depth | Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget. | --- ## Changeset Plan | CS | Scope | Files | |----|-------|-------| | CS1 | Extraction | `packages.go`: `extractableRelPath()` adds `star/`, `*.star`. Remove `_starlark_script` injection from installer. Add entry point validation. | | CS2 | Sandbox | `sandbox.go`: `ExecWithLoader()`. `LoadFunc` type. Existing `Exec()` wraps with nil loader. | | CS3 | Runner | `runner.go`: `packagesDir` field + setter. `loadScript()` disk-first. `packageLoader()` with security checks. `ExecPackage()` uses `ExecWithLoader()`. | | CS4 | Wiring | `main.go`: `SetPackagesDir()`. `build.sh`: add `star/` to archive. | | CS5 | Test | Integration test: install package with `script.star` + `star/` submodules, call `on_request`, verify `load()` resolved correctly. Install package with only `_starlark_script` in manifest, verify legacy path. | CS1-CS4 can land in a single session. CS5 validates. All five are backend-only — no FE changes. --- ## What This Enables - **Library packages** — a `gitea-client` with `script.star` as entry point, `star/repos.star`, `star/issues.star`, `star/ci.star` as submodules. Clean separation of concerns. - **Large extensions** — a workflow automation package with 20+ tool handlers doesn't have to be one 2000-line file. - **Shared utilities** — within a single package, common helpers (auth, validation, formatting) live in their own file and are loaded by multiple entry points. - **Testable modules** — each `.star` file can be loaded and tested independently in a dev sandbox. This does NOT enable cross-package loading. Package A cannot load Package B's files. That's what `lib.load()` is for (see DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level `load()` is intra-package. `lib.load()` is inter-package. Different mechanisms, different trust boundaries.