core/docs/DESIGN-MULTI-FILE-STARLARK.md

DESIGN — Multi-File Starlark Packages

Promotes Starlark scripts from a single inline blob to a proper file tree with load() support. Prerequisite for library packages.

---

Current State

Three problems:

1. Scripts live in the DB. The installer injects script.star contents into manifest["_starlark_script"]. The runner reads the script from the manifest JSONB column at runtime. This means the entire script — no matter how large — is serialized into a JSON string inside a JSON object inside a database row.

2. load() is disabled. The sandbox callback hard-returns an error. There's no way to split code across files.

3. Only static assets are extracted. extractableRelPath() allows js/, css/, assets/ — nothing else. Starlark files land in the DB, not on disk.

The hotfix in v0.37.14 (reading script.star from the archive and injecting it as _starlark_script) made single-file clean but didn't fix the underlying architecture.

---

Design

Archive Structure

gitea-client/
├── manifest.json          ← metadata only, no _starlark_script
├── script.star            ← entry point (required for starlark tier)
├── star/                  ← optional: additional modules
│   ├── repos.star
│   ├── issues.star
│   ├── auth.star
│   └── ci.star
├── js/                    ← optional: surface assets
│   └── main.js
└── css/
    └── main.css

Convention: script.star is always the entry point. Submodules live in star/. This parallels js/main.js as the JS entry point with supporting files alongside it.

Manifest Change

The manifest gains an optional entry_point field for packages that want a different entry script name. Default: script.star.

{
  "id": "gitea-client",
  "type": "library",
  "tier": "starlark",
  "entry_point": "script.star"
}

_starlark_script is no longer written to the manifest. The field is still read for backward compatibility (existing packages that have it in their DB row continue to work until reinstalled).

Extraction

extractableRelPath() adds star/ and bare *.star files:

var staticPrefixes = []string{"js/", "css/", "assets/", "star/"}

func extractableRelPath(name string) string {
    // Existing prefix matching for directories...

    // Also extract bare .star files at archive root
    base := filepath.Base(name)
    if strings.HasSuffix(base, ".star") {
        // Strip leading directory (package-id/) if present
        if idx := strings.Index(name, "/"); idx >= 0 {
            rest := name[idx+1:]
            if rest == base || strings.HasPrefix(rest, "star/") {
                return rest
            }
        }
        if name == base {
            return name
        }
    }

    return ""
}

After install, the packages directory looks like:

/data/packages/gitea-client/
├── script.star
├── star/
│   ├── repos.star
│   ├── issues.star
│   ├── auth.star
│   └── ci.star
├── js/
│   └── main.js
└── css/
    └── main.css

Installer Changes

packages.go InstallPackage():

1. Stop injecting _starlark_script into the manifest. Remove the hotfix code that reads script.star from the archive and stuffs it into the manifest JSON.

2. Let extractableRelPath() handle .star files naturally — they get extracted to disk alongside js/ and css/.

3. Validate: if tier == "starlark", confirm the archive contains script.star (or the manifest's entry_point value). Return 400 if missing.

Runner Changes

runner.go ExecPackage():

Replace manifest-based script loading with file-based:

func (r *Runner) ExecPackage(ctx context.Context, pkg *store.PackageRegistration, rc *RunContext) (*Result, error) {
    if pkg.Status != models.PackageStatusActive {
        return nil, fmt.Errorf("package %q is %s, not active", pkg.ID, pkg.Status)
    }
    if pkg.Tier != models.ExtTierStarlark {
        return nil, fmt.Errorf("package %q is tier %s, not starlark", pkg.ID, pkg.Tier)
    }

    // ── Load script from disk (primary) or manifest (legacy) ──
    script, err := r.loadScript(pkg)
    if err != nil {
        return nil, err
    }

    modules, err := r.buildModules(ctx, pkg.ID, pkg.Manifest, rc)
    if err != nil {
        return nil, fmt.Errorf("failed to build modules for %q: %w", pkg.ID, err)
    }

    // Build package-scoped load callback
    loader := r.packageLoader(pkg.ID, modules)

    log.Printf("  🔧 runner: exec %s (%d modules granted)", pkg.ID, len(modules))

    return r.sandbox.ExecWithLoader(ctx, pkg.ID+"/script.star", script, modules, loader)
}

func (r *Runner) loadScript(pkg *store.PackageRegistration) (string, error) {
    // Primary: read from disk
    if r.packagesDir != "" {
        entryPoint := "script.star"
        if ep, ok := pkg.Manifest["entry_point"].(string); ok && ep != "" {
            entryPoint = ep
        }
        path := filepath.Join(r.packagesDir, pkg.ID, entryPoint)
        data, err := os.ReadFile(path)
        if err == nil && len(data) > 0 {
            return string(data), nil
        }
        // Fall through to legacy
    }

    // Legacy: inline in manifest
    script, ok := pkg.Manifest["_starlark_script"].(string)
    if ok && script != "" {
        return script, nil
    }

    return "", fmt.Errorf("package %q: no script.star on disk and no _starlark_script in manifest", pkg.ID)
}

Sandbox Changes

sandbox.go gains ExecWithLoader() — same as Exec() but accepts a load callback:

// LoadFunc resolves load("path") calls to Starlark source.
// Returns the module's globals. The sandbox caches results per
// thread (Starlark handles this via thread.Load dedup).
type LoadFunc func(thread *starlark.Thread, module string) (starlark.StringDict, error)

func (s *Sandbox) ExecWithLoader(ctx context.Context, filename, source string,
    modules map[string]starlark.Value, loader LoadFunc) (*Result, error) {

    // ... same as Exec() but thread.Load = loader instead of error ...
}

The existing Exec() becomes a thin wrapper that passes a nil/error loader for backward compatibility.

Package-Scoped Loader

runner.go builds the load callback scoped to the package's directory:

func (r *Runner) packageLoader(pkgID string, modules map[string]starlark.Value) LoadFunc {
    if r.packagesDir == "" {
        return nil // no disk = no load support
    }

    pkgDir := filepath.Join(r.packagesDir, pkgID)
    cache := make(map[string]*loadEntry) // dedup + cycle detection

    return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
        // ── Security: reject traversal ──
        if strings.Contains(module, "..") || filepath.IsAbs(module) {
            return nil, fmt.Errorf("load: path traversal not allowed: %q", module)
        }

        // ── Resolve to package directory ──
        resolved := filepath.Join(pkgDir, module)
        if !strings.HasPrefix(filepath.Clean(resolved), filepath.Clean(pkgDir)) {
            return nil, fmt.Errorf("load: path escapes package directory: %q", module)
        }

        // ── Must be a .star file ──
        if !strings.HasSuffix(resolved, ".star") {
            return nil, fmt.Errorf("load: only .star files can be loaded: %q", module)
        }

        // ── Cache / cycle detection ──
        if entry, ok := cache[module]; ok {
            if entry.loading {
                return nil, fmt.Errorf("load: circular dependency: %q", module)
            }
            return entry.globals, nil
        }

        entry := &loadEntry{loading: true}
        cache[module] = entry

        // ── Read and execute ──
        data, err := os.ReadFile(resolved)
        if err != nil {
            return nil, fmt.Errorf("load: %q not found in package %q", module, pkgID)
        }

        globals, err := starlark.ExecFile(thread, module, string(data),
            r.predeclaredForLoad(modules))
        if err != nil {
            return nil, fmt.Errorf("load: error in %q: %w", module, err)
        }

        entry.globals = globals
        entry.loading = false
        return globals, nil
    }
}

type loadEntry struct {
    loading bool
    globals starlark.StringDict
}

Key constraints:

• Package-scoped. Can only load files from within the package's own directory. Path traversal (..) and absolute paths are rejected. One package cannot load another package's files.

• .star files only. Cannot load .js, .json, or anything else. This is a Starlark sandbox, not a file reader.

• Circular dependency detection. If A loads B loads A → error.

• Cached per invocation. If script.star and repos.star both load auth.star, it executes once. Starlark's thread-level load dedup handles this, plus our explicit cache.

• Same predeclared modules. Loaded files get the same db, http, json, settings, connections modules as the entry point. They run in the same permission context.

Starlark Usage

# script.star (entry point)
load("star/auth.star", "auth_headers", "auth_headers_json")
load("star/repos.star", "get_repos", "sync_repos")
load("star/issues.star", "get_issues", "create_issue", "cache_issue")
load("star/ci.star", "get_ci_status")

def on_request(req):
    # ... dispatch to imported functions ...

def on_tool_call(tool_name, params):
    # ... dispatch to imported functions ...

# star/auth.star
def auth_headers(conn):
    return {"Authorization": "token " + conn.get("api_token", "")}

def auth_headers_json(conn):
    h = auth_headers(conn)
    h["Content-Type"] = "application/json"
    return h

# star/repos.star
load("star/auth.star", "auth_headers")

def get_repos(conn):
    url = conn["base_url"] + "/api/v1/repos/search?limit=50"
    resp = http.get(url=url, headers=auth_headers(conn))
    if int(resp["status"]) >= 400:
        return None
    return json.decode(resp["body"])

def sync_repos(conn):
    repos = get_repos(conn)
    # ... db.insert into private tables ...

Submodules can load other submodules. The dependency graph is a DAG (cycles are rejected).

Build Script

packages/build.sh already includes script.star. Add star/:

local dirs=""
[ -d "$dir/js" ]          && dirs="$dirs js/"
[ -d "$dir/css" ]         && dirs="$dirs css/"
[ -d "$dir/assets" ]      && dirs="$dirs assets/"
[ -f "$dir/script.star" ] && dirs="$dirs script.star"
[ -d "$dir/star" ]        && dirs="$dirs star/"
[ -d "$dir/migrations" ]  && dirs="$dirs migrations/"

Runner Constructor

runner.go gains packagesDir:

type Runner struct {
    sandbox     *Sandbox
    stores      store.Stores
    packagesDir string              // NEW
    notifier    NotificationSender
    resolver    ProviderResolver
    db          *sql.DB
    dbPostgres  bool
}

func NewRunner(sb *Sandbox, stores store.Stores) *Runner {
    return &Runner{sandbox: sb, stores: stores}
}

func (r *Runner) SetPackagesDir(dir string) {
    r.packagesDir = dir
}

main.go wires it:

starlarkRunner := sandbox.NewRunner(starlarkSandbox, stores)
starlarkRunner.SetPackagesDir(cfg.StoragePath + "/packages")

---

Backward Compatibility

Scenario	Behavior
Existing package with _starlark_script in manifest, no files on disk	Works. loadScript() falls through to legacy path.
Package installed with v0.37.14 hotfix (script.star injected into manifest)	Works. Same legacy path.
New package with script.star on disk, no _starlark_script in manifest	Works. Primary path.
New package with script.star + star/ submodules	Works. Loader resolves load() calls.
Package with both _starlark_script AND script.star on disk	Disk wins. Manifest is legacy fallback only.
Package that calls load() but has no files on disk	Error: "load() not available (no package directory)".

No migration needed. Old packages work as-is. New packages benefit from the file-based path. Reinstalling an old package extracts its script.star to disk (via the updated extractableRelPath).

---

Security

Threat	Mitigation
Path traversal (load("../../etc/passwd"))	.. rejected. filepath.Clean + prefix check against package dir.
Absolute paths (load("/etc/shadow"))	filepath.IsAbs check → rejected.
Loading non-Starlark files (load("js/main.js"))	.star suffix required.
Cross-package load (load("../other-pkg/script.star"))	.. rejected. Resolution is always relative to the package's own directory.
Circular loads (A→B→A)	loadEntry.loading flag → error on re-entry.
Symlink escape	filepath.Clean resolves symlinks. Production: packages dir should be on a non-symlink-following mount.
Infinite load depth	Starlark's thread step limit applies across all loaded files. A package that loads 100 files still runs under the same 1M step budget.

---

Changeset Plan

CS	Scope	Files
CS1	Extraction	packages.go: extractableRelPath() adds star/, *.star. Remove _starlark_script injection from installer. Add entry point validation.
CS2	Sandbox	sandbox.go: ExecWithLoader(). LoadFunc type. Existing Exec() wraps with nil loader.
CS3	Runner	runner.go: packagesDir field + setter. loadScript() disk-first. packageLoader() with security checks. ExecPackage() uses ExecWithLoader().
CS4	Wiring	main.go: SetPackagesDir(). build.sh: add star/ to archive.
CS5	Test	Integration test: install package with script.star + star/ submodules, call on_request, verify load() resolved correctly. Install package with only _starlark_script in manifest, verify legacy path.

CS1-CS4 can land in a single session. CS5 validates. All five are backend-only — no FE changes.

---

What This Enables

• Library packages — a gitea-client with script.star as entry point, star/repos.star, star/issues.star, star/ci.star as submodules. Clean separation of concerns.

• Large extensions — a workflow automation package with 20+ tool handlers doesn't have to be one 2000-line file.

• Shared utilities — within a single package, common helpers (auth, validation, formatting) live in their own file and are loaded by multiple entry points.

• Testable modules — each .star file can be loaded and tested independently in a dev sandbox.

This does NOT enable cross-package loading. Package A cannot load Package B's files. That's what lib.load() is for (see DESIGN-EXT-CONNECTIONS-LIBRARIES.md). File-level load() is intra-package. lib.load() is inter-package. Different mechanisms, different trust boundaries.