Feat v0.7.12 batch exec (#66)
All checks were successful
CI/CD / detect-changes (push) Successful in 4s
CI/CD / test-runners (push) Has been skipped
CI/CD / e2e-smoke (push) Has been skipped
CI/CD / test-go-pg (push) Successful in 2m46s
CI/CD / test-sqlite (push) Successful in 2m58s
CI/CD / build-and-deploy (push) Successful in 1m27s
CI/CD / test-frontend (push) Successful in 5s

Co-authored-by: Jeffrey Smith <jasafpro@gmail.com>
Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
This commit was merged in pull request #66.
This commit is contained in:
2026-04-02 23:57:46 +00:00
committed by xcaliber
parent c2d52f50c5
commit 3b74774077
8 changed files with 1001 additions and 5 deletions

View File

@@ -2,6 +2,26 @@
All notable changes to Armature are documented here. All notable changes to Armature are documented here.
## v0.7.12 — Concurrent Execution Primitive
New `batch` sandbox module. Enables extensions to parallelize arbitrary
Starlark callables — including frozen library exports from `lib.require()`.
**batch module (permission: `batch.exec`)**
- `batch.exec(callables, timeout=10)` — runs up to 8 zero-arg callables concurrently, each in its own `starlark.Thread` with independent step budget. Returns `(results, errors)` tuple with ordered results. Per-branch timeout (130s, default 10).
- Nested `batch.exec()` calls are prohibited (prevents exponential goroutine growth).
- `lib.require()` not available inside branches — load libraries before the batch call.
- `print()` output from branches is discarded.
**Internal**
- New file: `sandbox/batch_module.go`.
- New permission constant: `ExtPermBatchExec`.
- Runner wiring: `buildModulesWithLibCtx` creates batch module when permission granted.
- Design doc: `docs/DESIGN-batch-exec.md`.
- 12 new tests (parallel ordering, partial failure, timeout, cancellation, cap enforcement, nesting prevention, frozen sharing, permission gating).
## v0.7.11 — Query & HTTP Ergonomics ## v0.7.11 — Query & HTTP Ergonomics
Four new Starlark sandbox builtins. No new permissions, no schema changes. Four new Starlark sandbox builtins. No new permissions, no schema changes.

View File

@@ -1,6 +1,6 @@
# Armature — Roadmap # Armature — Roadmap
## Current: v0.7.11Query & HTTP Ergonomics ## Current: v0.7.12Concurrent Execution Primitive
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
sandbox, storage, realtime, and ops are kernel primitives. Everything else sandbox, storage, realtime, and ops are kernel primitives. Everything else
@@ -371,10 +371,10 @@ fresh module instances (`db`, `http`, etc.) — same pattern as
| Step | Status | Description | | Step | Status | Description |
|------|--------|-------------| |------|--------|-------------|
| Design doc | | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. | | Design doc | done | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. |
| `batch.exec()` | | `batch.exec(callables)``(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. | | `batch.exec()` | done | `batch.exec(callables)``(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. |
| Runner wiring | | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. | | Runner wiring | done | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. |
| Tests | | Unit tests: parallel execution ordering, partial failure, cap enforcement, frozen library sharing, permission gating. | | Tests | done | 12 unit tests: parallel execution ordering, partial failure, timeout per-branch, parent cancellation, cap enforcement, empty list, non-callable, permission gating, frozen struct sharing, nested batch.exec, default timeout, timeout clamping. All pass with `-race`. |
--- ---

466
docs/DESIGN-batch-exec.md Normal file
View File

@@ -0,0 +1,466 @@
# DESIGN — Concurrent Execution Primitive (`batch.exec`)
**Version:** v0.7.12
**Status:** Implemented
**Author:** Jeff / Claude session 2026-04-02
---
## Problem
Starlark is single-threaded by design (`go.starlark.net` enforces one
thread per execution). Extensions that need to fan out — calling multiple
external APIs, invoking several library functions, or performing independent
I/O operations — must do so sequentially. For two `http.post()` calls
taking 200ms each, the extension blocks for 400ms regardless of whether
the calls are independent.
The v0.7.10 `http.batch()` primitive solves the narrow case of parallel
HTTP dispatch. But it doesn't help when the work is wrapped in library
functions. If a `jira-client` library exposes `create_issue()` and a
`confluence-client` library exposes `create_page()`, the extension author
must either:
1. Call them sequentially (slow), or
2. Decompose the library calls back into raw `http.post()` parameters
to use `http.batch()` (defeats the purpose of having libraries).
The platform needs a general-purpose concurrent execution primitive that
works with arbitrary Starlark callables — including library exports.
---
## Non-Goals
- **Shared mutable state between branches.** Each concurrent branch is
fully isolated. No channels, no mutexes, no shared dicts. If branches
need to coordinate, they don't belong in `batch.exec`.
- **Unlimited concurrency.** A hard cap prevents extensions from spawning
unbounded goroutines. This is a fan-out primitive, not a thread pool.
- **Automatic retry or circuit breaking.** Error handling is the caller's
responsibility. The kernel reports per-branch results and errors.
- **Nested `batch.exec()`.** A callable inside `batch.exec` cannot itself
call `batch.exec`. This prevents exponential goroutine growth and keeps
the concurrency model flat.
---
## Key Insight: Frozen Libraries Are Thread-Safe
The reason this works without exotic machinery is `lib.require()`.
When a library is loaded via `lib.require()`, its exports are wrapped in
a `starlarkstruct.FromStringDict()` — which produces a **frozen** struct.
Frozen Starlark values are immutable and safe to read from any number of
goroutines concurrently. This is a property of `go.starlark.net`, not
something we enforce.
The only mutable state in a Starlark execution is:
1. **The `starlark.Thread` itself** — step counter, print buffer, cancel
channel. Each branch gets its own thread.
2. **Module instances**`db`, `http`, `settings`, etc. contain
configuration and hold references to shared Go objects (`*sql.DB`,
`http.Client`). Each branch gets fresh module instances, but the
underlying Go resources (`*sql.DB` connection pool, etc.) are already
designed for concurrent access.
3. **Local variables** — thread-local by definition in Starlark.
So the construction is: one new `starlark.Thread` + one new module set
per branch, with frozen library structs shared read-only across all
branches. This is exactly what `triggers/schedule.go` already does for
scheduled task execution — `buildRestrictedModules` creates a fresh
module set for each cron fire. `batch.exec` generalizes that pattern.
---
## API
```python
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
lambda: slack.post_message(channel, msg),
])
# results[0] = return value of jira.create_issue(), or None on error
# errors[0] = None on success, or error string on failure
# All three ran concurrently.
```
### Signature
```
batch.exec(callables, timeout=10) → (results: list, errors: list)
```
**Parameters:**
| Param | Type | Description |
|-------|------|-------------|
| `callables` | `list[callable]` | Starlark callables (lambdas, named functions, bound methods). Max length: 8. |
| `timeout` | `int` (optional) | Per-branch timeout in seconds. Default 10. Max 30. Inherits parent context deadline if shorter. |
**Returns:** A 2-tuple of equal-length lists.
- `results[i]` — the return value of `callables[i]`, or `None` if it
errored.
- `errors[i]``None` if `callables[i]` succeeded, or a string error
message if it failed (timeout, step limit, runtime error).
**Errors (whole-call):**
- `callables` is empty → error
- `callables` length > 8 → error
- Any element is not callable → error
- Permission `batch.exec` not granted → error
### Permission
New extension permission: `batch.exec`. Declared in manifest:
```json
{
"permissions": ["batch.exec"]
}
```
This is a separate permission because concurrent execution has resource
implications (goroutines, module construction overhead). Extensions that
don't need it shouldn't pay for it. The permission doesn't imply any
other permissions — the branch inherits whatever modules the calling
package already has.
---
## Execution Model
```
batch.exec([fn_a, fn_b, fn_c])
├─── goroutine 1: Thread₁ + Modules₁ → fn_a() → result[0]
├─── goroutine 2: Thread₂ + Modules₂ → fn_b() → result[1]
└─── goroutine 3: Thread₃ + Modules₃ → fn_c() → result[2]
sync.WaitGroup.Wait()
return (results, errors)
```
### Per-Branch Construction
For each callable in the input list, the kernel:
1. Creates a new `sandbox.Sandbox` with the same `Config` as the parent
(same `MaxSteps` limit — each branch gets its own step budget, not a
shared one).
2. Calls `runner.buildModulesWithLibCtx()` with the **same** `packageID`,
`manifest`, and `RunContext` as the parent invocation. This produces
a fresh module set — new `DBModuleConfig`, new `HTTPModuleConfig`, etc.
— pointing at the same underlying Go resources (`*sql.DB`, etc.).
3. The `libContext` is **shared** (read path only — cached frozen exports).
Library exports are immutable. The `loading` map (cycle detection) is
not relevant because libraries are already loaded before `batch.exec`
runs. If a branch triggers a new `lib.require()`, it would need its
own `libContext` — see Open Questions.
4. Creates a new `starlark.Thread` with the branch's print handler,
step limit, and context-based cancellation.
5. Calls `starlark.Call(thread, callable, nil, nil)` — the callable
is a zero-arg lambda that closes over its arguments.
### Context & Cancellation
Each branch gets a child context derived from the parent with the
per-branch timeout applied:
```go
branchCtx, cancel := context.WithTimeout(parentCtx, branchTimeout)
defer cancel()
```
If the parent context is cancelled (e.g., HTTP request timeout), all
branches are cancelled. If one branch exceeds its timeout, only that
branch is cancelled — others continue.
### Goroutine Cap
Hard limit: **8 concurrent branches.** This is enforced at the API
boundary (list length check), not via a semaphore. Rationale:
- 8 covers the real-world fan-out patterns (2-5 API calls, small batch
operations). Nobody needs 50 concurrent Starlark branches.
- Each branch allocates a `starlark.Thread` + module instances. At 8
branches, overhead is bounded at ~8KB per thread + module construction
time (~50μs per module set).
- No semaphore means no queuing surprises. You get 8, period.
---
## Implementation
### New File: `sandbox/batch_module.go`
```go
// BuildBatchModule creates the "batch" module.
// Requires the Runner reference for per-branch module construction.
func BuildBatchModule(
ctx context.Context,
runner *Runner,
packageID string,
manifest map[string]any,
rc *RunContext,
lc *libContext,
) *starlarkstruct.Module
```
The module holds a reference to the `Runner` — same pattern as
`BuildLibModule`. It needs the runner to call `buildModulesWithLibCtx`
for each branch.
### Core Implementation Sketch
```go
func batchExec(ctx context.Context, runner *Runner, packageID string,
manifest map[string]any, rc *RunContext, parentLC *libContext,
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
return func(thread *starlark.Thread, b *starlark.Builtin,
args starlark.Tuple, kwargs []starlark.Tuple,
) (starlark.Value, error) {
var callableList *starlark.List
var timeout int = 10
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
"callables", &callableList,
"timeout?", &timeout,
); err != nil {
return nil, err
}
n := callableList.Len()
if n == 0 {
return nil, fmt.Errorf("batch.exec: callables list is empty")
}
if n > 8 {
return nil, fmt.Errorf("batch.exec: max 8 callables, got %d", n)
}
if timeout < 1 || timeout > 30 {
timeout = 10
}
// Validate all elements are callable.
callables := make([]starlark.Callable, n)
for i := 0; i < n; i++ {
c, ok := callableList.Index(i).(starlark.Callable)
if !ok {
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
i, callableList.Index(i).Type())
}
callables[i] = c
}
// Execute concurrently.
type branchResult struct {
index int
value starlark.Value
err error
}
results := make([]starlark.Value, n)
errors := make([]starlark.Value, n)
var wg sync.WaitGroup
for i, callable := range callables {
wg.Add(1)
go func(idx int, fn starlark.Callable) {
defer wg.Done()
// Per-branch context with timeout.
branchCtx, cancel := context.WithTimeout(ctx,
time.Duration(timeout)*time.Second)
defer cancel()
// Fresh module set for this branch.
modules, err := runner.buildModulesWithLibCtx(
branchCtx, packageID, manifest, rc, parentLC)
if err != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(err.Error())
return
}
// Fresh sandbox + thread.
sb := New(DefaultConfig())
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
if callErr != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(callErr.Error())
} else {
results[idx] = val
errors[idx] = starlark.None
}
}(i, callable)
}
wg.Wait()
return starlark.Tuple{
starlark.NewList(results),
starlark.NewList(errors),
}, nil
}
}
```
### Runner Wiring
In `buildModulesWithLibCtx`, add the permission case:
```go
case models.ExtPermBatchExec:
// Deferred — wired after module map is complete (needs runner ref).
hasBatchExec = true
```
After the module map is assembled:
```go
if hasBatchExec {
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
}
```
### Permission Constant
In `models/permissions.go`:
```go
ExtPermBatchExec = "batch.exec"
```
Add to `AllExtensionPermissions` slice.
---
## Callable Closure Semantics
The callables passed to `batch.exec` are typically lambdas that close
over variables from the calling scope:
```python
issue_data = {"summary": "Review Q3 report"}
page_data = {"title": "Q3 Report", "body": content}
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
])
```
The closed-over values (`issue_data`, `page_data`, `jira`, `confluence`)
are references to Starlark values in the calling thread's scope. Two
safety properties make this work:
1. **Library exports (`jira`, `confluence`) are frozen.** They were
returned by `lib.require()` as `starlarkstruct.FromStringDict()`
deeply immutable. Safe to read from any goroutine.
2. **Dict/list arguments may be mutable**, but Starlark's execution
model means the calling thread is **blocked** waiting for
`batch.exec` to return. No concurrent mutation is possible because
the caller can't execute while the branches are running.
This is the same safety model as Go's `sync.WaitGroup` pattern: the
goroutine that calls `wg.Wait()` cannot proceed until all goroutines
complete, so values passed to goroutines before `wg.Add` are safe to
read without locks.
---
## Open Questions
### 1. `lib.require()` Inside Branches
If a callable triggers a `lib.require()` that hasn't been cached yet,
the shared `libContext.cache` would be written from a goroutine. Options:
**A. Prohibit: branches cannot call `lib.require()`.** The branch gets
a nil `libContext`, so `lib` module is unavailable inside `batch.exec`.
Libraries must be loaded before the batch call. Simplest, safest.
**B. Per-branch `libContext` with shared read cache.** Each branch gets
its own `libContext` whose `cache` is pre-populated from the parent's
cache (snapshot). New loads go into the branch's local cache only.
Slightly wasteful if two branches load the same library (loaded twice),
but safe.
**C. Mutex-protected shared `libContext`.** Add a `sync.RWMutex` to
`libContext`. Reads use `RLock`, writes use `Lock`. Minimal overhead,
but makes `libContext` aware of concurrency — violates its current
assumptions.
**Recommendation: Option A for v0.7.11, Option B as follow-up if needed.**
In practice, extensions call `lib.require()` at module scope (top of
script), not inside request handlers. The lambdas passed to `batch.exec`
call methods on already-loaded library structs. Option A covers all
real-world patterns.
### 2. Print Output
Each branch has its own print buffer (the `output strings.Builder` in
`Sandbox.Call`). Options:
**A. Discard.** Branch print output is lost. Simple, avoids interleaving.
**B. Collect per-branch.** Return a third list: `(results, errors, outputs)`.
Useful for debugging but clutters the API.
**C. Merge into parent.** Append all branch output to the parent thread's
print buffer, prefixed with branch index. Requires passing the parent's
`outputMu` and `output` builder — invasive.
**Recommendation: Option A for v0.7.11.** `print()` in Starlark is a
debugging tool, not a production logging facility. Branch callables that
need to report status should return structured data. If debugging demand
emerges, Option B is a backward-compatible addition.
### 3. Step Limit Scope
Each branch gets its own `MaxSteps` budget (default 1M). Should the
total across all branches be capped?
**No.** The per-branch cap is sufficient. 8 branches × 1M steps = 8M
total, which completes in under a second on any modern hardware. The
wall-clock timeout (per-branch, max 30s) is the real resource guard.
Adding a cross-branch step budget creates coupling between independent
execution paths — branch A's step count shouldn't affect branch B's
ability to complete.
---
## Testing
| Test | Description |
|------|-------------|
| Parallel ordering | 3 callables with different sleep durations. Results in input order, not completion order. |
| Partial failure | 3 callables, middle one raises error. results = [val, None, val], errors = [None, "err msg", None]. |
| Timeout per-branch | One callable sleeps beyond timeout. Others succeed. Timed-out branch returns error. |
| Parent cancellation | Cancel parent context mid-execution. All branches cancelled. |
| Cap enforcement | List of 9 callables → immediate error, nothing executed. |
| Empty list | `batch.exec([])` → error. |
| Non-callable element | `batch.exec([1, 2])` → error, nothing executed. |
| Permission gating | Package without `batch.exec` permission → module not available. |
| Frozen library sharing | Two branches call same frozen library function concurrently. No race. |
| Nested batch.exec | Callable inside batch.exec attempts batch.exec → error (module not injected in branch). |
| db module isolation | Two branches insert into same table concurrently. Both succeed, no corruption. |
| http module isolation | Two branches make HTTP calls with different headers. No cross-contamination. |
---
## Migration
No schema changes. No new tables. No new migrations.
New permission constant `batch.exec` added to `models/permissions.go`.
Extensions must declare the permission in their manifest to access the
`batch` module.

View File

@@ -242,6 +242,39 @@ instances = workflow.list_instances(workflow_id, status="active")
# Returns list of instance dicts # Returns list of instance dicts
``` ```
### batch
**Permission:** `batch.exec`
Run multiple callables concurrently. Each callable gets its own
execution thread with an independent step budget.
```python
jira = lib.require("jira-client")
confluence = lib.require("confluence-client")
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
lambda: send_notification(user_id),
], timeout=15)
# results[i] = return value of callables[i], or None on error
# errors[i] = None on success, or error string on failure
# All three ran concurrently.
```
**Constraints:**
- Max 8 callables per call. Dispatched concurrently via goroutines.
- `timeout` (optional): 130 seconds per branch (default 10).
- `lib.require()` is not available inside branch callables.
Load libraries before the `batch.exec` call.
- `batch.exec()` cannot be called from within a branch (no nesting).
- `print()` output from branches is discarded.
---
## Example: automated stage hook ## Example: automated stage hook
A simple hook that reads a setting, queries data, and advances: A simple hook that reads a setting, queries data, and advances:

View File

@@ -36,6 +36,7 @@ const (
ExtPermConnectionsRead = "connections.read" ExtPermConnectionsRead = "connections.read"
ExtPermTriggersRegister = "triggers.register" ExtPermTriggersRegister = "triggers.register"
ExtPermRealtimePublish = "realtime.publish" ExtPermRealtimePublish = "realtime.publish"
ExtPermBatchExec = "batch.exec"
) )
// ValidExtensionPermissions is the set of recognized permission keys. // ValidExtensionPermissions is the set of recognized permission keys.
@@ -50,6 +51,7 @@ var ValidExtensionPermissions = map[string]bool{
ExtPermConnectionsRead: true, ExtPermConnectionsRead: true,
ExtPermTriggersRegister: true, ExtPermTriggersRegister: true,
ExtPermRealtimePublish: true, ExtPermRealtimePublish: true,
ExtPermBatchExec: true,
} }
// ── Extension Permission Model ─────────────── // ── Extension Permission Model ───────────────

View File

@@ -0,0 +1,140 @@
// Package sandbox — batch_module.go
//
// Permission: batch.exec
//
// Starlark API:
// results, errors = batch.exec([fn_a, fn_b, fn_c], timeout=10)
//
// Runs a list of zero-arg callables concurrently, each in its own
// starlark.Thread with an independent step budget. Results and errors
// are returned in input order. Max 8 callables per call.
//
// Each branch gets a fresh Sandbox (thread + step counter) and a
// child context with the per-branch timeout. The callable's closed-over
// bindings (frozen library structs, module builtins) are used as-is —
// the Go resources underneath (*sql.DB, http.Client) are concurrent-safe.
//
// lib.require() is not available inside branches (Option A).
// print() output from branches is discarded.
// Nested batch.exec() is prohibited via an atomic flag.
package sandbox
import (
"context"
"fmt"
"sync"
"sync/atomic"
"time"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
)
const (
batchMaxCallables = 8
batchDefaultTimeout = 10
batchMaxTimeout = 30
)
// BuildBatchModule creates the "batch" module.
// The runner/packageID/manifest/rc/lc parameters are stored for future
// expansion (Option B: per-branch lib.require) but are not used in v1.
func BuildBatchModule(
ctx context.Context,
runner *Runner,
packageID string,
manifest map[string]any,
rc *RunContext,
lc *libContext,
) *starlarkstruct.Module {
var running atomic.Bool
return MakeModule("batch", starlark.StringDict{
"exec": starlark.NewBuiltin("batch.exec", batchExec(ctx, &running)),
})
}
// batchExec returns the batch.exec builtin implementation.
func batchExec(
ctx context.Context,
running *atomic.Bool,
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
return func(
thread *starlark.Thread, b *starlark.Builtin,
args starlark.Tuple, kwargs []starlark.Tuple,
) (starlark.Value, error) {
// Nesting guard: parent thread is blocked in wg.Wait(), so only
// a branch goroutine could re-enter. The captured builtin reference
// makes this reachable even though branches don't get fresh modules.
if !running.CompareAndSwap(false, true) {
return nil, fmt.Errorf("batch.exec: nested calls are not allowed")
}
defer running.Store(false)
var callableList *starlark.List
var timeout int = batchDefaultTimeout
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
"callables", &callableList,
"timeout?", &timeout,
); err != nil {
return nil, err
}
n := callableList.Len()
if n == 0 {
return nil, fmt.Errorf("batch.exec: callables list is empty")
}
if n > batchMaxCallables {
return nil, fmt.Errorf("batch.exec: max %d callables, got %d", batchMaxCallables, n)
}
if timeout < 1 || timeout > batchMaxTimeout {
timeout = batchDefaultTimeout
}
// Validate all elements are callable before spawning goroutines.
callables := make([]starlark.Callable, n)
for i := 0; i < n; i++ {
c, ok := callableList.Index(i).(starlark.Callable)
if !ok {
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
i, callableList.Index(i).Type())
}
callables[i] = c
}
// Execute concurrently.
results := make([]starlark.Value, n)
errors := make([]starlark.Value, n)
var wg sync.WaitGroup
for i, callable := range callables {
wg.Add(1)
go func(idx int, fn starlark.Callable) {
defer wg.Done()
branchCtx, cancel := context.WithTimeout(ctx,
time.Duration(timeout)*time.Second)
defer cancel()
sb := New(DefaultConfig())
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
if callErr != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(callErr.Error())
} else {
results[idx] = val
errors[idx] = starlark.None
}
}(i, callable)
}
wg.Wait()
return starlark.Tuple{
starlark.NewList(results),
starlark.NewList(errors),
}, nil
}
}

View File

@@ -0,0 +1,324 @@
package sandbox
import (
"context"
"strings"
"testing"
"time"
"go.starlark.net/starlark"
)
// execWithBatch runs a Starlark script with the batch module available.
func execWithBatch(t *testing.T, script string) (*Result, error) {
t.Helper()
ctx := context.Background()
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
sb := New(DefaultConfig())
return sb.Exec(ctx, "test.star", script, map[string]starlark.Value{
"batch": batchMod,
})
}
// ─── Parallel Ordering ──────────────────────
func TestBatchExec_ParallelOrdering(t *testing.T) {
result, err := execWithBatch(t, `
results, errors = batch.exec([
lambda: 10,
lambda: 20,
lambda: 30,
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
// Verify results are in input order.
globals := result.Globals
results := globals["results"].(*starlark.List)
errors := globals["errors"].(*starlark.List)
if results.Len() != 3 {
t.Fatalf("expected 3 results, got %d", results.Len())
}
for i, want := range []int{10, 20, 30} {
v, _ := starlark.AsInt32(results.Index(i))
got := int(v)
if got != want {
t.Errorf("results[%d] = %d, want %d", i, got, want)
}
if errors.Index(i) != starlark.None {
t.Errorf("errors[%d] = %v, want None", i, errors.Index(i))
}
}
}
// ─── Partial Failure ────────────────────────
func TestBatchExec_PartialFailure(t *testing.T) {
result, err := execWithBatch(t, `
def ok_a():
return "a"
def bad():
return 1 // 0
def ok_c():
return "c"
results, errors = batch.exec([ok_a, bad, ok_c])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
errors := result.Globals["errors"].(*starlark.List)
// results[0] = "a", results[2] = "c"
if s, _ := starlark.AsString(results.Index(0)); s != "a" {
t.Errorf("results[0] = %v, want 'a'", results.Index(0))
}
if s, _ := starlark.AsString(results.Index(2)); s != "c" {
t.Errorf("results[2] = %v, want 'c'", results.Index(2))
}
// results[1] = None (failed)
if results.Index(1) != starlark.None {
t.Errorf("results[1] = %v, want None", results.Index(1))
}
// errors[0] and errors[2] = None (succeeded)
if errors.Index(0) != starlark.None {
t.Errorf("errors[0] = %v, want None", errors.Index(0))
}
if errors.Index(2) != starlark.None {
t.Errorf("errors[2] = %v, want None", errors.Index(2))
}
// errors[1] should be a string containing the error
errStr, ok := starlark.AsString(errors.Index(1))
if !ok || errStr == "" {
t.Errorf("errors[1] = %v, want non-empty error string", errors.Index(1))
}
}
// ─── Timeout Per-Branch ─────────────────────
func TestBatchExec_TimeoutPerBranch(t *testing.T) {
// Use a tight timeout. The slow lambda burns steps in an infinite loop,
// which will hit either the step limit or the context timeout.
result, err := execWithBatch(t, `
def fast():
return "done"
def slow():
x = 0
for i in range(999999999):
x = x + 1
return x
results, errors = batch.exec([fast, slow], timeout=1)
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
errors := result.Globals["errors"].(*starlark.List)
// Fast branch should succeed.
if s, _ := starlark.AsString(results.Index(0)); s != "done" {
t.Errorf("results[0] = %v, want 'done'", results.Index(0))
}
if errors.Index(0) != starlark.None {
t.Errorf("errors[0] = %v, want None", errors.Index(0))
}
// Slow branch should fail (step limit or timeout).
if results.Index(1) != starlark.None {
t.Errorf("results[1] = %v, want None", results.Index(1))
}
errStr, _ := starlark.AsString(errors.Index(1))
if errStr == "" {
t.Errorf("errors[1] should contain error string")
}
}
// ─── Parent Cancellation ────────────────────
func TestBatchExec_ParentCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel() // cancel immediately
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
sb := New(DefaultConfig())
result, err := sb.Exec(ctx, "test.star", `
results, errors = batch.exec([lambda: 1, lambda: 2])
`, map[string]starlark.Value{"batch": batchMod})
// Either the Exec itself fails (context cancelled before script runs)
// or the batch.exec branches all fail.
if err != nil {
// Script-level cancellation — acceptable.
return
}
errors := result.Globals["errors"].(*starlark.List)
allErrored := true
for i := 0; i < errors.Len(); i++ {
if errors.Index(i) == starlark.None {
allErrored = false
}
}
if !allErrored {
t.Errorf("expected all branches to error on cancelled context, errors=%v", errors)
}
}
// ─── Cap Enforcement ────────────────────────
func TestBatchExec_CapEnforcement(t *testing.T) {
_, err := execWithBatch(t, `
fns = [lambda: i for i in range(9)]
batch.exec(fns)
`)
if err == nil {
t.Fatal("expected error for 9 callables, got nil")
}
if !strings.Contains(err.Error(), "max 8") {
t.Errorf("error = %q, want to contain 'max 8'", err.Error())
}
}
// ─── Empty List ─────────────────────────────
func TestBatchExec_EmptyList(t *testing.T) {
_, err := execWithBatch(t, `batch.exec([])`)
if err == nil {
t.Fatal("expected error for empty list, got nil")
}
if !strings.Contains(err.Error(), "empty") {
t.Errorf("error = %q, want to contain 'empty'", err.Error())
}
}
// ─── Non-Callable Element ───────────────────
func TestBatchExec_NonCallableElement(t *testing.T) {
_, err := execWithBatch(t, `batch.exec([1, 2])`)
if err == nil {
t.Fatal("expected error for non-callable, got nil")
}
if !strings.Contains(err.Error(), "not callable") {
t.Errorf("error = %q, want to contain 'not callable'", err.Error())
}
}
// ─── Permission Gating ──────────────────────
func TestBatchExec_PermissionGating(t *testing.T) {
sb := New(DefaultConfig())
_, err := sb.Exec(context.Background(), "test.star",
`batch.exec([lambda: 1])`,
map[string]starlark.Value{}, // no batch module
)
if err == nil {
t.Fatal("expected error when batch module not available")
}
if !strings.Contains(err.Error(), "batch") {
t.Errorf("error = %q, want to mention 'batch'", err.Error())
}
}
// ─── Frozen Struct Sharing ──────────────────
func TestBatchExec_FrozenStructSharing(t *testing.T) {
// Two branches read the same frozen tuple. Should not race.
result, err := execWithBatch(t, `
data = (1, 2, 3) # tuples are frozen/immutable
results, errors = batch.exec([
lambda: data[0] + data[1],
lambda: data[1] + data[2],
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
got0, _ := starlark.AsInt32(results.Index(0))
got1, _ := starlark.AsInt32(results.Index(1))
if got0 != 3 {
t.Errorf("results[0] = %d, want 3", got0)
}
if got1 != 5 {
t.Errorf("results[1] = %d, want 5", got1)
}
}
// ─── Nested batch.exec ──────────────────────
func TestBatchExec_NestedBatchExec(t *testing.T) {
result, err := execWithBatch(t, `
results, errors = batch.exec([
lambda: batch.exec([lambda: 1]),
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
// The inner batch.exec should fail with nesting error.
errors := result.Globals["errors"].(*starlark.List)
errStr, _ := starlark.AsString(errors.Index(0))
if !strings.Contains(errStr, "nested") {
t.Errorf("errors[0] = %q, want to contain 'nested'", errStr)
}
}
// ─── Default Timeout ────────────────────────
func TestBatchExec_DefaultTimeout(t *testing.T) {
// Call without explicit timeout — should succeed.
result, err := execWithBatch(t, `
results, errors = batch.exec([lambda: 42])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
got, _ := starlark.AsInt32(results.Index(0))
if got != 42 {
t.Errorf("results[0] = %d, want 42", got)
}
}
// ─── Timeout Clamping ───────────────────────
func TestBatchExec_TimeoutClamp(t *testing.T) {
// timeout=0 should be clamped to default, not crash.
start := time.Now()
result, err := execWithBatch(t, `
results, errors = batch.exec([lambda: "ok"], timeout=0)
`)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
if s, _ := starlark.AsString(results.Index(0)); s != "ok" {
t.Errorf("results[0] = %v, want 'ok'", results.Index(0))
}
// Should complete quickly (clamped to default 10s, but lambda is instant).
if elapsed > 2*time.Second {
t.Errorf("took %v, expected fast completion", elapsed)
}
}

View File

@@ -338,6 +338,7 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
// Track db permission level: 0=none, 1=read, 2=write // Track db permission level: 0=none, 1=read, 2=write
dbLevel := 0 dbLevel := 0
hasBatchExec := false
for _, perm := range granted { for _, perm := range granted {
switch perm { switch perm {
@@ -374,6 +375,9 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
if r.bus != nil { if r.bus != nil {
modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID) modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID)
} }
case models.ExtPermBatchExec:
hasBatchExec = true
} }
} }
@@ -404,5 +408,12 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc) modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc)
} }
// Concurrent execution primitive — wired after all other modules
// so the runner reference can construct per-branch module sets if
// needed in future (Option B).
if hasBatchExec {
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
}
return modules, nil return modules, nil
} }