Feat v0.7.12 batch.exec concurrent execution primitive

Add batch.exec(callables, timeout=10) — a general-purpose concurrent
execution primitive that runs arbitrary Starlark callables in parallel,
each in its own thread with independent step budget. Enables extensions
to fan out library function calls (frozen exports from lib.require)
without decomposing them back into raw http.post parameters.

New permission: batch.exec. Max 8 callables, timeout 1-30s.
Nesting prohibited via atomic flag. 12 new tests, all pass with -race.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-02 23:52:01 +00:00
parent c2d52f50c5
commit 8957e99610
7 changed files with 981 additions and 5 deletions

View File

@@ -1,6 +1,6 @@
# Armature — Roadmap # Armature — Roadmap
## Current: v0.7.11Query & HTTP Ergonomics ## Current: v0.7.12Concurrent Execution Primitive
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
sandbox, storage, realtime, and ops are kernel primitives. Everything else sandbox, storage, realtime, and ops are kernel primitives. Everything else
@@ -371,10 +371,10 @@ fresh module instances (`db`, `http`, etc.) — same pattern as
| Step | Status | Description | | Step | Status | Description |
|------|--------|-------------| |------|--------|-------------|
| Design doc | | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. | | Design doc | done | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. |
| `batch.exec()` | | `batch.exec(callables)``(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. | | `batch.exec()` | done | `batch.exec(callables)``(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. |
| Runner wiring | | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. | | Runner wiring | done | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. |
| Tests | | Unit tests: parallel execution ordering, partial failure, cap enforcement, frozen library sharing, permission gating. | | Tests | done | 12 unit tests: parallel execution ordering, partial failure, timeout per-branch, parent cancellation, cap enforcement, empty list, non-callable, permission gating, frozen struct sharing, nested batch.exec, default timeout, timeout clamping. All pass with `-race`. |
--- ---

466
docs/DESIGN-batch-exec.md Normal file
View File

@@ -0,0 +1,466 @@
# DESIGN — Concurrent Execution Primitive (`batch.exec`)
**Version:** v0.7.12
**Status:** Implemented
**Author:** Jeff / Claude session 2026-04-02
---
## Problem
Starlark is single-threaded by design (`go.starlark.net` enforces one
thread per execution). Extensions that need to fan out — calling multiple
external APIs, invoking several library functions, or performing independent
I/O operations — must do so sequentially. For two `http.post()` calls
taking 200ms each, the extension blocks for 400ms regardless of whether
the calls are independent.
The v0.7.10 `http.batch()` primitive solves the narrow case of parallel
HTTP dispatch. But it doesn't help when the work is wrapped in library
functions. If a `jira-client` library exposes `create_issue()` and a
`confluence-client` library exposes `create_page()`, the extension author
must either:
1. Call them sequentially (slow), or
2. Decompose the library calls back into raw `http.post()` parameters
to use `http.batch()` (defeats the purpose of having libraries).
The platform needs a general-purpose concurrent execution primitive that
works with arbitrary Starlark callables — including library exports.
---
## Non-Goals
- **Shared mutable state between branches.** Each concurrent branch is
fully isolated. No channels, no mutexes, no shared dicts. If branches
need to coordinate, they don't belong in `batch.exec`.
- **Unlimited concurrency.** A hard cap prevents extensions from spawning
unbounded goroutines. This is a fan-out primitive, not a thread pool.
- **Automatic retry or circuit breaking.** Error handling is the caller's
responsibility. The kernel reports per-branch results and errors.
- **Nested `batch.exec()`.** A callable inside `batch.exec` cannot itself
call `batch.exec`. This prevents exponential goroutine growth and keeps
the concurrency model flat.
---
## Key Insight: Frozen Libraries Are Thread-Safe
The reason this works without exotic machinery is `lib.require()`.
When a library is loaded via `lib.require()`, its exports are wrapped in
a `starlarkstruct.FromStringDict()` — which produces a **frozen** struct.
Frozen Starlark values are immutable and safe to read from any number of
goroutines concurrently. This is a property of `go.starlark.net`, not
something we enforce.
The only mutable state in a Starlark execution is:
1. **The `starlark.Thread` itself** — step counter, print buffer, cancel
channel. Each branch gets its own thread.
2. **Module instances**`db`, `http`, `settings`, etc. contain
configuration and hold references to shared Go objects (`*sql.DB`,
`http.Client`). Each branch gets fresh module instances, but the
underlying Go resources (`*sql.DB` connection pool, etc.) are already
designed for concurrent access.
3. **Local variables** — thread-local by definition in Starlark.
So the construction is: one new `starlark.Thread` + one new module set
per branch, with frozen library structs shared read-only across all
branches. This is exactly what `triggers/schedule.go` already does for
scheduled task execution — `buildRestrictedModules` creates a fresh
module set for each cron fire. `batch.exec` generalizes that pattern.
---
## API
```python
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
lambda: slack.post_message(channel, msg),
])
# results[0] = return value of jira.create_issue(), or None on error
# errors[0] = None on success, or error string on failure
# All three ran concurrently.
```
### Signature
```
batch.exec(callables, timeout=10) → (results: list, errors: list)
```
**Parameters:**
| Param | Type | Description |
|-------|------|-------------|
| `callables` | `list[callable]` | Starlark callables (lambdas, named functions, bound methods). Max length: 8. |
| `timeout` | `int` (optional) | Per-branch timeout in seconds. Default 10. Max 30. Inherits parent context deadline if shorter. |
**Returns:** A 2-tuple of equal-length lists.
- `results[i]` — the return value of `callables[i]`, or `None` if it
errored.
- `errors[i]``None` if `callables[i]` succeeded, or a string error
message if it failed (timeout, step limit, runtime error).
**Errors (whole-call):**
- `callables` is empty → error
- `callables` length > 8 → error
- Any element is not callable → error
- Permission `batch.exec` not granted → error
### Permission
New extension permission: `batch.exec`. Declared in manifest:
```json
{
"permissions": ["batch.exec"]
}
```
This is a separate permission because concurrent execution has resource
implications (goroutines, module construction overhead). Extensions that
don't need it shouldn't pay for it. The permission doesn't imply any
other permissions — the branch inherits whatever modules the calling
package already has.
---
## Execution Model
```
batch.exec([fn_a, fn_b, fn_c])
├─── goroutine 1: Thread₁ + Modules₁ → fn_a() → result[0]
├─── goroutine 2: Thread₂ + Modules₂ → fn_b() → result[1]
└─── goroutine 3: Thread₃ + Modules₃ → fn_c() → result[2]
sync.WaitGroup.Wait()
return (results, errors)
```
### Per-Branch Construction
For each callable in the input list, the kernel:
1. Creates a new `sandbox.Sandbox` with the same `Config` as the parent
(same `MaxSteps` limit — each branch gets its own step budget, not a
shared one).
2. Calls `runner.buildModulesWithLibCtx()` with the **same** `packageID`,
`manifest`, and `RunContext` as the parent invocation. This produces
a fresh module set — new `DBModuleConfig`, new `HTTPModuleConfig`, etc.
— pointing at the same underlying Go resources (`*sql.DB`, etc.).
3. The `libContext` is **shared** (read path only — cached frozen exports).
Library exports are immutable. The `loading` map (cycle detection) is
not relevant because libraries are already loaded before `batch.exec`
runs. If a branch triggers a new `lib.require()`, it would need its
own `libContext` — see Open Questions.
4. Creates a new `starlark.Thread` with the branch's print handler,
step limit, and context-based cancellation.
5. Calls `starlark.Call(thread, callable, nil, nil)` — the callable
is a zero-arg lambda that closes over its arguments.
### Context & Cancellation
Each branch gets a child context derived from the parent with the
per-branch timeout applied:
```go
branchCtx, cancel := context.WithTimeout(parentCtx, branchTimeout)
defer cancel()
```
If the parent context is cancelled (e.g., HTTP request timeout), all
branches are cancelled. If one branch exceeds its timeout, only that
branch is cancelled — others continue.
### Goroutine Cap
Hard limit: **8 concurrent branches.** This is enforced at the API
boundary (list length check), not via a semaphore. Rationale:
- 8 covers the real-world fan-out patterns (2-5 API calls, small batch
operations). Nobody needs 50 concurrent Starlark branches.
- Each branch allocates a `starlark.Thread` + module instances. At 8
branches, overhead is bounded at ~8KB per thread + module construction
time (~50μs per module set).
- No semaphore means no queuing surprises. You get 8, period.
---
## Implementation
### New File: `sandbox/batch_module.go`
```go
// BuildBatchModule creates the "batch" module.
// Requires the Runner reference for per-branch module construction.
func BuildBatchModule(
ctx context.Context,
runner *Runner,
packageID string,
manifest map[string]any,
rc *RunContext,
lc *libContext,
) *starlarkstruct.Module
```
The module holds a reference to the `Runner` — same pattern as
`BuildLibModule`. It needs the runner to call `buildModulesWithLibCtx`
for each branch.
### Core Implementation Sketch
```go
func batchExec(ctx context.Context, runner *Runner, packageID string,
manifest map[string]any, rc *RunContext, parentLC *libContext,
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
return func(thread *starlark.Thread, b *starlark.Builtin,
args starlark.Tuple, kwargs []starlark.Tuple,
) (starlark.Value, error) {
var callableList *starlark.List
var timeout int = 10
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
"callables", &callableList,
"timeout?", &timeout,
); err != nil {
return nil, err
}
n := callableList.Len()
if n == 0 {
return nil, fmt.Errorf("batch.exec: callables list is empty")
}
if n > 8 {
return nil, fmt.Errorf("batch.exec: max 8 callables, got %d", n)
}
if timeout < 1 || timeout > 30 {
timeout = 10
}
// Validate all elements are callable.
callables := make([]starlark.Callable, n)
for i := 0; i < n; i++ {
c, ok := callableList.Index(i).(starlark.Callable)
if !ok {
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
i, callableList.Index(i).Type())
}
callables[i] = c
}
// Execute concurrently.
type branchResult struct {
index int
value starlark.Value
err error
}
results := make([]starlark.Value, n)
errors := make([]starlark.Value, n)
var wg sync.WaitGroup
for i, callable := range callables {
wg.Add(1)
go func(idx int, fn starlark.Callable) {
defer wg.Done()
// Per-branch context with timeout.
branchCtx, cancel := context.WithTimeout(ctx,
time.Duration(timeout)*time.Second)
defer cancel()
// Fresh module set for this branch.
modules, err := runner.buildModulesWithLibCtx(
branchCtx, packageID, manifest, rc, parentLC)
if err != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(err.Error())
return
}
// Fresh sandbox + thread.
sb := New(DefaultConfig())
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
if callErr != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(callErr.Error())
} else {
results[idx] = val
errors[idx] = starlark.None
}
}(i, callable)
}
wg.Wait()
return starlark.Tuple{
starlark.NewList(results),
starlark.NewList(errors),
}, nil
}
}
```
### Runner Wiring
In `buildModulesWithLibCtx`, add the permission case:
```go
case models.ExtPermBatchExec:
// Deferred — wired after module map is complete (needs runner ref).
hasBatchExec = true
```
After the module map is assembled:
```go
if hasBatchExec {
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
}
```
### Permission Constant
In `models/permissions.go`:
```go
ExtPermBatchExec = "batch.exec"
```
Add to `AllExtensionPermissions` slice.
---
## Callable Closure Semantics
The callables passed to `batch.exec` are typically lambdas that close
over variables from the calling scope:
```python
issue_data = {"summary": "Review Q3 report"}
page_data = {"title": "Q3 Report", "body": content}
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
])
```
The closed-over values (`issue_data`, `page_data`, `jira`, `confluence`)
are references to Starlark values in the calling thread's scope. Two
safety properties make this work:
1. **Library exports (`jira`, `confluence`) are frozen.** They were
returned by `lib.require()` as `starlarkstruct.FromStringDict()`
deeply immutable. Safe to read from any goroutine.
2. **Dict/list arguments may be mutable**, but Starlark's execution
model means the calling thread is **blocked** waiting for
`batch.exec` to return. No concurrent mutation is possible because
the caller can't execute while the branches are running.
This is the same safety model as Go's `sync.WaitGroup` pattern: the
goroutine that calls `wg.Wait()` cannot proceed until all goroutines
complete, so values passed to goroutines before `wg.Add` are safe to
read without locks.
---
## Open Questions
### 1. `lib.require()` Inside Branches
If a callable triggers a `lib.require()` that hasn't been cached yet,
the shared `libContext.cache` would be written from a goroutine. Options:
**A. Prohibit: branches cannot call `lib.require()`.** The branch gets
a nil `libContext`, so `lib` module is unavailable inside `batch.exec`.
Libraries must be loaded before the batch call. Simplest, safest.
**B. Per-branch `libContext` with shared read cache.** Each branch gets
its own `libContext` whose `cache` is pre-populated from the parent's
cache (snapshot). New loads go into the branch's local cache only.
Slightly wasteful if two branches load the same library (loaded twice),
but safe.
**C. Mutex-protected shared `libContext`.** Add a `sync.RWMutex` to
`libContext`. Reads use `RLock`, writes use `Lock`. Minimal overhead,
but makes `libContext` aware of concurrency — violates its current
assumptions.
**Recommendation: Option A for v0.7.11, Option B as follow-up if needed.**
In practice, extensions call `lib.require()` at module scope (top of
script), not inside request handlers. The lambdas passed to `batch.exec`
call methods on already-loaded library structs. Option A covers all
real-world patterns.
### 2. Print Output
Each branch has its own print buffer (the `output strings.Builder` in
`Sandbox.Call`). Options:
**A. Discard.** Branch print output is lost. Simple, avoids interleaving.
**B. Collect per-branch.** Return a third list: `(results, errors, outputs)`.
Useful for debugging but clutters the API.
**C. Merge into parent.** Append all branch output to the parent thread's
print buffer, prefixed with branch index. Requires passing the parent's
`outputMu` and `output` builder — invasive.
**Recommendation: Option A for v0.7.11.** `print()` in Starlark is a
debugging tool, not a production logging facility. Branch callables that
need to report status should return structured data. If debugging demand
emerges, Option B is a backward-compatible addition.
### 3. Step Limit Scope
Each branch gets its own `MaxSteps` budget (default 1M). Should the
total across all branches be capped?
**No.** The per-branch cap is sufficient. 8 branches × 1M steps = 8M
total, which completes in under a second on any modern hardware. The
wall-clock timeout (per-branch, max 30s) is the real resource guard.
Adding a cross-branch step budget creates coupling between independent
execution paths — branch A's step count shouldn't affect branch B's
ability to complete.
---
## Testing
| Test | Description |
|------|-------------|
| Parallel ordering | 3 callables with different sleep durations. Results in input order, not completion order. |
| Partial failure | 3 callables, middle one raises error. results = [val, None, val], errors = [None, "err msg", None]. |
| Timeout per-branch | One callable sleeps beyond timeout. Others succeed. Timed-out branch returns error. |
| Parent cancellation | Cancel parent context mid-execution. All branches cancelled. |
| Cap enforcement | List of 9 callables → immediate error, nothing executed. |
| Empty list | `batch.exec([])` → error. |
| Non-callable element | `batch.exec([1, 2])` → error, nothing executed. |
| Permission gating | Package without `batch.exec` permission → module not available. |
| Frozen library sharing | Two branches call same frozen library function concurrently. No race. |
| Nested batch.exec | Callable inside batch.exec attempts batch.exec → error (module not injected in branch). |
| db module isolation | Two branches insert into same table concurrently. Both succeed, no corruption. |
| http module isolation | Two branches make HTTP calls with different headers. No cross-contamination. |
---
## Migration
No schema changes. No new tables. No new migrations.
New permission constant `batch.exec` added to `models/permissions.go`.
Extensions must declare the permission in their manifest to access the
`batch` module.

View File

@@ -242,6 +242,39 @@ instances = workflow.list_instances(workflow_id, status="active")
# Returns list of instance dicts # Returns list of instance dicts
``` ```
### batch
**Permission:** `batch.exec`
Run multiple callables concurrently. Each callable gets its own
execution thread with an independent step budget.
```python
jira = lib.require("jira-client")
confluence = lib.require("confluence-client")
results, errors = batch.exec([
lambda: jira.create_issue(issue_data),
lambda: confluence.create_page(page_data),
lambda: send_notification(user_id),
], timeout=15)
# results[i] = return value of callables[i], or None on error
# errors[i] = None on success, or error string on failure
# All three ran concurrently.
```
**Constraints:**
- Max 8 callables per call. Dispatched concurrently via goroutines.
- `timeout` (optional): 130 seconds per branch (default 10).
- `lib.require()` is not available inside branch callables.
Load libraries before the `batch.exec` call.
- `batch.exec()` cannot be called from within a branch (no nesting).
- `print()` output from branches is discarded.
---
## Example: automated stage hook ## Example: automated stage hook
A simple hook that reads a setting, queries data, and advances: A simple hook that reads a setting, queries data, and advances:

View File

@@ -36,6 +36,7 @@ const (
ExtPermConnectionsRead = "connections.read" ExtPermConnectionsRead = "connections.read"
ExtPermTriggersRegister = "triggers.register" ExtPermTriggersRegister = "triggers.register"
ExtPermRealtimePublish = "realtime.publish" ExtPermRealtimePublish = "realtime.publish"
ExtPermBatchExec = "batch.exec"
) )
// ValidExtensionPermissions is the set of recognized permission keys. // ValidExtensionPermissions is the set of recognized permission keys.
@@ -50,6 +51,7 @@ var ValidExtensionPermissions = map[string]bool{
ExtPermConnectionsRead: true, ExtPermConnectionsRead: true,
ExtPermTriggersRegister: true, ExtPermTriggersRegister: true,
ExtPermRealtimePublish: true, ExtPermRealtimePublish: true,
ExtPermBatchExec: true,
} }
// ── Extension Permission Model ─────────────── // ── Extension Permission Model ───────────────

View File

@@ -0,0 +1,140 @@
// Package sandbox — batch_module.go
//
// Permission: batch.exec
//
// Starlark API:
// results, errors = batch.exec([fn_a, fn_b, fn_c], timeout=10)
//
// Runs a list of zero-arg callables concurrently, each in its own
// starlark.Thread with an independent step budget. Results and errors
// are returned in input order. Max 8 callables per call.
//
// Each branch gets a fresh Sandbox (thread + step counter) and a
// child context with the per-branch timeout. The callable's closed-over
// bindings (frozen library structs, module builtins) are used as-is —
// the Go resources underneath (*sql.DB, http.Client) are concurrent-safe.
//
// lib.require() is not available inside branches (Option A).
// print() output from branches is discarded.
// Nested batch.exec() is prohibited via an atomic flag.
package sandbox
import (
"context"
"fmt"
"sync"
"sync/atomic"
"time"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
)
const (
batchMaxCallables = 8
batchDefaultTimeout = 10
batchMaxTimeout = 30
)
// BuildBatchModule creates the "batch" module.
// The runner/packageID/manifest/rc/lc parameters are stored for future
// expansion (Option B: per-branch lib.require) but are not used in v1.
func BuildBatchModule(
ctx context.Context,
runner *Runner,
packageID string,
manifest map[string]any,
rc *RunContext,
lc *libContext,
) *starlarkstruct.Module {
var running atomic.Bool
return MakeModule("batch", starlark.StringDict{
"exec": starlark.NewBuiltin("batch.exec", batchExec(ctx, &running)),
})
}
// batchExec returns the batch.exec builtin implementation.
func batchExec(
ctx context.Context,
running *atomic.Bool,
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
return func(
thread *starlark.Thread, b *starlark.Builtin,
args starlark.Tuple, kwargs []starlark.Tuple,
) (starlark.Value, error) {
// Nesting guard: parent thread is blocked in wg.Wait(), so only
// a branch goroutine could re-enter. The captured builtin reference
// makes this reachable even though branches don't get fresh modules.
if !running.CompareAndSwap(false, true) {
return nil, fmt.Errorf("batch.exec: nested calls are not allowed")
}
defer running.Store(false)
var callableList *starlark.List
var timeout int = batchDefaultTimeout
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
"callables", &callableList,
"timeout?", &timeout,
); err != nil {
return nil, err
}
n := callableList.Len()
if n == 0 {
return nil, fmt.Errorf("batch.exec: callables list is empty")
}
if n > batchMaxCallables {
return nil, fmt.Errorf("batch.exec: max %d callables, got %d", batchMaxCallables, n)
}
if timeout < 1 || timeout > batchMaxTimeout {
timeout = batchDefaultTimeout
}
// Validate all elements are callable before spawning goroutines.
callables := make([]starlark.Callable, n)
for i := 0; i < n; i++ {
c, ok := callableList.Index(i).(starlark.Callable)
if !ok {
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
i, callableList.Index(i).Type())
}
callables[i] = c
}
// Execute concurrently.
results := make([]starlark.Value, n)
errors := make([]starlark.Value, n)
var wg sync.WaitGroup
for i, callable := range callables {
wg.Add(1)
go func(idx int, fn starlark.Callable) {
defer wg.Done()
branchCtx, cancel := context.WithTimeout(ctx,
time.Duration(timeout)*time.Second)
defer cancel()
sb := New(DefaultConfig())
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
if callErr != nil {
results[idx] = starlark.None
errors[idx] = starlark.String(callErr.Error())
} else {
results[idx] = val
errors[idx] = starlark.None
}
}(i, callable)
}
wg.Wait()
return starlark.Tuple{
starlark.NewList(results),
starlark.NewList(errors),
}, nil
}
}

View File

@@ -0,0 +1,324 @@
package sandbox
import (
"context"
"strings"
"testing"
"time"
"go.starlark.net/starlark"
)
// execWithBatch runs a Starlark script with the batch module available.
func execWithBatch(t *testing.T, script string) (*Result, error) {
t.Helper()
ctx := context.Background()
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
sb := New(DefaultConfig())
return sb.Exec(ctx, "test.star", script, map[string]starlark.Value{
"batch": batchMod,
})
}
// ─── Parallel Ordering ──────────────────────
func TestBatchExec_ParallelOrdering(t *testing.T) {
result, err := execWithBatch(t, `
results, errors = batch.exec([
lambda: 10,
lambda: 20,
lambda: 30,
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
// Verify results are in input order.
globals := result.Globals
results := globals["results"].(*starlark.List)
errors := globals["errors"].(*starlark.List)
if results.Len() != 3 {
t.Fatalf("expected 3 results, got %d", results.Len())
}
for i, want := range []int{10, 20, 30} {
v, _ := starlark.AsInt32(results.Index(i))
got := int(v)
if got != want {
t.Errorf("results[%d] = %d, want %d", i, got, want)
}
if errors.Index(i) != starlark.None {
t.Errorf("errors[%d] = %v, want None", i, errors.Index(i))
}
}
}
// ─── Partial Failure ────────────────────────
func TestBatchExec_PartialFailure(t *testing.T) {
result, err := execWithBatch(t, `
def ok_a():
return "a"
def bad():
return 1 // 0
def ok_c():
return "c"
results, errors = batch.exec([ok_a, bad, ok_c])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
errors := result.Globals["errors"].(*starlark.List)
// results[0] = "a", results[2] = "c"
if s, _ := starlark.AsString(results.Index(0)); s != "a" {
t.Errorf("results[0] = %v, want 'a'", results.Index(0))
}
if s, _ := starlark.AsString(results.Index(2)); s != "c" {
t.Errorf("results[2] = %v, want 'c'", results.Index(2))
}
// results[1] = None (failed)
if results.Index(1) != starlark.None {
t.Errorf("results[1] = %v, want None", results.Index(1))
}
// errors[0] and errors[2] = None (succeeded)
if errors.Index(0) != starlark.None {
t.Errorf("errors[0] = %v, want None", errors.Index(0))
}
if errors.Index(2) != starlark.None {
t.Errorf("errors[2] = %v, want None", errors.Index(2))
}
// errors[1] should be a string containing the error
errStr, ok := starlark.AsString(errors.Index(1))
if !ok || errStr == "" {
t.Errorf("errors[1] = %v, want non-empty error string", errors.Index(1))
}
}
// ─── Timeout Per-Branch ─────────────────────
func TestBatchExec_TimeoutPerBranch(t *testing.T) {
// Use a tight timeout. The slow lambda burns steps in an infinite loop,
// which will hit either the step limit or the context timeout.
result, err := execWithBatch(t, `
def fast():
return "done"
def slow():
x = 0
for i in range(999999999):
x = x + 1
return x
results, errors = batch.exec([fast, slow], timeout=1)
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
errors := result.Globals["errors"].(*starlark.List)
// Fast branch should succeed.
if s, _ := starlark.AsString(results.Index(0)); s != "done" {
t.Errorf("results[0] = %v, want 'done'", results.Index(0))
}
if errors.Index(0) != starlark.None {
t.Errorf("errors[0] = %v, want None", errors.Index(0))
}
// Slow branch should fail (step limit or timeout).
if results.Index(1) != starlark.None {
t.Errorf("results[1] = %v, want None", results.Index(1))
}
errStr, _ := starlark.AsString(errors.Index(1))
if errStr == "" {
t.Errorf("errors[1] should contain error string")
}
}
// ─── Parent Cancellation ────────────────────
func TestBatchExec_ParentCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel() // cancel immediately
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
sb := New(DefaultConfig())
result, err := sb.Exec(ctx, "test.star", `
results, errors = batch.exec([lambda: 1, lambda: 2])
`, map[string]starlark.Value{"batch": batchMod})
// Either the Exec itself fails (context cancelled before script runs)
// or the batch.exec branches all fail.
if err != nil {
// Script-level cancellation — acceptable.
return
}
errors := result.Globals["errors"].(*starlark.List)
allErrored := true
for i := 0; i < errors.Len(); i++ {
if errors.Index(i) == starlark.None {
allErrored = false
}
}
if !allErrored {
t.Errorf("expected all branches to error on cancelled context, errors=%v", errors)
}
}
// ─── Cap Enforcement ────────────────────────
func TestBatchExec_CapEnforcement(t *testing.T) {
_, err := execWithBatch(t, `
fns = [lambda: i for i in range(9)]
batch.exec(fns)
`)
if err == nil {
t.Fatal("expected error for 9 callables, got nil")
}
if !strings.Contains(err.Error(), "max 8") {
t.Errorf("error = %q, want to contain 'max 8'", err.Error())
}
}
// ─── Empty List ─────────────────────────────
func TestBatchExec_EmptyList(t *testing.T) {
_, err := execWithBatch(t, `batch.exec([])`)
if err == nil {
t.Fatal("expected error for empty list, got nil")
}
if !strings.Contains(err.Error(), "empty") {
t.Errorf("error = %q, want to contain 'empty'", err.Error())
}
}
// ─── Non-Callable Element ───────────────────
func TestBatchExec_NonCallableElement(t *testing.T) {
_, err := execWithBatch(t, `batch.exec([1, 2])`)
if err == nil {
t.Fatal("expected error for non-callable, got nil")
}
if !strings.Contains(err.Error(), "not callable") {
t.Errorf("error = %q, want to contain 'not callable'", err.Error())
}
}
// ─── Permission Gating ──────────────────────
func TestBatchExec_PermissionGating(t *testing.T) {
sb := New(DefaultConfig())
_, err := sb.Exec(context.Background(), "test.star",
`batch.exec([lambda: 1])`,
map[string]starlark.Value{}, // no batch module
)
if err == nil {
t.Fatal("expected error when batch module not available")
}
if !strings.Contains(err.Error(), "batch") {
t.Errorf("error = %q, want to mention 'batch'", err.Error())
}
}
// ─── Frozen Struct Sharing ──────────────────
func TestBatchExec_FrozenStructSharing(t *testing.T) {
// Two branches read the same frozen tuple. Should not race.
result, err := execWithBatch(t, `
data = (1, 2, 3) # tuples are frozen/immutable
results, errors = batch.exec([
lambda: data[0] + data[1],
lambda: data[1] + data[2],
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
got0, _ := starlark.AsInt32(results.Index(0))
got1, _ := starlark.AsInt32(results.Index(1))
if got0 != 3 {
t.Errorf("results[0] = %d, want 3", got0)
}
if got1 != 5 {
t.Errorf("results[1] = %d, want 5", got1)
}
}
// ─── Nested batch.exec ──────────────────────
func TestBatchExec_NestedBatchExec(t *testing.T) {
result, err := execWithBatch(t, `
results, errors = batch.exec([
lambda: batch.exec([lambda: 1]),
])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
// The inner batch.exec should fail with nesting error.
errors := result.Globals["errors"].(*starlark.List)
errStr, _ := starlark.AsString(errors.Index(0))
if !strings.Contains(errStr, "nested") {
t.Errorf("errors[0] = %q, want to contain 'nested'", errStr)
}
}
// ─── Default Timeout ────────────────────────
func TestBatchExec_DefaultTimeout(t *testing.T) {
// Call without explicit timeout — should succeed.
result, err := execWithBatch(t, `
results, errors = batch.exec([lambda: 42])
`)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
got, _ := starlark.AsInt32(results.Index(0))
if got != 42 {
t.Errorf("results[0] = %d, want 42", got)
}
}
// ─── Timeout Clamping ───────────────────────
func TestBatchExec_TimeoutClamp(t *testing.T) {
// timeout=0 should be clamped to default, not crash.
start := time.Now()
result, err := execWithBatch(t, `
results, errors = batch.exec([lambda: "ok"], timeout=0)
`)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("exec error: %v", err)
}
results := result.Globals["results"].(*starlark.List)
if s, _ := starlark.AsString(results.Index(0)); s != "ok" {
t.Errorf("results[0] = %v, want 'ok'", results.Index(0))
}
// Should complete quickly (clamped to default 10s, but lambda is instant).
if elapsed > 2*time.Second {
t.Errorf("took %v, expected fast completion", elapsed)
}
}

View File

@@ -338,6 +338,7 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
// Track db permission level: 0=none, 1=read, 2=write // Track db permission level: 0=none, 1=read, 2=write
dbLevel := 0 dbLevel := 0
hasBatchExec := false
for _, perm := range granted { for _, perm := range granted {
switch perm { switch perm {
@@ -374,6 +375,9 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
if r.bus != nil { if r.bus != nil {
modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID) modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID)
} }
case models.ExtPermBatchExec:
hasBatchExec = true
} }
} }
@@ -404,5 +408,12 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc) modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc)
} }
// Concurrent execution primitive — wired after all other modules
// so the runner reference can construct per-branch module sets if
// needed in future (Option B).
if hasBatchExec {
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
}
return modules, nil return modules, nil
} }