Feat v0.7.12 batch exec (#66)
All checks were successful
CI/CD / detect-changes (push) Successful in 4s
CI/CD / test-runners (push) Has been skipped
CI/CD / e2e-smoke (push) Has been skipped
CI/CD / test-go-pg (push) Successful in 2m46s
CI/CD / test-sqlite (push) Successful in 2m58s
CI/CD / build-and-deploy (push) Successful in 1m27s
CI/CD / test-frontend (push) Successful in 5s
All checks were successful
CI/CD / detect-changes (push) Successful in 4s
CI/CD / test-runners (push) Has been skipped
CI/CD / e2e-smoke (push) Has been skipped
CI/CD / test-go-pg (push) Successful in 2m46s
CI/CD / test-sqlite (push) Successful in 2m58s
CI/CD / build-and-deploy (push) Successful in 1m27s
CI/CD / test-frontend (push) Successful in 5s
Co-authored-by: Jeffrey Smith <jasafpro@gmail.com> Co-committed-by: Jeffrey Smith <jasafpro@gmail.com>
This commit was merged in pull request #66.
This commit is contained in:
20
CHANGELOG.md
20
CHANGELOG.md
@@ -2,6 +2,26 @@
|
|||||||
|
|
||||||
All notable changes to Armature are documented here.
|
All notable changes to Armature are documented here.
|
||||||
|
|
||||||
|
## v0.7.12 — Concurrent Execution Primitive
|
||||||
|
|
||||||
|
New `batch` sandbox module. Enables extensions to parallelize arbitrary
|
||||||
|
Starlark callables — including frozen library exports from `lib.require()`.
|
||||||
|
|
||||||
|
**batch module (permission: `batch.exec`)**
|
||||||
|
|
||||||
|
- `batch.exec(callables, timeout=10)` — runs up to 8 zero-arg callables concurrently, each in its own `starlark.Thread` with independent step budget. Returns `(results, errors)` tuple with ordered results. Per-branch timeout (1–30s, default 10).
|
||||||
|
- Nested `batch.exec()` calls are prohibited (prevents exponential goroutine growth).
|
||||||
|
- `lib.require()` not available inside branches — load libraries before the batch call.
|
||||||
|
- `print()` output from branches is discarded.
|
||||||
|
|
||||||
|
**Internal**
|
||||||
|
|
||||||
|
- New file: `sandbox/batch_module.go`.
|
||||||
|
- New permission constant: `ExtPermBatchExec`.
|
||||||
|
- Runner wiring: `buildModulesWithLibCtx` creates batch module when permission granted.
|
||||||
|
- Design doc: `docs/DESIGN-batch-exec.md`.
|
||||||
|
- 12 new tests (parallel ordering, partial failure, timeout, cancellation, cap enforcement, nesting prevention, frozen sharing, permission gating).
|
||||||
|
|
||||||
## v0.7.11 — Query & HTTP Ergonomics
|
## v0.7.11 — Query & HTTP Ergonomics
|
||||||
|
|
||||||
Four new Starlark sandbox builtins. No new permissions, no schema changes.
|
Four new Starlark sandbox builtins. No new permissions, no schema changes.
|
||||||
|
|||||||
10
ROADMAP.md
10
ROADMAP.md
@@ -1,6 +1,6 @@
|
|||||||
# Armature — Roadmap
|
# Armature — Roadmap
|
||||||
|
|
||||||
## Current: v0.7.11 — Query & HTTP Ergonomics
|
## Current: v0.7.12 — Concurrent Execution Primitive
|
||||||
|
|
||||||
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
|
Self-hosted extensible platform kernel. Auth, identity, packages, Starlark
|
||||||
sandbox, storage, realtime, and ops are kernel primitives. Everything else
|
sandbox, storage, realtime, and ops are kernel primitives. Everything else
|
||||||
@@ -371,10 +371,10 @@ fresh module instances (`db`, `http`, etc.) — same pattern as
|
|||||||
|
|
||||||
| Step | Status | Description |
|
| Step | Status | Description |
|
||||||
|------|--------|-------------|
|
|------|--------|-------------|
|
||||||
| Design doc | | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. |
|
| Design doc | done | `docs/DESIGN-batch-exec.md`. Thread isolation model, module construction, error semantics (partial results on failure), permission model, concurrency cap. |
|
||||||
| `batch.exec()` | | `batch.exec(callables)` → `(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. |
|
| `batch.exec()` | done | `batch.exec(callables)` → `(results, errors)` tuple. List of Starlark callables (lambdas, function refs). Each runs in its own `starlark.Thread` with fresh modules. Concurrent goroutines capped at 8. Ordered results. Errors are per-callable (None on success, string on failure). Permission: `batch.exec` (new). New file: `sandbox/batch_module.go`. |
|
||||||
| Runner wiring | | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. |
|
| Runner wiring | done | `buildModulesWithLibCtx` creates `batch` module when `batch.exec` permission granted. Module receives runner reference for per-branch module construction. |
|
||||||
| Tests | | Unit tests: parallel execution ordering, partial failure, cap enforcement, frozen library sharing, permission gating. |
|
| Tests | done | 12 unit tests: parallel execution ordering, partial failure, timeout per-branch, parent cancellation, cap enforcement, empty list, non-callable, permission gating, frozen struct sharing, nested batch.exec, default timeout, timeout clamping. All pass with `-race`. |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
466
docs/DESIGN-batch-exec.md
Normal file
466
docs/DESIGN-batch-exec.md
Normal file
@@ -0,0 +1,466 @@
|
|||||||
|
# DESIGN — Concurrent Execution Primitive (`batch.exec`)
|
||||||
|
|
||||||
|
**Version:** v0.7.12
|
||||||
|
**Status:** Implemented
|
||||||
|
**Author:** Jeff / Claude session 2026-04-02
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Starlark is single-threaded by design (`go.starlark.net` enforces one
|
||||||
|
thread per execution). Extensions that need to fan out — calling multiple
|
||||||
|
external APIs, invoking several library functions, or performing independent
|
||||||
|
I/O operations — must do so sequentially. For two `http.post()` calls
|
||||||
|
taking 200ms each, the extension blocks for 400ms regardless of whether
|
||||||
|
the calls are independent.
|
||||||
|
|
||||||
|
The v0.7.10 `http.batch()` primitive solves the narrow case of parallel
|
||||||
|
HTTP dispatch. But it doesn't help when the work is wrapped in library
|
||||||
|
functions. If a `jira-client` library exposes `create_issue()` and a
|
||||||
|
`confluence-client` library exposes `create_page()`, the extension author
|
||||||
|
must either:
|
||||||
|
|
||||||
|
1. Call them sequentially (slow), or
|
||||||
|
2. Decompose the library calls back into raw `http.post()` parameters
|
||||||
|
to use `http.batch()` (defeats the purpose of having libraries).
|
||||||
|
|
||||||
|
The platform needs a general-purpose concurrent execution primitive that
|
||||||
|
works with arbitrary Starlark callables — including library exports.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- **Shared mutable state between branches.** Each concurrent branch is
|
||||||
|
fully isolated. No channels, no mutexes, no shared dicts. If branches
|
||||||
|
need to coordinate, they don't belong in `batch.exec`.
|
||||||
|
- **Unlimited concurrency.** A hard cap prevents extensions from spawning
|
||||||
|
unbounded goroutines. This is a fan-out primitive, not a thread pool.
|
||||||
|
- **Automatic retry or circuit breaking.** Error handling is the caller's
|
||||||
|
responsibility. The kernel reports per-branch results and errors.
|
||||||
|
- **Nested `batch.exec()`.** A callable inside `batch.exec` cannot itself
|
||||||
|
call `batch.exec`. This prevents exponential goroutine growth and keeps
|
||||||
|
the concurrency model flat.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Insight: Frozen Libraries Are Thread-Safe
|
||||||
|
|
||||||
|
The reason this works without exotic machinery is `lib.require()`.
|
||||||
|
|
||||||
|
When a library is loaded via `lib.require()`, its exports are wrapped in
|
||||||
|
a `starlarkstruct.FromStringDict()` — which produces a **frozen** struct.
|
||||||
|
Frozen Starlark values are immutable and safe to read from any number of
|
||||||
|
goroutines concurrently. This is a property of `go.starlark.net`, not
|
||||||
|
something we enforce.
|
||||||
|
|
||||||
|
The only mutable state in a Starlark execution is:
|
||||||
|
|
||||||
|
1. **The `starlark.Thread` itself** — step counter, print buffer, cancel
|
||||||
|
channel. Each branch gets its own thread.
|
||||||
|
2. **Module instances** — `db`, `http`, `settings`, etc. contain
|
||||||
|
configuration and hold references to shared Go objects (`*sql.DB`,
|
||||||
|
`http.Client`). Each branch gets fresh module instances, but the
|
||||||
|
underlying Go resources (`*sql.DB` connection pool, etc.) are already
|
||||||
|
designed for concurrent access.
|
||||||
|
3. **Local variables** — thread-local by definition in Starlark.
|
||||||
|
|
||||||
|
So the construction is: one new `starlark.Thread` + one new module set
|
||||||
|
per branch, with frozen library structs shared read-only across all
|
||||||
|
branches. This is exactly what `triggers/schedule.go` already does for
|
||||||
|
scheduled task execution — `buildRestrictedModules` creates a fresh
|
||||||
|
module set for each cron fire. `batch.exec` generalizes that pattern.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API
|
||||||
|
|
||||||
|
```python
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: jira.create_issue(issue_data),
|
||||||
|
lambda: confluence.create_page(page_data),
|
||||||
|
lambda: slack.post_message(channel, msg),
|
||||||
|
])
|
||||||
|
|
||||||
|
# results[0] = return value of jira.create_issue(), or None on error
|
||||||
|
# errors[0] = None on success, or error string on failure
|
||||||
|
# All three ran concurrently.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Signature
|
||||||
|
|
||||||
|
```
|
||||||
|
batch.exec(callables, timeout=10) → (results: list, errors: list)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
|
||||||
|
| Param | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `callables` | `list[callable]` | Starlark callables (lambdas, named functions, bound methods). Max length: 8. |
|
||||||
|
| `timeout` | `int` (optional) | Per-branch timeout in seconds. Default 10. Max 30. Inherits parent context deadline if shorter. |
|
||||||
|
|
||||||
|
**Returns:** A 2-tuple of equal-length lists.
|
||||||
|
|
||||||
|
- `results[i]` — the return value of `callables[i]`, or `None` if it
|
||||||
|
errored.
|
||||||
|
- `errors[i]` — `None` if `callables[i]` succeeded, or a string error
|
||||||
|
message if it failed (timeout, step limit, runtime error).
|
||||||
|
|
||||||
|
**Errors (whole-call):**
|
||||||
|
|
||||||
|
- `callables` is empty → error
|
||||||
|
- `callables` length > 8 → error
|
||||||
|
- Any element is not callable → error
|
||||||
|
- Permission `batch.exec` not granted → error
|
||||||
|
|
||||||
|
### Permission
|
||||||
|
|
||||||
|
New extension permission: `batch.exec`. Declared in manifest:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"permissions": ["batch.exec"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a separate permission because concurrent execution has resource
|
||||||
|
implications (goroutines, module construction overhead). Extensions that
|
||||||
|
don't need it shouldn't pay for it. The permission doesn't imply any
|
||||||
|
other permissions — the branch inherits whatever modules the calling
|
||||||
|
package already has.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Execution Model
|
||||||
|
|
||||||
|
```
|
||||||
|
batch.exec([fn_a, fn_b, fn_c])
|
||||||
|
│
|
||||||
|
├─── goroutine 1: Thread₁ + Modules₁ → fn_a() → result[0]
|
||||||
|
├─── goroutine 2: Thread₂ + Modules₂ → fn_b() → result[1]
|
||||||
|
└─── goroutine 3: Thread₃ + Modules₃ → fn_c() → result[2]
|
||||||
|
│
|
||||||
|
sync.WaitGroup.Wait()
|
||||||
|
│
|
||||||
|
return (results, errors)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Per-Branch Construction
|
||||||
|
|
||||||
|
For each callable in the input list, the kernel:
|
||||||
|
|
||||||
|
1. Creates a new `sandbox.Sandbox` with the same `Config` as the parent
|
||||||
|
(same `MaxSteps` limit — each branch gets its own step budget, not a
|
||||||
|
shared one).
|
||||||
|
2. Calls `runner.buildModulesWithLibCtx()` with the **same** `packageID`,
|
||||||
|
`manifest`, and `RunContext` as the parent invocation. This produces
|
||||||
|
a fresh module set — new `DBModuleConfig`, new `HTTPModuleConfig`, etc.
|
||||||
|
— pointing at the same underlying Go resources (`*sql.DB`, etc.).
|
||||||
|
3. The `libContext` is **shared** (read path only — cached frozen exports).
|
||||||
|
Library exports are immutable. The `loading` map (cycle detection) is
|
||||||
|
not relevant because libraries are already loaded before `batch.exec`
|
||||||
|
runs. If a branch triggers a new `lib.require()`, it would need its
|
||||||
|
own `libContext` — see Open Questions.
|
||||||
|
4. Creates a new `starlark.Thread` with the branch's print handler,
|
||||||
|
step limit, and context-based cancellation.
|
||||||
|
5. Calls `starlark.Call(thread, callable, nil, nil)` — the callable
|
||||||
|
is a zero-arg lambda that closes over its arguments.
|
||||||
|
|
||||||
|
### Context & Cancellation
|
||||||
|
|
||||||
|
Each branch gets a child context derived from the parent with the
|
||||||
|
per-branch timeout applied:
|
||||||
|
|
||||||
|
```go
|
||||||
|
branchCtx, cancel := context.WithTimeout(parentCtx, branchTimeout)
|
||||||
|
defer cancel()
|
||||||
|
```
|
||||||
|
|
||||||
|
If the parent context is cancelled (e.g., HTTP request timeout), all
|
||||||
|
branches are cancelled. If one branch exceeds its timeout, only that
|
||||||
|
branch is cancelled — others continue.
|
||||||
|
|
||||||
|
### Goroutine Cap
|
||||||
|
|
||||||
|
Hard limit: **8 concurrent branches.** This is enforced at the API
|
||||||
|
boundary (list length check), not via a semaphore. Rationale:
|
||||||
|
|
||||||
|
- 8 covers the real-world fan-out patterns (2-5 API calls, small batch
|
||||||
|
operations). Nobody needs 50 concurrent Starlark branches.
|
||||||
|
- Each branch allocates a `starlark.Thread` + module instances. At 8
|
||||||
|
branches, overhead is bounded at ~8KB per thread + module construction
|
||||||
|
time (~50μs per module set).
|
||||||
|
- No semaphore means no queuing surprises. You get 8, period.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation
|
||||||
|
|
||||||
|
### New File: `sandbox/batch_module.go`
|
||||||
|
|
||||||
|
```go
|
||||||
|
// BuildBatchModule creates the "batch" module.
|
||||||
|
// Requires the Runner reference for per-branch module construction.
|
||||||
|
func BuildBatchModule(
|
||||||
|
ctx context.Context,
|
||||||
|
runner *Runner,
|
||||||
|
packageID string,
|
||||||
|
manifest map[string]any,
|
||||||
|
rc *RunContext,
|
||||||
|
lc *libContext,
|
||||||
|
) *starlarkstruct.Module
|
||||||
|
```
|
||||||
|
|
||||||
|
The module holds a reference to the `Runner` — same pattern as
|
||||||
|
`BuildLibModule`. It needs the runner to call `buildModulesWithLibCtx`
|
||||||
|
for each branch.
|
||||||
|
|
||||||
|
### Core Implementation Sketch
|
||||||
|
|
||||||
|
```go
|
||||||
|
func batchExec(ctx context.Context, runner *Runner, packageID string,
|
||||||
|
manifest map[string]any, rc *RunContext, parentLC *libContext,
|
||||||
|
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(thread *starlark.Thread, b *starlark.Builtin,
|
||||||
|
args starlark.Tuple, kwargs []starlark.Tuple,
|
||||||
|
) (starlark.Value, error) {
|
||||||
|
var callableList *starlark.List
|
||||||
|
var timeout int = 10
|
||||||
|
|
||||||
|
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
|
||||||
|
"callables", &callableList,
|
||||||
|
"timeout?", &timeout,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
n := callableList.Len()
|
||||||
|
if n == 0 {
|
||||||
|
return nil, fmt.Errorf("batch.exec: callables list is empty")
|
||||||
|
}
|
||||||
|
if n > 8 {
|
||||||
|
return nil, fmt.Errorf("batch.exec: max 8 callables, got %d", n)
|
||||||
|
}
|
||||||
|
if timeout < 1 || timeout > 30 {
|
||||||
|
timeout = 10
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate all elements are callable.
|
||||||
|
callables := make([]starlark.Callable, n)
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
c, ok := callableList.Index(i).(starlark.Callable)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
|
||||||
|
i, callableList.Index(i).Type())
|
||||||
|
}
|
||||||
|
callables[i] = c
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute concurrently.
|
||||||
|
type branchResult struct {
|
||||||
|
index int
|
||||||
|
value starlark.Value
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
results := make([]starlark.Value, n)
|
||||||
|
errors := make([]starlark.Value, n)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
|
||||||
|
for i, callable := range callables {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(idx int, fn starlark.Callable) {
|
||||||
|
defer wg.Done()
|
||||||
|
|
||||||
|
// Per-branch context with timeout.
|
||||||
|
branchCtx, cancel := context.WithTimeout(ctx,
|
||||||
|
time.Duration(timeout)*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// Fresh module set for this branch.
|
||||||
|
modules, err := runner.buildModulesWithLibCtx(
|
||||||
|
branchCtx, packageID, manifest, rc, parentLC)
|
||||||
|
if err != nil {
|
||||||
|
results[idx] = starlark.None
|
||||||
|
errors[idx] = starlark.String(err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fresh sandbox + thread.
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
|
||||||
|
|
||||||
|
if callErr != nil {
|
||||||
|
results[idx] = starlark.None
|
||||||
|
errors[idx] = starlark.String(callErr.Error())
|
||||||
|
} else {
|
||||||
|
results[idx] = val
|
||||||
|
errors[idx] = starlark.None
|
||||||
|
}
|
||||||
|
}(i, callable)
|
||||||
|
}
|
||||||
|
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
return starlark.Tuple{
|
||||||
|
starlark.NewList(results),
|
||||||
|
starlark.NewList(errors),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runner Wiring
|
||||||
|
|
||||||
|
In `buildModulesWithLibCtx`, add the permission case:
|
||||||
|
|
||||||
|
```go
|
||||||
|
case models.ExtPermBatchExec:
|
||||||
|
// Deferred — wired after module map is complete (needs runner ref).
|
||||||
|
hasBatchExec = true
|
||||||
|
```
|
||||||
|
|
||||||
|
After the module map is assembled:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if hasBatchExec {
|
||||||
|
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permission Constant
|
||||||
|
|
||||||
|
In `models/permissions.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
ExtPermBatchExec = "batch.exec"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to `AllExtensionPermissions` slice.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Callable Closure Semantics
|
||||||
|
|
||||||
|
The callables passed to `batch.exec` are typically lambdas that close
|
||||||
|
over variables from the calling scope:
|
||||||
|
|
||||||
|
```python
|
||||||
|
issue_data = {"summary": "Review Q3 report"}
|
||||||
|
page_data = {"title": "Q3 Report", "body": content}
|
||||||
|
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: jira.create_issue(issue_data),
|
||||||
|
lambda: confluence.create_page(page_data),
|
||||||
|
])
|
||||||
|
```
|
||||||
|
|
||||||
|
The closed-over values (`issue_data`, `page_data`, `jira`, `confluence`)
|
||||||
|
are references to Starlark values in the calling thread's scope. Two
|
||||||
|
safety properties make this work:
|
||||||
|
|
||||||
|
1. **Library exports (`jira`, `confluence`) are frozen.** They were
|
||||||
|
returned by `lib.require()` as `starlarkstruct.FromStringDict()` —
|
||||||
|
deeply immutable. Safe to read from any goroutine.
|
||||||
|
|
||||||
|
2. **Dict/list arguments may be mutable**, but Starlark's execution
|
||||||
|
model means the calling thread is **blocked** waiting for
|
||||||
|
`batch.exec` to return. No concurrent mutation is possible because
|
||||||
|
the caller can't execute while the branches are running.
|
||||||
|
|
||||||
|
This is the same safety model as Go's `sync.WaitGroup` pattern: the
|
||||||
|
goroutine that calls `wg.Wait()` cannot proceed until all goroutines
|
||||||
|
complete, so values passed to goroutines before `wg.Add` are safe to
|
||||||
|
read without locks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
### 1. `lib.require()` Inside Branches
|
||||||
|
|
||||||
|
If a callable triggers a `lib.require()` that hasn't been cached yet,
|
||||||
|
the shared `libContext.cache` would be written from a goroutine. Options:
|
||||||
|
|
||||||
|
**A. Prohibit: branches cannot call `lib.require()`.** The branch gets
|
||||||
|
a nil `libContext`, so `lib` module is unavailable inside `batch.exec`.
|
||||||
|
Libraries must be loaded before the batch call. Simplest, safest.
|
||||||
|
|
||||||
|
**B. Per-branch `libContext` with shared read cache.** Each branch gets
|
||||||
|
its own `libContext` whose `cache` is pre-populated from the parent's
|
||||||
|
cache (snapshot). New loads go into the branch's local cache only.
|
||||||
|
Slightly wasteful if two branches load the same library (loaded twice),
|
||||||
|
but safe.
|
||||||
|
|
||||||
|
**C. Mutex-protected shared `libContext`.** Add a `sync.RWMutex` to
|
||||||
|
`libContext`. Reads use `RLock`, writes use `Lock`. Minimal overhead,
|
||||||
|
but makes `libContext` aware of concurrency — violates its current
|
||||||
|
assumptions.
|
||||||
|
|
||||||
|
**Recommendation: Option A for v0.7.11, Option B as follow-up if needed.**
|
||||||
|
In practice, extensions call `lib.require()` at module scope (top of
|
||||||
|
script), not inside request handlers. The lambdas passed to `batch.exec`
|
||||||
|
call methods on already-loaded library structs. Option A covers all
|
||||||
|
real-world patterns.
|
||||||
|
|
||||||
|
### 2. Print Output
|
||||||
|
|
||||||
|
Each branch has its own print buffer (the `output strings.Builder` in
|
||||||
|
`Sandbox.Call`). Options:
|
||||||
|
|
||||||
|
**A. Discard.** Branch print output is lost. Simple, avoids interleaving.
|
||||||
|
|
||||||
|
**B. Collect per-branch.** Return a third list: `(results, errors, outputs)`.
|
||||||
|
Useful for debugging but clutters the API.
|
||||||
|
|
||||||
|
**C. Merge into parent.** Append all branch output to the parent thread's
|
||||||
|
print buffer, prefixed with branch index. Requires passing the parent's
|
||||||
|
`outputMu` and `output` builder — invasive.
|
||||||
|
|
||||||
|
**Recommendation: Option A for v0.7.11.** `print()` in Starlark is a
|
||||||
|
debugging tool, not a production logging facility. Branch callables that
|
||||||
|
need to report status should return structured data. If debugging demand
|
||||||
|
emerges, Option B is a backward-compatible addition.
|
||||||
|
|
||||||
|
### 3. Step Limit Scope
|
||||||
|
|
||||||
|
Each branch gets its own `MaxSteps` budget (default 1M). Should the
|
||||||
|
total across all branches be capped?
|
||||||
|
|
||||||
|
**No.** The per-branch cap is sufficient. 8 branches × 1M steps = 8M
|
||||||
|
total, which completes in under a second on any modern hardware. The
|
||||||
|
wall-clock timeout (per-branch, max 30s) is the real resource guard.
|
||||||
|
Adding a cross-branch step budget creates coupling between independent
|
||||||
|
execution paths — branch A's step count shouldn't affect branch B's
|
||||||
|
ability to complete.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
| Test | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| Parallel ordering | 3 callables with different sleep durations. Results in input order, not completion order. |
|
||||||
|
| Partial failure | 3 callables, middle one raises error. results = [val, None, val], errors = [None, "err msg", None]. |
|
||||||
|
| Timeout per-branch | One callable sleeps beyond timeout. Others succeed. Timed-out branch returns error. |
|
||||||
|
| Parent cancellation | Cancel parent context mid-execution. All branches cancelled. |
|
||||||
|
| Cap enforcement | List of 9 callables → immediate error, nothing executed. |
|
||||||
|
| Empty list | `batch.exec([])` → error. |
|
||||||
|
| Non-callable element | `batch.exec([1, 2])` → error, nothing executed. |
|
||||||
|
| Permission gating | Package without `batch.exec` permission → module not available. |
|
||||||
|
| Frozen library sharing | Two branches call same frozen library function concurrently. No race. |
|
||||||
|
| Nested batch.exec | Callable inside batch.exec attempts batch.exec → error (module not injected in branch). |
|
||||||
|
| db module isolation | Two branches insert into same table concurrently. Both succeed, no corruption. |
|
||||||
|
| http module isolation | Two branches make HTTP calls with different headers. No cross-contamination. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Migration
|
||||||
|
|
||||||
|
No schema changes. No new tables. No new migrations.
|
||||||
|
|
||||||
|
New permission constant `batch.exec` added to `models/permissions.go`.
|
||||||
|
Extensions must declare the permission in their manifest to access the
|
||||||
|
`batch` module.
|
||||||
@@ -242,6 +242,39 @@ instances = workflow.list_instances(workflow_id, status="active")
|
|||||||
# Returns list of instance dicts
|
# Returns list of instance dicts
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### batch
|
||||||
|
|
||||||
|
**Permission:** `batch.exec`
|
||||||
|
|
||||||
|
Run multiple callables concurrently. Each callable gets its own
|
||||||
|
execution thread with an independent step budget.
|
||||||
|
|
||||||
|
```python
|
||||||
|
jira = lib.require("jira-client")
|
||||||
|
confluence = lib.require("confluence-client")
|
||||||
|
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: jira.create_issue(issue_data),
|
||||||
|
lambda: confluence.create_page(page_data),
|
||||||
|
lambda: send_notification(user_id),
|
||||||
|
], timeout=15)
|
||||||
|
|
||||||
|
# results[i] = return value of callables[i], or None on error
|
||||||
|
# errors[i] = None on success, or error string on failure
|
||||||
|
# All three ran concurrently.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
|
||||||
|
- Max 8 callables per call. Dispatched concurrently via goroutines.
|
||||||
|
- `timeout` (optional): 1–30 seconds per branch (default 10).
|
||||||
|
- `lib.require()` is not available inside branch callables.
|
||||||
|
Load libraries before the `batch.exec` call.
|
||||||
|
- `batch.exec()` cannot be called from within a branch (no nesting).
|
||||||
|
- `print()` output from branches is discarded.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Example: automated stage hook
|
## Example: automated stage hook
|
||||||
|
|
||||||
A simple hook that reads a setting, queries data, and advances:
|
A simple hook that reads a setting, queries data, and advances:
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ const (
|
|||||||
ExtPermConnectionsRead = "connections.read"
|
ExtPermConnectionsRead = "connections.read"
|
||||||
ExtPermTriggersRegister = "triggers.register"
|
ExtPermTriggersRegister = "triggers.register"
|
||||||
ExtPermRealtimePublish = "realtime.publish"
|
ExtPermRealtimePublish = "realtime.publish"
|
||||||
|
ExtPermBatchExec = "batch.exec"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ValidExtensionPermissions is the set of recognized permission keys.
|
// ValidExtensionPermissions is the set of recognized permission keys.
|
||||||
@@ -50,6 +51,7 @@ var ValidExtensionPermissions = map[string]bool{
|
|||||||
ExtPermConnectionsRead: true,
|
ExtPermConnectionsRead: true,
|
||||||
ExtPermTriggersRegister: true,
|
ExtPermTriggersRegister: true,
|
||||||
ExtPermRealtimePublish: true,
|
ExtPermRealtimePublish: true,
|
||||||
|
ExtPermBatchExec: true,
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Extension Permission Model ───────────────
|
// ── Extension Permission Model ───────────────
|
||||||
|
|||||||
140
server/sandbox/batch_module.go
Normal file
140
server/sandbox/batch_module.go
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
// Package sandbox — batch_module.go
|
||||||
|
//
|
||||||
|
// Permission: batch.exec
|
||||||
|
//
|
||||||
|
// Starlark API:
|
||||||
|
// results, errors = batch.exec([fn_a, fn_b, fn_c], timeout=10)
|
||||||
|
//
|
||||||
|
// Runs a list of zero-arg callables concurrently, each in its own
|
||||||
|
// starlark.Thread with an independent step budget. Results and errors
|
||||||
|
// are returned in input order. Max 8 callables per call.
|
||||||
|
//
|
||||||
|
// Each branch gets a fresh Sandbox (thread + step counter) and a
|
||||||
|
// child context with the per-branch timeout. The callable's closed-over
|
||||||
|
// bindings (frozen library structs, module builtins) are used as-is —
|
||||||
|
// the Go resources underneath (*sql.DB, http.Client) are concurrent-safe.
|
||||||
|
//
|
||||||
|
// lib.require() is not available inside branches (Option A).
|
||||||
|
// print() output from branches is discarded.
|
||||||
|
// Nested batch.exec() is prohibited via an atomic flag.
|
||||||
|
package sandbox
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"go.starlark.net/starlark"
|
||||||
|
"go.starlark.net/starlarkstruct"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
batchMaxCallables = 8
|
||||||
|
batchDefaultTimeout = 10
|
||||||
|
batchMaxTimeout = 30
|
||||||
|
)
|
||||||
|
|
||||||
|
// BuildBatchModule creates the "batch" module.
|
||||||
|
// The runner/packageID/manifest/rc/lc parameters are stored for future
|
||||||
|
// expansion (Option B: per-branch lib.require) but are not used in v1.
|
||||||
|
func BuildBatchModule(
|
||||||
|
ctx context.Context,
|
||||||
|
runner *Runner,
|
||||||
|
packageID string,
|
||||||
|
manifest map[string]any,
|
||||||
|
rc *RunContext,
|
||||||
|
lc *libContext,
|
||||||
|
) *starlarkstruct.Module {
|
||||||
|
var running atomic.Bool
|
||||||
|
|
||||||
|
return MakeModule("batch", starlark.StringDict{
|
||||||
|
"exec": starlark.NewBuiltin("batch.exec", batchExec(ctx, &running)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// batchExec returns the batch.exec builtin implementation.
|
||||||
|
func batchExec(
|
||||||
|
ctx context.Context,
|
||||||
|
running *atomic.Bool,
|
||||||
|
) func(*starlark.Thread, *starlark.Builtin, starlark.Tuple, []starlark.Tuple) (starlark.Value, error) {
|
||||||
|
return func(
|
||||||
|
thread *starlark.Thread, b *starlark.Builtin,
|
||||||
|
args starlark.Tuple, kwargs []starlark.Tuple,
|
||||||
|
) (starlark.Value, error) {
|
||||||
|
// Nesting guard: parent thread is blocked in wg.Wait(), so only
|
||||||
|
// a branch goroutine could re-enter. The captured builtin reference
|
||||||
|
// makes this reachable even though branches don't get fresh modules.
|
||||||
|
if !running.CompareAndSwap(false, true) {
|
||||||
|
return nil, fmt.Errorf("batch.exec: nested calls are not allowed")
|
||||||
|
}
|
||||||
|
defer running.Store(false)
|
||||||
|
|
||||||
|
var callableList *starlark.List
|
||||||
|
var timeout int = batchDefaultTimeout
|
||||||
|
|
||||||
|
if err := starlark.UnpackArgs(b.Name(), args, kwargs,
|
||||||
|
"callables", &callableList,
|
||||||
|
"timeout?", &timeout,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
n := callableList.Len()
|
||||||
|
if n == 0 {
|
||||||
|
return nil, fmt.Errorf("batch.exec: callables list is empty")
|
||||||
|
}
|
||||||
|
if n > batchMaxCallables {
|
||||||
|
return nil, fmt.Errorf("batch.exec: max %d callables, got %d", batchMaxCallables, n)
|
||||||
|
}
|
||||||
|
if timeout < 1 || timeout > batchMaxTimeout {
|
||||||
|
timeout = batchDefaultTimeout
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate all elements are callable before spawning goroutines.
|
||||||
|
callables := make([]starlark.Callable, n)
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
c, ok := callableList.Index(i).(starlark.Callable)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("batch.exec: element %d is %s, not callable",
|
||||||
|
i, callableList.Index(i).Type())
|
||||||
|
}
|
||||||
|
callables[i] = c
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute concurrently.
|
||||||
|
results := make([]starlark.Value, n)
|
||||||
|
errors := make([]starlark.Value, n)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
|
||||||
|
for i, callable := range callables {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(idx int, fn starlark.Callable) {
|
||||||
|
defer wg.Done()
|
||||||
|
|
||||||
|
branchCtx, cancel := context.WithTimeout(ctx,
|
||||||
|
time.Duration(timeout)*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
val, _, callErr := sb.Call(branchCtx, fn, nil, nil)
|
||||||
|
|
||||||
|
if callErr != nil {
|
||||||
|
results[idx] = starlark.None
|
||||||
|
errors[idx] = starlark.String(callErr.Error())
|
||||||
|
} else {
|
||||||
|
results[idx] = val
|
||||||
|
errors[idx] = starlark.None
|
||||||
|
}
|
||||||
|
}(i, callable)
|
||||||
|
}
|
||||||
|
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
return starlark.Tuple{
|
||||||
|
starlark.NewList(results),
|
||||||
|
starlark.NewList(errors),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
324
server/sandbox/batch_module_test.go
Normal file
324
server/sandbox/batch_module_test.go
Normal file
@@ -0,0 +1,324 @@
|
|||||||
|
package sandbox
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"go.starlark.net/starlark"
|
||||||
|
)
|
||||||
|
|
||||||
|
// execWithBatch runs a Starlark script with the batch module available.
|
||||||
|
func execWithBatch(t *testing.T, script string) (*Result, error) {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
return sb.Exec(ctx, "test.star", script, map[string]starlark.Value{
|
||||||
|
"batch": batchMod,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Parallel Ordering ──────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_ParallelOrdering(t *testing.T) {
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: 10,
|
||||||
|
lambda: 20,
|
||||||
|
lambda: 30,
|
||||||
|
])
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify results are in input order.
|
||||||
|
globals := result.Globals
|
||||||
|
results := globals["results"].(*starlark.List)
|
||||||
|
errors := globals["errors"].(*starlark.List)
|
||||||
|
|
||||||
|
if results.Len() != 3 {
|
||||||
|
t.Fatalf("expected 3 results, got %d", results.Len())
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, want := range []int{10, 20, 30} {
|
||||||
|
v, _ := starlark.AsInt32(results.Index(i))
|
||||||
|
got := int(v)
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("results[%d] = %d, want %d", i, got, want)
|
||||||
|
}
|
||||||
|
if errors.Index(i) != starlark.None {
|
||||||
|
t.Errorf("errors[%d] = %v, want None", i, errors.Index(i))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Partial Failure ────────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_PartialFailure(t *testing.T) {
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
def ok_a():
|
||||||
|
return "a"
|
||||||
|
|
||||||
|
def bad():
|
||||||
|
return 1 // 0
|
||||||
|
|
||||||
|
def ok_c():
|
||||||
|
return "c"
|
||||||
|
|
||||||
|
results, errors = batch.exec([ok_a, bad, ok_c])
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
results := result.Globals["results"].(*starlark.List)
|
||||||
|
errors := result.Globals["errors"].(*starlark.List)
|
||||||
|
|
||||||
|
// results[0] = "a", results[2] = "c"
|
||||||
|
if s, _ := starlark.AsString(results.Index(0)); s != "a" {
|
||||||
|
t.Errorf("results[0] = %v, want 'a'", results.Index(0))
|
||||||
|
}
|
||||||
|
if s, _ := starlark.AsString(results.Index(2)); s != "c" {
|
||||||
|
t.Errorf("results[2] = %v, want 'c'", results.Index(2))
|
||||||
|
}
|
||||||
|
|
||||||
|
// results[1] = None (failed)
|
||||||
|
if results.Index(1) != starlark.None {
|
||||||
|
t.Errorf("results[1] = %v, want None", results.Index(1))
|
||||||
|
}
|
||||||
|
|
||||||
|
// errors[0] and errors[2] = None (succeeded)
|
||||||
|
if errors.Index(0) != starlark.None {
|
||||||
|
t.Errorf("errors[0] = %v, want None", errors.Index(0))
|
||||||
|
}
|
||||||
|
if errors.Index(2) != starlark.None {
|
||||||
|
t.Errorf("errors[2] = %v, want None", errors.Index(2))
|
||||||
|
}
|
||||||
|
|
||||||
|
// errors[1] should be a string containing the error
|
||||||
|
errStr, ok := starlark.AsString(errors.Index(1))
|
||||||
|
if !ok || errStr == "" {
|
||||||
|
t.Errorf("errors[1] = %v, want non-empty error string", errors.Index(1))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Timeout Per-Branch ─────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_TimeoutPerBranch(t *testing.T) {
|
||||||
|
// Use a tight timeout. The slow lambda burns steps in an infinite loop,
|
||||||
|
// which will hit either the step limit or the context timeout.
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
def fast():
|
||||||
|
return "done"
|
||||||
|
|
||||||
|
def slow():
|
||||||
|
x = 0
|
||||||
|
for i in range(999999999):
|
||||||
|
x = x + 1
|
||||||
|
return x
|
||||||
|
|
||||||
|
results, errors = batch.exec([fast, slow], timeout=1)
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
results := result.Globals["results"].(*starlark.List)
|
||||||
|
errors := result.Globals["errors"].(*starlark.List)
|
||||||
|
|
||||||
|
// Fast branch should succeed.
|
||||||
|
if s, _ := starlark.AsString(results.Index(0)); s != "done" {
|
||||||
|
t.Errorf("results[0] = %v, want 'done'", results.Index(0))
|
||||||
|
}
|
||||||
|
if errors.Index(0) != starlark.None {
|
||||||
|
t.Errorf("errors[0] = %v, want None", errors.Index(0))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slow branch should fail (step limit or timeout).
|
||||||
|
if results.Index(1) != starlark.None {
|
||||||
|
t.Errorf("results[1] = %v, want None", results.Index(1))
|
||||||
|
}
|
||||||
|
errStr, _ := starlark.AsString(errors.Index(1))
|
||||||
|
if errStr == "" {
|
||||||
|
t.Errorf("errors[1] should contain error string")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Parent Cancellation ────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_ParentCancellation(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel() // cancel immediately
|
||||||
|
|
||||||
|
batchMod := BuildBatchModule(ctx, nil, "test-pkg", nil, nil, nil)
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
result, err := sb.Exec(ctx, "test.star", `
|
||||||
|
results, errors = batch.exec([lambda: 1, lambda: 2])
|
||||||
|
`, map[string]starlark.Value{"batch": batchMod})
|
||||||
|
|
||||||
|
// Either the Exec itself fails (context cancelled before script runs)
|
||||||
|
// or the batch.exec branches all fail.
|
||||||
|
if err != nil {
|
||||||
|
// Script-level cancellation — acceptable.
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
errors := result.Globals["errors"].(*starlark.List)
|
||||||
|
allErrored := true
|
||||||
|
for i := 0; i < errors.Len(); i++ {
|
||||||
|
if errors.Index(i) == starlark.None {
|
||||||
|
allErrored = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !allErrored {
|
||||||
|
t.Errorf("expected all branches to error on cancelled context, errors=%v", errors)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Cap Enforcement ────────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_CapEnforcement(t *testing.T) {
|
||||||
|
_, err := execWithBatch(t, `
|
||||||
|
fns = [lambda: i for i in range(9)]
|
||||||
|
batch.exec(fns)
|
||||||
|
`)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for 9 callables, got nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "max 8") {
|
||||||
|
t.Errorf("error = %q, want to contain 'max 8'", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Empty List ─────────────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_EmptyList(t *testing.T) {
|
||||||
|
_, err := execWithBatch(t, `batch.exec([])`)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for empty list, got nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "empty") {
|
||||||
|
t.Errorf("error = %q, want to contain 'empty'", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Non-Callable Element ───────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_NonCallableElement(t *testing.T) {
|
||||||
|
_, err := execWithBatch(t, `batch.exec([1, 2])`)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for non-callable, got nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "not callable") {
|
||||||
|
t.Errorf("error = %q, want to contain 'not callable'", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Permission Gating ──────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_PermissionGating(t *testing.T) {
|
||||||
|
sb := New(DefaultConfig())
|
||||||
|
_, err := sb.Exec(context.Background(), "test.star",
|
||||||
|
`batch.exec([lambda: 1])`,
|
||||||
|
map[string]starlark.Value{}, // no batch module
|
||||||
|
)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error when batch module not available")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "batch") {
|
||||||
|
t.Errorf("error = %q, want to mention 'batch'", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Frozen Struct Sharing ──────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_FrozenStructSharing(t *testing.T) {
|
||||||
|
// Two branches read the same frozen tuple. Should not race.
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
data = (1, 2, 3) # tuples are frozen/immutable
|
||||||
|
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: data[0] + data[1],
|
||||||
|
lambda: data[1] + data[2],
|
||||||
|
])
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
results := result.Globals["results"].(*starlark.List)
|
||||||
|
|
||||||
|
got0, _ := starlark.AsInt32(results.Index(0))
|
||||||
|
got1, _ := starlark.AsInt32(results.Index(1))
|
||||||
|
if got0 != 3 {
|
||||||
|
t.Errorf("results[0] = %d, want 3", got0)
|
||||||
|
}
|
||||||
|
if got1 != 5 {
|
||||||
|
t.Errorf("results[1] = %d, want 5", got1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Nested batch.exec ──────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_NestedBatchExec(t *testing.T) {
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
results, errors = batch.exec([
|
||||||
|
lambda: batch.exec([lambda: 1]),
|
||||||
|
])
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The inner batch.exec should fail with nesting error.
|
||||||
|
errors := result.Globals["errors"].(*starlark.List)
|
||||||
|
errStr, _ := starlark.AsString(errors.Index(0))
|
||||||
|
if !strings.Contains(errStr, "nested") {
|
||||||
|
t.Errorf("errors[0] = %q, want to contain 'nested'", errStr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Default Timeout ────────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_DefaultTimeout(t *testing.T) {
|
||||||
|
// Call without explicit timeout — should succeed.
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
results, errors = batch.exec([lambda: 42])
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
results := result.Globals["results"].(*starlark.List)
|
||||||
|
got, _ := starlark.AsInt32(results.Index(0))
|
||||||
|
if got != 42 {
|
||||||
|
t.Errorf("results[0] = %d, want 42", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Timeout Clamping ───────────────────────
|
||||||
|
|
||||||
|
func TestBatchExec_TimeoutClamp(t *testing.T) {
|
||||||
|
// timeout=0 should be clamped to default, not crash.
|
||||||
|
start := time.Now()
|
||||||
|
result, err := execWithBatch(t, `
|
||||||
|
results, errors = batch.exec([lambda: "ok"], timeout=0)
|
||||||
|
`)
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("exec error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
results := result.Globals["results"].(*starlark.List)
|
||||||
|
if s, _ := starlark.AsString(results.Index(0)); s != "ok" {
|
||||||
|
t.Errorf("results[0] = %v, want 'ok'", results.Index(0))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should complete quickly (clamped to default 10s, but lambda is instant).
|
||||||
|
if elapsed > 2*time.Second {
|
||||||
|
t.Errorf("took %v, expected fast completion", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -338,6 +338,7 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
|
|
||||||
// Track db permission level: 0=none, 1=read, 2=write
|
// Track db permission level: 0=none, 1=read, 2=write
|
||||||
dbLevel := 0
|
dbLevel := 0
|
||||||
|
hasBatchExec := false
|
||||||
|
|
||||||
for _, perm := range granted {
|
for _, perm := range granted {
|
||||||
switch perm {
|
switch perm {
|
||||||
@@ -374,6 +375,9 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
if r.bus != nil {
|
if r.bus != nil {
|
||||||
modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID)
|
modules["realtime"] = BuildRealtimeModule(ctx, r.bus, packageID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
case models.ExtPermBatchExec:
|
||||||
|
hasBatchExec = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -404,5 +408,12 @@ func (r *Runner) buildModulesWithLibCtx(ctx context.Context, packageID string, m
|
|||||||
modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc)
|
modules["lib"] = BuildLibModule(ctx, r, packageID, rc, lc)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Concurrent execution primitive — wired after all other modules
|
||||||
|
// so the runner reference can construct per-branch module sets if
|
||||||
|
// needed in future (Option B).
|
||||||
|
if hasBatchExec {
|
||||||
|
modules["batch"] = BuildBatchModule(ctx, r, packageID, manifest, rc, lc)
|
||||||
|
}
|
||||||
|
|
||||||
return modules, nil
|
return modules, nil
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user