CLI Reference¶

Complete documentation for all Assay commands.

Planning note: see CLI Command Grouping RFC for the selective noun-verb grouping direction and migration contract.

Installation¶

# Rust
cargo install assay-cli --version 6.19.1 --locked
# Or via installer scripts (see Home)

Verify installation:

assay --version
# assay 6.19.1

Commands Overview¶

Global Options¶

Common top-level options. --color is global: it parses before or after the subcommand. --quiet and --non-interactive are top-level only: place them before the subcommand (assay --non-interactive doctor --fix ...); after a subcommand each is a clap usage error (except where a command defines its own local spelling: sandbox, monitor, and mcp tool verify define their own local --quiet with its own meaning, and setup keeps a hidden --non-interactive alias for the same posture).

Caller posture precedence (#2573, slices S1–S2)¶

Rules:

• Flag beats env beats convention beats detection. --color beats NO_COLOR; --quiet never beats diagnostics — warnings, reason codes, fatal errors, and stdout documents are always emitted, with no exceptions: neither the top-level flag nor ASSAY_QUIET reaches another command's local quiet (sandbox, monitor, mcp tool verify keep theirs).
• --non-interactive is fail-closed: with the flag or env set, any code path that would prompt refuses with exit 2 and names the remedy (--yes for confirms, OPENAI_API_KEY for the embedder secret), even on a real terminal — like certbot's --non-interactive ("run without ever asking for user input"), unlike zypper/debconf non-interactive mode, which assumes defaults. assay watch (a rerun loop with no one-shot meaning) refuses at startup under the flag. Explicit consent still works: --yes / --dry-run proceed as before.
• NO_COLOR keeps its long-standing meaning: present (even empty) disables decoration under --color auto.
• FORCE_COLOR and CLICOLOR_FORCE are deliberately not honoured (agent CI images export FORCE_COLOR; ANSI must not reach parsed output) — like anstream, which reads no FORCE_COLOR variable.
• The colour rule above governs the operator-diagnostic sites (run/ci/watch/replay failures); validate and demo still render format_terminal() unconditionally and the assay-core legacy policy path still emits raw ANSI — all pre-existing, all not yet covered (follow-up). (Machine-output aliases arrive in a later slice.)

Quick Examples¶

Run Tests¶

# Basic run
assay run --config eval.yaml

# Strict mode (fail on any violation)
assay run --config eval.yaml --strict

# Specific trace file
assay run --config eval.yaml --trace-file traces/golden.jsonl

# Machine-readable run report on stdout
assay run --config eval.yaml --trace-file traces/golden.jsonl --format json > results.json

# CI reports
assay ci --config eval.yaml --trace-file traces/golden.jsonl --sarif sarif.json --junit junit.xml

Generate Policy (With Diff Preview)¶

# Generate policy from trace
assay policy generate --input traces/session.jsonl --output policy.yaml

# Preview changes against existing policy file
assay policy generate --input traces/session.jsonl --output policy.yaml --diff --dry-run

Replay Bundles¶

# Create bundle from latest run artifacts
assay bundle create

# Verify bundle safety/integrity
assay bundle verify --bundle .assay/bundles/12345.tar.gz

# Replay from bundle (offline default)
assay replay --bundle .assay/bundles/12345.tar.gz

# Replay live with seed override
assay replay --bundle .assay/bundles/12345.tar.gz --live --seed 42

Migrate Config¶

# Upgrade to v1 format
assay migrate --config old-eval.yaml

# Preview changes without writing
assay migrate --config old-eval.yaml --dry-run

Validate Config And Trace¶

# Positional config path
assay validate eval.yaml --trace-file traces/golden.jsonl

# Equivalent explicit config flag
assay validate --config eval.yaml --trace-file traces/golden.jsonl

MCP Runtime¶

# Enforcing mode
assay mcp wrap --policy assay.yaml -- <real-mcp-command> [args...]

# Dry-run mode
assay mcp wrap --policy assay.yaml --dry-run -- <real-mcp-command> [args...]

# Discover local MCP servers
assay mcp discover --format json

# Review and explicitly promote one complete observed tool manifest
assay mcp manifest candidate --from-observed observed.json --out candidate.json
assay mcp manifest promote --candidate candidate.json --source observed.json --out declared.json

# Sign or verify MCP tool definitions
assay mcp tool sign tool.json --key private.pem --out signed.json

Diagnose and Watch¶

# Diagnose and auto-fix known issues
assay doctor --config eval.yaml --trace-file traces/dev.jsonl --fix --yes

# Live re-run loop on local edits
assay watch --config eval.yaml --trace-file traces/dev.jsonl --strict

Exit Codes¶

Environment Variables¶

Configuration File¶

Most run/ci commands read from eval.yaml by default:

version: 1
suite: my-agent
model: gpt-4o-mini
tests:
  - id: args_valid
    input:
      prompt: "Summarize this task."
    expected:
      type: args_valid
      policy: policies/default.yaml

See Configuration for full reference.

Command Details¶