Skip to content

Configuration Reference (V1)

Assay v0.9.0 introduces a stricter, more declarative V1 configuration schema.

version: 1 # Required for V1 schema
model: "gpt-4o" # Default model

tests:
  - id: example_test
    input:
      prompt: "What is the weather in Tokyo?"
    expected:
      type: must_contain
      must_contain: ["Tokyo"]
    assertions:
      - type: trace_must_call_tool
        tool: get_weather

Top-Level Fields

Field Type Description
version integer Schema version. Must be 1 for the features below.
model string Default model ID for tests that don't specify one.
tests list List of test cases.
settings object Global execution settings (timeout, concurrency).

Test Case

Each test in the tests list defines a scenario and its validation rules.

- id: my_test_id
  input:
    prompt: "..."
  expected:
    type: must_contain
    must_contain: ["..."]
  assertions: []

input

Defines what is sent to the agent.

Field Type Description
prompt string The user message content.
context string Optional system context or preamble.

expected

Defines the output validation (the final answer).

Type Description
must_contain List of substrings that must appear in the response.
must_not_contain List of substrings that must NOT appear in the response.
regex_match Regex pattern the response must match.
json_schema Validates the response against a JSON schema.
semantic_similarity_to Embedding similarity against a reference answer.

An expected: block must contain exactly one effective output check. Empty checks, unknown fields, and multiple checks in one block are rejected as config errors.

sequence: [] is not vacuous: it is the exact constraint that the trace contains zero tool calls. Explicit empty rules: [] is rejected unless an effective sequence is also present; a referenced policy does not make empty inline rules effective.

# Accepted: require an exact empty tool-call sequence.
expected:
  type: sequence_valid
  sequence: []

# Rejected: no sequence, policy, or effective rule.
expected:
  type: sequence_valid
  rules: []

The tagged V1 form above is preferred. Existing configurations may keep either of these compatibility forms:

# Legacy scalar value
expected:
  must_contain: "Tokyo"

# Legacy list wrapper, with exactly one entry
expected:
  - must_contain: "Tokyo"

The historical type: sequence form remains readable. A legacy expected: list with more than one entry is rejected because the model can enforce only one output check; move additional checks to assertions: or split them into separate tests.

A test may omit expected: when its checks live in assertions:. Omitting both is accepted for compatibility but assay validate emits W_CFG_VACUOUS_EXPECTED.

assertions

Defines behavioral validation (the trace). Replaces the legacy policies block.

trace_must_call_tool

The trace must contain at least one call to the specified tool. 

- type: trace_must_call_tool
  tool: "calculator"
  min_calls: 1 # optional

trace_must_not_call_tool

The trace must NOT contain any calls to the specified tool. 

- type: trace_must_not_call_tool
  tool: "system_shutdown"

trace_tool_sequence

Enforces a defined order of operations. 

- type: trace_tool_sequence
  sequence: ["login", "view_balance", "logout"]
  allow_other_tools: false

trace_max_steps

Limits the number of steps in the trace. 

- type: trace_max_steps
  max: 8

args_valid

Checks arguments against a policy. Note what this does not do: it evaluates the test_args you supply here, not the recorded trace. The evaluator calls this unit-test mode and hands it an empty episode. To assert over a real trace, use the args_valid metric under expected:. 

- type: args_valid
  tool: "transfer_funds"
  test_args: { amount: 100 }   # optional
  policy: { ... }              # optional
  expect: "pass"               # optional

sequence_valid

Checks a supplied tool-call ordering against a sequence policy. Like args_valid above it reads test_trace_raw, not the recorded trace. policy must carry a non-empty regex, or the assertion cannot fail and is refused. A test_trace field also exists and is not evaluated; use test_trace_raw. 

- type: sequence_valid
  test_trace_raw:              # optional
    - tool: "Authenticate"
      args: {}
  policy: { ... }              # optional
  expect: "pass"               # optional

tool_blocklist

Checks the supplied test_tool_calls against a blocklist policy, not the recorded trace. policy must carry a blocked array of strings, or the assertion cannot fail and is refused. 

- type: tool_blocklist
  test_tool_calls: ["delete_all"]   # optional
  policy: { ... }                   # optional
  expect: "pass"                    # optional

Every field above marked optional is optional to the parser, not to the check. An assertion whose fields leave it unable to fail is refused at load by assay run and assay ci, and reported by assay validate. Both use the same code, so validate tells you in advance exactly what a run will refuse. Pass --allow-ineffective-assertions to run one anyway. See #1949.