Assay Codebase Overview¶

Version: 2.15.0 (February 2026) SOTA Status: Bleeding Edge (Judge Reliability, MCP Auth, OTel GenAI, Replay Bundle)

What is Assay?¶

Assay is a Policy-as-Code engine for Model Context Protocol (MCP) that validates AI agent behavior. It provides:

• Deterministic testing: Replay recorded traces without LLM API calls (milliseconds, $0 cost, 0% flakiness)
• Runtime security: Kernel-level enforcement on Linux to block unauthorized tool access
• Compliance gates: Validate tool arguments, sequences, and blocklists before production

Assay replaces flaky, network-dependent evals with deterministic replay testing. Record agent behavior once, then validate every PR in milliseconds.

High-Level Architecture¶

Assay is a Rust monorepo with multiple crates, a Python SDK, and comprehensive documentation.

Core Crates¶

Python SDK¶

Located in assay-python-sdk/python/assay/:

• client.py: AssayClient for recording traces to JSONL
• coverage.py: Coverage for analyzing policy coverage
• explain.py: Human-readable explanations of policy violations
• pytest_plugin.py: Pytest integration for automatic trace capture

GitHub Action¶

Repository: https://github.com/Rul1an/assay/tree/main/assay-action

- uses: Rul1an/assay/assay-action@v2

Features: - Zero-config evidence bundle discovery - SARIF integration with GitHub Security tab - PR comments (only when findings) - Baseline comparison via cache - Artifact upload

See ADR-014 for design details.

Documentation & Examples¶

• docs/: Concepts, use cases, integration guides, reference documentation
• examples/: Concrete YAML configs, traces, and scenarios (RAG, baseline gate, negation safety)

Core Components in Detail¶

assay-core Structure¶

The core crate is organized into these main modules:

Engine (engine/)¶

• Runner: Central orchestrator
• run_suite(): Parallel test execution with semaphore
• run_test_with_policy(): Retries, policy checks, quarantine, agent assertions
• run_test_once(): Fingerprinting, cache lookup, LLM call/replay, metrics evaluation, baseline check

Storage (storage/)¶

• Store: SQLite wrapper for runs, results, attempts, embeddings, judge cache
• Schema: runs, results, attempts, embeddings, episodes/steps (for trace ingestion)
• Methods: create_run(), insert_result_embedded(), get_last_passing_by_fingerprint()

Trace (trace/)¶

• ingest: JSONL traces → database
• precompute: Pre-compute embeddings and judge results for deterministic, fast runs
• verify, upgrader, otel_ingest: Schema validation, version migration, OpenTelemetry ingest

MCP (mcp/)¶

• JSON-RPC parsing, tool call mapping to policies, audit logging
• mapper_v2: Maps MCP tool calls to policy checks
• proxy: Intercepts and validates tool calls, ProxyConfig with logging paths
• identity: Tool identity management (Phase 9) - tool metadata hashing and pinning
• policy: McpPolicy with tool_pins for integrity verification
• jcs: JCS canonicalization (RFC 8785) for deterministic JSON
• signing: Ed25519 tool signing with DSSE PAE encoding (sign_tool, verify_tool)
• trust_policy: Trust policy loading (require_signed, trusted_key_ids)
• decision: DecisionEmitter for tool.decision events, reason codes (P_, M_, S_*)
• lifecycle: LifecycleEmitter for mandate.used/revoked events (CloudEvents)
• tool_call_handler: Central handler integrating policy + mandate authorization

Runtime (runtime/)¶

• mandate_store: SQLite-backed mandate consumption tracking
• AuthzReceipt with was_new flag for idempotent retries
• RevocationRecord for mandate cancellation
• Deterministic use_id computation (content-addressed SHA256)
• Tables: mandates, mandate_uses, nonces, mandate_revocations
• authorizer: 7-step authorization flow per SPEC-Mandate §7.6-7.8
• Validity window check (with ±30s skew)
• Revocation check (no skew - hard cutoff)
• Scope and kind verification
• transaction_ref verification for commit tools
• Atomic consumption
• schema: SQLite DDL for mandate runtime tables (schema v3)

Report (report/)¶

• Output formatters: console (summary), json, junit, sarif
• RunArtifacts: Container for run_id, suite, results
• Summary (summary.rs): Machine-readable run summary with schema_version, reason_code_version, exit_code, reason_code, seeds (required; Seeds with order_seed/judge_seed as string or null via serde_seed), judge_metrics (optional; abstain_rate, flip_rate, etc.). Summary::with_seeds() injects seeds; written to summary.json and reflected in run.json.
• print_run_footer (console.rs): Prints one line Seeds: seed_version=1 order_seed=… judge_seed=… and judge metrics line to stderr (CI job summary visibility). Called from assay-cli after run/ci.
• run.json / summary.json: Contract per SPEC-PR-Gate-Outputs-v1 (§3.3.1 Seeds, §3.3.2 Judge metrics). Seeds are decimal strings or null for JS/TS precision safety.

Providers & Metrics API¶

• providers/: LLM clients (OpenAI, fake, trace replay), embedders, strict mode wrappers
• metrics_api.rs: Trait definitions that assay-metrics implements

Other Key Modules¶

• baseline/: Compares new scores with historical baselines
• quarantine.rs: Marks and skips flaky tests
• agent_assertions/: Enforces sequence and structural expectations on traces (e.g., tool call order)

assay-cli Flow¶

1. Entry: main.rs parses Cli args → calls dispatch()
2. Command handling: dispatch() matches command → calls handler (e.g., cmd_run())
3. Runner construction: build_runner() creates Runner:
4. Opens Store (SQLite)
5. Creates VcrCache
6. Selects LLM client (trace replay or live)
7. Loads metrics from assay-metrics
8. Configures embedder/judge/baseline if provided
9. Execution: Runner::run_suite() → parallel run_test_with_policy() → run_test_once() → LLM call → metric evaluation → store results
10. Reporting: RunArtifacts → formatters (console/JSON/JUnit/SARIF)

assay-metrics Metrics¶

Metrics are composable building blocks:

• Content metrics: MustContain, MustNotContain, RegexMatch
• Semantic metrics: SemanticSimilarity, Faithfulness, Relevance (using embedder/judge)
• Structure/usage: ArgsValid, SequenceValid, ToolBlocklist, Usage
• JSON validation: JsonSchema for argument validation

Integration: CLI loads a standard set via default_metrics(), and policies reference these metrics per testcase.

MCP, Policies, Monitor & LSM¶

Policy Compilation (assay-policy)¶

• Policies are compiled into a CompiledPolicy with:
• Tier 1: Kernel/LSM rules (exact paths, CIDRs, ports)
• Tier 2: Userspace rules (glob/regex, complex constraints)

Monitor & eBPF (assay-monitor, assay-common, assay-ebpf)¶

• eBPF programs run in kernel
• Userspace monitor reads events and applies Tier 1 rules
• assay-common contains no_std-compatible structs for event types, keys, etc.

MCP Server (assay-mcp-server)¶

• Runs as MCP proxy via stdio (JSON-RPC)
• Inspects tool calls, applies policies, makes deny/allow decisions
• Handles rate limiting, audit logging

Execution Flow (CLI → Core)¶

User Command
    ↓
CLI (main.rs)
    ↓
dispatch() → Command Handler
    ↓
build_runner()
    ├─→ Store (SQLite)
    ├─→ VcrCache
    ├─→ LLM Client (trace replay or live)
    ├─→ Metrics (from assay-metrics)
    ├─→ Embedder (optional)
    ├─→ Judge (optional)
    └─→ Baseline (optional)
    ↓
Runner::run_suite()
    ↓
Parallel run_test_with_policy()
    ↓
run_test_once()
    ├─→ Fingerprinting
    ├─→ Cache lookup
    ├─→ LLM call (or replay)
    ├─→ Metrics evaluation
    └─→ Baseline check
    ↓
Store results
    ↓
Report (console/JSON/JUnit/SARIF; SARIF truncation at 25k results by default, with sarif.omitted in run/summary when truncated — PR #160)

Key Design Principles¶

1. Determinism: Same input + same policy = same result (zero flakiness)
2. Statelessness: Validation requires only policy file + trace list
3. Policy-as-Code: Uses logic, not LLMs, for evaluation
4. Separation of Concerns: CLI handles UX/config, core handles evaluation logic
5. Extensibility: Metrics, providers, and policies are pluggable via traits

SOTA Features (January 2026)¶

Exit Codes & Reason Codes¶

Exit Codes (Coarse, Stable)¶

Reason Codes (Fine-Grained)¶

Compatibility¶

• --exit-codes=v2 (default): New exit code mapping
• --exit-codes=v1: Legacy mapping (exit 3 = trace not found)
• Environment: ASSAY_EXIT_CODES=v1|v2

Note: Always use reason_code in summary.json for programmatic handling, not exit codes.

Extension Points¶

New Metrics¶

• Implement in crates/assay-metrics/src/ following the Metric trait
• Register in factory (e.g., default_metrics()) so policies can use them

New CLI Commands¶

• Add to assay-cli (CLI structure, command handler)
• Wire to build_runner() / Runner if needed

New Policy Features¶

• Extend policy engine in assay-core (parser/validator, constraints)
• Map to assay-policy for Tier ½ compilation

New Python SDK Features¶

• Add thin wrappers in assay-python-sdk/python/assay/ around existing CLI/core functionality

Related Documentation¶

• User Flows - How users interact with the system
• Interdependencies - Crate relationships and interfaces
• Architecture Diagrams - Visual architecture representations
• Entry Points - All interaction points

Architecture Decision Records¶

Key ADRs for understanding the codebase:

See ADR Index for the complete list.