Assay Codebase Overview¶

Authority: ADR-042 defines product scope; Product Support records capability status. This page is a navigation map, not release proof.

What is Assay?¶

Assay publishes an open evidence profile for privileged MCP tool actions. The enforcing proxy is its reference producer. Decisions, observations and external outcomes retain their distinct evidence boundaries. Supporting surfaces include:

• Protocol policy enforcement: Deterministic MCP tool decisions without LLM calls in CI
• Evidence bundles: Offline-verifiable artifacts for audit, review, and replay
• Trust compiler artifacts: Trust Basis and Trust Card outputs with explicit evidence levels
• External receipt lanes: Bounded receipt imports for selected eval, decision, inventory, and score-event surfaces

Derived reports and imported receipts do not establish provider-side execution, an aggregate trust score, compliance or a safe-agent claim.

High-Level Architecture¶

Assay is a Rust monorepo with multiple crates, a Python SDK, and comprehensive documentation.

Core Crates¶

Python SDK¶

Located in assay-python-sdk/python/assay/:

• client.py: AssayClient for recording traces to JSONL
• coverage.py: Coverage for analyzing policy coverage
• explain.py: Human-readable explanations of policy violations
• pytest_plugin.py: Pytest integration for automatic trace capture

GitHub Action¶

Repository: https://github.com/Rul1an/assay-action

- uses: Rul1an/assay-action@v3

Features: - Zero-config evidence bundle discovery - SARIF integration with GitHub Security tab - PR comments (only when findings) - Baseline comparison via cache - Artifact upload

See ADR-014 for design details.

Documentation & Examples¶

• docs/: Concepts, use cases, integration guides, reference documentation
• examples/: Concrete YAML configs, traces, and scenarios (RAG, baseline gate, negation safety)

Core Components in Detail¶

assay-core Structure¶

The core crate is organized into these main modules:

Engine (engine/)¶

• Runner: Central orchestrator
• run_suite(): Parallel test execution with semaphore
• run_test_with_policy(): Retries, policy checks, quarantine, agent assertions
• run_test_once(): Fingerprinting, cache lookup, LLM call/replay, metrics evaluation, baseline check

Storage (storage/)¶

• Store: SQLite wrapper for runs, results, attempts, embeddings, judge cache
• Schema: runs, results, attempts, embeddings, episodes/steps (for trace ingestion)
• Methods: create_run(), insert_result_embedded(), get_last_passing_by_fingerprint()

Trace (trace/)¶

• ingest: JSONL traces → database
• precompute: Pre-compute embeddings and judge results for deterministic, fast runs
• verify, upgrader, otel_ingest: Schema validation, version migration, OpenTelemetry ingest

MCP (mcp/)¶

• JSON-RPC parsing, tool call mapping to policies, audit logging
• mapper_v2: Maps MCP tool calls to policy checks
• proxy: Intercepts and validates tool calls, ProxyConfig with logging paths
• identity: Tool identity management (Phase 9) - tool metadata hashing and pinning
• policy: McpPolicy with tool_pins for integrity verification
• jcs: JCS canonicalization (RFC 8785) for deterministic JSON
• signing: Ed25519 tool signing with DSSE PAE encoding (sign_tool, verify_tool)
• trust_policy: Trust policy loading (require_signed, trusted_key_ids)
• decision: DecisionEmitter for tool.decision events, reason codes (P_, M_, S_*)
• lifecycle: LifecycleEmitter for mandate.used/revoked events (CloudEvents)
• tool_call_handler: Central handler integrating policy + mandate authorization

Runtime (runtime/)¶

• mandate_store: SQLite-backed mandate consumption tracking
• AuthzReceipt with was_new flag for idempotent retries
• RevocationRecord for mandate cancellation
• Deterministic use_id computation (content-addressed SHA256)
• Tables: mandates, mandate_uses, nonces, mandate_revocations
• authorizer: 7-step authorization flow per SPEC-Mandate §7.6-7.8
• Validity window check (with ±30s skew)
• Revocation check (no skew - hard cutoff)
• Scope and kind verification
• transaction_ref verification for commit tools
• Atomic consumption
• schema: SQLite DDL for mandate runtime tables (schema v3)

Report (report/)¶

• Output formatters: console (summary), json, junit, sarif
• RunArtifacts: Container for run_id, suite, results
• Summary (summary.rs): Machine-readable run summary with schema_version, reason_code_version, exit_code, reason_code, seeds (required; Seeds with order_seed/judge_seed as string or null via serde_seed), judge_metrics (optional; abstain_rate, flip_rate, etc.). Summary::with_seeds() injects seeds; written to summary.json and reflected in run.json.
• print_run_footer (console.rs): Prints one line Seeds: seed_version=1 order_seed=… judge_seed=… and judge metrics line to stderr (CI job summary visibility). Called from assay-cli after run/ci.
• run.json / summary.json: Contract per SPEC-PR-Gate-Outputs-v1 (§3.3.1 Seeds, §3.3.2 Judge metrics). Seeds are decimal strings or null for JS/TS precision safety.

Providers & Metrics API¶

• providers/: LLM clients (OpenAI, fake, trace replay), embedders, strict mode wrappers
• metrics_api.rs: Trait definitions that assay-metrics implements

Other Key Modules¶

• baseline/: Compares new scores with historical baselines
• quarantine.rs: Marks and skips flaky tests
• agent_assertions/: Enforces sequence and structural expectations on traces (e.g., tool call order)

assay-cli Flow¶

1. Entry: main.rs parses Cli args → calls dispatch()
2. Command handling: dispatch() matches command → calls handler (e.g., cmd_run())
3. Runner construction: build_runner() creates Runner:
4. Opens Store (SQLite)
5. Creates VcrCache
6. Selects LLM client (trace replay or live)
7. Loads metrics from assay-metrics
8. Configures embedder/judge/baseline if provided
9. Execution: Runner::run_suite() → parallel run_test_with_policy() → run_test_once() → LLM call → metric evaluation → store results
10. Reporting: RunArtifacts → formatters (console/JSON/JUnit/SARIF)

assay-metrics Metrics¶

Metrics are composable building blocks:

• Content metrics: MustContain, MustNotContain, RegexMatch
• Semantic metrics: SemanticSimilarity, Faithfulness, Relevance (using embedder/judge)
• Structure/usage: ArgsValid, SequenceValid, ToolBlocklist, Usage
• JSON validation: JsonSchema for argument validation

Integration: CLI loads a standard set via default_metrics(), and policies reference these metrics per testcase.

MCP, Policies, Monitor & LSM¶

Policy Compilation (assay-policy)¶

• Policies are compiled into a CompiledPolicy with:
• Tier 1: Kernel/LSM rules (exact paths, CIDRs, ports)
• Tier 2: Userspace rules (glob/regex, complex constraints)

Monitor & eBPF (assay-monitor, assay-common, assay-ebpf)¶

• eBPF programs run in kernel
• Userspace monitor reads events and applies Tier 1 rules
• assay-common contains no_std-compatible structs for event types, keys, etc.

MCP Server (assay-mcp-server)¶

• Runs as MCP proxy via stdio (JSON-RPC)
• Inspects tool calls, applies policies, makes deny/allow decisions
• Handles rate limiting, audit logging

Execution Flow (CLI → Core)¶

User Command
    ↓
CLI (main.rs)
    ↓
dispatch() → Command Handler
    ↓
build_runner()
    ├─→ Store (SQLite)
    ├─→ VcrCache
    ├─→ LLM Client (trace replay or live)
    ├─→ Metrics (from assay-metrics)
    ├─→ Embedder (optional)
    ├─→ Judge (optional)
    └─→ Baseline (optional)
    ↓
Runner::run_suite()
    ↓
Parallel run_test_with_policy()
    ↓
run_test_once()
    ├─→ Fingerprinting
    ├─→ Cache lookup
    ├─→ LLM call (or replay)
    ├─→ Metrics evaluation
    └─→ Baseline check
    ↓
Store results
    ↓
Report (console/JSON/JUnit/SARIF; SARIF truncation at 25k results by default, with sarif.omitted in run/summary when truncated — PR #160)

Key Design Principles¶

1. Deterministic policy: Explicit policy evaluation is separate from judge, host and provider variability
2. Statelessness: Validation requires only policy file + trace list
3. Deterministic policy: Uses explicit logic, not LLMs, for policy decisions
4. Separation of Concerns: CLI handles UX/config, core handles evaluation logic
5. Extensibility: Metrics, providers, and policies are pluggable via traits

Capability Status¶

Use the support reference, rather than a second roadmap table here. The local stdio server and proxies reject unsupported transport-auth configuration; a historical OAuth/JWKS plan is not a supported mode. See the editor recipe before choosing between review-tool installation and enforcement of a target server.

Exit Codes & Reason Codes¶

Exit Codes (Coarse, Stable)¶

Reason Codes (Fine-Grained)¶

Compatibility¶

• --exit-codes=v2 (default): New exit code mapping
• --exit-codes=v1: Legacy mapping (exit 3 = trace not found)
• Environment: ASSAY_EXIT_CODES=v1|v2

Note: Always use reason_code in summary.json for programmatic handling, not exit codes.

Extension Points¶

New Metrics¶

• Implement in crates/assay-metrics/src/ following the Metric trait
• Register in factory (e.g., default_metrics()) so policies can use them

New CLI Commands¶

• Add to assay-cli (CLI structure, command handler)
• Wire to build_runner() / Runner if needed

New Policy Features¶

• Extend policy engine in assay-core (parser/validator, constraints)
• Map to assay-policy for Tier ½ compilation

New Python SDK Features¶

• Add thin wrappers in assay-python-sdk/python/assay/ around existing CLI/core functionality

Related Documentation¶

• User Flows - How users interact with the system
• Interdependencies - Crate relationships and interfaces
• Architecture Diagrams - Visual architecture representations
• Entry Points - All interaction points

Architecture Decision Records¶

Key ADRs for understanding the codebase:

See ADR Index for the complete list.