CATT 0.2.1 釋出 CI 測試代理/服務工具

Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.

Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.

Contributing | Security | Changelog | MIT license

A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented, unofficial, assumed or unknown. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.

What you can test today

The connector lane records the basis for each Muse-specific behavior. Tool classes and approval prompts follow documented guidelines. The muse-cli projection and template client follow unofficial mirrors; the native projection is an assumed generic baseline. The neutral cli projection supports both targets as a CATT treatment. Instinct runs have a separate source record and label the shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter; no public Instinct MCP integration contract was found.

Run the local booking example under Instinct without a model or account:

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json \
  --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"

Set emulation_target: instinct in your scenario or pass --emulation-target instinct. Both targets use native or cli projection and the generic MCP client. Instinct defaults to plain-text replies and does not run Muse guideline checks. Its local wallet fixture covers payment behavior; live payment rails, trusted-network protocol, Vault and Files remain outside the shared contract.

Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.

Setup

You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), uv, and Node.js 20 or newer with npm (the installed packages declare >=20; tested with 25.9). The recorded release evidence covers macOS. From a clone of this repository, install the Python and Node dependencies and the Chromium build the browser lanes use:

uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium

The connector lane, the readiness checks and the scripted runs need only uv sync. Node and Chromium are for the MCP Apps and browser lanes.

On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.

sudo apt-get update && sudo apt-get install -y nodejs npm   # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
  node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium

Three features rely on macOS Seatbelt (sandbox-exec) and report unsupported or blocked on Linux rather than running unconfined: the --offline readiness mode, the network-denied validator workers behind run --surface api-tools, and the local-model memory profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it.

Check the setup with doctor:

uv run consumer-agent-testing-tool doctor

It checks Node against the floors the installed packages declare, the tooling/payment-ui packages against their pinned versions, and Chromium and its headless shell under .cache/ms-playwright. It reports the optional protocol tooling, and checks Docker only for development lanes or configured stdio MCP and CLI targets. It makes no model requests, even when an existing configuration or --model-profile selects a model. Each failed or blocked check prints one line on stderr with its reason and, for a setup problem, the command that fixes it.

To explicitly run the provider preflight, which sends two short model requests and may incur charges, add --model-preflight:

uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight

Scripted runs need no model account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a model profile and an API key; Model providers shows how to create one. Supply credentials through environment variables, and keep them out of scenarios and shared evidence. consumer-agent-testing-tool.yaml, the default settings file, is local and git-ignored, and no profile is shipped.

Quickstart: connector lane

This quickstart stays on loopback and needs no account. Complete Setup, then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none

Replay the example's scripted calls, with no model:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json

The script searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's private: values, which the model never sees, and the scripted user answers each approval card from the scenario's permissions: rules. After the run, the host checks the fixture's private oracle (one confirmed booking, no duplicate charges) and the booking's status through get_booking. It also checks that each sent call matched its approval card (approval.dispatches_match, approval.disclosure_bound) and that the final answer made no false claim and relayed the total and the confirmation (answer.no_false_claims, answer.relay_complete).

The command prints the run directory and Connector report: .../report.html, and exits 0 when every required check passed. Runs are sealed under runs/connector-<id>/. Share the output of export, not the run directory: its private/ folder holds host-private values.

For an agent run, create agent.yaml with the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

The model preflight runs first and prints its own evidence path. The agent then sees the four allowed tools as native functions, with the host-private email removed from their schemas, and writes the other arguments itself. This uses paid model inference. To change the treatment without editing the scenario, --projection cli offers the same tools through one exec command for either target, --exposure search-only loads their parameters through tool_search, --output-schema strict refuses results that break a tool's outputSchema as the official MCP SDK does, and --trials N runs independent trials and prints pass^k. The run lock records the projection, exposure, client-profile and output-schema overrides under overrides.

connector-run exits 0 when every required check passed; 1 when a check failed, the answer made a false claim or an agent limit was reached; 2 for a setup, harness or reporting error; and 3 when the trial is blocked or unsupported, for example by an uncertain write to reconcile, an unobservable check or a client-profile limit. To start from your own server, run scaffold; to replay a recorded, shareable agent trial without a model, turn it into a calls file with scaffold --from-run. The connector lane reference covers scenarios, flags and evidence. The examples list each shipped scenario with its fixture command, and lane extensions covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions.

Quickstart: a popular MCP server

The connector lane works with any MCP server over Streamable HTTP, not only the bundled test service. This quickstart points it at "Everything" (@modelcontextprotocol/server-everything, MIT), the reference server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no account or key. Start it in a second terminal; the first run downloads the pinned release from npm:

PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp

See what the host infers from the live catalog. scaffold runs the handshake and tools/list, calls no tool, and writes a scenario skeleton with suggested tool classes from the server's annotations:

uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines

The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --projection muse-cli --client-profile muse-template-unofficial-2026-10 \
  --driver scripted --script examples/connector/everything-weather-cli.calls.json

Both exit 0. The scenario offers the model three of the server's tools; a reviewed allowlist keeps the rest off its list, including get-env, which returns the server process's environment variables, and gzip-file-as-resource, which fetches any URL it is given. The server has no private oracle, so the outcome check calls get-structured-content itself after the agent stops, and the answer checks look for the temperature and conditions the server returned. With a model profile, let an agent do the same task three times and print pass^k (paid inference). The scenario declares no reset URL, so connector-run warns that the trials share the server's state; this weather tool keeps none:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --config agent.yaml --trials 3

Readiness checks

check also runs readiness suites that read catalogs and documents instead of running an agent. With the connector-lane quickstart's fixture still running, review its catalog and the example manifest against the pinned Muse connector guidelines:

uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \
  --manifest examples/connector/readiness.yaml

The guidelines review lists the server's tools and calls none; --drift-check fetches the live guidelines, and only when given. Poppy discovery only reads. Its probes run only with --manifest and act on the company's own hosts: they start sessions, sign in the declared test account and confirm one declared operation. The gadget suite reads one local file. The contract suite checks a target named in the configuration file (--config). For a kind: mcp target without declared contract_cases, its operation-contracts check is unsupported and the run exits 3.

A clean readiness result is not an admission. Meta reviews connectors by hand, Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms say it is not a developer platform. Readiness checks describes the rules, manifests, --offline behavior and the mcp-contract-2026-07-28 profile.

Quickstart: Coverage Cat sandbox

From a source checkout or unpacked source distribution, complete Setup, then run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted

The example creates a fictional shopper's draft in Coverage Cat's hosted sandbox at https://www.coveragecat.com/api/consumer/mcp. It needs no Coverage Cat account, operator credentials or model API key.

The first umbrella_consumer_prefill call sets sandbox: true, as specified in the consumer MCP manifest. That argument activates the server's sandbox. The scenario's test_only: true flag merely records the test author's intent.

The host fixes the tool arguments and keeps the returned access token private. It permits two tools, umbrella_consumer_prefill and consumer_intake_issues, and checks that:

1. Prefill returns sandbox: true.
2. Follow-up retains sandbox: true.
3. Follow-up returns the original application ID.
4. The server reports application_submitted: false.

Each run creates a new draft and stops before submission. A failed or missing sandbox check stops the workflow. These checks cover draft creation and follow-up, not payment widgets, browser behavior or independent backend checks.

To let a model choose the steps, create agent.yaml with the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \
  --driver agent --config agent.yaml

The agent driver is a scripted action menu: Spark Open picks one ready scenario step per turn, and the host runs it with the scenario's arguments. The model never sees the server's tools, schemas or raw results, only the result fields the scenario exposes, and the evidence is labeled scripted action menu. The host supplies the test inputs and checks the results. This version uses paid model inference through Together, with ZDR enabled and fallback disabled by default.

Verified 15 October 2026: both scripted and Spark Open agent runs passed all four sandbox checks. Both model responses identified Together; both requests used the configured ZDR policy. See the recorded quickstart results.

Each command prints an outcome line, such as Surface run completed (exit 0), then Surface report: .../report.html. Open the report to inspect the workflow and its checks. The same directory contains result.json for the structured result and events.jsonl for the redacted event timeline. Agent runs also store model and tool records under agent/.

Replace the placeholders below with the run directories printed by your commands:

uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID

Share the generated export. Keep the full run folder private. While a run can resume, checkpoint.private.json holds the credentials, tool results and host inputs it needs; a completed run, or one that failed with exit 1 and left no write unconfirmed, keeps only a scrubbed record.

To test your own MCP service, copy the example and replace its endpoint, allowed origins, tool policies, test data and response checks. Surface workflows explains UI steps, the agent driver and resuming runs.

Extend through submission and sandbox checkout

The checkout example adds review, submission, Monoline offer selection, consent, bind follow-up, document upload and simulated payment-method setup. Its 14 steps contain 62 assertions. All seven tools use the consumer umbrella MCP, and every service step must return the same sandbox intake.

Current integration status, 15 October 2026: blocked at submission. Coverage Cat returns quote fields that its MCP output schema excludes. It also returns minimum_auto_limits as a string where the schema requires an object. The host rejects the response. The draft quickstart passes, but the full checkout test has not passed end to end. See the failure report and separate diagnostics.

Run this command to reproduce the failure or retest after the service contract is corrected:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
  --driver scripted \
  --replies examples/coveragecat-sandbox-checkout-replies.json

The reply files supply a fictional shopper's answers: approval to submit, credit consent, the Monoline $2M offer at $389, sufficient auto limits, declined excess UM/UIM, and “No” to both carrier questions. These fixtures do not authorize transactions for a real shopper. A changed quote or missing or negative approval stops the next write. The two generated declarations PDFs are labelled SYNTHETIC TEST ONLY.

Once submission works, the example pauses with exit code 3 at sandbox_document_review_poll. Only at that checkpoint, wait at least the reported 60 seconds and resume the printed run:

sleep 60
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
  --driver scripted --resume runs/surface-YOUR_RUN_ID \
  --replies examples/coveragecat-sandbox-checkout-after-review.json

The host pauses for your reply but does not measure how long you wait. The after-review fixture approves the simulated payment callback.

For an agent-driven run, replace --driver scripted with --driver agent --config agent.yaml on both commands. A resumed run must keep its original scenario, driver and model configuration.

The sandbox supplies mock payment credentials. The example sends the returned seti_sandbox_... with the fixture pm_sandbox_consumer_agent_testing_tool, then requires payment_method_on_file, waiting_on_carrier, and no charge confirmation. Billing checks inspect setup metadata rather than rendered fields. No Stripe form opens, no card details are collected, and no fallback URL is used. This test establishes neither premium collection nor active insurance. Test the embedded payment UI and Stripe test-mode setup separately.

Test your own service

The quickstarts above are the pieces of one workflow. For an MCP server you own:

1. Write a scenario. scaffold URL --output my-connector.yaml reads the live catalog (--auth-env VAR for an API key, or --auth-from FILE with a scenario that holds an OAuth block) and writes a skeleton. Replace every TODO(scaffold): the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your documentation settles, approval rules for the scripted user, host-private values such as the user's email, and checks that read the real outcome, from a private state URL (an oracle) or one of your read tools.
2. Check readiness. check URL --suite muse-guidelines --manifest FILE reviews the catalog and your declarations against the pinned connector guidelines. A clean result is a documented constraint, not an admission.
3. Pin the happy path. Replay a scripted calls file with --driver scripted; it needs no model and belongs in CI. After an agent trial you trust, scaffold --from-run RUN --output FILE turns it into one.
4. Run agent trials. --trials N with a model profile gives pass^k. Run the same scenario under each treatment a Muse-like host may use: --projection muse-cli, --exposure search-only, --client-profile muse-template-unofficial-2026-10 and --output-schema strict. The run lock records each one.
5. Add realism. Lane extensions add competing connectors and connect-on-request, payment with single-use cards, background jobs, untrusted content, channel limits and a simulated user.
6. Compare builds. diff compares two sealed runs and exits 1 on an outcome regression. Share the output of export; the run directory's private/ folder holds host-private values.

For an MCP app with its own UI, start from the Coverage Cat scenario and surface-run. For a website, use browse with a browser policy, or the connector lane's browser extension. For an HTTP API, write a mapping for run --surface api-tools, or test whether an agent can build its own client from your docs with the custom_connector extension. Run a consumer test covers the other lanes.

Model providers

Create a profile with init. This one uses Muse Spark Open (meta/muse-spark-open) through OpenRouter, routed only to Together, with zero data retention (ZDR) required and fallback disabled; set OPENROUTER_API_KEY in your shell before an agent run:

uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open \
  --routing-provider together --config agent.yaml

init also stages, beside the configuration, the API scenarios, the browser scenarios and the mapped-API example. It stages only scenarios the supported CLI can run, and refuses providers that have only a configuration template (openai, anthropic, gemini, azure-openai, vertex-gemini, openai-compatible) unless --development-extensions comes before init. init --provider none --config scripted.yaml writes a configuration for scripted commands that need one. init refuses to overwrite an existing file.

Profiles that init writes are labeled specified/unverified. doctor --model-preflight, run and the agent drivers of connector-run, surface-run and browse print a warning for any profile not labeled supported, and the run lock records the label. Model IDs that contain contributor select a mode in which the provider may train on your prompts; such a profile needs allow_training_on_prompts: true (init --allow-training-on-prompts). See model providers for every adapter, its checks and its failure labels.

OpenRouter model and privacy settings

Consumer Agent Testing Tool requests zero data retention (ZDR) by default, even when routing.zdr is omitted. To change this, edit the selected profile in consumer-agent-testing-tool.yaml or the file passed to --config:

model_profiles:
  openrouter-dev:
    provider: openrouter
    api: chat-completions
    model_id: meta/muse-spark-open
    auth:
      mode: api_key
      env: OPENROUTER_API_KEY
    routing:
      only: [together]
      allow_fallbacks: false
      require_parameters: true
      data_collection: deny
      zdr: true  # Set to false to disable this tool's per-request ZDR requirement.
defaults:
  model_profile: openrouter-dev

Use unquoted YAML booleans: true or false. Consumer Agent Testing Tool applies the setting to every OpenRouter request, including requests that continue after a tool result, and records it in the routing evidence.

Setting zdr: false removes this request-level requirement. It leaves data_collection: deny, the pinned provider and the ban on fallbacks in place. Your OpenRouter account or guardrails may still enforce ZDR. Request logs show the policy sent; they cannot independently verify the provider's retention practices. See OpenRouter's ZDR documentation.

Checked 15 October 2026: OpenRouter listed Together for Spark Open in both its model and ZDR endpoint catalogs. A live tool-call test passed with the routing settings above. Both responses identified Together, and the model used the tool result correctly. See the test record.

To choose another provider, set routing.only or pass --routing-provider to init. Requests never switch providers or models silently.

Runs use Inspect's dedicated openrouter provider, which replays earlier reasoning in structured form rather than as text. Each response must name the pinned provider. Responses do not name endpoint variants, so a pin such as nebius/fp4 is recorded as variant unverified. Streaming is not supported. See model providers.

With zdr: true, only ZDR-listed endpoints are eligible. On 15 October 2026, Muse Spark 1.3 and Muse Wonder were served only by Meta on OpenRouter and were not ZDR-listed, so no ZDR-compliant OpenRouter route existed for them. A request for them is expected to fail, but no live run has recorded the error. When OpenRouter's error cites the data policy, it is labeled no endpoint satisfies the routing/ZDR policy. With routing.only: [together], the init default, it may instead be labeled routing policy: no allowed provider.

Sources: Meta announcement, OpenRouter model, provider endpoints, and ZDR endpoints.

The standalone smoke test and older consumer helper read the same --config and --model-profile, including privacy settings. Profiles that init writes leave reasoning effort unset.

.venv/bin/python scripts/openrouter_smoke.py --config agent.yaml \
  --model-profile openrouter-dev

This command tests the model connection with synthetic tool data. It needs an API key and an available route to the pinned provider. It makes no requests to the service you plan to test.

Meta Model API

init --provider meta writes a profile for Meta's Model API:

uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml

The profile uses the Responses API at https://api.meta.ai/v1 and reads the key from MODEL_API_KEY or META_API_KEY; if both are set, they must agree. Add --api chat-completions for Chat Completions, or --key-env NAME to use another variable, which is then the only one read. The adapter uses the OpenAI SDK from the openrouter extra.

The host refuses requests that Meta rejects, such as a forced tool choice, before sending them (exit 3). Meta adds steering context to every prompt on its servers, so captured requests are incomplete, and every Meta run records this as an observation gap. Streaming is not supported, and the API is pay-as-you-go. See Meta Model API. No live Meta Model API run has been recorded; the adapter is tested against local fixtures.

Run a consumer test

This section covers the embedded MCP apps (surface-run), browser (browse) and HTTP API lanes. To let an agent use your MCP server's own tools, use the connector lane.

A scenario defines the endpoint, allowed origins, tools, private arguments, UI actions, consent requirements and result checks, and optionally an API key for the MCP connection (auth). Use endpoints and synthetic transactions you have permission to test.

consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver scripted
consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver agent --config agent.yaml
consumer-agent-testing-tool surface-run path/to/scenario.yaml --resume runs/existing-run
consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json

The host declares MCP Apps support as a client, finds each tool's UI through its tools/list definition and renders it in a native browser. With --driver agent, the model picks among the scenario's steps (a scripted action menu) and never sees the server's tools, raw results or private inputs. Service checks determine the result; a rendered form or a model's claim of success cannot prove that a transaction completed.

browse drives a restricted browser through accessibility snapshots and element references, with no selectors, scripts or CDP. Its policy file sets the start url, the permitted origins and request rules, snapshot_max_chars (default 20,000), history_snapshots (2) and dialogs (agent, accept or dismiss), as in examples/browser-policy.json. Long snapshots are paged with {"kind": "snapshot", "offset": N}. Refused or failed actions come back to the agent as observations with a status, and dialogs and file choosers are answered with kind: dialog or kind: upload. With --driver agent, the provider preflight runs before the browser opens. The result's observations block counts truncated snapshots and refused actions. When the agent requests a takeover, the run pauses (exit 3). The CLI has no handback, so control does not return to the agent in that run. In native mode, WebSockets opened by dedicated workers and WebRTC TCP relays are not enforced. The policy field webmcp: true, off by default, lists page-registered WebMCP tools in the snapshot and lets the agent call them with kind: webmcp; that lane is labeled "generic WebMCP page tools, not Muse" (basis: assumed), and its evidence notes that Meta documents WebMCP for its AI glasses, a Meta product that is not Muse. See restricted browser behavior.

HTTP API tests use explicit mappings with the scenario runner. Mappings can send declared request headers and put the target credential in a custom API-key header. In API and browser trials, invalid tool proposals are returned to the model as tool errors, and exhausted limits fail the trial; see API scenarios. To test whether an agent can build its own client from an API's documentation instead, use the connector lane's custom_connector extension. Prism, Schemathesis and generated-OpenAPI tools are available only as development extensions. Their five packages with unresolved original license notices are excluded from the supported release, not treated as cleared.

Outcomes and exit codes

surface-run follows the exit codes in spec §18:

Before Surface report: ... it prints one line: Surface run <status> (exit N) at <stage> [<class>]: <reason>. result.json records the same failure_class and failure_stage, and whether the run is resumable. A completed run, or one that failed with exit 1, is final unless the failure left a write unconfirmed: that write may have taken effect, so the run stays resumable to reconcile it. Blocked runs and setup failures can resume.

run and browse use the same four codes, except that exhausting a turn, tool-call or time limit is a failure (exit 1). Only Inspect operator and custom limits, and in run a deadline that passed before an attempt's first generation, block instead (exit 3). run prints each evidence path on stdout. browse prints the final observation and its evidence path when the run completes or pauses for takeover, and nothing on stdout otherwise. For any outcome other than completed, both also print one line on stderr: <label> <status> (<class>) at <check>: <reason>. Evidence: <path>. The label names the step, for example Model preflight, Trial 2/10 of stale_total, Resumed trial of stale_total or Browse (doctor uses Doctor check). A run with a failed or blocked trial, or with --trials above 1, also prints a summary line with the counts. With --trials above 1 it adds the suite pass^k, the mean over scenarios, for example Run completed: 0 failed, 0 blocked and 4 completed of 4 trials; pass^1 1.00, pass^2 1.00. Evidence: .... The suite's metrics also break the trials down by scenario (statistical decisions). run DIR refuses (exit 2), before the preflight, a directory whose scenario files share an id.

connector-run prints the same stderr line, labeled for example Connector trial 1/1 of fixture-booking-happy-path; its codes are listed in the connector-lane quickstart. With --trials N, one code stands for all trials: 2 when any trial's evidence is not shareable, else 1 when any trial failed, else 2 when any hit a setup, harness or verification error, else 3 when any was blocked, else 0. The Connector trials: line on stderr gives pass^k and names the trials that are not shareable or ended in an error. The readiness suites' codes are in Readiness checks.

What the MCP Apps host implements

The embedded-app lane is generic MCP Apps host evidence (MCP Apps 2026-01-26), for hosts such as ChatGPT or Claude. It is not Muse evidence: Muse documents no third-party UI. Each result.json says so under mcp_apps_host.

The host implements:

• the client capability (io.modelcontextprotocol/ui with text/html;profile=mcp-app), UI discovery from the tool's tools/list definition, and HTML resources as text or base64 blobs;
• tool-input with the call's arguments, tool-result, and a hostContext (theme, display mode, locale, time zone, platform, container width and maximum height, and tool information);
• a content security policy built from the resource's _meta.ui.csp, or the specification's restrictive default when the resource declares none, and the declared camera, microphone, geolocation and clipboard-write permissions;
• in the native renderer, a sandbox proxy on an origin separate from the host page. With browser_origins, the host page also has Chromium's local-network-access permission, every redirect hop is checked again (GET or HEAD to an allowlisted origin only), and each direct write is recorded as an unconsented host_action before it is sent;
• view tools/call requests for the step's allowlisted tools, and resources/read and resources/list for the step's callback_resources;
• ui/message, delivered once to agent runs in the next observation a model reads, as app_messages; the view is told when the host does not deliver it. ui/update-model-context, ui/open-link, display-mode, log and teardown requests are recorded, and the latest model-context update reaches agent runs as app_context;
• size changes, ui/resource-teardown before the view closes, and tool-cancelled when the view never received the call's result;
• _meta.ui.visibility, for view calls and for tool steps;
• MCP 2026-07-28 servers, when the scenario sets version_negotiation: auto. The default is the 2025 initialize handshake;
• an API key on the MCP connection (auth), as a bearer token, a custom header or a query parameter;
• a legacy window.openai bridge.

It does not implement a model turn for a view's message, resource templates, _meta.ui.domain, display modes other than inline, partial tool input, host-context change notifications, OAuth on the MCP connection, or consent for a view's direct writes in the native renderer. The legacy bridge and the isolated renderer have no sandbox proxy. See MCP Apps host.

Inspect retained evidence

consumer-agent-testing-tool report runs/existing-run
consumer-agent-testing-tool export runs/existing-run
consumer-agent-testing-tool replay runs/existing-run
consumer-agent-testing-tool diff runs/baseline runs/candidate

Replay checks the evidence seals and stops on a mismatch. It never substitutes a live request. To test the service again, start a new run. Reports mark missing measurements as unknown and keep partial totals distinct from complete totals.

Reports open with the verdict, which names the failing step and check, then the checks and host capabilities, and a per-step timeline that shows what the model received in full. Then come a connectors table for runs with several connectors, the surface views (what the service published, what the model was offered, what it received at each step, and what only the developer sees), the metrics, and the viewers and linked evidence: agent and preflight runs, their Inspect logs and the inspect view command, and, locally, a captured Playwright trace. Gaps and limits follow, then the run lock and the full recorded result, each collapsed; see report layout. report on an agent sub-run of a sealed run returns its sealed report unchanged.

Evidence files and exports hide credentials, registered secrets, contact details and other personal fields such as names and addresses. Model views keep the user's own data visible: they hide only credentials and declared secrets, plus browser password fields and, in a resumed browser scenario, text typed in an earlier attempt. The provider boundary refuses any model request that still contains a registered credential. Export also applies the run's own private values, which are never exported, and refuses (exit 2) when a staged file, or a member of an Inspect log or other archive, still holds one. See redaction.

diff compares two sealed runs, such as surface, browser or connector runs. A change to the package source counts as a run-lock change, so compare runs made from the same checkout. It exits 0 when they are comparable with no outcome regression, and 1 when an outcome regressed: a worse status, or a passing check that no longer passes or is missing. It exits 3 when they are not comparable, because of a run-lock change outside --treatment or an unsealed or unverifiable run. Generated IDs and service timestamps are normalized; run locks are compared exactly. --ignore POINTER (repeatable) excludes a JSON pointer under /outcomes, /capabilities or /milestones, and --treatment /server declares an intentional server build change. See replay and diff.

For more detail, see the documentation index, connector lane, lane extensions, readiness checks, consumer tests, browser diagnostics, measurement coverage, model timing, and release extensions, which includes development history.

Use it in CI

Scripted connector runs need no model and no key, and they replay the same calls every time, so they work as a pull-request check. connector-run can start the service itself and write reports a CI system reads:

uv run consumer-agent-testing-tool connector-run my-connector.yaml \
  --driver scripted --script my-connector.calls.json \
  --start "python -m my_service --port {port}" \
  --junit reports/connector.xml --summary-json reports/connector.json

• --start COMMAND runs the service on a free loopback port. The command is split like a shell command but not run in a shell.
  • {port} in the command is that port. The service also gets it as PORT and the scenario as ${PORT}.
  • The tool waits until --ready-url, or the scenario's first connector endpoint, answers HTTP. It waits at most --ready-timeout seconds (default 30).
  • When the run ends, the service's process group gets SIGTERM, then SIGKILL after 5 seconds.
  • Its output goes to runs/service-<id>.log, outside the sealed run. That log isn't redacted, so pass secrets to the service through the environment. The run lock records the command with credentials redacted.
  • A service that exits early or never becomes ready exits 2.
• --junit FILE writes one test suite per trial and one test case per check. Blocked checks are <skipped>. check --suite muse-guidelines accepts it too.
• --summary-json FILE writes the verdict, exit code, checks, trials, pass^k and run paths. Its format is schemas/summary-v1.schema.json. --summary-json - prints only that JSON on stdout and moves the other lines to stderr.
• Both reports are also written when the command stops with an error. See CI outputs and services.

The exit codes in Outcomes and exit codes decide the step. A 3 (blocked or unsupported) is not a pass. examples/ci/connector-lane.yml is a GitHub Actions template with two jobs. On pull requests, it runs the scripted calls and the readiness review. Nightly or on demand, it runs agent trials under each Muse-like treatment, and only when a model key secret is set. .github/workflows/ci.yml runs this repository's tests and shipped examples on macOS and Linux. Coding agents should start with AGENTS.md. All matrix jobs feed the required CI check; Linux failures are not allowed to pass silently. Workflow validation, tracked-file hygiene, secret scanning and schema freshness are checked too.

Running the tests

Plain pytest runs the supported collection:

uv run pytest -q

tests/conftest.py gives each session a temporary directory, home and cache under .cache/pytest/, removes them after a passing session, and removes the surface and browser run directories its tests create under runs/. The full collection takes about twenty minutes and makes no network request beyond loopback and no model request. pytest --development-extensions also collects the experimental tests, which may need other runtimes, tools and external evidence. Async cleanup and unhandled thread exceptions fail tests instead of remaining warnings. See CONTRIBUTING.md for the review workflow and the launch checklist for the private-to-public transition.

Release audit

The supported release is local-consumer-v1: the connector lane and its extensions, the readiness checks, consumer MCP apps, the restricted browser, mapped HTTP APIs, transactional consent, continuation, evidence and replay, measurements, the distribution, and the OpenRouter and Meta Model API adapters. Real-Muse calibration, cloud provisioning, local Glimmer, browser OS isolation and live Bedrock qualification are outside it; see release scope for the full boundary.

Release evidence is generated per release and is not committed. docs/release-gates.json is the committed policy: the gates, the scope and the exclusions. The release workflow runs on a v* tag. It records every gate on macOS, requires the audit to pass, and attaches three things to the GitHub release:

• the wheel and the source archive, with signed build provenance;
• release-evidence-<commit>.tar.gz, which holds release/release-gates.json (the recorded ledger) and release/evidence/.

Use consumer-agent-testing-tool release-audit to check release readiness. It reads release/release-gates.json when evidence has been recorded or extracted there. Without it, it reads the committed policy and reports every gate unverified. Exit code 0 means release-ready for local-consumer-v1; exit code 3 means at least one required gate needs attention. The evidence binds a digest of the source, so a source change or an expired gate (seven days after recording) cannot inherit an earlier passing result. To audit a published release, check out its tag, extract its evidence archive at the repository root and run the audit before the gates expire.

Day to day, you don't need any of this. Run the tests for what you change, and let CI run the full collection on each pull request.

A separate live Spark Open test passed through Together with ZDR enabled. It verified native tool calls, continuation after a tool result, and the requested routing policy. It did not test streaming, independently verify retention practices, or establish compatibility with every consumer workflow. The Coverage Cat checkout failure described above remains an open integration issue.

The audit requires five gates:

To record evidence locally instead, for example before tagging, run the same steps as the workflow, in this order. verify_distribution.py regenerates the dependency notices, which are part of the source fingerprint, so the distribution comes before the regression. The regression takes about twenty minutes. Everything is written under release/, which git ignores:

.venv/bin/python scripts/verify_distribution.py
.venv/bin/python scripts/record_distribution.py
.venv/bin/python scripts/run_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_release_tests.py release/evidence/regression-run.xml
consumer-agent-testing-tool release-audit

run_regression.py refuses an existing report path, so remove release/ or pick a new name to record again. It records workspace-relative paths and writes the home directory as ~ in the log.

Distribution checks use caches in the workspace and the installed browser. They need no Docker VM, cloud worker or model credentials. If a dependency cache is missing, populate it separately with scripts/verify_distribution.py --prepare-cache; that setup step uses the network. The installation evidence covers macOS only.

The audit verifies source fingerprints, artifact hashes, required checks and expiry dates. It returns 0 when every required gate passes, or 3 otherwise. Excluded capabilities remain labelled as excluded. Qualification inputs explains what each gate needs.

Development extensions

The repository retains worker, isolation, local-model, calibration and generated-OpenAPI experiments, and the CLI/skill lane for a developer's own CLI binary (run --surface cli-skill), which needs Docker and an oracle that returns the reservation fixture's snapshot schema. The normal CLI rejects these paths. To use them, put --development-extensions before the subcommand and install the separate development-extensions dependency group and any required tooling. These experiments are outside the supported release. The supported way to give an agent a CLI over an MCP server is connector-run --projection muse-cli.

The optional Node API tools have known dependency advisories. Do not use them with untrusted inputs while those findings remain open.

The original specification and earlier test records document the project's history; they do not define the scope of this release.

License

MIT; see LICENSE. The license does not cover the Coverage Cat and CATT names or the logo; see TRADEMARKS.md. Third-party components keep their own licenses; the distribution ships their notices under third_party/.