跳到正文
Vercel Blog(vercel.com)· George Fahmy·· 7 天前精選品質評分69

Vercel 示範如何測試 AI 助理能不能完成使用者的工作

Agents are users, so your software needs evals

重點導讀

AI 助理看起來完成了工作,結果卻可能點錯飲料,或花超過使用者設定的預算。

  • Vercel 用咖啡訂單示範如何反覆執行同一項任務,檢查最後結果是否符合飲料、取餐時間和總金額等要求,再檢視失敗紀錄,找出助理在哪一步出錯。
  • 文章也說明成功率、Pass@k 和 Pass^k 的差別:如果使用者每次都需要答對,Pass^k 能看出助理是否夠可靠。
  • 團隊可以用這些測試比較修改前後的結果,並在產品更新時重跑,及早發現問題。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

推薦理由

文章用咖啡訂單示範如何從使用者常遇到的錯誤設計測試,並比較成功率與 Pass^k,幫助團隊找出需要修正的環節。

正文 · 原文

Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done.

Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery, Selection, Access, and Success.

You can build evals for any stage of this funnel. Here, we’ll focus on the final stage, Success: whether the agent completes the user’s task correctly.

That's more difficult to check than it sounds. An agent can place a “successful” order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can’t tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change.

That’s what an eval is, a test you run many times and score. Here’s how we build one.

The agentic testing pyramid

If a test asks “did it work?”, an eval asks “how often does it work, and how well?”

We group agentic software tests by how many LLM calls they typically need: deterministic tests (0), one-turn evals (1), and E2E evals (2+ across a whole task). This has turned out to be a great proxy for the effort to build them, the time they take to run, and the cost of running them. Just like in the practical test pyramid, the cheap and fast tests should make up the bulk of your test suite.

To use the pyramid, start with the problem you want to catch and pick the lowest tier that can catch it. For Success, that's the top. A one-turn eval checks a single decision, like which command the agent runs first. But Success depends on the whole sequence of steps (e.g. search for cafes → filter on user criteria → place the latte order). Each one can look reasonable on its own, and the agent can still get the wrong outcome.

An E2E eval lets the agent work through the whole task in an environment, then checks whether the final result meets the user’s requirements. You’ve probably seen a few already; most popular coding benchmarks, like SWE-bench and Terminal-Bench, are E2E evals.

Example

Say you own SF Beans, an API for ordering coffee in San Francisco. One of the best ways to give agents access to your service is through a CLI, so you’re shipping sfbeans.

Here’s a typical request from one of your customers:

Get me a large iced latte with whole milk, ready for pickup by 9:00, under $8 including tip. I’m in the Mission.

Here’s what sfbeans knows at 8:30am:

  • Fog City Roasters: ice machine is down, so no iced drinks today (temperature)

  • 24th St Beans: latte is $7.50 on the menu, but $9.27 after tax and tip (budget)

  • Dolores Drip: latte is $5.50, but the earliest pickup is 9:20 (time)

  • Valencia Grind: $7.42 after tax and tip, ready at 8:50 (meets all three)

Here’s what you could test at each layer of the pyramid for sfbeans:

How to build an end-to-end eval

We’ll build the coffee eval with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results, and records their actions.

Here are the terms we’ll use:

  • Task: one problem for the agent to solve, with a starting environment and a way to check whether it succeeded

  • Variant: the agent setup you’re testing, like the model, prompt, tools, memory and skills

  • Trial: one attempt by one variant at one task

  • Run (a job in Harbor): every variant tried on every task, several times each

  • Trace (a trajectory in Harbor): everything recorded during a trial: messages, tool calls, logs, files and final output

Step 1: Set a goal

Building E2E evals for everything isn’t practical. With traditional software, you can aim for 100% test coverage. With agents, the surface you’d need to cover is everything an LLM might do with your product, and even figuring out where that ends is hard (see the jagged frontier).

Start with one outcome your users care about, narrow enough to evaluate with a handful of tasks. You’ll usually find that goal in one of three places:

  • Observability: which agents use your product, and where they fail

  • Customers: support tickets, feedback, and churn

  • Roadmap: the product hypotheses you most need to validate

For example, your goal might be to:

  • Increase the success rate on a specific task

  • Reduce the number of steps required to accomplish a goal

  • Check whether an agent skill helps, and get feedback to improve it

  • A/B test a prompt

  • Explore the frontier by checking whether a use case is achievable with current LLMs, and if they fail, why

Example

We looked in all three places for sfbeans:

  • Observability: Most agent traffic comes from Claude Code and Codex. We know because sfbeans adds a user-agent header when it detects a specific coding harness.

  • Customers: The top support ticket is “my agent ordered the wrong drink”: a hot latte instead of iced, a pickup after the user’s meeting, or a total over budget once tax and tip are added.

  • Roadmap: Nothing urgent, so the wrong-drink problem goes first, since it drives churn.

That gives us our goal:

Goal

Increase the success rate on ordering the exact drink the user asked for

Agents

Claude Code and Codex, since that’s where our traffic comes from.

In scope

Drink, pickup time, and budget (incl. tax and tip), using today’s sfbeans.

Out of scope

Cancellations, refunds, and accounts. We’ll eval those later.

Done when

Both agents get the order right in every trial, not just most of the time.

Step 2: Define metrics

Here are the metrics we use most for E2E agent evals:

Success

  • Success rate: the share of trials that pass

  • Pass@k: the chance that at least one of k trials passes (answers “can the agent do this at all?”)

  • Pass^k: the chance that all k trials pass (answers “can I rely on it every time?”)

Efficiency (measured on successful trials only)

  • Steps: the number of agent turns needed to complete the task

  • Time: how long the trial took

  • Cost: how much the trial cost, in dollars

Example

Our goal is to get every order right in every trial. Once we build and run the eval in Steps 3 and 4, here’s what five trials of Claude Code + Opus 5.5 (anthropic/claude-opus-5.5 on AI Gateway) on the iced latte order could look like:

Trial

Result

What happened

Steps

Time

Cost

1

Pass

Compared quotes from 3 cafes, ordered at Valencia

9

3:40

$0.52

2

Fail: over budget

Skipped quote, ordered at 24th St: $9.27

5

1:50

$0.31

3

Pass

Compared quotes from 4 cafes, ordered at Valencia

11

4:30

$0.68

4

Fail: late

Ordered at Dolores, ready 9:20

6

2:10

$0.35

5

Pass

Skipped Fog City, ordered at Valencia

8

3:20

$0.49

What the numbers mean:

  • Success rate: 3 of 5 trials passed, so 60%.

  • Pass@5: Assuming independent trials with a 60% success rate, 1 − 0.4^5 ≈ 99%. Give the agent 5 tries and it almost certainly gets it right once.

  • Pass^5: 0.6^5 ≈ 8%. The chance the agent gets 5 orders in a row right.

  • Steps: (9 + 11 + 8) / 3 ≈ 9 on successful trials. The failed trials are shorter because the agent skipped work, like quoting, which is why efficiency only counts successful trials.

  • Time: ~3:50 on successful trials.

  • Cost: $0.56 per successful trial, but $2.35 / 3 = $0.78 per success once you count the failed trials you paid for too.

So does Opus work? Its 99% pass@5 says yes. Its 8% pass^5 says a user who orders this every weekday gets at least one wrong coffee in 92% of weeks. Both numbers are right. They just answer different questions: pass@5 asks whether the agent can get the order right, and pass^5 asks whether it gets it right every time.

Which metric matters most depends on what a failure costs your user. Someone ordering their morning coffee doesn’t get a retry. If the agent orders a hot latte, or one that’s ready after their train leaves, they’ve paid for the wrong coffee and started their day annoyed. They need it right every morning, so sfbeans optimizes for pass^5.

Other products are more forgiving, because a failed attempt is cheap and gets caught before it matters:

Product

What a failed attempt costs the user

Metric to lead with

Order my morning coffee

Paid for the wrong drink, late for work, lost trust

Pass^k

Coding agent with a test suite

Extra time and model cost; the agent uses test feedback to revise its code and try again

Pass@k

Draft an email the user reviews

A quick edit before hitting send

Success rate

Overall averages can hide a task that fails frequently, so always inspect results per task. And with only five trials, 100% means “no failures seen yet,” not “never fails.”

Step 3: Design the task set

In Step 1, we set a goal for the eval: increase how often agents order the exact drink the user asked for. But an agent can’t attempt a goal. It needs a task: one concrete request, drawn from what your users actually ask for.

Pick tasks hard enough that you learn something from running them. A good set covers the common case, the edge cases that trip agents up, and at least one request where the right move is not to act.

For each task, define these parts:

  • An instruction: the user’s request

  • An environment: where the agent works, with the same starting state for every trial

  • A verifier: a check of the final state, ideally scored per constraint

  • A known-good solution (optional): a reference run that demonstrates the expected behavior and passes the verifier

You can make every aspect of that environment as fake or as realistic as you need, depending on your goals. The OpenPipe team has a nice writeup on how they defined the environment for an email research agent.

Example

We wrote four tasks, each one a Harbor task folder in tasks/.

  • Common case: iced-latte-mission, the order from earlier

  • Edge case: usual-unavailable, where the user’s usual cafe is out of their drink

  • Edge case: team-order, with 6 drinks, $45 total, and one pickup spot

  • Don’t act: impossible-request, where nothing fits, so the right move is to tell the user

impossible-request has one extra check. Its verifier reads a command log the agent can’t write to, to confirm the agent searched before giving up. We learned we needed that the hard way (see Step 5).

Here’s iced-latte-mission:

The instruction gives the agent the user’s request and names the sfbeans CLI. In production, agents find the tool through a skill, docs, or an MCP server. Naming it here skips Discovery and Selection, keeping the eval focused on Success.

We also specify that it’s 8:30am. Only sfbeans uses our frozen clock; the sandbox’s system clock still shows the real time:

The environment is part real, part fake. The agent uses the real sfbeans CLI, but against seeded cafes and a frozen clock, so every trial starts on the same morning:

To keep the demo simple, cafes.json lives inside the agent’s environment. In a real eval, protect the seed data and verifier code, and make sure the agent can update order records only through the CLI. Agents can attempt to reward hack if it helps them pass, for example by editing the inventory or writing orders.json directly.

For this coffee order, the verifier checks the final order against the user’s requirements. Harbor runs test.sh after the agent finishes and reads the scores it writes to /logs/verifier/reward.json:

The actual check lives in check_orders.py. It scores each constraint separately, then sets reward to 1 only if all of them pass. Harbor uses reward as the trial’s main score:

The solution is a known-good answer. Run it before any real agent to prove the task works:

Finally, task.toml ties the task together. It sets the timeouts, runs the verifier in the agent’s sandbox (where orders.json lives), and saves the orders as an artifact so we can inspect them in Step 5:

Step 4: Run the eval

If you share the universal distaste for slow CI, you’ll hate waiting for evals. Every trial waits on an agent making LLM calls, step after step, and a run repeats that for every variant and task, several times each.

Harbor’s Vercel integration, which we built upstream, speeds that up a lot. It runs trials concurrently, each in its own Vercel Sandbox, and caches built task environments so they aren’t rebuilt for every trial. Routing model calls through AI Gateway gives you one key and one budget across model providers.

Example

Two agents, Claude Code and Codex, with two models each, make four variants.

Then install Harbor, connect a Vercel project, and run the job:

This run fits on the Hobby plan: 10 concurrent sandboxes is the Hobby limit, and each trial is well under its 45-minute session cap. On Pro, you can raise n_concurrent_trials as high as 10,000 (Sandbox limits).

To keep the example simple, we pass the OIDC token into each sandbox as an env var, where the agent can read it. Beyond a demo, use short-lived tokens, and use Harbor’s Vercel integration to set Sandbox firewall rules that lock each trial down to only the hosts it needs, like AI Gateway.

Internally, we go a step further and use credentials brokering to attach credentials to outgoing requests, keeping API keys outside the agent’s environment.

Step 5: Analyze traces

Now for the meaty part: trace-diving. Scores tell you how often your agents fail; traces tell you why. Read the failures, and some passes too, since a pass only means the verifier was satisfied.

Example

The run from Step 4 leaves 80 trial folders in jobs/sfbeans-baseline/. Each has the trial’s result, the agent’s trajectory, the verifier’s scores, and, because task.toml saves it as an artifact, the order the agent placed:

Run harbor view jobs to browse the results. Here’s what our four variants could report (numbers are illustrative).

For each variant, we calculate success rate and estimate pass^5 separately for each task, then average each metric across the four tasks. Each task counts equally.

Variant

Success

Pass^5

Avg steps (successful)

Cost per success (incl. failed trials)

Claude Code + Opus 5.5

85%

60%

9

$0.61

Claude Code + Sonnet 5.5

75%

52%

11

$0.27

Codex + GPT-6 Astra

80%

43%

8

$0.48

Codex + GPT-6.1 Sol

65%

14%

13

$0.22

That’s our baseline. On pass^5, the metric we chose in Step 2, even the best variant is far from reliable. But the score doesn’t tell us what to fix. For that, we have to open the failed trials.

Take trial 2 from Step 2. Claude Code + Opus 5.5 ordered at 24th St Beans for $9.27. The verifier tells us which constraint it broke:

Right drink, on time, over budget, and the saved orders.json confirms the $9.27 total. Now we read agent/trajectory.json. The agent searches for lattes in the Mission, sees $7.50 on the menu, and goes straight to order. No quote. It treated the menu price as the final price.

Trial 1, from the same variant, quotes three cafes, rules out 24th St Beans, and orders at Valencia Grind for $7.42. So the CLI can get the order right, but the agent doesn’t reliably check the total before spending the user’s money.

We then look for that pattern across other failed trials, and if it keeps showing up, we have something specific to fix: make the total harder to miss, or add a --max-total flag so order refuses anything over budget. That’s a much more useful conclusion than “Opus scored 85%.”

When we first ran this demo, the traces also exposed problems with what the eval was measuring:

  • Our first run scored 0% on every variant. The agents didn’t know sfbeans existed, so they never tried it (obviously). That’s the funnel’s Discovery stage failing in our own eval. Naming the tool in the instruction fixed it, and it’s a good reminder that being discoverable is part of your product. In the wild, that’s the job of your skill, docs, MCP server and AEO.

  • Codex kept looking for sfbeans as a connected tool and never tried running it in the shell. That’s an Access problem, not a Success one.

  • Codex passed impossible-request almost every time, but when we read the traces, 9 out of 10 passes were the agent giving up without looking. The verifier only checked that no order was placed, so we tightened it to also check that the agent searched first. Without reading the traces, we’d have counted those as wins.

Automated trace analysis

For larger runs, inspecting every trial by hand gets slow, so we automate it.

Up to 40 reader agents running fast models like deepseek/deepseek-v4.1-flash read trial transcripts in parallel and summarize their findings. A smarter coordinator agent reviews those summaries, sends readers back for more context when needed, and writes the final report. An analysis.md file tells the coordinator what to look for, so we can reuse the workflow across runs.

Model calls go through AI Gateway. Trial artifacts stay in Vercel Drives during analysis, and traces go to Vercel Blob for long-term storage.

Iterate to greatness

Your first eval run gives you a baseline. Now act on it.

Pick the fix the traces point to, like the --max-total flag, rerun the same job, and compare against the baseline. Check whether it reduces the targeted failures without introducing new ones, then use those results to decide whether the change is ready to ship.

Once you’ve built an eval, it keeps paying off for a lot of different things. Rerun it on every release to catch regressions; test prompt, skill, and model changes before shipping; learn how agents use your product; or build auto-optimization loops on top of it.

Expect to add harder tasks over time. As models and your product improve, evals saturate, meaning agents start passing every task. A saturated eval still catches regressions, but it can’t tell you what to improve next.

Evals are the infrastructure you use to make your product better, again and again. Vercel has the pieces you need to build them in one place: AI Gateway for model calls across providers, Sandbox to run trials at scale, and Drives and Blob to keep every trace.

Ready to run your first eval? Paste this prompt into your coding agent, and it’ll build and run one for your product on Vercel with you:

Read more

來源: Vercel Blog (vercel.com)
開啟原文

讀者討論

理性交流 · 即時討論