Carnegie Mellon 與 JetBrains 的研究追蹤 404 名工程師五週的 coding agent token 使用量與提交紀錄,發現 token 支出增加不等同於產出同比增加。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
研究把 404 名工程師連續五週的 coding agent token 使用量與提交紀錄對照,呈現 token 支出和產出之間的落差。
New Carnegie Mellon and JetBrains paper finds that token spend shows how much engineers use AI agents, not how much they ship.
Companies now track engineers' token spend to set budgets and judge productivity, and some even rank top users. No study had followed both spend and output for the same engineers over time.
They linked 5 weeks of coding-agent token use to the commits of 404 JetBrains engineers. They also analyzed a company survey of 412 employees, where technical staff reported using agents beyond coding in nearly 9 of 10 evaluations.
For the same engineer, doubling spend went with only about 10% more commits. Among weeks with similar commit counts, the 90th-percentile week used 39 to 240 times the tokens of the 10th-percentile week.
Use token spend only for budgeting, and judge agent-assisted work by evidence about tasks and outcomes.