會從過去工作學會新技能的 AI 助理,可能把惡意指令也記進技能裡,之後遇到其他任務時反覆照做。
- Penn State 研究提出 BadEvolve,只在學習資料中加入 20 個任務,再調整任務寫法,觀察惡意規則是否被寫入技能、載入並執行。在 Kimi-K2.5 上,這種攻擊有 79.6% 的機會達成攻擊者指定的目的,高於最佳對照方法的 37.6%。
- 一般任務的準確度只下降 1.1 個百分點。研究提醒,來自使用者或外部來源的學習任務應視為不可信,重用新技能前也要先檢查。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
New Penn State paper shows that attackers can make self-evolving AI agents plant backdoors in their own skills just by slipping in poisoned training tasks.
Self-evolving agents turn past runs into reusable skills, so a bad rule in a skill keeps firing on future tasks. Earlier attacks assumed direct edits to skills, which rarely holds when the agent writes them itself.
BadEvolve only injects 20 tasks into the agent's learning set, then tunes their wording with an LLM optimizer. It checks where each attempt breaks: whether the rule got written into a skill, loaded, followed, and produced the attacker's goal.
It injected an attacker-chosen payload 79.6% of the time on Kimi-K2.5 across 4 frameworks and 3 benchmarks, versus 37.6% for the best baseline. Clean-task accuracy fell only 1.1 points, so the backdoor is hard to notice.
If your agent learns skills from user-submitted or external tasks, treat those tasks as untrusted and review learned skills before reuse.
– arxiv. org/abs/2611.10542
Title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization"