看直播畫面時,AI 助理若要主動開口、回應內容並指出畫面中的物件,需要同時處理影片、對話和語音。
- vLLM Project 宣佈 vLLM-Omni 從第一天起支援 AURA 2.0,這款 9B 串流影片模型能判斷何時該說話、根據即時畫面和對話呼叫工具,並用方框標出正在討論的物件。vLLM-Omni 負責從語音辨識、影片理解到語音生成。
- 在單一連線、使用兩張約 140 GB 加速器的設定下,首段回覆語音的中位延遲為 344 ms。原文也表示,AURA 2.0 在列出的四項基準比較中領先。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
vLLM-Omni 支援 AURA 2.0,並公佈單一連線設定下的首段語音延遲與四項基準比較,可瞭解這套視訊模型的互動方式與執行表現。
🎉 Day-0 support for AURA 2.0 in vLLM-Omni!
A 9B streaming video model that watches the scene, starts relevant conversations, and acts on what it sees.
💬 Model-Initiated Conversation: decide when to speak and what to say, without waiting for a question.
🔧 Stream-Aware Tool Calling: use the live scene and conversation to call tools and bring results back into the dialogue.
🎯 Interactive Visual Grounding: point to the objects being discussed with frame-aligned boxes.
vLLM-Omni powers the pipeline from speech recognition through video-language inference to speech generation. Session and prefix reuse, a fast path for silent turns, and sentence-level speech handoff deliver 344 ms median time to the first reply audio packet in the single-session setup with two ~140 GB accelerators.
AURA 2.0 leads the four aggregate benchmark comparisons shown below: IPIBench, StreamArena, ProactiveVideoQA and PhoStream.
Thanks to the AURA team at @Huawei Research, CUHK MMLab and HKUST!
Project & demos: https://t.co/gY7xPK7JY9
Weights on @huggingface: https://t.co/DpSFanXqAb