跳到正文
𝕏 vLLM(x.com)·· 5 天前精選品質評分74
重點導讀

看直播畫面時,AI 助理若要主動開口、回應內容並指出畫面中的物件,需要同時處理影片、對話和語音。

  • vLLM Project 宣佈 vLLM-Omni 從第一天起支援 AURA 2.0,這款 9B 串流影片模型能判斷何時該說話、根據即時畫面和對話呼叫工具,並用方框標出正在討論的物件。vLLM-Omni 負責從語音辨識、影片理解到語音生成。
  • 在單一連線、使用兩張約 140 GB 加速器的設定下,首段回覆語音的中位延遲為 344 ms。原文也表示,AURA 2.0 在列出的四項基準比較中領先。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

推薦理由

vLLM-Omni 支援 AURA 2.0,並公佈單一連線設定下的首段語音延遲與四項基準比較,可瞭解這套視訊模型的互動方式與執行表現。

vLLM
vLLM@vllm_project
·5 天前
𝕏

🎉 Day-0 support for AURA 2.0 in vLLM-Omni!

A 9B streaming video model that watches the scene, starts relevant conversations, and acts on what it sees.

💬 Model-Initiated Conversation: decide when to speak and what to say, without waiting for a question.
🔧 Stream-Aware Tool Calling: use the live scene and conversation to call tools and bring results back into the dialogue.
🎯 Interactive Visual Grounding: point to the objects being discussed with frame-aligned boxes.

vLLM-Omni powers the pipeline from speech recognition through video-language inference to speech generation. Session and prefix reuse, a fast path for silent turns, and sentence-level speech handoff deliver 344 ms median time to the first reply audio packet in the single-session setup with two ~140 GB accelerators.

AURA 2.0 leads the four aggregate benchmark comparisons shown below: IPIBench, StreamArena, ProactiveVideoQA and PhoStream.

Thanks to the AURA team at @Huawei Research, CUHK MMLab and HKUST!

Project & demos: https://t.co/gY7xPK7JY9
Weights on @huggingface: https://t.co/DpSFanXqAb

推文來源 · X (Twitter)在 X 參與討論
來源: vLLM (x.com)
在 X 查看推文

讀者討論

理性交流 · 即時討論