Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond to unfolding events. Towards this vision, we focus on the innovative task where, given ego-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning.

This task embodies three key properties:

1. Proactive Coherence
2. Just-in-Time Responsiveness
3. Synchronized Efficiency.

To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric—a novel framework designed for their rigorous assessment.

Secondly, we propose a comprehensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises:

1. a data engine
2. a multi-stage training strategy
3. a proactive dynamic compression technique.

Our proposed model effectively addresses these critical properties while achieving state-of-the-art (SOTA) performance on the standard COIN benchmark.