AI Learns to Search Videos Like Humans

Video search is essential for everything from online platforms to scientific research, but current AI systems struggle with complex, varied queries. A new approach co-designs evaluation, data, and training to build a general-purpose video search model that excels across different formats and domains without needing specialized tuning.

Researchers developed a universal video retriever that handles text, images, and combined queries to find relevant videos accurately. This model, called the General Video Embedder (GVE), uses a pyramid curriculum to learn from diverse data, improving its ability to match queries to videos in scenarios like fine-grained detail retrieval and long-context understanding.

The team created the Universal Video Retrieval Benchmark (UVRB) to measure performance across nine datasets covering coarse-grained, fine-grained, long-context, and composed queries. Using this benchmark, they diagnosed gaps in existing models and generated a synthetic dataset of 1.55 million high-quality examples via the V-SynFlow workflow. This data, combined with a training curriculum that prioritizes foundational skills before complex tasks, enabled the model to learn interconnected representations effectively.

Extensive testing showed GVE outperforms 14 state-of-the-art models on UVRB, achieving a mean score of 0.573 across categories. For instance, it improved compositional query performance by 27% over baselines and excelled in fine-grained spatial and temporal tasks, with scores up to 0.779. The model's design avoids overfitting to specific datasets, making it robust for real-world applications like content recommendation and security analysis where queries vary widely.

This advancement matters because it allows for more accurate and private video searches in fields like education and entertainment, where users need to find clips based on detailed descriptions without sharing sensitive information. It also reduces the need for multiple specialized models, saving computational resources and improving efficiency.

Limitations include the model's focus on visual data only, excluding audio or metadata, and its fixed inference setup that may not adapt well to extremely long videos. Future work could address these to enhance applicability in domains like healthcare or surveillance.