AI Gains a Better Ear for Real Conversations

The next generation of AI assistants aims to converse like humans—listening and speaking at the same time, handling interruptions, and picking up on subtle cues like backchanneling. But training these full-duplex speech language models has hit a major roadblock: a severe shortage of high-quality conversational data that captures the messy reality of real dialogue. Existing large-scale speech datasets are mostly single-speaker or lack the overlapping speech and rapid turn-taking that define natural interaction. Standard processing tools often fail with these complexities, introducing errors that degrade AI performance. Now, researchers have developed an open-source pipeline called Sommelier to tackle this data scarcity, enabling the creation of training corpora from raw, in-the-wild audio like podcasts and radio shows.

The core breakthrough of Sommelier is its ability to preserve and process the chaotic dynamics of human conversation rather than stripping them away. The pipeline transforms raw audio into clean, structured data suitable for training full-duplex models by handling overlapping speech, identifying speakers accurately, and generating reliable transcripts. In validation experiments, fine-tuning the full-duplex model Moshi on just 83 hours of Sommelier-processed data led to measurable improvements. On the Full-Duplex-Bench 1.0 benchmark, the fine-tuned model showed better performance in backchanneling, smooth turn-taking, and user interruption handling compared to the base model. For example, in backchanneling tasks, the model reduced the turn-over rate where it incorrectly seized the floor, and in interruption handling, it achieved higher relevance scores from GPT-4o, indicating more contextually appropriate responses.

Ology behind Sommelier is a modular, scalable pipeline designed to process web-scale audio efficiently. It begins by standardizing audio formats and splitting long recordings into manageable chunks using voice activity detection. A key component is the speaker diarization module, which uses the Sortformer model instead of the commonly used Pyannote 3.1. Sortformer demonstrated superior performance, reducing diarization error rates, particularly for short utterances and turn-taking regions, as shown in Table 2 where it achieved a 7.16% DER compared to Pyannote's 8.40% on the VoxConverse dataset. For overlapping speech, the pipeline employs a separation module that disentangles simultaneous speakers using cosine similarity matching with speaker embeddings, preserving full utterance information rather than discarding overlaps. This approach maintained high audio quality, with UTMOS scores close to the oracle upper bound in tests, as detailed in Table 3.

To ensure transcript accuracy, Sommelier uses an ensemble of three automatic speech recognition models—Whisper, Canary, and Parakeet—with a voting mechanism to reduce hallucinations. This ensemble cut word error rates significantly, from 6.26% to 3.92% on noisy LibriSpeech data, as reported in Table 4. The pipeline also includes optional background music removal and denoising steps, though these can be toggled based on use cases. In terms of scalability, the system achieves a real-time factor of 0.1746 on an A100 GPU, meaning processing 10,000 hours of audio with eight GPUs would take about 55 hours, making it practical for industrial applications. The researchers validated the pipeline's utility by fine-tuning Moshi, observing that data selection criteria, such as limiting turns to 10 seconds, were crucial for stable training.

Of this work extend beyond academic research, offering a practical tool for developing more natural AI assistants. By providing an open-source pipeline, Sommelier addresses the community-wide data scarcity that has bottlenecked progress in full-duplex systems. This could accelerate the deployment of AI that engages in fluid, real-time dialogue, enhancing applications in customer service, education, and entertainment. The pipeline's focus on preserving conversational authenticity—like overlaps and backchanneling—means models trained on its output may better mimic human interaction patterns, reducing the robotic feel of current assistants. Moreover, the release of the pipeline under permissive licenses encourages widespread adoption and innovation, though the researchers emphasize ethical considerations, such as preventing misuse for voice cloning.

Despite its advancements, Sommelier has limitations. The pipeline is optimized for speech data and does not account for non-speech acoustic events or general sound scenes, limiting its scope compared to omni-modal audio approaches. Additionally, while the overlap separation module effectively disentangles speakers, the resulting audio fidelity is slightly inferior to datasets recorded with isolated channels, as artificial separation can introduce minor artifacts. The researchers also note that the pipeline's performance depends on the quality of input audio; highly noisy or low-quality recordings may still pose s. Future work could integrate broader audio understanding capabilities and further optimize latency for even larger-scale processing.