Abstract
Spoken Dialogue Models (SDMs) have advanced rapidly, yet their ability to sustain genuinely interactive multi-turn conversations remains underexplored, as most benchmarks focus on single-turn exchanges. We introduce Multi-Bench, the first benchmark explicitly designed to evaluate SDMs in multi-turn interactive dialogue with an emphasis on emotional intelligence. Multi-Bench employs a hierarchical structure with a basic track for emotion understanding and reasoning and an advanced track for emotion support and application. It comprises five carefully designed tasks and about 4K samples, ranging from emotion recognition to complex reasoning and interactive dialogue, supported by a reproducible evaluation framework. We evaluate six representative SDMs on eight subsets of Multi-Bench. Results show that while current SDMs achieve good performance on basic understanding tasks, they still have room for improvement in advanced multi-turn interactive dialogue and reasoning-related tasks, particularly in emotion awareness and application.
Example Data
Overall, Multi-Bench comprises 3,212 samples covering tasks from basic emotion recognition to complex reasoning and interactive dialogue, drawing on datasets such as MELD, NVSpeech, PsyQA, PsyDTCorpus, and MultiDialog to span diverse topics from everyday conversation to psychological support in both single-turn and multi-turn settings.
Overview of the Multi-Turn Interactive Evaluation Framework
The evaluation process begins with the construction of a user profile, which specifies the scenario, goal, and user character. To build diverse and realistic profiles, we extract user attributes from English and Chinese dialogues using GPT4o and DeepSeek-r1, respectively. We ensure topic diversity by sampling dialogues from daily-life and emotional scenarios, using LLM-based topic annotation and stratified sampling. Each instance includes a user profile and an initial dialogue utterance. The first utterance is then transformed into an emotional audio signal using Step-Audio-TTS, which serves as the initial input to the SDMs and initiates the dialogue loop. During multi-turn interaction, the user’s text responses are generated by a chat LLM and subsequently converted into speech with emotional prompts via the TTS module. The SDM receives these audio inputs and generates both spoken and textual outputs, enabling an end-to-end audio-based conversational exchange that closely mirrors human–machine interaction. The process iterates until a termination condition is reached, such as explicit user termination, sufficient emotional relief, or repeated stagnation. To simulate the user, we adopt DeepSeek-V3.1 as the chat LLM and Step-Audio-TTS as the speech synthesizer. An additional LLM is employed to decide when the conversation should terminate, providing a dynamic and flexible evaluation loop. To improve contextual appropriateness and emotional expressiveness, we design an emotion conditioning mechanism. Given a user’s output sentence, another LLM determines the most suitable emotion for the context. According to this decision, we retrieve a matching audio prompt from a curated emotional speech dataset. Specifically, we recorded 38 emotional prompts spanning diverse categories, such as sadness, fear, happiness, relaxation, excitement, humor, hesitation, and empathy. The TTS model then conditions on these prompts to generate human-like emotional speech.
