DuraS2ST

Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation

Official repository of the EMNLP 2026 Main Conference paper.

DuraS2ST explicitly plans target wording and phonetic length before generating translated speech, balancing translation fidelity with duration consistency.

News

[2026-08-21] 🎉 DuraS2ST is accepted to the EMNLP 2026 Main Conference (15.4% acceptance rate for Main Conference papers).

Overview of the DuraS2ST framework

Video Demos

Compare the original clips with speech translated by DuraS2ST. The video and its timeline are preserved.

Translated audio is AI-generated by DuraS2ST. No endorsement by the speakers or source organizations is implied.
Chinese → English · Wang Gang introduces Jerry

Original video

Original Chinese speech · 18.5 seconds.

DuraS2ST · English translation

AI-generated English speech with the original video.

English → Chinese · Andrew Morgan: daily life & family

Original video

Original English speech · 16.7 seconds.

DuraS2ST · Chinese translation

AI-generated Chinese speech with the original video.

Chinese → English · Wang Gang: preparing mapo tofu

Original video

Original Chinese speech · 14.6 seconds.

DuraS2ST · English translation

AI-generated English speech with the original video.

English → Chinese · Christina Koch: lessons from space

Original video

Original English speech · 29.4 seconds.

DuraS2ST · Chinese translation

AI-generated Chinese speech with the original video.

Henan Chinese → English · Wikitongues

Original video

Chinese dialect source clip from the Wikitongues archive.

DuraS2ST · English translation

English speech mixed with a lowered background track.

Guiyangese → English · Wikitongues

Original video

Chinese dialect source clip from the Wikitongues archive.

DuraS2ST · English translation

English speech mixed with a lowered background track.

Media credits and licenses

English → Chinese

Each example shows the source, the model's duration-planning chain of thought, and the corresponding translated speech.

Example 1

Source

both works mentioned above are full of romanticism

Duration-planning CoT

Show full reasoning
As I begin to digest the sentence "Both works mentioned above are full of romanticism," I am struck by its evocative nature, suggesting a rich tapestry of emotion and artistry. The phoneme sequence length of 38 indicates a moderately paced sentence, neither too brief nor overly extended, allowing for a nuanced exploration of its themes. This duration suggests a need for careful balance in translation to maintain the rhythm and emotional weight. The challenge lies in translating key concepts like "romanticism" into Chinese while respecting the phonetic length. If I were to translate "romanticism" directly, the resulting phrase might be too concise, potentially losing the depth and nuance of the original. The word "mentioned" also requires careful consideration; a literal translation might not capture the implied connection between the works, risking a disjointed audio synthesis. To avoid compressing the audio, I must consider expanding certain clauses or choosing synonyms that naturally elongate the sentence without distorting its meaning. Looking at the phrase "both works," I realize that emphasizing the duality and shared essence of the works is crucial. By incorporating terms that highlight this connection, I can ensure the translation resonates with the same emotional impact. However, I must consider the pacing; a direct translation might rush through the sentiment, so I explore ways to subtly extend the phrasing, perhaps by adding context or descriptive elements. Ultimately, weighing these phonetic and semantic factors, the most natural fit takes shape as "上述两部作品都充满浪漫主义色彩。" This translation captures the essence of the original while aligning with the phonetic length. The target phoneme sequence "[sh, aŋ_4, sh, uu_4, l, i, aŋ_3, b, uu_4, z, uɔ_4, p, in_3, d, əu_1, ch, uŋ_1, m, an_3, l, aŋ_4, m, an_4, zh, uu_3, y, ii_4, s, ə_4, ts, ai_3]" and its length of 31 successfully achieve isochrony and semantic harmony, ensuring the translation flows naturally and retains the original's emotive power. The source phoneme sequence is: [b, əu, θ, w, ə, r, k, s, m, e, n, sh, ə, n, d, ə, b, ʌ, v, a, r, f, u, l, ʌ, v, r, əu, m, æ, n, t, ə, s, i, z, ə, m], with a length of 38. The target phoneme sequence is: [sh, aŋ_4, sh, uu_4, l, i, aŋ_3, b, uu_4, z, uɔ_4, p, in_3, d, əu_1, ch, uŋ_1, m, an_3, l, aŋ_4, m, an_4, zh, uu_3, y, ii_4, s, ə_4, ts, ai_3], with a length of 31.

DuraS2ST output

上述两部作品都充满浪漫主义色彩。

Example 2

Source

he is also a professional butcher

Duration-planning CoT

Show full reasoning
As I begin to process the sentence "He is also a professional butcher," I immediately sense its straightforward yet informative nature. The phoneme sequence length of 24 suggests a moderate pace, neither too brisk nor overly drawn out, which is crucial for maintaining the natural rhythm in translation. The phrase carries a certain weight, emphasizing not just the profession but also the expertise implied by "professional." This nuance must be preserved in the Chinese translation. The challenge lies in balancing the vocabulary with the phonetic duration. The word "also" indicates an additional role or skill, which in Chinese could be translated as "也" (yě). However, if I translate "professional" directly as "专业" (zhuānyè), I must ensure that the resulting phrase doesn't feel rushed or overly compact compared to the original's 24 phonemes. The word "butcher" translates to "屠夫" (túfū), which fits well semantically but needs careful integration to maintain the sentence's flow. To avoid compressing the audio and losing the sentence's natural cadence, I consider expanding the structure slightly. By introducing "是一名" (shì yī míng) before "专业屠夫," I can elongate the phrase without altering its meaning, ensuring it aligns more closely with the source's phonetic length. This addition not only matches the pacing but also enhances the clarity of the profession being described. Weighing these phonetic and semantic factors, the most natural fit takes shape as "他也是一名专业屠夫。" This translation respects the original's intent and rhythm, with a target phoneme sequence of "[t, a_1, y, ie_3, sh, iii_4, y, ii_4, m, iŋ_2, zh, u, an_1, y, ie_4, t, uu_2, f, uu_1]" and a length of 19. This choice successfully achieves isochrony and semantic harmony, ensuring the translation feels as natural and informative as the English source. The source phoneme sequence is: [h, ii, i, z, ɔ, l, s, əu, ə, p, r, ə, f, e, sh, ə, n, ə, l, b, u, ch, ə, r], with a length of 24. The target phoneme sequence is: [t, a_1, y, ie_3, sh, iii_4, y, ii_4, m, iŋ_2, zh, u, an_1, y, ie_4, t, uu_2, f, uu_1], with a length of 19.

DuraS2ST output

他也是一名专业屠夫。

Example 3

Source

bill wyatt australian basketball player

Duration-planning CoT

Show full reasoning
As I begin to process the phrase "Bill Wyatt, Australian basketball player," I immediately sense its concise yet informative nature. The phoneme sequence length of 30 suggests a brief but complete introduction, not overly drawn out, yet packed with essential details. This brevity implies a need for precision in translation, ensuring that each element is conveyed without unnecessary embellishment. The challenge lies in maintaining this balance while translating into Chinese, where the phonetic structure and rhythm can differ significantly. The name "Bill Wyatt" is straightforward, but translating names often requires careful consideration to preserve their identity across languages. In Chinese, names are typically transliterated, which can affect the phonetic length. The term "Australian" is another focal point; it carries cultural significance and must be translated accurately to convey the same geographical context. However, the direct translation might be shorter, potentially disrupting the pacing. To address this, I consider expanding the phrase slightly or choosing synonyms that maintain the original's rhythm. "Basketball player" is a key concept that must be translated with clarity. The Chinese equivalent should not only match the semantic meaning but also fit within the phonetic constraints. If translated too literally, it might result in a rushed delivery. By slightly adjusting the syntax or adding a descriptive element, I can ensure the translation flows naturally and aligns with the source's duration. Weighing these phonetic and semantic factors, the most natural fit takes shape as "比尔·怀亚特,澳大利亚篮球运动员。" This translation respects the original's intent and maintains a balanced phonetic length. The target phoneme sequence, "[b, ii_2, ər_3, h, u, ai_2, y, a_4, t, ə_4, au_4, d, a_4, l, ii_4, y, a_4, l, an_2, q, i, əu_2, yu, n_4, d, uŋ_4, yu, an_2]," with a length of 28, achieves a harmonious isochrony with the source, ensuring that the translation is both semantically accurate and phonetically aligned. The source phoneme sequence is: [b, i, l, w, ai, ə, t, ɔ, s, t, r, ei, l, y, ə, n, b, æ, s, k, ə, t, b, ɔ, l, p, l, ei, ə, r], with a length of 30. The target phoneme sequence is: [b, ii_2, ər_3, h, u, ai_2, y, a_4, t, ə_4, au_4, d, a_4, l, ii_4, y, a_4, l, an_2, q, i, əu_2, yu, n_4, d, uŋ_4, yu, an_2], with a length of 28.

DuraS2ST output

比尔·怀亚特,澳大利亚篮球运动员。

Example 4

Source

he is now head coach of rostov

Duration-planning CoT

Show full reasoning
As I begin to process the sentence "He is now head coach of Rostov," I immediately sense its straightforwardness and authority. The phrase is concise, with a phoneme length of 20, suggesting a brisk, decisive delivery. This brevity implies a need for precision in translation, ensuring that the target language captures both the immediacy and the significance of the role being described. The challenge lies in maintaining this succinctness while translating into Chinese, a language that often requires more syllables to convey the same meaning. The word "head coach" stands out as a pivotal concept. Translating it directly might result in a shorter Chinese phrase, potentially disrupting the balance with the original's phonetic length. To address this, I consider expanding the translation slightly to encapsulate the full weight of the position, perhaps by emphasizing the leadership aspect inherent in "head coach." Similarly, "Rostov" must be handled with care, as proper nouns can vary significantly in phonetic length across languages. The name "Rostov" in Chinese, "罗斯托夫," naturally aligns well with the source, but I must ensure it doesn't elongate the sentence unnecessarily. Weighing these phonetic and semantic factors, the most natural fit takes shape as "他现在是罗斯托夫的主教练。" This translation respects the original's intent and maintains a balanced phonetic footprint. The target phoneme sequence, "[t, a_1, x, i, an_4, z, ai_4, sh, iii_4, l, uɔ_2, s, iii_1, t, uɔ_1, f, uu_1, d, ə_0, zh, uu_3, j, i, au_4, l, i, an_4]," with a length of 27, achieves a harmonious isochrony with the source. This slight increase in phoneme count allows for a natural flow in Chinese, ensuring that the translation is neither rushed nor overly extended, preserving the original's clarity and impact. The source phoneme sequence is: [h, ii, i, z, n, au, h, e, d, k, əu, ch, ʌ, v, r, a, s, t, a, v], with a length of 20. The target phoneme sequence is: [t, a_1, x, i, an_4, z, ai_4, sh, iii_4, l, uɔ_2, s, iii_1, t, uɔ_1, f, uu_1, d, ə_0, zh, uu_3, j, i, au_4, l, i, an_4], with a length of 27.

DuraS2ST output

他现在是罗斯托夫的主教练。

Example 5

Source

computing in parallel

Duration-planning CoT

Show full reasoning
As I begin to process the phrase "Computing in parallel," I immediately sense its technical and precise nature. The phrase is concise, with a phoneme length of 18, suggesting a quick, efficient delivery, much like the concept it describes. This brevity implies a need for precision in translation, ensuring that the target language captures both the technicality and the efficiency of the original. The challenge lies in maintaining this balance while translating into Chinese, a language that often requires more syllables to convey the same meaning. The word "Computing" is central here, carrying the weight of technology and calculation. Translating it directly to "计算" in Chinese seems appropriate, as it conveys the same sense of mathematical processing. However, the phrase "in parallel" presents a more complex challenge. A literal translation might be too brief, potentially leading to a rushed delivery that doesn't match the original's pacing. To address this, I consider expanding the phrase to "并行计算," which not only captures the idea of parallelism but also maintains the technical nuance by explicitly mentioning "计算." Weighing these phonetic and semantic factors, the most natural fit takes shape as "并行计算。" This choice ensures that the translation is neither too short nor too long, aligning well with the original's phonetic footprint. The target phoneme sequence, "[b, iŋ_4, x, iŋ_2, j, ii_4, s, u, an_4]," with a length of 9, achieves a harmonious balance. It mirrors the original's rhythm and maintains the technical integrity, ensuring that the translation is both accurate and naturally paced. The source phoneme sequence is: [k, ə, m, p, y, uu, t, i, ŋ, i, n, p, e, r, ə, l, e, l], with a length of 18. The target phoneme sequence is: [b, iŋ_4, x, iŋ_2, j, ii_4, s, u, an_4], with a length of 9.

DuraS2ST output

并行计算。

Chinese → English

The reasoning, translated text, and generated waveform in each example come from the same DuraS2ST prediction.

Example 1

Source

样条理论已成为函数逼近的有力工具。

Duration-planning CoT

Show full reasoning
As I begin to process the Chinese sentence "样条理论已成为函数逼近的有力工具。", I immediately sense its technical and precise nature. The phoneme sequence length of 34 suggests a moderately paced sentence, not too brief but not overly extended either. This gives me a clue about the rhythm and flow I need to maintain in the English translation. The phrase carries a weighty academic tone, hinting at a significant contribution to mathematical theory, which I must preserve in English. The challenge lies in balancing the vocabulary with the duration. The term "样条理论" translates to "spline theory," a direct and concise equivalent in English. However, the phrase "已成为" implies a transformation or evolution, which could be rendered as "has become." This choice maintains the semantic integrity while aligning with the source's pacing. The phrase "函数逼近" translates to "function approximation," a term that is both accurate and phonetically substantial, helping to fill the phonetic space without rushing the delivery. The final part, "的有力工具," meaning "a powerful tool," needs careful handling. A literal translation might be too brief, so opting for "a powerful tool" ensures the English version doesn't feel truncated. Weighing these phonetic and semantic factors, the most natural fit takes shape as "spline theory has become a powerful tool for function approximation." This translation respects the original's academic tone and pacing. The target phoneme sequence, "[s, p, l, ai, n, θ, i, r, ii, h, æ, z, b, i, k, ʌ, m, ə, p, au, ə, r, f, ə, l, t, uu, l, f, ɔ, r, f, ʌ, ŋ, k, sh, ə, n, ə, p, r, a, k, s, ə, m, ei, sh, ə, n]," with a length of 50, successfully achieves isochrony and semantic harmony with the original, ensuring a smooth and natural audio synthesis. The source phoneme sequence is: [y, aŋ_4, t, i, au_2, l, ii_3, l, u, ən_4, y, ii_3, ch, əŋ_2, w, ei_4, h, an_2, sh, uu_4, b, ii_1, j, in_4, d, ə_0, y, əu_3, l, ii_4, g, uŋ_1, j, yu_4], with a length of 34. The target phoneme sequence is: [s, p, l, ai, n, θ, i, r, ii, h, æ, z, b, i, k, ʌ, m, ə, p, au, ə, r, f, ə, l, t, uu, l, f, ɔ, r, f, ʌ, ŋ, k, sh, ə, n, ə, p, r, a, k, s, ə, m, ei, sh, ə, n], with a length of 50.

DuraS2ST output

spline theory has become a powerful tool for function approximation

Example 2

Source

这个拱门作为白金汉宫的北门。

Duration-planning CoT

Show full reasoning
As I begin to digest the Chinese sentence "这个拱门作为白金汉宫的北门。", I am struck by its straightforward yet regal tone, evoking images of grandeur and history. The phoneme sequence length of 26 suggests a moderate pace, neither too brisk nor overly drawn out, which implies a need for careful balance in the translation process. The phrase carries a sense of importance, as it describes a specific architectural feature, the arch, and its role as an entrance to a prestigious location, Buckingham Palace. In translating this into English, I must consider how each word contributes to the overall meaning and rhythm. The word "拱门" (arch) is central, and translating it directly as "arch" seems appropriate, but I must ensure the rest of the sentence maintains the same weight and duration. The phrase "作为白金汉宫的北门" translates to "as the north gate of Buckingham Palace," but simply using "north gate" might compress the audio too much, losing the original's pacing. To avoid this, I consider expanding the phrase slightly, perhaps by adding "is" to create "is the north gate of," which helps match the phonetic footprint while preserving semantic fidelity. Weighing these phonetic and semantic factors, the most natural fit takes shape as "this arch is the north gate of buckingham palace." This translation maintains the original's regal tone and pacing, ensuring the listener receives the full impact of the sentence. The target phoneme sequence "[ð, i, s, a, r, ch, i, z, ð, ə, n, ɔ, r, θ, g, ei, t, ʌ, v, b, ʌ, k, i, ŋ, h, æ, m, p, æ, l, ə, s]" with a length of 32 successfully achieves isochrony and semantic harmony with the original, allowing the translation to flow naturally and resonate with the intended grandeur. The source phoneme sequence is: [zh, ei_4, g, ə_4, g, uŋ_3, m, ən_2, z, uɔ_4, w, ei_2, b, ai_2, j, in_1, h, an_4, g, uŋ_1, d, ə_0, b, ei_3, m, ən_2], with a length of 26. The target phoneme sequence is: [ð, i, s, a, r, ch, i, z, ð, ə, n, ɔ, r, θ, g, ei, t, ʌ, v, b, ʌ, k, i, ŋ, h, æ, m, p, æ, l, ə, s], with a length of 32.

DuraS2ST output

this arch is the north gate of buckingham palace

Example 3

Source

斯里兰卡的总统是国家元首。

Duration-planning CoT

Show full reasoning
As I begin to process the Chinese sentence "斯里兰卡的总统是国家元首。", I immediately sense its formal tone and the weight of its content. The phoneme sequence length of 25 suggests a moderately paced sentence, neither too brief nor overly extended. This gives me a clue about the rhythm and flow I need to maintain in the English translation. The phrase speaks of leadership and authority, which are crucial elements to convey accurately. The first challenge is translating "斯里兰卡" to "Sri Lanka." This part is straightforward, but I must ensure it doesn't disrupt the pacing. The word "总统" translates to "president," a key term that carries significant weight. However, if I translate it too literally without considering the phonetic length, the English version might feel rushed. The phrase "是国家元首" means "is the head of state," which is essential to maintain the sentence's authoritative tone. Here, I need to be careful; translating it directly could result in a shorter English phrase, potentially leading to a mismatch in audio duration. To avoid compressing the audio and to ensure the translation feels natural, I consider expanding the English sentence slightly. By introducing "the president of Sri Lanka is the head of state," I can maintain the original's semantic depth while aligning with the phonetic length. Weighing these phonetic and semantic factors, the most natural fit takes shape as "the president of Sri Lanka is the head of state." Now, validating this choice, I introduce the target phoneme sequence "[ð, ə, p, r, e, z, i, d, ə, n, t, ʌ, v, sh, r, ii, l, æ, ŋ, k, ə, i, z, ð, ə, h, e, d, ʌ, v, s, t, ei, t]" with a length of 34. This length successfully achieves isochrony and semantic harmony with the original, ensuring the translation is both accurate and naturally paced. The source phoneme sequence is: [s, iii_1, l, ii_3, l, an_2, k, a_3, d, ə_0, z, uŋ_2, t, uŋ_3, sh, iii_4, g, uɔ_2, j, i, a_1, yu, an_2, sh, əu_3], with a length of 25. The target phoneme sequence is: [ð, ə, p, r, e, z, i, d, ə, n, t, ʌ, v, sh, r, ii, l, æ, ŋ, k, ə, i, z, ð, ə, h, e, d, ʌ, v, s, t, ei, t], with a length of 34.

DuraS2ST output

the president of sri lanka is the head of state

Example 4

Source

没看过这么弱的天才

Duration-planning CoT

Show full reasoning
As I begin to digest the Chinese phrase "没看过这么弱的天才," I immediately sense its sharp, almost dismissive tone. The phoneme sequence length of 19 suggests a concise, impactful statement rather than a meandering thought. This brevity implies a need for precision in translation, ensuring the emotional weight is preserved without rushing through the delivery. The phrase seems to convey a sense of disbelief or disappointment, perhaps at the perceived inadequacy of a genius. Looking at the word "弱," which means "weak," I consider how to convey this weakness in English without losing the original's intensity. If I translate it directly, the English might feel too abrupt, potentially missing the nuanced disappointment embedded in the source. The word "天才," meaning "genius," is straightforward, but the challenge lies in maintaining the balance between the source's phonetic length and the target's natural flow. However, I must consider the pacing; translating "没看过" as "never seen" captures the sense of incredulity, but I need to ensure the English version doesn't feel rushed compared to the source's 19 phonemes. To avoid compressing the audio, expanding the clause slightly by adding "such a" before "weak" helps match the phonetic footprint while preserving semantic fidelity. Ultimately, weighing these phonetic and semantic factors, the most natural fit takes shape as "never seen such a weak genius." This choice aligns well with the target phoneme sequence "[n, e, v, ə, r, s, ii, n, s, ʌ, ch, ə, w, ii, k, zh, ii, n, y, ə, s]" and its length of 21, achieving a harmonious balance between the original's brevity and the target's expressive clarity. The source phoneme sequence is: [m, ei_2, k, an_1, g, uɔ_4, zh, ə_4, m, ə_0, ʒ, uɔ_4, d, ə_0, t, i, an_1, ts, ai_2], with a length of 19. The target phoneme sequence is: [n, e, v, ə, r, s, ii, n, s, ʌ, ch, ə, w, ii, k, zh, ii, n, y, ə, s], with a length of 21.

DuraS2ST output

never seen such a weak genius

Example 5

Source

未出嫁的女子戴圆形的耳环。

Duration-planning CoT

Show full reasoning
As I begin to digest the Chinese phrase "未出嫁的女子戴圆形的耳环," I am struck by its cultural and emotional resonance. The phrase carries a sense of tradition and expectation, with "未出嫁的女子" evoking the image of unmarried women and "圆形的耳环" suggesting a specific style of jewelry. The phoneme sequence length of 25 indicates a moderately paced sentence, not too brief, allowing for a nuanced expression of the idea. This duration suggests that the translation should neither rush nor drag, but rather flow naturally. In translating this into English, I face the challenge of maintaining the semantic richness while respecting the phonetic length. The word "未出嫁" directly translates to "unmarried," but simply using "unmarried" might not capture the full cultural context. If I opt for "unmarried women," it aligns well with the source's intent and helps maintain the pacing. The phrase "戴圆形的耳环" translates to "wear round earrings," which is straightforward, yet I must ensure it doesn't result in a rushed delivery compared to the source's 25 phonemes. By choosing "wear round earrings," I preserve the imagery and rhythm, ensuring the translation doesn't feel compressed or overly extended. Weighing these phonetic and semantic factors, the most natural fit takes shape as "unmarried women wear round earrings." This choice respects the original's cultural nuances and maintains a balanced phonetic footprint. The target phoneme sequence "[ə, n, m, e, r, ii, d, w, i, m, ə, n, w, e, r, r, au, n, d, i, r, i, ŋ, z]" with a length of 24 achieves a harmonious isochrony with the source, ensuring the translation flows seamlessly and retains the intended meaning. The source phoneme sequence is: [w, ei_4, ch, uu_1, j, i, a_4, d, ə_0, n, yu_2, z, iii_3, d, ai_4, yu, an_2, x, iŋ_2, d, ə_0, ər_3, h, u, an_2], with a length of 25. The target phoneme sequence is: [ə, n, m, e, r, ii, d, w, i, m, ə, n, w, e, r, r, au, n, d, i, r, i, ŋ, z], with a length of 24.

DuraS2ST output

unmarried women wear round earrings