Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-End Speech Synthesis

Authors: Guanghui Xu, Wei Song, Zhengchen Zhang, Chao Zhang, Xiaodong He, Bowen Zhou

Abstract: Despite prosody is related to the linguistic information up to the discourse structure, most text-to-speech (TTS) systems only take into account that within each sentence, which makes it challenging when converting a paragraph of texts into natural and expressive speech. In this paper, we propose to use the text embeddings of the neighboring sentences to improve the prosody generation for each utterance of a paragraph in an end-to-end fashion without using any explicit prosody features. More specifically, cross-utterance (CU) context vectors, which are produced by an additional CU encoder based on the sentence embeddings extracted by a pre-trained BERT model, are used to augment the input of the Tacotron2 decoder. Two types of BERT embeddings are investigated, which leads to the use of different CU encoder structures. Experimental results on a Mandarin audiobook dataset and the LJ-Speech English audiobook dataset demonstrate the use of CU information can improve the naturalness and expressiveness of the synthesized speech. Subjective listening testing shows most of the participants prefer the voice generated using the CU encoder over that generated using standard Tacotron2. It is also found that the prosody can be controlled indirectly by changing the neighbouring sentences.

Paper link:

Wave Demo

Ground truth means ground truth.
Copy-syns means synthesized from ground truth mel-spectragram using waveRNN vocoder.
Tacotron2 means standard tacotron2.
CSE-based Tacotron2 means tacotron2 with CSE cross-utterance encoder.
PSE-based Tacotron2 means tacotron2 with PSE cross-utterance encoder.


Mandarin

Cases from the same audiobook but not included in the training set

1. 顷刻间寸寸爆裂而开。
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
2. 旋即一道火红影子快若闪电般的掠进场中。
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
4. 狩猎，算是青阳镇几大势力另外的一种博弈之处。
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
5. 只不过有些奇异的是。
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2

Cases in open domain paragraph(4 consecutive sentences). Synthesized by griffin-lim algorithm

1. novel: 过了一阵，他们看到一个涂着红黄白等各色油彩的小丑跑了过来，先是往天上扔出一个个网球，接着在人们的注意力被吸引到半空时，不知从哪里变出了一束花，一束塞维亚菊。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
2. novel: 张家口是南北通道，塞外皮毛集散之地，人烟稠密，市肆繁盛。郭靖手牵红马，东张西望，他从未到过这般大城市，但见事事透着新鲜。来到一家大酒店之前，腹中饥饿，便把马系在门前马桩之上。进店入座，要了一盘牛肉，两斤面饼，大口吃了起来。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
3. introduction: 长城修筑的历史可上溯到西周时期，发生在首都镐京的著名的典故“烽火戏诸侯”就源于此。春秋战国时期列国争霸，互相防守，长城修筑进入第一个高潮，但此时修筑的长度都比较短。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
4. introduction: 桥的两端紧连着街市，车水马龙，热闹非凡。一艘准备驶过拱桥的巨大漕船的细节描绘，一直为人们所称道：船正在放倒桅杆准备过桥，船夫们呼唤叫喊，握篙盘索。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
5. prose: 山朗润起来了，水长起来了，太阳的脸红起来了。小草偷偷地从土里钻出来，嫩嫩的，绿绿的。园子里，田野里，瞧去，一大片一大片满是的。坐着，躺着，打两个滚，踢几脚球，赛几趟跑。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
6. prose: 叶子出水很高，像亭亭的舞女的裙。层层的叶子中间，零星地点缀着些白花，有袅娜地开着的，有羞涩地打着朵儿；正如一粒粒的明珠，又如碧天里的星星，又如刚出浴的美人。
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2

English

Cases from the same audiobook but not included in the training set

1. It must be said that it is in no way like the transition type of Subiaco.
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
2. and who are therefore lodged apart from all other districts of the jail. End quote.
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
3. instruct the prisoners, quote, give spiritual advice and administer religious consolation, end quote. to all who might need them;
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
4. I shall have occasion to speak again, and more at length, of Mr. Cope's careless and perfunctory discharge.
Ground truth Copy-syns Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2

Cases in open domain paragraph(2 consecutive sentences). Synthesized by griffin-lim algorithm

20000 Leagues Under the Sea
1. You know as well as I do, Professor, that man can live under water providing he carries with him a sufficient supply of breathable air. In submarine works, the workman, clad in an impervious dress, with his head in a metal helmet, receives air from above by means of forcing pumps and regulators.
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
2. The sea was distinctly visible for a mile all round the Nautilus. What a spectacle! What pen can describe it? Who could paint the effects of the light through those transparent sheets of water, and the softness of the successive gradations from the lower to the superior strata of the ocean?
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
The Scarlet Letter
3. Such an exhibition however was but to be pictured in fancy; not to be anticipated nor desired. What I saw in him, as evidently as the indestructible ramparts of Old Ticonderoga, already cited as the most appropriate simile, was the features of stubborn and ponderous endurance, which might well have amounted to obstinacy in his earlier days;
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2
4. The somewhat dim coal fire has an essential Influence in producing the effect which I would describe. It throws its unobtrusive tinge throughout the room, with a faint ruddiness upon the walls and ceiling, and a reflected gleam upon the polish of the furniture.
Tacotron2 CSE-based Tacotron2 PSE-based Tacotron2

Prosody control

We used the Griffin-Lim algorithm to generate speech signals, and dropout was turned off at test-time.

  Who called Mary? Tom called Mary     What did Tom do with Mary? Tom called Mary.