Abstract: Despite prosody is related to the linguistic information up to the discourse structure, most text-to-speech (TTS) systems only take into account that within each sentence, which makes it challenging when converting a paragraph of texts into natural and expressive speech. In this paper, we propose to use the text embeddings of the neighboring sentences to improve the prosody generation for each utterance of a paragraph in an end-to-end fashion without using any explicit prosody features. More specifically, cross-utterance (CU) context vectors, which are produced by an additional CU encoder based on the sentence embeddings extracted by a pre-trained BERT model, are used to augment the input of the Tacotron2 decoder. Two types of BERT embeddings are investigated, which leads to the use of different CU encoder structures. Experimental results on a Mandarin audiobook dataset and the LJ-Speech English audiobook dataset demonstrate the use of CU information can improve the naturalness and expressiveness of the synthesized speech. Subjective listening testing shows most of the participants prefer the voice generated using the CU encoder over that generated using standard Tacotron2. It is also found that the prosody can be controlled indirectly by changing the neighbouring sentences.
Paper link:
Wave Demo
Ground truth means ground truth.
Copy-syns means synthesized from ground truth mel-spectragram using waveRNN vocoder.
Tacotron2 means standard tacotron2.
CSE-based Tacotron2 means tacotron2 with CSE cross-utterance encoder.
PSE-based Tacotron2 means tacotron2 with PSE cross-utterance encoder.
Mandarin
Cases from the same audiobook but not included in the training set
1. 顷刻间寸寸爆裂而开。
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
2. 旋即一道火红影子快若闪电般的掠进场中。
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
4. 狩猎,算是青阳镇几大势力另外的一种博弈之处。
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
5. 只不过有些奇异的是。
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
Cases in open domain paragraph(4 consecutive sentences). Synthesized by griffin-lim algorithm
Cases from the same audiobook but not included in the training set
1. It must be said that it is in no way like the transition type of Subiaco.
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
2. and who are therefore lodged apart from all other districts of the jail. End quote.
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
3. instruct the prisoners, quote, give spiritual advice and administer religious consolation, end quote. to all who might need them;
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
4. I shall have occasion to speak again, and more at length, of Mr. Cope's careless and perfunctory discharge.
Ground truth
Copy-syns
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
Cases in open domain paragraph(2 consecutive sentences). Synthesized by griffin-lim algorithm
20000 Leagues Under the Sea
1. You know as well as I do, Professor, that man can live under water providing he carries with him a sufficient supply of breathable air. In submarine works, the workman, clad in an impervious dress, with his head in a metal helmet, receives air from above by means of forcing pumps and regulators.
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
2. The sea was distinctly visible for a mile all round the Nautilus. What a spectacle! What pen can describe it? Who could paint the effects of the light through those transparent sheets of water, and the softness of the successive gradations from the lower to the superior strata of the ocean?
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
The Scarlet Letter
3. Such an exhibition however was but to be pictured in fancy; not to be anticipated nor desired. What I saw in him, as evidently as the indestructible ramparts of Old Ticonderoga, already cited as the most appropriate simile, was the features of stubborn and ponderous endurance, which might well have amounted to obstinacy in his earlier days;
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
4. The somewhat dim coal fire has an essential Influence in producing the effect which I would describe. It throws its unobtrusive tinge throughout the room, with a faint ruddiness upon the walls and ceiling, and a reflected gleam upon the polish of the furniture.
Tacotron2
CSE-based Tacotron2
PSE-based Tacotron2
Prosody control
We used the Griffin-Lim algorithm to generate speech signals, and dropout was turned off at test-time.