105. The processing by the speech decoder 106 generates a speech signal having the desired content according to the user-input text data with the desired speaking style according to the prosody data. In other words, the speech decoder 106 decodes the speech feature embedding and prosody feature embedding to generate a corresponding expressive speech signal. The speech signal may be a waveform encoded in any suitable digital audio format.

The prosody of the generated speech signal may be adjusted by a user simply by providing a different speaking style input to the system. New prosody data may be generated or the existing prosody data may be modified based upon the input different speaking style. The prosody encoder 105 may process the prosody data to generate a prosody feature embedding corresponding to the different speaking style. The speech decoder 106 processes the new prosody feature embedding and the speech feature embedding to generate a speech signal having the different speaking style but with the same linguistic/para-linguistic content corresponding to the user-input text data.

The speech feature embedding and the prosody feature embedding may be combined and provided as input to the speech decoder 106. The speech decoder 106 may be configured to process the combined embedding to generate the speech signal. The combining operation may comprise any operation resulting in a single embedding. For example, the combination may be performed by a concatenation, an addition, an averaging, a dot product, or a Hadamard product. The speech feature embedding and the prosody feature embedding may be vectors having dimensions adapted for combination. For example, in some embodiments the embedding converter 104 and the prosody encoder 105 may generate vector outputs of the same dimension, which may be combined by a suitable operation such as addition.

The speech decoder 106 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

In some implementations, the speech audio generation system 102 may comprise a duration predictor 107. The duration predictor 107 is configured to process the text feature embedding to generate a predicted spoken duration for the user-specified text data. For example, the duration predictor 107 may determine a number of audio frames that a particular element of the user-input text data should last for. The predicted spoken duration may be used to modify the text feature embedding to reflect the predicted duration of each element of the user-input text data. The embedding converter 104 may be configured to process the duration modified text feature embedding to generate the speech feature embedding.

The duration predictor 107 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

The training system 108 is used or otherwise accessed when training the speech audio generation system 102. The training system 108 comprises a speech encoder 109, training data 110 comprising a plurality of training examples, and a model trainer 111. The training system 108 may further comprise a prosody analyzer 112.

During training, model trainer 111 is configured to receive one or more training examples from training data 109, and to process the training examples to update parameters of the speech audio generation system 102. The parameters of the speech encoder 109 are also updated during trained by the model trainer 111 at the same time. After training however, the speech encoder 109 is not required in order to generate a speech signal due to the use of the embedder converter 104 to convert from the text embedding space to the speech embedding space.

Each training example comprises a reference speech signal and reference text data corresponding to the reference speech signal. The training example may further comprise prosody data corresponding to the reference speech signal. Alternatively, the prosody analyzer 112 may be configured to process the reference speech signal to generate prosody data corresponding to the reference speech signal. The prosody analyzer 112 may determine prosodic statistical features by analyzing the reference speech signal and computing the appropriate statistics of the signal. Alternatively, or in addition, the prosody analyzer 112 may comprise a machine-learned model that has previously been trained to extract prosodic features. In this regard, the machine-learned model may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

During training, the prosody encoder 105 of the speech audio generation system 102 is configured to generate a prosody feature embedding from the prosody data corresponding to the reference speech signal.

The speech encoder 109 is configured to process the reference speech signal to generate a speech feature embedding. The speech encoder 109 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

During training, the speech decoder 106 of the speech audio generation system 102 is configured to process the speech feature embedding and the prosody feature embedding to generate a reconstruction of the reference speech signal. The model trainer 111 is configured to use the reconstructed speech signal as part of a loss function for training the speech audio generation system 102. For example, the loss function may comprise a reconstruction loss term based upon a measure of difference between the reconstructed speech signal and the reference speech signal. The measure of difference may be any appropriate measure such as an L1-norm (absolute difference) or L2-norm (Euclidean distance). The measure of difference may compare the waveforms of the reconstructed and reference speech signals directly or the measure of difference may compare a representation derived from the two speech signals such as a linear or mel-spectrogram.

During training, the text encoder 103 is configured to process the reference text data to generate a text feature embedding. The embedding converter 104 is configured to perform one or more embedding conversions. That is, the embedding converter 104 may be configured to process the text feature embedding generated from the reference text data to generate a (text converted) speech feature embedding. Additionally or alternatively, the embedding converter 104 may be configured to process the speech feature embedding generated from the reference speech signal to generate a (speech converted) text feature embedding. The model trainer 111 is configured to use the one or more embedding conversions in a loss function for training speech audio generation system 102. For example, the loss function may comprise a conversion loss term based upon a Kullback-Liebler (KL) divergence computed using the one or more embedding conversions.

The model trainer 111 is configured to determine a loss value using the loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions. The model trainer 111 is configured to update one or more of the speech decoder 106, the prosody encoder 105, the speech encoder 109, the embedding converter 104, or the text encoder 103 based upon the determined loss value. As described in more detail below, the particular components that are to be updated may be dependent on the particular form of the loss function and which components contribute towards the determination of the loss value. The model trainer 111 may update the components based upon a gradient optimization technique such as stochastic gradient descent. In implementations where the speech audio generation system 102 comprises a duration predictor 107, the duration predictor 107 may be configured to operate during training in a similar manner discussed above for generating a speech signal from user input text-data. The model trainer 111 may also be configured to update the duration predictor 107 as appropriate. As discussed above, training is carried out in one-stage, that is the components of the system are trained jointly, end-to-end.

In some implementations, the speech audio generation system 102 and the speech encoder 109 may be based upon a variational auto-encoder framework. For example, the text encoder 103 may comprise a machine-learning model having parameters that represents a probability distribution conditioned on the user-input text data. The conditional probability distribution parameterized by the text encoder 103 may form a “prior encoder” of a variational auto-encoder. The text feature embedding may be generated by sampling from this conditional probability distribution. In implementations comprising a duration predictor 107, the text encoder 103 and the duration predictor 107 may together parameterize the conditional probability distribution.

Likewise, the speech encoder 109 may comprise a machine-learning model having parameters that represents a probability distribution conditioned on a speech signal. The conditional probability distribution parameterized by the speech encoder 109 may form a “posterior encoder” of a variational auto-encoder. A speech feature embedding (of the input speech signal) may be generated by sampling from this conditional probability distribution.

The speech decoder 106 may comprise a machine-learning model having parameters that represents a probability distribution conditioned on a speech feature embedding and a prosody feature embedding. The conditional probability distribution parameterized by the speech decoder 106 may form a “decoder” of a variational auto-encoder. The speech signal may be generated by sampling from this conditional probability distribution.

All of the above discussed conditional probability distributions may take any appropriate form. For example, the conditional probability distributions may be based upon a Gaussian distribution, in particular, with diagonal covariance.

Where a variational auto-encoder is implemented, the loss function may be based upon an evidence-based lower bound (ELBO). For example, the loss function may comprise a conversion loss term based upon the KL-divergence between the conditional probability distributions parameterised by the posterior encoder and the prior encoder.

FIG. 2 illustrates an example method 200 of generating expressive speech audio. The method 200 can be implemented using the system described above with reference to FIG. 1.

Text-data 201 is received and processed by a text encoder 202 to generate a text feature embedding 203. As discussed above, the received text data is indicative of the desired content of the speech audio to be generated. For example, the text data may comprise data indicative of words to be spoken and/or any paralinguistic utterances such as sighs, yawns, moans, laughs, grunts, etc. The text data may be encoded by a sequence of vectors with each vector representing a word, character, phoneme or any other sub-word unit or paralinguistic utterance. For example, a character may be a letter, a number, and/or a tag indicating a paralinguistic utterance. The elements of a character vector may correspond with one character out of a set of possible characters, with each character represented by a character vector with only one non-zero element (also known as a one-hot vector). Additionally or alternatively, the speech content may be represented by continuous embeddings, e.g. character embeddings and/or word embeddings.

The text feature embedding 203 may comprise one or more vectors in a learned embedding space. The embedding space may disentangle aspects of the input text data 201 and may provide a representation or encoding of the input text data 201 that is more amenable for the subsequent processing to generate speech audio. The text encoder 202 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

As discussed above, the text encoder 202 may parameterize a conditional probability distribution. In such implementations, the text feature embedding 203 may be generated by sampling from this conditional probability distribution.

An embedding converter 204 processes the text feature embedding 203 to generate a speech feature embedding 205. The embedding converter 204 is trained with the aim of generating a speech feature embedding that would have been generated by a speech encoder from a speech signal corresponding to the user-input text data. The speech feature embedding may comprise one or more vectors in a learned embedding space that disentangles aspects of a speech signal and the embedding converter 204 provides a conversion from the learned text embedding space to the learned speech embedding space in order for speech audio corresponding to the user-input text data 201 to be generated. The embedding converter 204 may comprise a non-linear invertible transform such as a flow model and may be implemented using one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

Prosody data 206 is received and processed by a prosody encoder 207 to generate a prosody feature embedding 208. As discussed above, the prosody data is obtained from user-input speech style data. The speech style data may comprise data indicative of a particular speech expression. For example, the data may indicate a speech expression as natural language, e.g. “Surprised”, “In disbelief”, “Accusingly”, etc. In some cases, the speech expression data may implicitly indicate a speech expression, for example by indicating a particular context of the video game such as “Running away from demons”. In some implementations, the prosody data may be the speech style data itself.

The prosody data may relate to (or represent) the volume and/or pitch of the desired speech audio. The prosody data may comprise prosodic statistical features. For example, the prosody data may comprise one or more statistical features of a pitch contour and/or a volume contour for the generated speech signal. The one or more statistical features may comprise: a mean, a variance, a maximum and a minimum of a pitch contour for the speech audio; and/or a mean, a variance, and a maximum of a volume contour for the speech audio. Statistical features provide a low-dimensional representation of a desired speech style and can allow for more control of various speech style attributes when generating expressive speech audio. Processing statistical features may use fewer computational resources and/or be processed faster than high dimensional features, such as spectrogram data. Statistical features may also require less storage space than other types of speech style features. In implementations where the speech style features are transmitted via a network, transmitting statistical features may also use fewer network resources and consume less network bandwidth. In some implementations, a user may directly provide prosody data as the user-input speech style data.

The prosody feature embedding 208 is a representation or encoding of the prosody data 206 in a learned prosody embedding space. The prosody feature embedding 208 may comprise one or more vectors in the learned embedding space. The prosody encoder 207 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

A speech decoder 209 processes the speech feature embedding 205 and the prosody feature embedding 208 to generate a speech signal 210. As discussed above, the processing by the speech decoder 209 generates a speech signal 210 having the desired content according to the user-input text data 201 with the desired speaking style according to the prosody data 206. In other words, the speech decoder 209 decodes the speech feature embedding 205 and prosody feature embedding 208 to generate a corresponding expressive speech signal 210. The speech signal 210 may be a waveform encoded in any suitable digital audio format.

The speech feature embedding 205 and the prosody feature embedding 208 may be combined to generate a combined embedding. In these implementations, the speech decoder 209 processes the combined embedding to generate the speech signal 210. The combining operation may comprise any operation resulting in a single encoding. For example, the combination may be performed by a concatenation, an addition, an averaging, a dot product, or a Hadamard product. The speech feature embedding 205 and the prosody feature embedding 208 may be vectors having dimensions adapted for combination. For example, in some embodiments the embedding converter 204 and the prosody encoder 207 may generate vector outputs of the same dimension, which may be combined by a suitable operation such as addition.

The speech decoder 209 may be comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

As discussed above, the speech decoder 209 may parameterize a conditional probability distribution. In such implementations, the speech signal 210 may be generated by sampling from this conditional probability distribution.

In some implementations, a duration predictor 211 processes the text feature embedding 203 to generate a predicted spoken duration for the user-specified text data 201. For example, the duration predictor 211 may determine a number of audio frames that a particular element of the user-input text data 201 should last for. The predicted spoken duration may be used to modify the text feature embedding 203 to reflect the predicted duration of each element of the user-input text data 201. The embedding converter 204 may process the duration modified text feature embedding 203 to generate the speech feature embedding 205.

The duration predictor 211 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

FIG. 3 illustrates an example method 300 of training a speech audio generation system. The method 300 can be implemented using the system described above with reference to FIG. 1.

A training example 301 comprising a reference speech signal 302 and corresponding reference text data 303 is received. The training example 301 may be part of a training data set and may be obtained from local storage or received over a network connection via any appropriate communications method.

A speech encoder 304 processes the reference speech signal 302 to generate a speech feature embedding 305. The speech feature embedding 305 may comprise one or more vectors in a learned embedding space that disentangles aspects of the reference speech signal 302. The speech encoder 304 may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

As discussed above, the speech encoder 304 may parameterize a conditional probability distribution. In such implementations, the speech feature embedding 305 may be generated by sampling from this conditional probability distribution.

Prosody data 306 corresponding to the reference speech signal 302 is received and processed a prosody encoder 307 to generate a prosody feature embedding 308. The prosody data 306 may relate to (or represent) the volume and/or pitch of the reference speech signal 302. The prosody data 306 may comprise prosodic statistical features. For example, the prosody data 306 may comprise one or more statistical features of a pitch contour and/or a volume contour of the reference speech signal 302. The one or more statistical features may comprise: a mean, a variance, a maximum and a minimum of a pitch contour for the speech audio; and/or a mean, a variance, and a maximum of a volume contour for the speech audio.

The training example 301 may further comprise the prosody data 306 corresponding to the reference speech signal 302. Alternatively, a prosody analyzer may process the reference speech signal 302 to generate prosody data 306 corresponding to the reference speech signal 302. The prosody analyzer may determine prosodic statistical features by analyzing the reference speech signal 302 and computing the appropriate statistics of the signal. Alternatively, or in addition, the prosody analyzer may comprise a machine-learned model that has previously been trained to extract prosodic features. In this regard, the machine-learned model may comprise one or more neural network layers. For example, the neural network layers may comprise feedforward layers, e.g. fully connected layers and/or convolutional layers. Additionally or alternatively, the neural network layers may comprise recurrent layers, e.g. LSTM layers and/or bidirectional LSTM layers. Additionally or alternatively, the neural network layers may comprise layers configured based upon a Transformer block. For example, the neural network layers may implement an attention mechanism such as self-attention or cross-attention.

A speech decoder 309 processes the speech feature embedding 305 generated from the reference speech signal 302 and the prosody feature embedding 308 to generate a reconstructed speech signal 310. That is, the speech decoder 309 decodes the speech feature embedding 305 and the prosody feature embedding 308 to attempt to reconstruct the original reference speech signal 302. As described in more detail below, the reconstructed speech signal 310 and the reference speech signal 302 are used by a model trainer 315 as part of a loss function for the training method.

As discussed above, the speech decoder 309 may parameterize a conditional probability distribution. In such implementations, the reconstructed speech signal 310 may be generated by sampling from this conditional probability distribution.

A text encoder 311 processes the reference text data 303 of the training example 301 to generate a text feature embedding 312. As discussed above, the text encoder 311 may parameterize a conditional probability distribution. In such implementations, the text feature embedding 312 may be generated by sampling from this conditional probability distribution.

An embedding converter 313 generates one or more embedding conversions 314. The one or more embedding conversions 314 may comprise a speech converted text feature embedding generated by processing the speech feature embedding 305 using the embedding converter 313, and/or a text converted speech feature embedding generated by processing the text feature embedding 312 using the embedding converter 313. That is, the one or more embedding conversions 314 may comprise a single conversion in a single direction, i.e. from the speech feature embedding space to the text feature embedding space or from the text feature embedding space to the speech feature embedding space, or may comprise conversions in both directions. As described in more detail below, the model trainer 315 uses the one or more embedding conversions 314 as part of a loss function for the training method.

The model trainer 315 determines a loss value using a loss function based upon the reconstructed speech signal 310, the reference speech signal 302 and the one or more embedding conversions 314.

For example, the loss function may comprise a reconstruction loss term based upon a measure of difference between the reconstructed speech signal 310 and the reference speech signal 302. The measure of difference may be any appropriate measure such as an L1-norm (absolute difference) or L2-norm (Euclidean distance). The measure of difference may compare the waveforms of the reconstructed and reference speech signals directly or the measure of difference may compare a representation derived from the two speech signals such as a linear or mel-spectrogram.

In another example, the loss function may comprise a conversion loss term based upon a Kullback-Liebler (KL) divergence computed using the one or more embedding conversions. In implementations where a variational auto-encoder is implemented, the conversion loss term may be based upon the KL-divergence between the conditional probability distributions parameterised by the posterior encoder and the prior encoder. For example, where a speech feature embedding generated from a reference speech signal is converted to a text feature embedding, computing the KL-divergence may comprise computing the probability of the speech feature embedding under the conditional probability distribution parameterized by the speech encoder (posterior encoder), computing the probability of the speech converted text feature embedding under the conditional probability distribution parameterized by the text encoder (prior encoder), and comparing the computed probabilities.

Likewise, where a text feature embedding generated from reference text data is converted to a speech feature embedding, computing the KL-divergence may comprise computing the probability of the text feature embedding generated from the reference text data under the conditional probability distribution parameterized by the text encoder, computing the probability of the text converted speech feature embedding under the conditional probability distribution parameterized by the speech encoder, and comparing the computed probabilities. Where both types of conversions are used, the conversion loss term may comprise a combination of the individual KL-divergence values.

As an alternative to the KL-divergence, the conversion loss term may be based upon a measure of difference between the text converted speech feature embedding and the speech feature embedding generated from the reference speech signal and/or a measure of difference between the speech converted text feature embedding and the text feature embedding generated from the reference text data. The measure of difference may be any appropriate measure of difference such as an L1-norm (absolute distance) or L2-norm (Euclidean distance).

In some implementations, the loss function comprises a reconstruction loss term and a conversion loss term. Where a variational auto-encoder is implemented, the loss function may be based upon an evidence-based lower bound (ELBO).

The model trainer 315 updates one or more of the speech decoder 309, the prosody encoder 307, the embedding converter 313, the speech encoder 304 or the text encoder 311 based upon the determined loss value. The particular components that are to be updated may be dependent on the particular form of the loss function and which components contribute towards the determination of the loss value. For example, where the loss function comprises a reconstruction loss term based upon a measure of a difference between the reconstructed speech signal 310 and the reference speech signal 302, the components that contributed towards generating the reconstructed speech signal 310, i.e. the speech encoder 304, the prosody encoder 307 and the speech decoder 309 may be updated. Where the loss function comprises a conversion loss term, the text encoder 311, the speech encoder 304 and the embedding converter 313 may be updated.

The model trainer 315 may update the components based upon a gradient optimization technique such as stochastic gradient descent and may include include momentum-based methods such as Adam, RMSProp, and AdaGrad. An update may be based upon a single training example or may be based upon a batch of training examples. For example, in batch updating, a plurality of training examples may be processed to determine an update value for each individual training example of the batch. The actual update to the components may be based upon a combination of the individual update values such as an average over the individual update values.

The training method 300 may be repeated for further training examples/batches in the training data set and the training method may be performed over a plurality of passes over the training data set. The training may continue until a stopping criterion is reached. For example, where the loss function value reaches a particular threshold value or when a fixed number of training iterations have been performed.

In some implementations, the training method 300 may also utilize adversarial training. In general, adversarial training utilizes a discriminator machine learning model that is trained at the same time as the generative system. The purpose of the discriminator is to classify whether a particular input example was generated by the system or whether the particular input example is a reference example. Thus, in adversarial training, the system attempts to generate examples that are good enough to fool the discriminator whilst the discriminator is trained to detect the output of the system. With respect to the training method 300, the loss function may further comprise an adversarial loss term based upon an ability of a discriminator machine learning model to differentiate between the reconstructed speech signal 310 and the reference speech signal 302.

In some implementations, a duration predictor 316 may process the text feature embedding 312 to generate a predicted spoken duration for the reference text data 303. The predicted spoken duration may be used to modify the text feature embedding 312 to reflect the predicted duration of each element of the reference text data 303. Where the text feature embedding 312 is used in the method 300, the duration modified text feature embedding 312 may be used, for example, by the embedding converter 313 to generate the one or more embedding conversions 314. The loss function may comprise a duration prediction term based upon the predicted spoken duration and a reference spoken duration. The reference spoken duration may be provided as part of the training example 301 or may be determined from the reference speech signal 302.

In some implementations, where a text feature embedding 312 generated from the reference text data 303 is converted to a speech feature embedding using the embedding converter 313, the speech decoder 309 may also process the text converted speech feature embedding to generate a second speech signal. This mimics the text-to-speech generation process and the loss function may further comprise a generative loss term based upon the second speech signal and the reference speech signal 302. For example, the generative loss term may be in the same form as the reconstruction loss term or adversarial training may be used for the generative loss term or any other appropriate measure of difference may be used.

It will be appreciated that whilst the loss function may comprise a plurality of terms and only a subset of components may be updated based upon particular terms of the loss function, the components of the system are jointly trained and updated together as a one-stage system.

It will be further appreciated that even with the inclusion of prosody control via the prosody encoder and the speech decoder, the system can still be trained using existing training frameworks such as a variational auto-encoder framework without necessarily requiring any prosody specific modification of the loss functions used in the framework.

FIG. 4 is a flow diagram illustrating an example method 400 of generating expressive speech audio. FIG. 4 displays a general overview of the methods described in relation to FIG. 2.

In step 4.1, user-input text data is received. The received text data is indicative of the desired content of the speech audio to be generated. For example, the text data may comprise data indicative of words to be spoken and/or any paralinguistic utterances such as sighs, yawns, moans, laughs, grunts, etc. The text data may be encoded by a sequence of vectors with each vector representing a word, character, phoneme or any other sub-word unit or paralinguistic utterance.

In step 4.2, prosody data obtained from user-input speech style data is received. The prosody data may comprise prosodic statistical features. For example, the prosody data may comprise one or more statistical features of a pitch contour and/or a volume contour for the generated speech signal.

In step 4.3, the user-input text data is processed by a text encoder to generate a text feature embedding. The text feature embedding may comprise one or more vectors in a learned embedding space. Optionally, the text feature embedding may be processed by a duration predictor to generate a predicted spoken duration for the user-specified text data. The text feature embedding may be modified based upon the predicted spoken duration.

In step 4.4, a speech feature embedding is generated by an embedding converter based upon the text feature embedding. The embedding converter is trained with the aim of generating a speech feature embedding that would have been generated by a speech encoder from a speech signal corresponding to the user-input text data. The speech feature embedding may comprise one or more vectors in a learned embedding space.

In step 4.5, the prosody data is processed by a prosody encoder to generate a prosody feature embedding. The prosody feature embedding is a representation or encoding of the prosody data in a learned prosody embedding space. The prosody feature embedding may comprise one or more vectors in the learned embedding space.