106 or the native language associated with a recipient user 104b that speaks a different native language than the native language in which the original utterance 106 was spoken. There are a number of ways to determine the language of the transliterated script 121. For instance, a user's language preference may be set explicitly by the user when executing a speech recognition program on their user device. Likewise, the user providing the utterance may explicitly set/input the native language of the recipient user in the context of language translation. In additional examples, the user's language preference may be based on a geographical region in which the user device 102 is located. Alternatively, a language identification system may identify the language of the originating utterance on a per utterance basis so that the speech recognition result in the target script can be transliterated back to the originating language spoken by the user of the utterance.

The multilingual E2E ASR model 300 may implement any type of sequence-to-sequence neural network architecture. For instance, the multilingual E2E ASR model 300 implements a Listen, Attend, Spell (LAS) neural network architecture. In some implementations, the multilingual E2E ASR model 300 uses a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to the latency constraints associated with interactive applications. Referring to FIG. 3, an example multilingual E2E ASR model 300 includes an encoder network 310, a prediction network 320, and a joint network 330. The encoder network 310, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 (FIG. 1)) vectors x=(x₁, x₂, . . . , x_T), where x_t∈R_d, and produces at each time step a higher-order feature representation. This higher-order feature representation is denoted as h₁^enc, . . . , h_T^enc.

Similarly, the prediction network 320 is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output by a final Softmax layer 340 so far, y₀, . . . , y_ui−1, into a dense representation h_u_i^dec. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction networks 310, 320 are combined by the joint network 330. The joint network then predicts P(y₁|x₁, . . . , x_t_i, y₀, . . . , y_ui−1), which is a distribution over the next output symbol. The Softmax layer 340 may employ any technique to select the output symbol with the highest probability in the distribution as the next output symbol predicted by the model 300. In this manner, the multilingual RNN-T model 300 does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The multilingual RNN-T model 300 does assume an output symbol is independent of future acoustic frames 110, which allows a multilingual RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network 310 of the multilingual RNN-T model 300 is made up of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 320 may have two 2,048-dimensional LSTM layers, each of which is also followed by 640-dimensional projection layer. Finally, the joint network 330 may also have 640 hidden units. The softmax layer may be composed of a unified grapheme set from all languages, i.e., 988 graphemes in total, that is generated using all unique graphemes in a plurality of training data sets 202 (FIG. 2).

As opposed to most state-of-the-art multilingual models that require the encoding of language information with audio inputs during training, implementations herein are directed toward the multilingual E2E ASR model 300 being language-agnostic such that no language information (e.g., embedding, vectors, tags, etc.) is provided with the input acoustic frames 110 to identify the language(s) associated with the input acoustic frames 110. Moreover, and discussed in greater detail below, the multi-lingual E2E ASR model 300 is not conditioned on any language information during training such that the model 300 is configured to receive training audio in any natural language and learn to predict speech recognition results 120 in a target script for the audio that match a corresponding reference transcription in the same target script independent of the respective natural language associated with the audio. As will become apparent, training the multilingual E2E ASR model 300 to be language-agnostic, permits all parameters of the model to be shared across all natural languages representing the input acoustic frames 110. Not only does this data and parameter sharing by the model improve computational costs, improve latency, and reduce memory constraints of the model 300, the model 300 is also able to provide benefits for data-scarce languages and enable training of the model 300 on new or different languages at any time, thereby providing a scalable and uniform model for multilingual speech recognition in a multitude of different multicultural societies where several languages are frequency used together (but often rendered with different writing systems). That is, by not depending on language information limits, the language-agnostic multilingual E2E ASR model 300 can be extended to newer languages and be adaptable to accepting codemixed utterances spoken in languages used during training.

Moreover, for Indic languages, code-switching in conversation provides additional challenges due to a considerable amount of variability in the usage of a second language (e.g., typically English) alongside native languages such as Tamil, Bengali, or Hindi. As a result, it is difficult to model context under which code switching occurs, and the language to which a spoken word should be assigned. This problem is further compounded by inconsistent transcriptions and text normalization. While Indic languages often overlap in acoustic and lexical content due to their language family relations and/or the geographic and cultural proximity of the native speakers, the respective writing systems occupy different Unicode blocks that result in inconsistent transcriptions. That is, a common word, workpiece, or phoneme can be realized with multiple variants in the native language writing systems, leading to increased confusions and inefficiency in data sharing when training the model 300.

Referring to FIG. 2, an example training process 200 for building/training the language-agnostic, multilingual E2E ASR model 300 includes transforming all languages used to train the model into one writing system (e.g., a target script) through a many-to-one transliteration module 400. By transliterating into one common writing system, the ASR model 300 will be able to map similar sounding acoustics to a single, canonical target sequence of graphemes, effectively separating modeling and rendering problems over traditional language-dependent multilingual models. As used herein, transliteration refers to a sequence-to-sequence mapping problem that aims to convert text/script from one writing system to another

A computing device 201, such as a remote server executing on a distributed system in a cloud computing environment, may execute the training process 200 and later push the trained language-agnostic, multilingual E2E ASR model 300 to user devices 102 for generating speech recognition results on-device. Additionally or alternatively, the trained model 300 may execute on the computing device 201 for generating speech recognition results 120 in the target script based on acoustic frames 110 received from user devices 102.

The training process 200 obtains a plurality of training data sets 202, 202a-n each associated with a respective native language that is different than the respective native languages of the other training data sets 202. Here, each training data set 202 includes a plurality of respective training data samples 204, 204a-n, whereby each training sample 204 includes audio 210 (e.g., an audible utterance) spoken in the respective native language and a corresponding transcription 220 of the audio 210 in a respective native script representing the respective native language.

For each respective training data sample 204 of each training data set 202 in the respective native language, the training process 200 transliterates the corresponding transcription 220 in the respective native script into corresponding transliterated text 221 representing the respective native language of the corresponding audio 210 in a target script. That is, the training process transliterates the native script of the transcriptions in all of the different native languages into the same target script, whereby the target script is associated with a different writing system than the writing systems associated with each of the native scripts. In some examples, the target script includes Latin script representing the Latin writing system. In the example shown, the training process 200 uses a many-to-one transliteration module 400 to transliterate the transcriptions 220 in the native scripts into the corresponding transliterated texts 221 in the target script.

Referring to FIGS. 2 and 4, in some implementations, the transliteration module 400 includes multiple transliteration transducers 400, 400a-n each associated with a respective native language for transliterating the respective native script representing the respective native language into the transliterated text 221 in the target script. For instance, FIG. 4 shows each transliteration transducer 400a-n associated with a respective native script and including a composition of three transducers: I° POO, where I includes an input transducer configured to map Unicode symbols to symbols in a pair language model, P includes a bigram pair language model transducer configured to map between symbols in the respective native script and the target script (e.g., Bengali-Latin; Hindi-Latin; Kannada-Latin; and Tamil-Latin), and O includes an output transducer configured to map the pair language model symbols to the target output symbols of the target script (e.g., Latin). Each pair language model transducer P includes an n-gram model over “pair′ symbols having an input Unicode code point paired with an output Unicode code point. Thus, as with grapheme-to-phoneme conversion, given an input lexicon including native script words and Latin script realizations of those words (e.g., known as Romanizations), expectation maximization is used to derive pairwise alignments between symbols in both the native and Latin scripts. FIG. 5 shows an example transliteration transducer transliterating Devanagari writing script into Latin script. The conditional probability of the transliterated word (e.g., Browser) is obtained by dividing a joint probability from the transliteration transducer by a marginalization sum over all input and output sequences. This computation is efficiently implemented by computing a shortest path in the transliteration transducer 400.

As set forth above, the input for training each pair language model transducer P of each transliteration transducer 400 includes respective transliteration pairs formed from native script words and possible Latin script Romanizations. As used herein, a “transliteration pair” (interchangeably referred to as a “transliterated pair” or “a native-transliterated word pair”) refers to a word in a native language script (e.g., a respective one of Bengali, Hindi, Kannada; or Tamil) paired with a corresponding spelling of the word in the target script (e.g., Latin script Romanization). However, the possible Latin script romanizations can result in the spelling of words in a variety of different ways since there is no standard orthography in the Latin script. Table 1 shows native script spellings of the English word “discovery” in each of the four Indic languages of Bengali, Hindi, Kannada, and Tamil with attested Romanizations of that word in transducer training data.

TABLE 1

Bengali
Hindi
Kannada
Tamil

[Image Omitted]

discoveri
discovery
discovary
tiskavari

discovery

discovery
discovery

diskovary

discoveri

diskovery

discowery

diskoveri

Table 1 shows that while the actual spelling of the word in English is attested in all four of the Indic native languages, annotators in each language may vary in the number and kind of Romanization they suggest. This variance by the annotators may be driven by many factors, including differences in pronunciation or simply individual variation. Unfortunately, spelling inconsistency across languages in transliterated text 221 creates confusion and diminishes the intended sharing of knowledge across languages when training the multilingual ASR model 300 with the transliterated text 221. To mitigate these inconsistencies where the transliteration transducer transliterates multiple different target script spellings for a same word, an agreement-based data pre-processing technique or a frequency-based data pre-processing technique can be employed.

In agreement-based data pre-processing, each transliteration transducer 400 associated with a respective native language is configured to only process transliteration pairs which have at least one common spelling in the target script of the transliterated text 221. For instance, in the above example where the target script spelling of “Discovery” is common across each of the four Indic languages of Bengali, Hindi, Kannada, and Tamil, the transliteration transducers 400 associated with each of the four Indic languages may be trained to only process the target script with the spelling “Discovery” while leaving all other spellings unprocessed. That is, in agreement-based pre-processing, the transliteration transducer 400a for transliterating Bengali to Latin is trained to only process the target script spelling “Discovery” without processing the other possible spellings of “discoveri”, “diskovary”, “diskovery”, and “diskoveri”. Table 2 below provides an example algorithm for training the transliteration transducers 400 on the agreement-based pre-processing technique.

TABLE 2

Algorithm 1 Agreement-based pre-processing

HiWords: Mapping from native Hindi words to Latin transliterated forms;

BnWords: Mapping from native Bengali words to Latin transliterated

forms;

TaWords: Mapping from native Tamil words to Latin transliterated forms;

KnWords: Mapping from native Kannada words to Latin transliterated

forms;

common_latin ← Latin(HiWords) ∩ Latin(BnWords) ∩ Latin(TaWords)

∩ Latin(KnWords)

for all mapping in {HiWords, BnWords, TaWords, KnWords} do

for all native_word in Native(mapping) do

agreed_latin ← mapping[native_word] ∩ common_latin

if agreed_latin ≠ ∅ then

mapping[native_word] ← agreed_latin

end if

end for

end for

In addition to the native-transliterate word pair, the training data also contains a frequency of occurrences of all transliterated forms for a word in the respective native script. By utilizing these frequencies of occurrence, the frequency-based data pre-processing technique transforms all of the transliteration pairs for each language. Moreover, the frequency-based data pre-processing may also rely on an empirical observation that the most frequent transliterated pairs formed usually correlated to commonly used spellings of proper nouns and/or actual dictionary spellings of the English words. Accordingly, when the training data includes multiple different spellings in the target text of the transliterated text 221 for a given native word, each respective transliteration transducer 400 is configured to only process/retain the target script with spellings that meet a frequency threshold and discard the rest. In some examples, the frequency threshold includes an average transliteration frequency per native word in the training data. Table 3 below provides an example algorithm for training the transliteration transducers 400 on the frequency-based pre-processing technique.

TABLE 3

Algorithm 2 Frequency-based pre-processing

Mappings: For each language, mapping from native words to

transliterated forms

for all mapping in Mappings do

for all native_word in Native(mapping) do

translits ← mapping[word]

avg_freq \leftarrow \frac{1}{❘ translits ❘} * \sum_{t \in translits} Freq (t)

mapping[native_word] ← {t|t∈translits, Freq(t) ≥ avg_freq}

end for

end for

Referring back to FIG. 2, in additional implementations, the many-to-one transliteration module 400 includes a language-independent transliteration transducer configured to transliterate each corresponding transcription 220 in each respective native script into the corresponding transliterated text 221 in the target script. As such, separate transliteration transducers 400 each associated with a respective language would not have to be trained individually.

After transliterating the corresponding transcription 220 in the respective native script into corresponding transliterated text 221 representing the respective native language of the corresponding audio 210 in a target script, FIG. 2 shows the training process 200 associating the corresponding transliterated text 221 in the target script with the corresponding audio 210 in the respective native language to generate a respective normalized training data sample 240. Here, a normalizer 230 receives the audio 210 spoken in the respective native language from the respective training data sample 204 and the corresponding transliterated text 221 output from the transliteration module 400 to generate the respective normalized training data sample 240. While the example shows the transliterated text 221 in the target script replacing the corresponding transcription 220 in the respective native script, the normalized training data sample 240 may also include the transcription 220 in addition to the corresponding audio 210 and the transliterated text 221. Thereafter, data storage 260 (e.g., residing on memory hardware of the computing system 201) may store normalized training sets 205, 205a-n corresponding to respective ones of the received training data sets 202, 202a-n. That is, each normalized training set 205 includes a plurality of respective normalized training samples 240, 240a-n, whereby each respective normalized training sample 240 includes the audio 210 (e.g., an audible utterance) spoken in the respective native language and the corresponding transliterated text 221 representing the respective native language of the audio 210 in the target scrip.

In the example shown, the training process 200 trains, using the normalized training data samples 240 generated from each respective training data sample 204 of each training data set 202 and without providing any language information, the multilingual E2E ASR model 300 to predict speech recognition results 120 in the target script (e.g., Latin) for corresponding speech utterances 106 spoken in any of the different native languages (e.g., the Indic languages of Bengali, Hindi, Kannada, and Tamil) associated with the plurality of training data sets 202. As set forth above, the model 300 is trained without being conditioned on any language information associated with the normalized training data samples 240 provided as input such that the model 300 is agnostic to the natural languages of the audio 210 provided as input. In some examples, training the multilingual E2E ASR model 300 includes shuffling the normalized training data samples 240 such that a sequence of normalized training data samples 240 received as training inputs includes randomly selected audio 210 in any combination and order of natural languages. In doing so, multilingual training of the model 300 may be optimized so that the model 300 does not learn to apply weights favoring one particular language at a time as in the case if the model were trained by grouping the normalized training data samples 240 according to their respective native languages.

Training of the multilingual E2E ASR model 300 generally includes using a stochastic optimization algorithm, such as stochastic gradient decent, to train a neural network architecture of the model 300 through backpropagation. Here, the stochastic optimization algorithm defines a loss function (e.g., a cross-entropy loss function) based on a difference between actual outputs (e.g., recognition results 120 in the target script) of the neural network and desired outputs (e.g., the transliterated text 221 representing the respective native language of the audio 210 in the target scrip). For instance, the loss function is computed for a batch of training examples, and then differentiated with respect to each weight in the model 300.

Moreover, the training process 200 takes into account data imbalance across the plurality of data sets 202. Data imbalance is a natural consequence of the varied distribution of speakers across the world's languages. Languages with more speakers tend to produce transcribed data more easily. While some ASR systems may only train the AM on transcribed speech data, all components in a multilingual E2E model are trained on transcribed speech data. As a result, multilingual E2E models may be more sensitive to data imbalance. That is, the multilingual E2E ASR model 300 tends to be more influenced by over-represented native languages in the training data sets 202. The magnitude of over influence is more pronounced in the instant case when no language information/identifier is provided (e.g., no language identifiers encoded with the training audio or language models incorporated).

In some implementations, to address data imbalance across the plurality of data sets 202, the training process first augments the plurality of training data sets 202 with diverse noise styles. In these implementations, the degree of data augmentation for each language is determined empirically by observing a count of noisy copies in the training data set 202 associated with lowest-resource language (e.g., Kannada) that causes the model 300 to degrade in performance. Based on the count of noisy copies, the training data sets 202 associated with the remaining native languages are augmented with a target number of noise styles to result in equal amounts of data for each of the native languages used for training the model 300.

FIG. 6 provides a flowchart of an example arrangement of operations for a method 600 of training a language-agnostic, multilingual E2E ASR model 300. At operation 602, the method 600 includes obtaining a plurality of training data sets 202 each associated with a respective native language that is different than the respective native languages associated with the other training data sets 202. Here, each training data set 202 includes a plurality of respective training data samples 204 that each include audio 210 spoken in the respective native language and a corresponding transcription 220 of the audio in a respective native script representing the respective native language.

For each respective training data sample 204 of each training data set in the respective native language, the method 600 includes, at operation 604, transliterating the corresponding transcription 220 in the respective native script into corresponding transliterated text 221. Here the transliterated text 221 represents the respective native language of the corresponding audio 210 in a target script. Thereafter, for each respective training data sample 204 of each training data set 202 in the respective native language, the method 600 includes, at operation 606, associating the corresponding transliterated text 221 in the target script with the corresponding audio 210 in the respective native language to generate a respective normalized training data sample 240. Here, the respective normalized training data sample 240 includes the audio 210 spoken in the respective native language and the corresponding transliterated text 221 in the target script.

At operation 608, the method 600 includes training, using the normalized training data samples generated from each respective training data sample of each training data set and without providing any language information, the multilingual E2E ASR model