340 not shared with any of the other mask-conformer blocks 300 in the stack of mask-conformer blocks 300.

Referring now specifically to FIG. 3A, during the first pass 301, each mask-conformer block 300 includes the stack of self-attention layers 310, the convolution layer 330, and the mask-predict decoder 340. That is, during the first pass 301 the cross-attention layer 320 (FIG. 3B) is disabled. The stack of self-attention layers 310 is configured to receive, as input, the first-pass audio encodings 222 generated by an immediately preceding mask-conformer block 300 in the stack of mask-conformer blocks 300. Since there is no mask-conformer block 300 that precedes an initial mask-conformer block 300 in the stack of mask-conformer blocks 300, the initial mask-conformer block 300 is configured to receive and process the convolution output 212 generated by the one or more convolution subsampling layers 210 (FIG. 2) in lieu of the first pass audio encodings 222. The stack of self-attention layers 310 is configured to process the first-pass audio encodings 222 generated by the immediately preceding mask-conformer block 300 to generate a corresponding self-attention output 312. For example, a second mask-conformer block 300 receives the first pass-audio encodings 222 generated by a first mask-conformer block 300, a third mask-conformer block 300 receives the first-pass audio encodings 222 generated by the second mask-conformer block 300, and so on. The stack of self-attention layers 310 may include conformer or transformer layers. Thereafter, the mask-conformer block 300 uses a first concatenation operator 303 to generate a first concatenation 314 of the self-attention output 312 and the first-pass audio encodings 222.

The convolution layer 330 is configured to receive, as input, the first concatenation 314 and generate, as output, corresponding first-pass audio encodings 222 based on the first concatenation 314. As such, the corresponding first-pass audio encodings 222 are sent to the speech recognition decoder 230 and an immediately subsequent mask-conformer block 300 in the stack of mask-conformer blocks 300 (i.e., forward propagation). Thus, because the speech recognition decoder 230 is connected to an output of each mask-conformer block 300, the speech recognition decoder 230 generates the first-pass transcription 120a (FIG. 2) based on the first-pass audio encodings 222 generated by each mask-conformer block 300 and back-propagates the first-pass transcription 120a to the immediately subsequent mask-conformer block 300. For instance, the first mask-conformer block 300 generates corresponding first-pass audio encodings 222 which the speech recognition decoder 230 decodes to produce a corresponding first-pass transcription 120a that is back-propagated to the second mask-conformer block 300 (e.g., immediately subsequent mask-conformer block 300 with respect to the first mask-conformer block 300). Moreover, the mask-conformer block 300 uses a second concatenation operator 304 to generate a second concatenation 332 of the first-pass audio encodings 222 and the first concatenation 314.

The mask-predict decoder 340 is configured to generate a first-pass masked output sequence 344 by scoring each speech recognition token 122 of the sequence of speech recognition tokens 122 from the first-pass transcription 120a generated by the immediately preceding mask-conformer block 300. In particular, for each respective speech recognition token 122, the mask-predict decoder 340 determines a corresponding confidence value 342 representing a likelihood that the respective speech recognition token 122 from the first-pass transcription 120a is correct. Thereafter, the mask-predict decoder 340 determines whether the corresponding confidence value 342 satisfies a confidence values threshold. Based on determining that the corresponding confidence value 342 of a respective speech recognition token 122 fails to satisfy the confidence value threshold, the mask-predict decoder 340 replaces the respective speech recognition token 122 with a masked token. The masked token may represent a null value or blank value. On the other hand, based determining that the corresponding confidence value 342 of a respective speech recognition token 122 satisfies the confidence value threshold, the mask-predict decoder 340 outputs the respective speech recognition token 122 (e.g., the mask-predict decoder 340 determines to not replace the respective speech recognition token 122). The first-pass masked output sequence 344 is output to the stack of mask-conformer blocks 300 for processing during the second pass 302 (FIG. 3B). Accordingly, the first-pass masked output sequence 344 includes a partially masked output sequence. In some implementations, the mask-predict decoder 340 does not replace any of the speech recognition tokens 122 with the masked token. Instead, in these implementations, the mask-predict decoder 340 generates the first-pass masked output sequence 344 that includes the sequence of speech recognition tokens 122 and the corresponding confidence values 342.

Referring now specifically to FIG. 3B, during the second pass 302, each mask-conformer block 300 includes the stack of self-attention layers 310, the cross-attention layer 320, the convolution layer 330, and the mask-predict decoder 340. Notably, the cross-attention layer 320 is enabled during the second pass 302 in contrast to the first pass 301 (FIG. 3A). The stack of self-attention layers 310 is configured to receive, as input, the second-pass audio encodings 224 generated by an immediately preceding mask-conformer block 300 in the stack of mask-conformer blocks 300. Since there is no mask-conformer block 300 that precedes an initial mask-conformer block 300 in the stack of mask-conformer blocks 300, the initial mask-conformer block 300 is configured to receive and process the convolution output 212 generated by the one or more convolution subsampling layers 210 (FIG. 2) in lieu of the second-pass audio encodings 224. The stack of self-attention layers 310 is configured to process the second-pass audio encodings 224 generated by the immediately preceding mask-conformer block 300 to generate a corresponding self-attention output 312. Thereafter, the mask-conformer block 300 uses a third concatenation operator 306 to generate a third concatenation 316 of the self-attention output 312 and the second-pass audio encodings 224.

The cross-attention layer 320 is configured to receive, as input, the third concatenation 316 and the first-pass masked output sequence 344 and generate, as output, a corresponding cross-attention output 322. In some examples, the cross-attention layer 320 receives the first-pass masked output sequence 344 from an immediately preceding mask-conformer block 300. For instance, the second mask-conformer block 300 receives the first-pass masked output sequence 344 generated by the first mask-conformer block 300. In other examples, the cross-attention layer 320 receives the first-pass masked output sequence 344 from the same mask-conformer block 300. For instance, during the second pass 302, the second mask-conformer block 300 receives the first-pass masked output sequence 344 generated by the second mask-conformer block 300 during the first pass 301 (FIG. 3A). In some implementations, the cross-attention layer 320 receives a second-pass masked output sequence 346 from an immediately preceding mask-conformer block 300 in addition to, or in lieu of, the first-pass masked output sequence 344 (not shown). During the second pass 302, the cross-attention layer 320 is conditioned on the first-pass masked output sequence 344 whereby the conditioned cross-attention layer 320 generates the cross-attention output 322 based on the third concatenation 316. Moreover, the mask-conformer block 300 uses a fourth concatenation operator 308 to generate a fourth concatenation 324 of the cross-attention output 322 and the second-pass audio encodings 224.

The convolution layer 330 is configured to receive, as input, the fourth concatenation 324 and generate, as output, corresponding second-pass audio encodings 222 based on the fourth concatenation 324. As such, the corresponding second-pass audio encoding 224 is sent to the speech recognition decoder 230 and an immediately subsequent mask-conformer block 300 in the stack of mask-conformer blocks 300. Thus, because the output of each mask-conformer block 300 is connected the speech recognition decoder 230, the speech recognition decoder 230 generates the second-pass transcription 120b (FIG. 2) based on the second-pass audio encodings 224 generated by each mask-conformer block 300 and back-propagates the second-pass transcription 120b to the immediately subsequent mask-conformer block 300. For instance, the first mask-conformer block 300 generates a corresponding second-pass audio encoding 224 which the speech recognition decoder 230 decodes to produce a corresponding second-pass transcription 120b that is back-propagated to the second mask-conformer block 300 (e.g., immediately subsequent mask-conformer block 300 with respect to the first mask-conformer block 300). Moreover, the mask-conformer block 300 uses a fifth concatenation operator 309 to generate a fifth concatenation 334 of the second-pass audio encodings 224 and the fourth concatenation 324.

The mask-predict decoder 340 is configured to generate a second-pass masked output sequence 346 by scoring each speech recognition token 122 of the sequence of speech recognition tokens 122 from the second-pass transcription 120b generated by the immediately preceding mask-conformer block 300. In particular, for each respective speech recognition token 122, the mask-predict decoder 340 determines a corresponding confidence value 342 representing a likelihood that the respective speech recognition token 122 from the second-pass transcription 120b is correct. Thereafter, the mask-predict decoder 340 determines whether the corresponding confidence value 342 satisfies a confidence values threshold. Based on determining that the corresponding confidence value 342 of a respective speech recognition token 122 fails to satisfy the confidence value threshold, the mask-predict decoder 340 replaces the respective speech recognition token 122 with a masked token. The masked token may represent a null value or blank value. On the other hand, based determining that the corresponding confidence value 342 of a respective speech recognition token 122 satisfies the confidence value threshold, the mask-predict decoder 340 outputs the respective speech recognition token 122 (e.g., the mask-predict decoder 340 determines to not replace the respective speech recognition token 122). The second-pass masked output sequence 346 may be output to the next mask-conformer block 300 in the stack of mask-conformer blocks 300. Accordingly, the second-pass masked output sequence 346 includes a partially masked output sequence.

In some implementations, the mask-predict decoder 340 does not replace any of the speech recognition tokens 122 with the masked token. Instead, in these implementations, the mask-predict decoder 340 generates the second-pass masked output sequence 346 that includes the sequence of speech recognition tokens 122 and the corresponding confidence values 342.

In some implementations, the ASR model 200 is trained on a training dataset. The training dataset includes one or more training utterances each including a sequence of training speech recognition tokens. In some examples, a training process randomly replaces a first predetermined number of training speech recognition tokens with the masked token where the masked positions are sampled from a Bernoulli distribution. Moreover, the training process may replace a second predetermined number of training speech recognition tokens with random training speech recognition tokens. Here, the first predetermined number of training speech recognition tokens replaced with the masked token do not overlap with the second predetermined number of training speech recognition tokens replaced with a random token. By replacing the second predetermined number of training speech recognition tokens with a random training speech recognition token, the training process simulates the possibility of unmasked incorrect tokens during the two-pass speech recognition process. That is, in some scenarios, the mask-predict decoder 340 fails to replace a respective one of the speech recognition tokens 122 that is incorrect with the masked token. As such, the second-pass transcription 120b may correct incorrect speech recognition tokens 122 that were not masked in addition to speech recognition tokens 122 that were masked during the first pass 301.

FIG. 4 illustrates an example two-pass decoding process 400 of the ASR model 200. The ASR model 200 receives, as input, an input sequence (O=o_tε [Image Omitted] ^F|t=1,T′) and generates, as output, a transcription 120 (W=w_nεV|n=1, . . . , N). Here, T′ represents a length of the input sequence (e.g., sequence of acoustic frames 110), N represents a length of the transcription 120, o_trepresents an F-dimensional acoustic frame at frame t, w_nis an output token (e.g., speech recognition token 122) at position n, and V represents a vocabulary. The process 400 performs operations 402-406 during the first-pass 301 (FIG. 3A) of the two-pass decoding process 400. At operation 402, the process 400 uses the stack of mask-conformer blocks 300 to generate a first-pass audio encoding (H_cfm) 222 based on the sequence of acoustic frames 110. At operation 404, the process 400 uses the decoder 230 to decode the first-pass audio encoding 222 to generate a corresponding first-pass transcription (Ŵ) 120a according to a beam size K. At operation 406, the process 400 uses the mask-predict decoder 340 to determine a confidence value 342 for each respective speech recognition token (ŵ_n) 122 from the first-pass transcription 120a. Moreover, the mask-predict decoder 340 determines whether each confidence value 342 satisfies a confidence value threshold (P_thres). When the confidence value 342 fails to satisfy the confidence value threshold, the mask-predict decoder 340 replaces the corresponding speech recognition token 122 with a masked token. Otherwise, the mask-predict decoder 340 determines to not replace the speech recognition token 122. As a result, the mask-predict decoder 340 outputs the first-pass masked output sequence 344. The process 400 performs operations 408 and 410 during the second-pass 302 (FIG. 3B). In particular, at operation 408, the process 400 includes generating second-pass audio encodings (H_cfm-ca) 224 based on the sequence of acoustic frames 110 and the first-pass masked output sequence 344 generated during the first-pass 301. Finally, at operation 410, the process 400 includes using the decoder 230 to decode the second-pass audio encodings 224 to generate a corresponding second-pass transcription 120b.

For example, the ASR model 200 may receive a respective sequence of acoustic frames 110 for an utterance 106 of “it is cold today” and generate a first-pass transcription 120a of “it his cold today” from a corresponding first-pass audio encoding 222. Notably, the first-pass transcription 120a includes the incorrect term of “his” instead of “is.” Thereafter, the mask-predict decoder 340 may determine a confidence value 342 for each respective speech recognition token 122 from the first-pass transcription 120a. In this example, the mask-predict decoder 340 determines that the confidence value 342 for the speech recognition token 122 of “his” fails to satisfy the confidence value threshold and replace the speech recognition token 122 of “his” with a masked token (e.g., [MASK]). As a result, the mask-predict decoder 340 generates the first-pass masked output sequence 344 of “it [MASK] cold today”. During the second-pass 302, the stack of mask-conformer blocks 300 is conditioned on the first-pass masked output sequence 344 and a corresponding second-pass audio encoding which is decoded by the speech recognition decoder 230 to generate the second-pass transcription 120b of “it is cold today.”

FIG. 5 is a flowchart of an exemplary arrangement of operations for a computer-implemented method 500 of performing speech recognition using an acoustic encoder that has a stack of mask-conformer blocks. The method may execute on data processing hardware 610 (FIG. 6) using instructions stored on memory hardware 620 (FIG. 6). The data processing hardware 610 and the memory hardware 620 may reside on the user device 102 and/or the remote computing device 201 of FIG. 1 each corresponding to the computing device 600 (FIG. 6).

At operation 502, the method 500 includes receiving a sequence of acoustic frames 110 characterizing an utterance 106. During a first pass 301, the method 500 performs operations 504-508. At operation 504, the method 500 includes generating, using a stack of mask-conformer blocks 300 of an acoustic encoder 220, first-pass audio encodings 222 based on the sequence of acoustic frames 110. In some examples, the stack of mask-conformer blocks 300 generates the first-pass audio encodings 222 based on a convolution output 212 in addition to, or in lieu of, the sequence of acoustic frames 110. At operation 506, the method 500 includes generating, using a speech recognition decoder 230, a first-pass transcription 120a of the utterance 106 based on the first-pass audio encodings 222. The first-pass transcription 120a includes a sequence of speech recognition tokens 122 such as words, wordpieces, graphemes, etc. At operation 508, the method 500 includes generating, using a mask-predict decoder 340 of the acoustic encoder 220, a first-pass masked output sequence 344 by masking one or more of the speech recognition tokens 122 from the sequence of speech recognition tokens 122 with a masked token.

The method 500 performs operations 510 and 512 during a second pass 302. At operation 510, the method 500 includes generating, using the stack of mask-conformer blocks 300 of the acoustic encoder 220, second-pass audio encodings 224 by performing cross-attention on the sequence of acoustic frames 110 and the first-pass masked output sequence 344 generated during the first pass 301. At operation 512, the method 500 includes generating, using the speech recognition decoder 230, a second-pass transcription 120b of the utterance 106 based on the second-pass audio encodings 224.

FIG. 6 is a schematic view of an example computing device 600 that may be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device 600 includes a processor 610, memory 620, a storage device 630, a high-speed interface/controller 640 connecting to the memory 620 and high-speed expansion ports 650, and a low speed interface/controller 660 connecting to a low speed bus 670 and a storage device 630. Each of the components 610, 620, 630, 640, 650, and 660, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630 to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display 680 coupled to high speed interface 640. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices