10 that may be combined (e.g., averaged or otherwise accumulated) to form the respective TD reference vector 252. Here, the predetermined term spoken by each enrolled user 10 may be the predetermined keyword or another predetermined term. The TD verifier 210 stores the TD reference vector 252 in connection with the respective user profile 250 associated with the user 10 that spoke the enrollment utterance. In some examples, in addition to, or in lieu of, storing the TD reference vector 252 the TD verifier 210 stores the TD reference audio data 253 in connection with the respective user profile 250 associated with the user 10 that spoke the enrollment utterance. That is, instead of generating a reference vector 252 from the enrollment utterances, the TD verifier 210 stores the TD reference audio data 253 directly.

In some examples, after a user has performed the enrollment process, the TD verifier 210 performs speaker identification on the audio data 120 to identify the identity 205 of the particular user that spoke the utterance. The TD verifier 210 identifies the user 10 that spoke the utterance 106 by first extracting, from the first portion 121 of the audio data 120 that characterizes the predetermined keyword spoken by the user, a TD evaluation vector (e.g., TD-E) 214 representing voice characteristics of the utterance of the keyword. Here, the TD verifier 210 may execute the TD speaker verification model 212 configured to receive the first portion 121 (e.g., characterizing the portion of the utterance corresponding to the keyword) of the audio data as input and generate, as output, the TD evaluation vector 214. The TD speaker verification model 212 may be a neural network model trained using machine or human supervision to output the TD evaluation vector 214.

Once the TD evaluation vector 214 is output from the TD speaker verification model 212, the TD verifier 210 determines whether the TD evaluation vector 214 matches any of the stored user profiles 250 (e.g., stored at the memory hardware 105 and/or the memory hardware 115) in connection with identities 205 of the enrolled users 10. In particular, the TD verifier 210 may compare the TD evaluation vector 214 to the TD reference vector 252 or the TD reference audio data 253. Here, each TD reference vector 252 may be used as a reference vector corresponding to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user 10 speaking the predetermined keyword.

In some implementations, the TD verifier 210 uses a TD scorer 216 that compares the TD evaluation vector 214 to the respective TD reference vector 252 associated with each enrolled user 10 of the user device 102. Here, the TD scorer 216 may generate a score for each comparison indicating a likelihood that the utterance 106 corresponds to an identity 205 of the respective enrolled user 10. Specifically, the TD scorer 216 generates a TD confidence score 217 for each enrolled user 10 of the user device 102. In some implementations, the TD scorer 216 determines the TD confidence score by determining a respective cosine distance between the TD evaluation vector 214 and each TD reference vector 252 to generate the TD confidence score 217 for each respective enrolled user 10.

Thereafter, the TD scorer 216 determines whether any of the TD confidence scores 217 satisfy a confidence threshold. When the TD confidence score 217 satisfies the confidence threshold, the TD scorer 216 outputs the identity 205 of the particular user that spoke the utterance and the associated user profile 250 to the keyword detector 400. On the other hand, when the TD confidence score fails to satisfy the confidence threshold, the TD scorer 216 does not output any identity or user profile 250 to the keyword detector 400.

Referring now to FIG. 2B, in some examples, a second example speech verification system 200, 200b includes a text-independent (TI) verifier 220 that has a TI speaker verification model 222 and a TI scorer 226. Moreover, the TI verifier 220 may store the user profiles 250, 250a-n in connection with the identities 205 of enrolled users. The TI speaker verification model 222 may generate one or more TI reference vectors (e.g., TI-RV) 254 from audio samples of enrollment phrases spoken by each enrolled user that may be combined (e.g., averaged or otherwise accumulated) to form the respective TI reference vector 254. Here, the enrollment phrases spoken may be free-form users including any speech the user wishes to speak. Thus, the enrollment phrases may be different than the keyword or any phrase the user wishes to speak. The TI verifier 220 stores the TI reference vector 254 in connection with the respective user profile 250 associated with the user 10 that spoke the enrollment utterance. In some examples, in addition to, or in lieu of, storing the TI reference vector 254 the TI verifier 220 stores the TI reference audio data 255 in connection with the respective user profile 250 associated with the user 10 that spoke the enrollment utterance. That is, instead of generating a TI reference vector 254 from the enrollment utterances, the TI verifier 220 stores the TI reference audio data 255 directly. Moreover, the TI verifier 220 may store the personalized keyword detection model 420 in connection with the respective user profile 250 associated with the user 10 that spoke the enrollment utterance.

In some examples, after a user has performed the enrollment process, the TI verifier 220 performs speaker identification on the audio data 120 to identify the identity 205 of the particular user that spoke the utterance. The TI verifier 220 identifies the user 10 that spoke the utterance 106 by first extracting, from the second portion 122 of the audio data 120 that characterizes the query including free-form speech or the query following the predetermined keyword spoken by the user, a TD evaluation vector (e.g., TD-E) 214 representing voice characteristics of the utterance. Here, the TI verifier 220 may execute the TD speaker verification model 212 configured to receive the first portion 121 of the audio data as input and generate, as output, the TD evaluation vector 214. The TI speaker verification model 222 may be a neural network model trained using machine or human supervision to output the TI evaluation vector 224.

Once the TI evaluation vector 224 is output from the TI speaker verification model 222, the TI verifier 220 determines whether the TI evaluation vector 224 matches any of the stored user profiles 250 (e.g., stored at the memory hardware 105 and/or the memory hardware 115) in connection with identities 205 of the enrolled users 10. In particular, the TI verifier 220 may compare the TI evaluation vector 224 to the TI reference vector 254 or the TI reference audio data 255. Here, each TI reference vector 254 may be used as a reference vector corresponding to a voiceprint or unique identifier representing characteristics of the voice of the respective enrolled user 10.

In some implementations, the TI verifier 220 uses a TI scorer 226 that compares the TI evaluation vector 224 to the respective TI reference vector 254 associated with each enrolled user 10 of the user device 102. Here, the TI scorer 226 may generate a score for each comparison indicating a likelihood that the utterance 106 corresponds to the identity 205 of the respective enrolled user 10. Specifically, the TI scorer 226 generates a TI confidence score 227 for each enrolled user 10 of the user device 102. In some implementations, the TI scorer 226 determines the TI confidence score 227 by determining a respective cosine distance between the TI evaluation vector 224 and each TI reference vector 254 to generate the TI confidence score 227 for each respective enrolled user 10.

Thereafter, the TI scorer 226 determines whether any of the TI confidence scores 227 satisfy a confidence threshold. When the TI confidence score 227 satisfies the confidence threshold, the TI scorer 226 outputs the identity 205 of the particular user that spoke the utterance and the associated user profile 250 to the keyword detector 400. On the other hand, when the TI confidence score 227 fails to satisfy the confidence threshold, the TI scorer 226 does not output any identity or user profile 250 to the keyword detector 400.

FIG. 3 shows an example speaker verification training process 300 for training the speaker verification system 200. The example speaker verification training process 300 (also referred to as simply “training process 300”) obtains a plurality of training datasets 310, 310A-N stored in data storage 301 and trains each of the TD speaker verification model 212 and the TI speaker verification model 222 on the training datasets 310. Each training dataset 310 may be associated with a different respective language or dialect and includes corresponding training utterances 320, 320Aa-Nn spoken in the respective language or dialect by different speakers. For instance, a first training dataset 310A may be associated with American English speakers that include corresponding training utterances 320Aa-An each spoken in English by speakers from the United States of America. That is, the training utterances 320Aa-An in the first training dataset 310A are all spoken in English with an American accent. On the other hand, a second training dataset 310B may be associated with British English speakers that includes corresponding training utterances 320Ba-Bn also spoken in English, but by speakers from Great Britain. Accordingly, the training utterances 320Ba-Bn in the second training data set 310B are spoken in English with a British accent, and are therefore associated with a different dialect (i.e., British Accent) than the training utterances 320Aa-An associated with the American accent dialect. Notably, an English speaker with a British accent may pronounce some words differently than another English speaker with an American accent. FIG. 3 also shows another training data set 310N associated with Korean that includes corresponding training utterances 320Na-Nn spoken by Korean speakers.

Each corresponding training utterance includes a text-dependent (TD) portion 321 and a text-independent (TI) portion 322. The TD portion 321 includes an audio segment characterizing a predetermined keyword (e.g., “Hey Google”) or a variant of the predetermined keyword (e.g., “Ok Google”) spoken in the training utterance 320. Here, the predetermined keyword and variant thereof may each be detectable by the keyword detector 400 when spoken in streaming audio 118 to trigger the user device to wake-up and initiate speech recognition on one or more terms following the predetermined hotword or variant thereof. In some examples, the fixed-length audio segment associated with the TD portion 321 of the corresponding training utterance 320 that characterizes the predetermined keyword is extracted by the keyword detector 400.

The TI portion 322 in each training utterance 320 includes an audio segment that characterizes a query statement spoken in the training utterance 320 following the predetermined hotword characterized by the TD portion 321. For instance, the corresponding training utterance 320 may include “Ok Google, What is the weather outside?” whereby the TD portion 321 characterizes the hotword “Ok Google” and the TI portion 322 characterizes the query statement “What is the weather outside?” While the TD portion 321 in each training utterance 320 is phonetically constrained by the same predetermined keyword or variation thereof, the lexicon of the query statement characterized by each TI portion 322 is not constrained such that the duration and phonemes associated with each query statement is variable. Notably, the language of the spoken query statement characterized by the TD portion 321 includes the respective language associated with the training dataset 310. For instance, the query statement “What is the weather outside” spoken in English translates to “Cual es el clima afuera” when spoken in Spanish. In some examples, the audio segment characterizing the query statement of each training utterance 320 includes a variable duration ranging from 0.24 seconds to 1.60 seconds.

With continued reference to FIG. 3, the training process 300 trains a first neural network 330 on the TD portions 321 of the training utterances 320, 320Aa-Nn spoken in the respective language or dialect associated with each training dataset 310, 310A-N. During training, additional information about the TD portions 321 may be provided as input to the first neural network 330. For instance, text-dependent (TD) targets 323 corresponding to ground-truth output labels for training the TD speaker verification model 212 to learn how to predict may be provided as input to the first neural network 330 during training with the TD portions 321. The TD targets 323 may be ground-truth labels for TD evaluation vectors 214 (e.g., when training on TD reference vectors 252) or ground-truth labels for TD audio (e.g., when training on TD reference audio data 253). Thus, one or more utterances of the predetermined keyword from each particular speaker may be paired with a particular TD target 323.

The first neural network 330 may include a deep neural network formed from multiple long short-term memory (LSTM) layers with a projection layer after each LSTM layer. In some examples, the first neural network uses 128 memory cells and the projection size is equal to 64. The TD speaker verification model 212 includes a trained version of the first neural network 330. The TD evaluation and reference vectors 214, 252 generated by the TD speaker verification model 212 may include d-vectors or i-vectors with an embedding size equal to the projection size of the last projection layer. The training process may use generalized end-to-end contrast loss for training the first neural network 330.

After training, the first neural network 330 generates the TD speaker verification model 212. The TD speaker verification model 212 may be pushed to a plurality of user device 102 distributed across multiple geographical regions and associated with users that speak different languages, dialects, or both. The user devices 102 may store and execute the TD speaker verification model 212 to perform text-dependent speaker verification on audio segments characterizing the predetermined keyword spoken by any of the enrolled users of the user device 102.

The training process 300 also trains a second neural network 340 on the TI portions 322 of the training utterances 320, 320Aa-Nn spoken in the respective language or dialect associated with each training dataset 310, 310A-N. Here, for the training utterance 320Aa, the training process 300 trains the second neural network 340 on the TI portion 322 characterizing the query statement “what is the weather outside” spoken in American English. Optionally, the training process 300 may also trains the second neural network 340 on the TD portion 321 (not shown) of at least one corresponding training utterance 320 in one or more of the training datasets 310 in addition to the TI portion 322 of the corresponding training utterance 320. For instance, using the training utterance 320Aa above, the training process 300 may train the second neural network 340 on the entire utterance “Ok Google, what is the weather outside” During training, additional information about the TI portions 322 may be provided as input to the second neural network 340. For instance, TI targets 324 corresponding to ground-truth output labels for training the TI speaker verification model 222 to learn how to predict may be provided as input to the second neural network 340 during training with the TI portions 322. The TI targets 324 may be ground-truth labels for TI evaluation vectors 224 (e.g., when training on TI reference vectors 254) or ground-truth labels for TI audio (e.g., when training on TI reference audio data 255). Thus, one or more utterances of query statements from each particular speaker may be paired with a particular TI target 324.

The second neural network 340 may include a deep neural network formed from LSTM layers with a projection layer after each LSTM layer. In some examples, the second neural network uses 384 memory cells and the projection size is equal to 128. The TI speaker verification model 222 includes a trained version of the second neural network 340. The TI evaluation and reference vectors 252, 254 generated by the TI speaker verification model 222 may include d-vectors or i-vectors with an embedding size equal to the projection size of the last projection layer. The training process 300 may use generalized end-to-end contrastive losses for training the first and second neural networks 330, 340.

FIG. 5 shows an example training process 500 for training the keyword detector 400. The training process 500 receives a plurality of sets of utterances 510 from data storage. Each respective set of utterances 510 includes audio data samples 520 of a corresponding utterance that is different than the corresponding utterance of each other set of utterances 510 of the plurality of sets of utterances. As such, the audio data samples 520 of each set of utterance 510 may characterize a particular keyword different than the particular keyword of each other set of utterances 510. In the example shown, the training process 500 receives a first set of utterances 510, 510A that includes four audio data samples 520Aa-Ad for the keyword “up,” a second set of utterances 510, 510B that includes four audio data samples 520Ba-Bd for the keyword “down,” and a third set of utterances 510, 510C that includes four audio data samples 520Ca-Cd for the keyword “over.” However, it is understood that the plurality of sets of utterances 510 may include any number of sets of utterances 510 and each set of utterances 510 may include any number of audio data samples 520 irrespective of the number of audio data samples 520 of other sets of utterances 510.

The corresponding utterance may include a user-defined custom keyword. For instance, the user 10 may provide the user-defined custom keyword during the enrollment process (FIGS. 2A and 2B) by speaking the custom keyword one or more times or providing the custom keyword via textual input. Thus, the audio data samples 520 may include at least one of non-synthetic audio data samples (e.g., spoken by a user) or synthetic audio data samples (e.g., generated by a text-to-speech model using a textual input). Moreover, each audio data samples 520 of a respective set of utterances 510 includes speech characteristics (e.g., pitch, prosody, accent, style, etc.) speaking the corresponding utterance different than at least one other audio data sample 520 of the respective set of utterances 510. For example, the first set of utterances 510 may include four audio data samples 520Aa-Ad of the term “up” each spoken by a different speaker with different speaker characteristics.

During each of a plurality of training iterations, the training process 500 may select one of the plurality of sets of utterances 510 to represent the user-defined keyword. For each iteration, the training process 500 trains the keyword detector 400 using the selected one of the plurality of sets of utterances 510 to represent the user-defined keyword. After each iteration, the training process 500 selects another one of the plurality of sets of utterances 510 to represent the user-defined keyword and trains the keyword detector 400 using the selected other one of the plurality of sets of utterances 510 to represent the user-defined keyword. In the example shown, the training process 500 selects the first set of utterances 510A to represent the user-defined keyword by way of example only.

For the selected one of the plurality of utterances (e.g., a respective one of the set of utterances) 510, the training process assigns one or more audio data samples 520 from the selected one of the plurality of utterances 510 to an enrollment subset and assigns each other audio data sample 520 from the selected one of the plurality of utterances 510 not assigned to the enrollment subset to a test subset. Here, the enrollment subset of audio data samples 520 represent audio data samples 520 spoken by the user 10 during the enrollment process to provide the user-defined keyword. On the other hand, the test subset of audio data samples 520 represent audio data samples 520 spoken by the user 10 during inference after the user 10 has completed the enrollment process to provide the user-defined keyword. Thus, by creating the enrollment subset and the test subset the training process 500 emulates the two-stage nature of the enrollment process of the user-defined keyword and subsequently receiving the user-defined keyword during training.

In the example shown, the training process 500 assigns a first and second audio data sample 520Aa, 520Ab from the first set of utterances 510 to the enrollment subset and a third and fourth audio data sample 520Ac, 520Ad to the test subset. Assigning the audio data samples 520 to the enrollment subset and the test subset may include randomly sampling the audio data samples. In some implementations, the training process 500 assigns the same number of audio data samples 520 to the enrollment subset and the test subset. In other implementations, the training process 500 assigns a different number of audio data samples 520 to the enrollment subset of the test subset.

For the selected one of the plurality of utterances (e.g., a respective one of the set of utterances) 510, the training process 500 determines, using the encoder 410, a keyword enrollment embedding 412 for the enrollment subset of the audio samples 520 of the selected one of the plurality of utterances 510 and determines, using the encoder 410, a corresponding matching keyword test embedding 414 for each respective audio data sample 520 of the test subset of the audio data samples 520. That is, the encoder 410 may determine a corresponding keyword enrollment embedding 412 for each respective audio data sample 520 of the enrollment subset and determine a centroid keyword enrollment embedding based on the corresponding keyword enrollment embedding 412 determined for each respective audio data sample 520 of the enrollment subset. Here, the centroid keyword enrollment embedding may serve as the keyword enrollment embedding 412 for the enrollment subset. The encoder 410 may determine the centroid keyword enrollment embedding according to:

$\begin{matrix} c_{i} = \frac{1}{Y / 2} \sum^{Y} \underset{j (\mod 2) \neq 0}{j = 1} e_{i j} & (1) \end{matrix}$

In Equation 1, c_irepresents the centroid keyword enrollment embedding and Y represents the number of phrases in the selected on of the sets of utterances 510.

In the example shown, the encoder 410 determines a corresponding keyword enrollment embedding 412 for the first audio data sample 520Aa and the second audio data sample 520Ab and determines the centroid keyword enrollment embedding based on the corresponding keyword enrollment embeddings determined for the first audio data sample 520Aa and the second audio data sample 520Ab. Thus, the centroid keyword enrollment embedding 412 serves as a single embedding that represents all the audio data samples 520 from the enrollment subset. Continuing with the example shown, the encoder 410 determines a corresponding matching keyword test embedding 414 based on the third audio data sample 520Ac and determines a corresponding matching keyword test embedding 414 based on the fourth audio data sample 520Ad. As such, the encoder 410 determines a corresponding matching keyword test embedding 414 for each audio data sample 520 in the test subset which may be in contrast to determining the single keyword enrollment embedding 412 for all the audio data samples 520 in the enrollment subset. As will become apparent, the matching keyword test embeddings 414 represent embeddings determined by the encoder 410 for speech that includes the user-defined keyword. Put another way, the encoder 410 determines the keyword enrollment embedding 412 and the matching keyword test embedding 414 based on audio data samples 520 that include the user-defined keyword.

For each respective audio data sample 520 of each other set of utterances 510 (e.g., the set of utterances 510 other than the respective one of the set of utterances 510), the training process 500 determines, using the encoder 410, a corresponding nonmatching keyword test embedding 416. In the example shown, the other set of utterances 510 include the second set of utterances 510B and the third set of utterances 510C. Thus, the encoder 410 determines a first corresponding nonmatching keyword test embedding 416, 416a for each respective audio data sample 520Ba-Bd of the second set of utterances 510B (e.g., four total first corresponding nonmatching keyword test embeddings 416a) and determines a second corresponding nonmatching keyword test embedding 416, 416b for each respective audio data sample 520Ca-Cd of the third set of utterances 510C (e.g., four total second corresponding nonmatching keyword test embeddings 416b).

The loss module 550 receives the keyword enrollment embedding 412, the matching keyword test embeddings 414, and the nonmatching keyword test embeddings 416 and determines an overall loss 555. The overall loss 555 may include a first loss 552 and a second loss 554. As such, the training process 500 may train the keyword detector 400 based on the overall loss 555 or specifically on the first loss 552 or the second loss 554. In some examples, training the keyword detector 400 includes updating parameters of the keyword detector 400 based on the loss. For instance, the training process 500 may update parameters of the encoder 410 of the keyword detector 400 based on the loss.

In some examples, the loss module 550 determines the first loss 552 based on the keyword enrollment embedding 412 and the matching keyword test embeddings 414. In particular, the loss module 550 may compare each matching keyword test embedding 414 to the keyword enrollment embedding 412 to determine the first loss 552. For instance, the loss module 550 may determine a cosine similarity between the keyword enrollment embedding 412 and each matching keyword test embedding 414. Thereafter, the loss module 550 determines the first loss 552 based on each cosine similarity determined between the keyword enrollment embedding 412 and the matching keyword test embeddings 414. The loss module 550 may determine the first loss 552 according to:

$\begin{matrix} p_{i} = {e_{i j} | j (\mod 2) \neq 1} & (2) \end{matrix}$

Since the encoder 410 determined the keyword enrollment embedding 412 and the matching keyword test embeddings 414 based on audio data samples 520 which correspond to the same utterance (e.g., user-defined keyword), the keyword enrollment embedding 412 and the matching keyword test embeddings 414 should be similar to one another. Thus, the training process 500 may aim to minimize the first loss 552 to teach the encoder 410 to determine similar embeddings for audio corresponding to the user-defined keyword regardless of whether the audio was spoken during the enrollment process or during inference. For example, the encoder 410 determined the keyword enrollment embedding 412 based on the audio data samples 520Aa, 520Ab each corresponding to the utterance “up” and determined the matching keyword test embedding 414 based on the audio data samples 520Ac, 520Ad each corresponding to the utterance “up.” As such, in this example, the training process 500 aims to minimize the first loss 552 between these embeddings 412, 414 each corresponding to the utterance up.

In some implementations, the loss module 550 determines the second loss 554 based on the keyword enrollment embedding 412 and the nonmatching keyword test embeddings 416. In particular, the loss module 550 may compare each nonmatching keyword test embedding 416 to the keyword enrollment embedding 412 to determine the second loss 554. For instance, the loss module 550 may determine a cosine similarity between the keyword enrollment embedding 412 and each nonmatching keyword test embedding 416. Thereafter, the loss module 550 determines the second loss 554 based on each cosine similarity determined between the keyword enrollment embedding 412 and the nonmatching keyword test embeddings 416. The loss module 550 may determine the second loss 554 according to:

$\begin{matrix} n_{i} = {e_{k j} | j (\mod 2) \neq 1, k = 1, 2, \dots, X and k \neq i} & (3) \end{matrix}$

Since the encoder 410 determined the keyword enrollment embedding 412 and the nonmatching keyword test embeddings 416 based on audio data samples 520 which correspond to different utterances, the keyword enrollment embedding 412 and the nonmatching keyword test embeddings 416 should not be similar to one another. Thus, the training process 500 may aim to maximize the second loss 554 to teach the encoder 410 to determine different embeddings for audio corresponding to the user-defined keyword and any other utterance regardless of whether the audio was spoken during the enrollment process or during inference. For example, the encoder 410 determined the keyword enrollment embedding 412 based on the audio data samples 520Aa, 520Ab each corresponding to the utterance “up,” determined the first nonmatching keyword test embeddings 416a based on the audio data samples 520Ba-Bd each corresponding to the utterance “down,” and determined the second nonmatching keyword test embeddings 416b based on the audio data samples 520Ca-Cd each corresponding to the utterance “over.” As such, in this example, the training process 500 aims to maximize the second loss 554 between the keyword enrollment embeddings 412 and the nonmatching keyword test embeddings 416. The loss module 350 may determine the overall loss 355 based on the first loss 352 and the second loss 354 according to:

$\begin{matrix} L (c_{i}) = \log \sum_{n \in n_{i}} \exp \cos (c_{i}, n) - \log \sum_{p \in p_{i}} \exp \cos (c_{i}, p) & (4) \end{matrix}$

Accordingly, FIG. 5 shows an example iteration of the training process 500 whereby the first set of utterances 510A are selected to represent the enrollment utterances for the user-defined keyword. Thereafter, in a subsequent iteration, the training process 500 may select another set of utterances 510 to represent the enrollment utterances for the user-defined keyword. For example, in the subsequent iteration, the training process 500 may select the second set of utterances 510B and assign the second set of utterances to the enrollment subset and the test subset such that the first and third set of utterances 510A, 510C are now the other set of utterances 510.

Referring now specifically to FIG. 1A, in some examples, for a first example system 100, 100a the first user 10a (e.g., John) speaks the utterance 106 of “Up, Play my music playlist.” Notably, the first user 10a is an enrolled user that the speaker verification system 200 generated first speaker characteristics information (e.g., user profile) 250. Thus, the speaker verification system 200 identifies a first identity 205a and a first user profile 250a associated with the first user 10a by processing the utterance 106. The keyword detector 400 receives the first identity 205a and the first user profile 150a associated with the first user 10a. The user profile 250 may indicate to the keyword detector 400 one or more user-defined custom keywords provided by the user 10 (e.g., via textual input or speech input) during the enrollment process. The one or more user-defined custom keywords may be used by the keyword detector 400 to generate the keyword indication 405 in addition to, or in lieu of, any generic keywords.

Described in greater detail with reference to FIG. 4, the keyword detector 400 may be conditioned on speaker characteristic information 250 (e.g., reference vectors 252, 254, and/or reference audio data 253, 255 (FIGS. 2A and 2B)) associated with the first user 1Oa to adapt the keyword detector 400 to detect the presence of the keyword in audio for the first user 10a. In the example shown, the first user 10a provided the user-defined custom keyword of “up.” To that end, the keyword detector 400 generates the keyword indication 405 when the first user 10a speaks the keyword of “up” to indicate to the ASR system 180 to process speech that follows the keyword. Thus, in this example, based on the keyword detector 400 detecting the presence of the custom keyword from the audio data 120, the keyword detector 400 outputs the keyword indication 405 to the ASR system 180.

In response to receiving the keyword indication 405, the ASR system 180 processes the second portion 122 of the utterance 106 of “Play my playlist” spoken by the first user 10a. In particular, the ASR system 180 includes an ASR model 182 configured to perform speech recognition on the second portion 122 of the audio data 120 that characterizes the query. The ASR system 180 also includes a natural language understanding module (NLU) 184 configured to perform query interpretation on the speech recognition result output by the ASR model 182. Generally, the NLU module 184 may perform semantic analysis on the speech recognition result to identify the action to perform that is specified by the query. In some examples, the NLU module 184 includes a large language model (LLM) capable of not only performing query interpretation on the speech recognition result output by the ASR model 182, but also performing text generation tasks based on the speech recognition result. Additionally or alternatively, the ASR model 182 may include an audio encoder and a text decoder that includes a LLM such that the LLM is capable of not only decoding audio encodings into text associated with speech recognition results, but also performing semantic analysis on the speech recognition results and/or downstream text generation tasks based on the speech recognition results. In some examples, the ASR system 180 receives the first identity 205a and the first user profile 250a associated with the first user 10a, and personalizes the speech recognition for the first user 10a. For instance, the ASR system 180 may determine the “music playlist” from the utterance 106 is referencing a music playlist associated with the first user 10a. Thereafter, the user device 102 may send the response including an audio track from John's music playlist for the user device 102 to play for audible output from a speaker.

Referring now specifically to FIG. 1B, in some examples, for a second example system 100, 100b the first user 10a (e.g., John) speaks the utterance 106 of “Down, Play my music playlist.” Notably, the first user 10a is an enrolled user that the speaker verification system 200 generated first speaker characteristics information (e.g., user profile) 250. Thus, the speaker verification system 200 identifies a first identity 205a and a first user profile 250a associated with the first user 10a by processing the utterance 106. The keyword detector 400 receives the first identity 205a and the first user profile 150a associated with the first user 10a. The user profile 250 may indicate to the keyword detector 400 one or more user-defined custom keywords provided by the user 10 (e.g., via textual input or speech input) during the enrollment process. The one or more user-defined custom keywords may be used by the keyword detector 400 to generate the keyword indication 405 in addition to, or in lieu of, any generic keywords.

Yet, in the example shown the term “down” is neither a custom keyword or a generic keyword. Thus, in this example, the keyword detector 400 does not detect the presence of the keyword and does not generate the keyword indication 405.

Consequently, the ASR system 180 does not process the second portion 122 of the audio data 120. That is, the ASR system 180 only processes the second portion 122 when the keyword indication 405 is received. Thus, the query spoken by the first user 10a is not processed by the ASR system 180.

FIG. 4 shows an example conditioning process 401 for conditioning the keyword detector 400 on speaker characteristic information 250. In some implementations, the conditioning process 401 occurs during the enrollment process described with reference to FIGS. 2A and 2B. That is, after generating the speaker characteristic information 250 for the user 10 that spoke the enrollment utterances, the conditioning process 401 may condition the keyword detector 400 such that the personal keyword detection model is pre-determined before the user 10 speaks any utterances 106 directed towards the user device 102. Advantageously, performing the conditioning process 401 in this manner limits computational resources (and therefore the observed latency) when the enrolled user 10 speaks the utterance 106 that is directed towards the user device 102