To generate the confidence score 140 for each of the one or more subsequences included in the initial output sequence 120, the scoring engine 130 generates a query embedding 131 for each subsequence, and then identifies one or more neighbor pre-generated text embeddings 137 from the databases 132, 134, 136, 138 using the query embedding 131.

Like the plurality of pre-generated embeddings stored in the databases, the query embedding 131 for each subsequence included in the initial output sequence 120 can be generated in any appropriate way. For example, the query embedding 131 can be the output hidden state of an intermediate layer in the language model neural network 110, or a combination of the output hidden states of two or more layers in the language model neural network 110, when the neural network is processing the input sequence 102 to generate the subsequence in the initial output sequence 120. As a particular example, the query embedding 131 can be the output hidden state generated by an attention block included in the language model neural network 110 of FIG. 1 when the neural network is processing the input sequence 102 to generate the subsequence in the initial output sequence 120.

Generally, identifying the one or more neighbor pre-generated text embeddings 137 can include computing, for each subsequence included in the initial output sequence 120, a respective distance between the query embedding 131 that has been generated for the subsequence and the plurality of pre-generated text embeddings included in each of the databases 132, 134, 136, 138, and then selecting one or more pre-generated text embeddings based on the respective distances, that is, selecting the one or more “neighbor pre-generated text embeddings 137.” For example, the system 100 can select, as the neighbor pre-generated text embeddings 137, one or more pre-generated text embeddings that have the smallest distances to the query embedding 131 or that satisfy a distance threshold, and the distance can be, e.g., a Euclidean distance, a Hamming distance, or other type of distance in the embedding space.

After identifying the one or more neighbor pre-generated text embeddings 137, the scoring engine 130 can compute the confidence score for each subsequence. In particular, it does this based on the index data that is stored in association with the one or more neighbor pre-generated text embeddings 137.

For each subsequence, when only one neighbor pre-generated text embedding 137 is identified, the scoring engine 130 can compute the confidence score 140 for the subsequence based on the level of confidence specified by the index that is stored in association with the neighbor pre-generated text embedding 137. For example, there can be a one-to-one mapping between different levels of confidence and different confidence scores, and the scoring engine 130 can use the confidence score mapped to the level of confidence specified by the index as the confidence score 140 for the subsequence.

In some implementations, the scoring engine 130 can compute the confidence score 140 based on the additional information specified by the index that is stored in association with the neighbor pre-generated text embedding 137. For example, the confidence score can be lower when the neighbor pre-generated text embedding is generated based on a source subsequence included in an electronic document at a first network location, while the confidence score can be higher when the neighbor pre-generated text embedding is generated based on a source subsequence included in an electronic document at a second network location.

As another example, the confidence score can be higher when the neighbor pre-generated text embedding is generated based on a source subsequence at a first location within an electronic document, while the confidence score can be lower when the neighbor pre-generated text embedding is generated based on a source subsequence at a second location within the electronic document. For example, assuming that the electronic document is a web page, then the first location can be the body content of the web page, while the second location can be the chat forum that follows the body content of the web page.

As another example, the confidence score can be lower when the neighbor pre-generated text embedding is obtained from an electronic document that is relatively recently modified, while the confidence score can be higher when the neighbor pre-generated text embedding is obtained from an electronic document that hasn't been modified for a certain amount of time, or vice versa.

Alternatively, when multiple neighbor pre-generated text embeddings 137 are identified for each subsequence, the scoring engine 130 can compute the confidence score 140 for the subsequence based on the multiple indices that are respectively stored in association with the multiple neighbor pre-generated text embeddings 137.

For example, the confidence score 140 for the subsequence can be a combination of confidence scores that can be computed individually for each neighbor pre-generated text embedding based on the level of confidence, and, in some implementations, on the additional information specified by the corresponding index that is stored in association with the neighbor pre-generated text embedding (as mentioned earlier). For example, the scoring engine 130 can combine, e.g., average or sum, the confidences scores, and use the combined confidence score as the confidence score for the subsequence, or can alternatively use one of the confidence scores, e.g., the lowest or highest confidence score, as the confidence score 140 for the subsequence.

The scoring engine 130 uses the confidence scores 140 to determine whether the initial output sequence 120 is suitable for providing for presentation to the user. Generally, the scoring engine 130 makes this determination by comparing the confidence scores 140 that have been computed for the one or more subsequences included in the initial output sequence 120 against a predetermined threshold confidence score.

For example, the scoring engine 130 can determine that the initial output sequence 120 does not include any factually incorrect, misleading, or otherwise nonsensical information, and is thus suitable for presentation to the user as-is, when the confidence score 140 generated for each of the one or more subsequences included in the initial output sequence 120 satisfies the predetermined threshold confidence score.

Alternatively, the scoring engine 130 can determine that the initial output sequence 120 includes at least some information that is factually incorrect, misleading, and/or otherwise nonsensical, and is thus unsuitable for presentation to the user, when the confidence score 140 generated for at least one subsequence fails to satisfy the predetermined threshold confidence score.

When the scoring engine 130 determines that the confidence score for a particular subsequence fails to satisfy the predetermined threshold confidence score, the system 100 can use an augmentation engine 150 to modify the particular subsequence to generate a modified subsequence that accounts for, e.g., mitigates or alleviates, a lack of confidence of the information represented by the particular subsequence.

To that end, the augmentation engine 150 can be implemented in any of a variety of ways. In some implementations, it can be configured as a prompt engine that generates a prompt that, when processed by the language model neural network 110, will cause or encourage it to generate a replacement subsequence that is more likely to be factually accurate. That is, the same language model neural network 110 is then used to generate another output sequence for the same input sequence 102 that includes different information than the initial output sequence 120. An example of such a prompt can be “Your previous statement that [ . . . ] is factually inaccurate. Please generate another answer that is more factually accurate.” In the example, the content within the brackets can be the particular subsequence that has a confidence score that fails to satisfy the confidence threshold.

In some of these implementations, the augmentation engine 150 provides the initial output sequence 120 (or the particular subsequence of the initial output sequence 120) and, optionally, the confidence scores 140, alongside the input sequence 102 as input to the language model neural network 110. In this way the language model neural network 110 is able to generate new output sentences conditioned on the previously generated output sequences, and, when provided, the confidence scores 140 that have been computed for these output sequences. This increase the likelihood that the new output sentences are more factually accurate.

In other implementations, the augmentation engine 150 can be configured as a text processing engine that can edit the initial output sequence 120 to generate an augmented output sequence 152. For example, the text processing engine can delete the particular subsequence from the initial output sequence 120, such that the augmented output sequence 152 includes no subsequences that have the confidence scores that fail to satisfy the predetermined threshold confidence score.

As another example, the text processing engine can generate a rephrasing of the particular subsequence for inclusion in the augmented output sequence 152 in place of the particular subsequence. The rephrasing generally accounts for the lack of confidence of the information represented by the particular subsequence. For example, such a rephrasing uses certain words to express a degree of uncertainly.

As yet another example, the text processing engine can add a textual indication to the particular subsequence to indicate to the user the potential lack of confidence with respect to the particular subsequence. For example, a textual indication that includes one of the following clauses can be prepended or appended to the particular subsequence within the initial output sequence 120: “There might be mistakes in this statement. Please double check the content for factual accuracy,” or “I'm not sure about the factual accuracy about this statement,” such that the augmented output sequence 152 includes the content of the initial output sequence 120, and also includes additional content that informs the user about the potential lack of confidence with respect to some of the content.

As a particular example for illustration, suppose that the system 100 receives an input sequence 102 that reads as follows: “Who were the survivors of the Titanic?” and uses the language model neural network 110 to generate an initial output sequence 120 that includes a total of three subsequences.

The first subsequence is: “Molly Brown is one of the famous passengers to survive the sinking of the Titanic.” The second subsequence is: “J. Bruce Ismay managed to survive the sinking of the Titanic. He was the highest-ranked official in the White Star Line Company.” The third subsequence is: “The sole survivor of the Titanic is Charles Joughin.”

Each of these subsequences included in the initial output sequence 120 is coherent and grammatically correct, but the third subsequence is nevertheless factually incorrect, because about 700 passengers survived the sinking of Titanic.

In this example, the one or more neighbor pre-generated text embeddings 137 for the third subsequence that has been identified by the scoring engine 130 have corresponding indices that specify relatively lower levels of confidence, and, correspondingly, the confidence score 140 computed by the scoring engine 130 for the third subsequence based on the indices will not satisfy the predetermined threshold confidence score.

In response to determining that the confidence score 140 computed for the third subsequence does not satisfy the predetermined threshold confidence score, the system 100 uses the augmentation engine 150 to modify the third subsequence to generate an augmented output sequence 152.

For example, the third subsequence is deleted, and the augmented output sequence 152 does not include the third subsequence. As another example, a textual indication which informs the user about the potential lack of confidence is added, and the augmented output sequence 152 includes the first, second, and third subsequence, as well as the textual indication. In either example, the system 100 mitigates the negative effects that might be caused by the false or misleading information included in the initial output sequence 120 of the language model neural network 110 to improve system performance and user experience.

FIG. 2 is a flow diagram of an example process 200 for generating an augmented output sequence. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an augmented output sequence generation system, e.g., the augmented output sequence generation system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

The system maintains a plurality of databases (step 202). Each database corresponds to an external knowledge base, and is associated with a respective level of confidence that is defined with respect to content of the knowledge base. The content of the external knowledge base includes a collection of electronic documents. Each electronic document includes a plurality of source subsequences. An example way of generating these databases is described below with reference to FIG. 3.

Each database stores a plurality of pre-generated embeddings that have been generated by using a neural network based on processing the source subsequences included in the collection of electronic documents in the knowledge base that corresponds to the database. In some implementations, the neural network can be the same as the neural network that is used below at step 204.

In particular, each database stores the plurality of pre-generated embeddings in association with index data that specifies the level of confidence of the database, and, in some implementations, additional contextual information about the source subsequences based on which the pre-generated embeddings are generated. For example, for a given pre-generated embedding, the index data can specify a network location of an electronic document from which the source subsequence is obtained, e.g., extracted. As another example, for a given pre-generated embedding, the index data can specify a location offset of the source subsequence within an electronic document that includes the source subsequence. As yet another example, for a given pre-generated embedding, the index data can specify a timestamp that an electronic document from which the source subsequence is obtained is last modified.

The system generates an initial output sequence by using a language model neural network (step 204). The initial output sequence can be generated by the language model neural network from processing an input sequence submitted by a user. The initial output sequence includes one or more subsequences.

For each of the one or more subsequences included in the initial output sequence, the system determines a confidence score that indicates a level of confidence of information represented by the subsequence (step 206). The system performs one iteration of following steps 208-212 to determine the confidence score for each subsequence.

The system generates a query embedding in the embedding space based on the subsequence (step 208). The query embedding can generally be derived from the output hidden states generated by one or more intermediate layers of the language model neural network while it is processing the input sequence to generate the subsequence in the initial output sequence.

In some implementations, the output hidden state of a layer of the language model neural network is directly used as the query embedding, while in other implementations, the output hidden state is further processed to generate the query embedding. For example, the system can apply a linear projection to the output hidden state using principal component analysis or another dimensionality reduction technique to reduce the dimensionality of the output hidden state. The projected output hidden state can then be used as the query embedding.

The system identifies, from the respective pluralities of pre-generated embeddings stored in the plurality of databases, one or more neighbor pre-generated text embeddings based on respective distances between the query embedding and each of the respective pluralities of pre-generated embeddings (step 210). For example, the system can select, as the neighbor pre-generated text embeddings, one or more pre-generated text embeddings that have the smallest distances to the query embedding or that satisfy a distance threshold.

In some implementations, this can be done by using a maximum inner product search technique (MIPS) to compute the inner product of the query embedding with the pre-generated text embeddings to identify pairs of query embedding and pre-generated text embedding that have the largest, or maximum, inner products (i.e., closest distance or, put another way, highest similarity).

When a large number of, e.g., one million, ten million, one billion, or more, pre-generated embeddings are stored in each database, however, computation of the inner products may be slow and processor resource intensive. Some implementations of the system thus use an approximate nearest neighbor matching technique, i.e., instead of a brute-force method, to enable faster computation time while retaining a high level of accuracy. For example, the system can use search space pruning, quantization for the MIPS, or both.

The system computes the confidence score for the subsequence based on the indices that are stored in association with the one or more neighbor pre-generated text embeddings (step 212). As mentioned above, the indices specify the levels of confidence of the databases that store the one or more neighbor pre-generated subsequences and, in some implementations, additional contextual information about the source subsequences based on which the neighbor pre-generated embedding are generated. Example ways of computing the confidence score based on the indices is described above with reference to FIG. 1.

The system determines that a particular subsequence in the one or more subsequences has a confidence score that fails to satisfy a predetermined threshold confidence score and, in response, generates an augmented output that modifies the particular subsequence (step 214). The system can make this determination by comparing the confidence scores that have been computed for each of the one or more subsequences against the predetermined threshold confidence score.

The system can generate the augmented output sequence by modifying the particular subsequence in any of a variety of ways. For example, the system can cause the language model neural network to generate a new output sequence, e.g., from processing a new prompt that includes the input sequence submitted by the user and the initial output sequence generated by the language model neural network. As another example, the augmented output sequence will include, in place of the particular subsequence, a rephrasing of the particular subsequence that accounts for a lack of confidence of the information represented by the particular subsequence. As another example, the augmented output sequence will exclude, i.e., will not include, the particular subsequence (but it may include the other subsequences included in the initial output sequence generated by the language model neural network that each satisfy the predetermined threshold confidence score).

After generating the augmented output sequence, the system can provide the augmented output sequence in response to the user who submitted the input sequence. For example, the system can provide the augmented output sequence to a client device of the user for presentation on a display of the client device.

FIG. 3 is a flow diagram of an example process 300 generating a plurality of databases. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

The system generates a database associated with a first level of confidence (step 302). The database stores a first plurality of pre-generated embeddings that have been generated, e.g., by using a neural network, from source subsequences extracted from electronic documents located at a first network location, e.g., a Uniform Resource Locator (URL) on the Internet. In particular, the first network location is known to the system, and the system associates the database with the first level of confidence at least partly because of the known, first network location.

In some implementations, the system generates an index for each of the first plurality of pre-generated embeddings that specifies the first level of confidence, and stores the indices in association with the plurality of pre-generated embeddings in the database associated with the first level of confidence.

The system generates a database associated with the second level of confidence (step 304). The database stores a second plurality of pre-generated embeddings that have been generated, e.g., by using the neural network, from source subsequences extracted from electronic documents located at a second network location, e.g., a URL on the Internet, that is different from the first network location. The second network location is also known to the system, and the system associates the database with the second level of confidence at least partly because of the known, second network location.

The system generates an additional database that stores a third plurality of pre-generated embeddings that have been generated, e.g., by using the neural network, from source subsequences extracted from electronic documents that is located at a third network location (step 306). The third network location is different from both the first and second network locations, and, in particular, unlike the first and second network locations, the third network location is unknown to system and therefore, the confidence of the content stored therein, is unknown to system.

To automatically determine the level of confidence for the additional database, the system determines (i) a first similarity of the third plurality of pre-generated embeddings relative to the first plurality of pre-generated embeddings and (ii) a second similarity of the third plurality of pre-generated embeddings relative to the second plurality of pre-generated embeddings, and associates the additional database with a corresponding level of confidence based on the first and second similarities.

In some implementations, the similarity can be computed as a distance in the embedding space, e.g., as a combination, e.g., sum or average, of the similarities between one or more pairs of third pre-generated embedding and first (or second) pre-generated embedding.

For example, if the first similarity is within a predetermined similarity threshold, then the corresponding level of confidence can be the first level of confidence (because the additional database stores pre-generated embeddings that are not too much different from the pre-generated embeddings stored in the database associated with the first level of confidence). Analogously, if the second similarity is within the predetermined similarity threshold, then the corresponding level of confidence can be the second level of confidence (because the additional database stores pre-generated embeddings that are not too much different from the pre-generated embeddings stored in the database associated with the second level of confidence).

As another example, if neither the first similarity nor the second similarity is within the predetermined similarity threshold, then the third level of confidence can be different from both the first and the second similarities. For example, the corresponding level of confidence can be in-between the first and the second levels of confidence.

The system can repeat step 306 until iterating through all of the databases. Each database stores a respective plurality of pre-generated embeddings that have been generated, e.g., by using the neural network, from source subsequences extracted from electronic documents located at a different network location. By repeatedly performing step 306 for each database, the system can determine the levels of confidence for all for all of the databases.

For example, the system can generate a further additional database that stores a fourth plurality of pre-generated embeddings that have been generated, e.g., by using the neural network, from source subsequences extracted from electronic documents that is located at a fourth network location (step 308).

For the further additional database, the system determines (i) a first similarity of the fourth plurality of pre-generated embeddings relative to the first plurality of pre-generated embeddings, (ii) a second similarity of the fourth plurality of pre-generated embeddings relative to the second plurality of pre-generated embeddings, and (iii) a third similarity of the fourth plurality of pre-generated embeddings relative to the third plurality of pre-generated embeddings.

The system then associates the further additional database with a corresponding level of confidence based on the first, second, and third similarities.

For example, when one of the first, second, and third similarities is within the predetermined similarity threshold, then the corresponding level of confidence of the further additional database can be same as one of the levels of confidence mentioned above.

Alternatively, when none of the first, second, and third similarities is within the predetermined similarity threshold, then the corresponding level of confidence can be different from any of the levels of confidence mentioned above, i.e., the corresponding level of confidence of the further additional database can be in-between the first and the third levels of confidence, or in-between the third and the second levels of confidence.

In this way, the system can automatically generate a large number of databases where each database corresponds to an external knowledge base, and is associated with a respective level of confidence that is defined with respect to content of the knowledge base. Generally, the greater the number of databases, the higher the likelihood that the system can generate an augmented output sequence that is factually accurate as a result of performing the process 200 which makes use of these databases.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.