105 associated with the portion of document data. In an example, the reference locations can be temporal locations, such as timestamps within a time-dimensioned document (e.g., audio recording, video recording, etc.).

In some embodiments, the inference document 105 has undergone optical character recognition (“OCR”). The OCR process can result in the generation of the document data and reference location tags for different portions of the document data. In other embodiments, the inference document 105 can simply be an image representation of an electronic document and no OCR is needed; instead, the image representation itself can be processed. For example, prompt generation model 120 can generate multimodal prompts that can instruct machine-learned model 125 to populate a portion of target schema 110 based on an input image. In some embodiments, the inference document 105 can be in plain text already, and not require any further processing.

Example content from an inference document is provided below:

Fictional Company LLC 07 | 03 | 23 | 04

INVOICE 75 | 03 | 93 | 06

Due Date: 67 | 17 | 75 | 18

Nov 3, 2019 83 | 17 | 92 | 18

Balance Due: 63 | 20 | 74 | 21

$26,211.90 83 | 20 | 92 | 21

In this example of content from an inference document, textual content obtained from the original document is provided at the beginning of the line. Reference location tags are appended to the line in the following format: “{x_min}|{y_min}|{x_max}|{y_max}”. It is to be understood that this example is provided for the sake of illustration only. Other formats can be used. For instance, reference location tags can be or include line numbers. Reference location tags can be or include locations in other coordinate systems. Reference location tags can include other values, such as height and width values for bounding boxes.

The target schema 110 can include a definition of a schema for extracted data, such as defining the structure of data entities (e.g., data entities 112, 114, 116) to be extracted from the inference document 105. The target schema 110 can describe the structure of the data entities to be extracted in a formal language, can describe various sub-entities for each data entity, can describe types of data to be extracted, can describe example values of data to be extracted, and the like. Data entities with sub-entities can be nested entities. Nested entities include a top-level data entity and one or more sub-entities, which in turn can have types of data to be extracted, can describe example values of data to be extracted, and the like.

Nesting can include multiple layers. For instance, a nested structure can include a top-level entity, zero, one, or more intermediate entities, and one or more leaf entities. In some embodiments, there can be multiple instances of a single top-level entity (e.g., multiple line items in an invoice). An example arrangement of top-level entities, intermediate entities, and leaf entities can include the following:

{

“top level entity”: [

“leaf entity”: “”,

“intermediate entity 1” : {

“leaf entity”: “”,

},

“intermediate entity 2” : {

“leaf entity”: “”,

},

],

}

In a sample embodiment, the target schema 110 can include data entities such as “Company Name,” “Invoice ID Number,” “Line Item” with nested entities “amount” and “description,” “Contact Information,” “Filing Status,” “Account Type,” and other suitable entities, with associated data types for the data entities, such as strings. For instance, an example target schema is illustrated below in JSON format:

{

“Company Name”: “”,

“Invoice ID Number”: “”,

“Line Item”:[

{

“amount”: “”,

“description”: “”,

}

]

“Contact Information”: “”,

“Filing Status”: “”,

“Account Type”: “”,

}

The inference controller 115 can receive inference document 105 and target schema 110. The inference controller 115 can include a prompt generation model 120. The prompt generation model 120 can take the inference document 105 and the target schema 110 as input and, using the inference document 105 and the target schema 110, generate a prompt for completion by the machine-learned model 125.

The prompt generation model 120 can include one or more machine-learned models. The prompt generation model 120 can include one or more data processing components that can receive and combine inputs. The prompt generation model 120 can receive an inference document 105 and a target schema 110 can combine data from each to generate an input for the machine-learned model 125.

The prompt generation model 120 can add additional content to the prompt. For instance, the prompt generation model 120 can add instructions for the machine-learned model 125 to execute based on the provided data. For instance, the prompt generation model 120 can insert description of the entity extraction task, description of the provided document(s), description of the schema, instructions for how to populate the schema, etc.

Extraction instructions can provide the machine-learned model 125 with directions for extracting data entities from the document portions and with rules for matching the data entities to the schema portions. A sample extraction instruction can be “the extraction must respect the schema,” indicating that any extracted data entity must be extracted to match the target schema 110 and not to create a new data entity within the target schema 110. More complex extraction instructions can provide valid value formats for extraction (e.g., “valid string value format “FORMAT_FOR_STRING”), invalid value formats, indications of values that will be present (e.g., “line tags appear at the end of each line.”), and the like.

This additional content can be generated by a machine-learned model (e.g., a model configured to generate instructions for entity extraction) or retrieved from a stored file.

For example, using the example inference document and target schema shown above, in one example the prompt generator 120 can output the following prompt:

For the keys in SCHEMA, extract values containing the

corresponding text and tags from DOCUMENT and provide the

key-value pairs in OUTPUT in JSON format.

SCHEMA:

{

“Company Name”: “”,

“Invoice ID Number”: “”,

“Line Item”:[

{

“amount”: “”,

“description”: “”,

}

]

“Due Date”: “”,

“Amount”: “”,

“Account Type”: “”,

}

DOCUMENT:

Fictional Company LLC 07 | 03 | 23 | 04

INVOICE 75 | 03 | 93 | 06

Due Date: 67 | 17 | 75 | 18

Nov 3, 2019 83 | 17 | 92 | 18

Balance Due: 63 | 20 | 74 | 21

$26,211.90 83 | 20 | 92 | 21

$200.00 15 | 25 | 20 | 26

Item of work 30 | 25 | 45 | 26

OUTPUT:

{

Machine-learned model 125 can receive and process the generated prompt. Machine-learned model 125 can be, for example, a model configured for processing natural language. Machine-learned model 125 can be a model configured for processing multimodal data, such as data in image and text format. Machine-learned model 125 can be or include various types of models, including deep neural networks, transformer models, etc. For instance, an example model can be referred to as a “large language model” (LLM).

The machine-learned model 125 can receive the generated prompt and generate an output based on the inference document 105 and the target schema 110. In some examples, the output can be a completion of the prompt. The output can be an answer to or otherwise a response to the prompt. For example, the machine-learned model 125 can receive a generated prompt asking the machine-learned model 125 to identify values in the inference document 105 related to invoice amounts, invoice identification numbers, or other data entities. In some examples, the machine-learned model 125 can encode the prompt into a context vector and then utilize the context vector to identify portions of the inference document 105 associated with the data entities.

The output of the machine-learned model 125 can be one or more data values for data entities defined in the schema. For example, based on an input of the inference document 105 and the target schema 110 requiring an invoice amount and invoice identification number, the machine-learned model 125 can output an invoice amount of “$5000” and an invoice identification number of “0123456” from the inference document 105.

An example completion for a prompt can be as follows (carrying forward the examples from above:

{

“Company Name”: “Fictional Company LLC 07 | 03 | 23 | 04”,

“Invoice ID Number”: “null”,

“Due Date”: “Nov 3, 2019 83 | 17 | 92 | 18”,

“Amount”: “26,211. 90 83 | 20 | 92 | 21”,

“Account Type”: “null”,

“Line Item”:[

{

“amount”: “200.00”,

“description”: “Item of work”,

}

]}

In some embodiments, the machine-learned model 125 may not be able to find a verbatim data entity in the inference document 105. For example, the target schema 110 can define an “Invoice ID” data entity, but the machine-learned model 125 may only be able to identify an “Invoice Number” in the inference document 105. The machine-learned model 125 can therefore support abstractive extraction, where the verbatim data entity is not present in the document, but the machine-learned model 125 is able to understand, based on the context, that the data entity is, in fact, present, under a similar or related name.

Similarly, a machine-learned model 125 can inferentially obtain computed entity values. For instance, if a requested entity is “Total order price,” and the document only contains “unit price” and “unit quantity,” machine-learned model 125 can output the product of the unit price value and the unit quantity value as the total order price entity value. The machine-learned model 125 can also output an indication of the component values used in the computation (along with their corresponding location tags) for downstream validation.

The inference controller 115 can receive the output from the machine-learned model 125 and validator 122 can validate the output from the machine-learned model 125. For instance, validator 122 can determine whether the output entity values have an inference location tag. Validator 122 can invalidate values lacking an inference location tag. Validator 122 can determine if an inference location tag for an extracted entity value points to a location in the inference document 105 that contains content matching the extracted entity value. If there is no match, validator 122 can invalidate the value. Validator 122 can validate extracted entity values that agree with the original content from the inference document 105.

For example validator 122 can find the reference spatial location in the original inference document 105 based on the inference spatial location extracted by the large language model 125. In one example, validator 122 can identify the reference spatial location in the original inference document using the inference spatial location, such as using one or more values of coordinates from the inference spatial location to identify the reference spatial location in the original inference document 105.

Validator 122 can perform a normalized string search on the line text at the reference spatial location in the original inference document 105 to determine if the extracted data entity value matches the value at the reference spatial location. If a match is found, validator 122 can obtain a text anchor for the matched value based on the reference spatial location. In some embodiments, the text anchor is obtained by obtaining a start index of the matching value on the line in the inference document 105 and then determining the full text anchor using Equation 1 provided below.

Equation
⁢

1

text
⁢

anchor

=

[

line_start
⁢
_index

+

match_start
⁢
_index

,

line_start
⁢
_index

+

match_start
⁢
_index

+

length
(

entity_value
⁢
_on
⁢
_line

)

+
1

]

.

The generated text anchor can then be associated with the extracted data entity. During confidence scoring (as described below), samples with text anchors can be used to determine a final extracted data entity output. Conversely, samples without a text anchor (e.g., did not match the normalized string search) can be discarded when determining a final extracted data entity output because the extracted values of the data entity cannot be grounded in the original document. For example, large language models can sometimes “hallucinate,” or generate false information. The use of text anchors for grounding and data validation enables extracted data values to be compared to the original inference document 205 to ensure that the extracted data values are actually present in the original inference document 205, and not simply “hallucinated” by the large language model 225.

Validator 122 can parse the returned output to determine if it is in a proper output format. For instance, validator 122 can determine if the output is in proper JSON, XML, CSV, or other desired formats. Validator 122 can apply formatting corrections as needed.

Validator 122 can compare the output against target schema 110 to confirm that all entity types are present, that nested entities are respected, that the data values for extracted entities are strings, and the like.

If the output fails validation, validator 122 can initiate remediation. Remediation can include re-processing the entity extraction task. For instance, remediation can include re-processing the task with different instructions that highlight or caution against that type of error that cause the initial output to fail validation. Remediation can include reprocessing the document with a modified schema, such as a schema only including entities that were missed in the first pass.

After the extracted data is validated, the inference controller 115 can output data output 130. Data output 130 can contain the extracted entity values corresponding to target schema 110. For instance, data output 130 can include extracted and validated entity data values 132, and 134, and 136 respectively corresponding to entities 112, 114, and 116 from the target schema 110. The data output 130 can then be used by other software applications, such as system controllers, security verification systems, communications systems, customer-relationship management software applications, human resources software applications, database software applications, and other software applications that utilize such data to perform various tasks and functions.

FIG. 2 depicts a block diagram of an example document entity extraction system 200 according to example embodiments of the present disclosure. Much like the system described in FIG. 1, the document entity extraction system 200 (which can be the same as or different from system 100) can receive an inference document 205 and a target schema 210. Inference document 205 can be the same as or different from inference document 105. Target schema 210 can be the same as or different from target schema 110. The inference document 205 and the target schema 210 can be provided to a prompt generation model 220 of an inference controller 215. Prompt generation model 220 can be the same as or different from prompt generation model 120. Inference controller 215 can be the same as or different from inference controller 115. The prompt generation model 220 can use the inference document 205 and target schema 210 to generate input(s) for machine-learned model 225. Machine-learned model(s) 225 can be the same as or different from machine-learned model(s) 125. The inference controller 215 can provide the input(s) to the machine-learned model 225 and can receive corresponding output(s) from the machine-learned model 225 that contain predicted entity extraction values based on target schema 110. Validator 222 can process the output(s) of the machine-learned model 225 to validate and assemble data output 230. Validator 222 can be the same as or different from validator 122.

A chunker 221 can process inference document 105 and target schema 210 to perform chunking. Chunking can include subdividing inputs into a number of sub-portions for processing by a machine-learned model. For instance, different machine-learned models can have different input dimensions, and as such have different capacities for processing different amounts of input data. The chunker 221 can subdivide the inference document 205 and the target schema 210 into portions that fit within the respective input capacity of machine-learned model 225. For instance, using chunker 221, prompt generation model 220 can output a prompt N 240 that contains a document input i 242 and schema input j 244, where prompt N 240 is sized to fit within an input dimension of machine-learned model 225.

To allow machine-learned model 225 to process each schema input j 244 in view of each document input i 242, prompt generation model 220 can generate all combinations of document inputs i and schema inputs j in N different prompts. For example, given 3 document inputs and 3 schema inputs, the prompt generator 220 can generate text prompts for each permutation of combinations of document inputs and schema inputs (e.g., (document input 1, schema input 1), (document input 1, schema input 2), (document input 1, schema input 3), (document input 2, schema input 1), and so on).

These N prompts can be fed to machine-learned model 225 to generate N outputs (e.g., with a prompt N 240 corresponding to an output N 250). For instance, data entities 212, 214, and 216 can be located in different schema inputs, and data 232, 234, and 236 can be located in different document inputs, but by processing all the combinations, machine-learned model 225 can be able to successfully populate entity 212 with data 232 responsive to at least one of the N prompts. For the other prompts which lack either entity 212 or data 232, machine-learned model 225 can return a null result. In this manner, for instance, validator 222 can merge the N outputs to obtain a complete data output 230.

For example, as noted above, an example input from prompt generator 220 can include a portion from inference document 205 and a portion from target schema 210. Accordingly, the portions can be selected such that the combination of the portions remains smaller than the input dimension of machine-learned model 225. Additional input capacity can be reserved for instructions added to the portions by prompt generator 220. The amount reserved can be fixed (e.g., 5%, 10%, etc.) or determined dynamically based on a context signal. For instance, complex schemas may be associated with longer instructions, so greater space may be reserved by chunking the inputs into smaller chunks. Similarly, for few-shot implementations, longer documents or more complex schemas may be associated with longer few-shot examples (or more examples), thus consuming more capacity. As such, chunker 221 can identify that capacity is needed for few-shot exemplars and reserve space accordingly by setting a smaller chunk size.

The chunker 221 can subdivide inference document 205 and the target schema 210 to create one or more document inputs and one or more schema inputs. The one or more document inputs can be portions of the inference document 205. The chunker 221 can subdivide the inference document into a number of document inputs by dividing, for example, the inference document 205 into sections, pages, paragraph, sentences, or other sub-units of the document that may or may not align with natural subdivisions of the document itself. For example, the chunker 221 can identify a maximum portion length for the document input(s) and determine a document subdivision that is near to, but does not exceed, the maximum portion length.

Similarly, the target schema 210 can be separated by the chunker 221 into portions for input into a language processing model. In some examples, the chunker 221 can subdivide the target schema 210 by splitting top-level entities in the target schema 210 into independent branches, including any subentities. Each independent branch can be a schema input. In some examples, the chunker 221 can subdivide the target schema 210 into a number of schema inputs by subdividing the target schema 210 based on data types associated with various data entities. For example, the chunker 221 can identify a maximum portion length for the schema input(s) and determine a schema subdivision that is near to, but does not exceed, the maximum portion length.

The chunker 221 can, in some embodiments, create chunks to optimize the performance of the machine-learned model. For example, some machine-learned models can have better performance with shorter inputs or outputs. For instance, some machine-learned models operate on a sequence-to-sequence basis, such that an output sequence is processed based on an input sequence. In some models, small portions of the input (e.g., instructions for performing a task) can have decreased efficacy as the output gets longer and longer (with content being generated in context further removed from the instructions). The chunker 221 can create document inputs and schema inputs of portion length(s) that are configured to improve an output performance. For instance, the portion length(s) can be smaller than necessary, such that the machine-learned model receives shorter inputs than the maximum input dimension. Smaller inputs can provide for faster processing or more accurate processing.

In some embodiments, the chunker 221 can optimize chunk size based on an available compute budget. For instance, large chunks can be processed by machine-learned models with large context windows. But machine-learned models with large context windows can be more expensive to operate. Smaller chunks can be processed by smaller machine-learned models with smaller context windows. But for a document of fixed size, smaller chunks can require more calls to the model to iterate through the full document and schema. Smaller chunks can also result in more combinations that need to be processed, potentially increasing a total input amount across all the iterations.

Accordingly, the chunker 221 can determine a chunk size that optimizes a net compute cost based on the cost of the model 225 to run inference and a count of how many inference runs will be used based on the number of prompts used to process all the combinations of document and schema inputs.

FIG. 3 depicts a block diagram of an example implementation of a document entity extraction system that validates based on a consistency metric. For instance, an inference controller 315 (which can be the same as or different from inference controller 115 or 125) can include a prompt generation model 320 (which can be the same as or different from prompt generation model 120 or 220) and a validator 322 (which can be the same as or different from validator 122 or 222). For each prompt N 340 output by prompt generation model 320, inference controller 315 can cause machine-learned model 325 (which can be the same as or different from machine-learned model 125 or 225) to process the prompt N 340 k times to generate k outputs. Validator 322 can receive each output N_k350 and compute a consistency metric across the set of received outputs for prompt N 340. The consistency metric can include a vote, such as a majority vote, plurality vote, etc. For example, validator 322 can incorporate top-k and/or top-p sampling.

Because of the randomness introduced by the use of top-p and top-k sampling for each of the number of samples for a generated prompt, the validator 322 can also determine a confidence score. In one embodiment, the confidence score for a particular extracted data value can be determined using Equation 2.

Confidence_score
=

number_samples
⁢
⁠
_with
⁢
⁠
_same
⁢

answer
/
total_number

⁢
_of
⁢

samples

Equation
⁢

2

Intuitively, the most confident answer can be the answer from the number of samples that the machine-learned model 325 most often identified to be the data value for the data entity, despite any randomness introduced during sampling. In some embodiments, if the most common answer has multiple elements but the occurrence type for the particular entity is only once in the schema, the validator 325 can select the most frequent-appearing value by text value as the most confident answer.

Based on the generated confidence scores, the validator 325 can identify a final data value for the extracted data entity. It is to be noted that alternative methods for computing confidence scores for data values can be used, such as utilizing weights for each sample received from the machine-learned model 325 (e.g., model-generated confidence scores), using raw logits exposed after decoding, or other voting methods.

In some embodiments, the target schema can include one or more nested entities. Nested entities can include a top-level entity and one or more sub-entities. Nesting can include multiple layers. For instance, a nested structure can include a top-level entity, zero, one, or more intermediate entities, and one or more leaf entities.

Confidence scoring of nested entities can be performed using various techniques based on the nested architecture. In some examples, data extracted for the top-level entity can come from one selected sample, rather than mixing and matching among multiple samples. For example, the confidences for each individual intermediate and leaf entity can be determined and aggregated on a per-sample basis to give a confidence value for the top level entity of that sample, and the resulting values can be selected based on the aggregate confidence for that sample (e.g., all the nested entities from the winning sample). In some embodiments, there can be multiple instances of the same type of top-level entity. Data extracted for each instance can come from one selected sample (e.g., based on the aggregate confidence of entities nested within that instance). Data extracted for all instances can come from one selected sample (e.g., based on the aggregate confidence of all instances). Data extracted for each instance can come from different samples (e.g., based on which sample provides the highest aggregate confidence for the respective instance).

In some examples, data extracted for the top level entity can be mixed and matched across multiple samples. For instance, in some implementations, each individual intermediate and leaf entity value can be selected based on the highest confidence sample for each individual intermediate and leaf entity. The resulting values can include the set of highest-confidence values as determined across all samples.

For each sub-entity in the nested entity, predictions for each sample can be extracted, and then confidence scoring can be performed using the values extracted from each sample. Confidence scoring for each sample can be performed, and in some embodiments, the sample with the highest confidence score can be selected.

FIG. 4 depicts a block diagram of an example implementation of a prompt generation model that retrieves examples for few-shot prompting. Prompt generation model 420 (which can be the same as or different from prompt generation model 120, 220, or 320) can process an inference document 405 (which can be the same as or different from inference document 105 or 205) and target schema 410 (which can be the same as or different from target schema 110 or 210) to generate a prompt 440 (which can be the same as or different from prompt 240).

To facilitate few-shot prompting, prompt generation model 420 can implement an example retriever 426. Example retriever 426 can process any one or both of inference document 405 and target schema 410 to query an annotated document corpus 460 to retrieve one or more documents and a set of corresponding annotations (e.g., a populated schema) that can operate as examples for how the machine-learned model should populate the target schema 410.

For instance, the example documents can have a template, such as a particular schema or structure. This template can match or be similar to, at least in part, the desired target schema 410. The example retriever 426 can select documents with similar templates to the target schema 410 and include portions of the example document(s) and associated completions of the portions of the example schema for inclusion in a prompt for processing by the machine-learned model. In other embodiments, however, the example retriever can select documents regardless of similarity to the target schema 410.

The selected example documents from the annotated document corpus 460 can optionally be provided to a chunker (e.g., chunker 221) for processing in a similar fashion to the inference document 205 for aligning input dimensions of the prompt with the input dimension of the machine-learned model.

In some embodiments, the selected documents are retrieved at random from the annotated document corpus 230, as long as a schema for the selected documents match or are similar to at least in part the target schema 410. In another embodiment, a nearest neighbor search can be performed using one or more embeddings associated with the annotated documents and the inference document 405 to determine one or more closest matches to the inference document 405 (e.g., documents having the same or substantially similar template). In a further embodiment, a first N search can be performed, which returns the first N documents in the corpus regardless of template. In yet another embodiment, a maximum entity coverage search can be performed. In a maximum entity coverage search, the annotated documents can be ordered by entity coverage (e.g. documents with the maximum number of schema entities being placed first, followed by documents that cover less entities, and then documents with no overlap in schema entities). The first N documents from this ordering can be retrieved.

An example prompt 440 can include at least a portion of an example document 442, at least a portion of an example populated schema 444, an inference document input 446, and a target schema input 448. For example, an example prompt 440 can be as follows:

For the keys in SCHEMA, extract values containing the

corresponding text and tags from DOCUMENT and provide the

key-value pairs in OUTPUT in JSON format.

SCHEMA:

{

“Company Name”: “”,

“Invoice ID Number”: “”,

“Line Item”:[

{

“amount”: “”,

“description”: “”,

}

]

“Due Date”: “”,

“Amount”: “”,

“Account Type”: “”,

}

DOCUMENT:

Fictional Company A LLC 07 | 03 | 23 | 04

INVOICE 75 | 03 | 93 | 06

Due Date: 67 | 17 | 75 | 18

Nov 3, 2019 83 | 17 | 92 | 18

Balance Due: 63 | 20 | 74 | 21

$26,211.90 83 | 20 | 92 | 21

$200.00 15 | 25 | 20 | 26

Item of work 30 | 25 | 45 | 26

OUTPUT:

{

“Company Name”: “Fictional Company A LLC 07 | 03 | 23 | 04”,

“Invoice ID Number”: “null”,

“Due Date”: “Nov 3, 2019 83 | 17 | 92 | 18” ,

“Amount”: “26,211.90 83 | 20 | 92 | 21” ,

“Account Type”: “null”,

“Line Item”:[

{

“amount”: “200.00”,

“description”: “Item of work”,

}

]

}

DOCUMENT:

Fictional Company B LLC 15 | 03 | 62 | 07

INVOICE 06 | 14 | 32 | 16

Due on: 06 | 52 | 18 | 53

Sep 18, 2023 06 | 54 | 29 | 55

Amount Due: 44 | 77 | 60 | 78

$2,000.00 64 | 64 | 72 | 65

OUTPUT:

{

Example Methods

FIG. 5 depicts a flow chart diagram of an example method 500 to perform according to example embodiments of the present disclosure. Although FIG. 4 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method 500 can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure.

At 502, a computing system receives an inference document and a target schema. As described above in reference to FIGS. 1 and 2, the inference document includes document data and reference spatial locations for document data, and the target schema defines data entities to be extracted from the document data. In some embodiments, the inference document is based on an output of an optical character recognition system, and the document data of the inference document includes data representing optically-recognized characters in the rendering of the inference document.

In some embodiments the computing system can also receive an image input and can use the image input to validate the output of the optical character recognition system.

In some embodiments, the inference document can be an image representation of an electronic document.

In some embodiments, the one or more reference location tags respectively indicating one or more reference spatial locations of the document data within the rendering of the inference document and can be indicative of one or more bounding boxes containing a portion of the document data.

At 504, the computing system generates one or more document inputs and one or more schema inputs. As described above in reference to FIG. 2, in one embodiment, the one or more document inputs and one or more schema inputs can be generated using chunking to separate the inference document into pages or other subportions and separate the target schema into high-level data entities and subentities.

In some embodiments, the computing system can subdivide the target schema into a plurality of independent branches, where each branch of the plurality of independent branches representing a data entity and subentities of the data entity. Each independent branch of the plurality of independent branches can then be used as a schema input of the target schema.

At 506, for each respective combination of the one or more document inputs and the one or more schema inputs, the computing system obtains an extraction input for a language processing model based on a combination of a document input and a schema input.

At 506a, the computing system generates a respective extraction input based on the respective combination of the document input and the schema input.

In some embodiments, the prompt for the respective extraction input can include one or more extraction instructions. The one or more extraction instructions can include a description of a spatial location.

In some embodiments, generating the respective extraction input can include retrieving at least one document from a document corpus and adding at least a portion of the at least one document to the prompt for at least one document input and schema input combination.

In some embodiments, the prompt can include an extraction representation of one or more data entities extracted from the portion of the at least one document.

At 506b, the computing system provides the extract input to the language processing model for completion of the extraction input. The language processing model analyzes the document input based on the extraction instructions and the schema input.

At 506c, the computing system receives a data output from the language processing model. In some embodiments, the data output includes entity data extracted according to the target schema and one or more inference location tags corresponding to one or more inference spatial locations of the entity data within the rendering of the inference document.

At 508, the computing system validates the extracted entity data using, for example, decoding, grounding, and confidence scores as described above with reference to FIG. 2-4. In some embodiments, validating the extracted entity data based on the reference spatial locations and the inference spatial locations comprises can include performing normalized string matching between the extracted entity data and document data at the reference spatial locations in the rendering of the inference document as indicated by the one or more inference location tags corresponding to one or more inference spatial locations of the entity data within the rendering of the inference document. Validating the data can also include determine if the extracted entity data matches the document data. In response to determining that the extracted entity data matches the document data, the computing system can validate the extracted entity data.

In some embodiments, the computing system can, in response to determining that the extracted entity data does not match the document data, discard the extracted entity data.

In some embodiments, the computing system can send the extraction input to the model for a plurality of iterations to obtain a plurality of outputs and, based on the plurality of outputs, the computing system can determine a representative value.

In some embodiments, determining the representative value can include determining a majority output from the plurality of outputs.

In some embodiments, a confidence score can be generated based on the majority output and the plurality of outputs.

In some embodiments, the representative value can be determined based at least in part on one or more received scores from the model.

At 510, the computing system outputs validated extracted entity data for use by other computer software applications.

Example Devices and Systems

FIG. 6A depicts a block diagram of an example computing system 600 that performs document entity extraction according to example embodiments of the present disclosure. The system 600 includes a user computing device 602, a server computing system 630, and a training computing system 650 that are communicatively coupled over a network 680.

The user computing device 602 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device 602 includes one or more processors 612 and a memory 614. The one or more processors 612 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 614 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 614 can store data 616 and instructions 618 which are executed by the processor 612 to cause the user computing device 602 to perform operations.

In some implementations, the user computing device 602 can store or include one or more machine-learned models 620 (e.g., machine learned model 125, 225, 325, etc.). For example, the machine-learned models 620 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example machine-learned models 620 are discussed with reference to FIGS. 1 to 5.

In some implementations, the one or more machine-learned models 620 can be received from the server computing system 630 over network 680, stored in the user computing device memory 614, and then used or otherwise implemented by the one or more processors 612. In some implementations, the user computing device 602 can implement multiple parallel instances of a single machine-learned model 620 (e.g., to perform parallel language processing).

More particularly, the one or more machine-learned models 620 can process text data using component machine-learned models, such as neural networks.

Additionally or alternatively, one or more machine-learned models 640 can be included in or otherwise stored and implemented by the server computing system 630 that communicates with the user computing device 602 according to a client-server relationship. For example, the machine-learned models 640 can be implemented by the server computing system 640 as a portion of a web service (e.g., a language processing service). Thus, one or more models 620 can be stored and implemented at the user computing device 602 and/or one or more models 640 can be stored and implemented at the server computing system 630.

The user computing device 602 can also include one or more user input components 622 that receives user input. For example, the user input component 622 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

The server computing system 630 includes one or more processors 632 and a memory 634. The one or more processors 632 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 634 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 634 can store data 636 and instructions 638 which are executed by the processor 632 to cause the server computing system 630 to perform operations.

In some implementations, the server computing system 630 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 630 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

As described above, the server computing system 630 can store or otherwise include one or more machine-learned models 640. For example, the models 640 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example models 540 are discussed with reference to FIGS. 1 to 5.

The user computing device 602 and/or the server computing system 630 can train the models 620 and/or 640 via interaction with the training computing system 650 that is communicatively coupled over the network 680. The training computing system 650 can be separate from the server computing system 630 or can be a portion of the server computing system 530.

The training computing system 650 includes one or more processors 652 and a memory 654. The one or more processors 652 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 654 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 654 can store data 656 and instructions 658 which are executed by the processor 652 to cause the training computing system 650 to perform operations. In some implementations, the training computing system 650 includes or is otherwise implemented by one or more server computing devices.

The training computing system 650 can include a model trainer 660 that trains the machine-learned models 620 and/or 640 stored at the user computing device 602 and/or the server computing system 630 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 660 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer 660 can train the machine-learned models 620 and/or 640 based on a set of training data 662. The training data 662 can include, for example, text data and data from other input modes to learn the statistical relationships between words, phrases, and sentences, which allows the machine-learned models 620 and/or 640 generate coherent and contextually relevant responses when given a prompt or a query.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 602. Thus, in such implementations, the model 620 provided to the user computing device 602 can be trained by the training computing system 650 on user-specific data received from the user computing device 602. In some instances, this process can be referred to as personalizing the model.

The model trainer 660 includes computer logic utilized to provide desired functionality. The model trainer 660 can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer 660 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 660 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

The network 680 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 680 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.