According to some embodiments, training component 220 trains neural network 230 to identify the unknown class based on the annotation of the set of samples. In some examples, training component 220 computes a multi-label loss based on the classification data and ground truth labels, where each of the ground truth labels describes a known class of the set of known classes. In some examples, training component 220 updates parameters of neural network 230 based on the multi-label loss.

According to some embodiments, training component 220 is configured to train neural network 230 based on the annotation. Training component 220 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 4. In some examples, training component 220 is part of another apparatus other than document classification apparatus 200.

Machine learning model 225 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 3. In one embodiment, machine learning model 225 includes neural network 230, clustering component 235, open-set metric component 240, annotation component 245, and user interface component 250.

According to some embodiments, neural network 230 receives an electronic document. In some examples, neural network 230 classifies the electronic document to obtain classification data, where neural network 230 is trained by iteratively selecting samples for annotation with an unknown class using an open-set metric based on predicted classification data. In some examples, neural network 230 generates the predicted classification data for a set of samples, where neural network 230 is trained to identify a set of known classes. In some examples, neural network 230 identifies a document type based on the classification data.

According to some embodiments, neural network 230 generates classification data for a set of samples, where neural network 230 is trained to identify a set of known classes. In some examples, neural network 230 generates a feature embedding corresponding to each of the set of samples, where the classification data is generated based on the feature embedding. In some examples, the classification data includes prediction logits, uncertainty measures, or both.

According to some embodiments, neural network 230 is trained to identify a set of known classes for a set of samples. In some examples, neural network 230 includes a transformer network that includes a classification head. Neural network 230 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 4.

According to some embodiments, clustering component 235 clusters the set of samples to obtain a set of clusters. Clustering component 235 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 4.

According to some embodiments, open-set metric component 240 selects a set of samples for annotation with the unknown class from the set of samples using the open-set metric based on the predicted classification data.

According to some embodiments, open-set metric component 240 selects a set of samples for annotation from the set of samples using an open-set metric based on the classification data, where the annotation includes an unknown class. In some examples, the open-set metric indicates that the set of known classes do not characterize the set of samples to a threshold level. In some examples, open-set metric component 240 selects a cluster from the set of clusters based on the open-set metric, where the set of samples includes samples from the cluster.

In some examples, open-set metric component 240 computes an activation logit value for each sample in the selected cluster. Open-set metric component 240 identifies a maximum activation logit value based on the activation logit value for each sample in the selected cluster, where the open-set metric is based on the maximum activation logit value. In some examples, the selected cluster minimizes the maximum activation logit value.

In some examples, open-set metric component 240 identifies a class of the set of known classes. Open-set metric component 240 excludes the class from the set of known classes to obtain a reduced set of known classes. Open-set metric component 240 computes the open-set metric based on the reduced set of known classes.

According to some embodiments, open-set metric component 240 is configured to identify a cluster of the plurality of clusters for annotation based on an open-set metric. Open-set metric component 240 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 4.

According to some embodiments, annotation component 245 displays the set of samples in an annotation interface. In some examples, annotation component 245 receives annotation input via the annotation interface, where the training is based on the annotation input. In some examples, annotation component 245 identifies a shared label of the set of samples based on the annotation input, where the annotation is based on the shared label. In some examples, annotation component 245 identifies a distinguishing label of the set of samples based on the annotation input, where the annotation is based on the distinguishing label.

According to some embodiments, annotation component 245 is configured to display samples of the identified cluster in an annotation interface.

According to some embodiments, user interface component 250 displays the electronic document via a customized user interface based on the classification data. In some examples, user interface component 250 selects an interface element associated with the document type, where the customized user interface includes the interface element.

According to some embodiments, user interface component 250 is configured to display a customized user interface based on classification data generated by neural network 230.

The described methods may be implemented or performed by devices that include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, a conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration). Thus, the functions described herein may be implemented in hardware or software and may be executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored in the form of instructions or code on a computer-readable medium.

Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates the transfer of code or data. A non-transitory storage medium may be any available medium that can be accessed by a computer. For example, non-transitory computer-readable media can comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk (CD) or other optical disk storage, magnetic disk storage, or any other non-transitory medium for carrying or storing data or code.

Also, connecting components may be properly termed computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave signals, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology are included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

FIG. 3 shows an example of an annotating process according to aspects of the present disclosure. The example shown includes machine learning model 300, annotator 305, document 310, and label 315. Machine learning model 300 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 2. Document 310 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 6 and 7. Label 315 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 6 and 7.

Training a classification model may depend on a large amount of labeled training data, which can be costly and time consuming to obtain. Accordingly, active learning is used to train a machine learning model with a limited annotation budget. Active learning refers to a machine learning model that is able to select training samples to learn from. In some cases, active learning models depend on uncertainty estimates based on information theory, for designing acquisition functions for active learning. In some examples, the document classification apparatus incorporates weak labeling techniques (i.e., presenting a portion of the annotation data generated by the classification model to a human annotator) are combined with active learning. The weak labeling techniques make use of annotator feedback (e.g., annotator's label assignment or corrections) during an active learning cycle. Weak labeling is used to reduce the annotation cost.

According to an embodiment, an unlabeled pool U includes a set of samples that do not have labels (e.g., not yet assigned a label by annotator 305). A sample from the unlabeled pool U is sent to annotator 305. Annotator 305 assigns a label to the sample. Then, the sample is added into the labeled training set L. The labeled training set L is used to train machine learning model 300. For example, the labeled training set includes known label(s) 315 such as contract, apartment lease, scientific publication, and NeurIPS. Additional unknown labels are to be discovered using the annotating process. Then the additional unknown labels, once discovered, are moved from unlabeled pool U to labeled training set L.

FIG. 4 shows an example of a machine learning model according to aspects of the present disclosure. The example shown includes neural network 400, clustering component 405, open-set metric component 410, and training component 415.

According to an embodiment, a set of samples is input to neural network 400. Then, neural network 400 generates classification data for the set of samples. Neural network 400 is trained to identify a set of known classes. Neural network 400 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 2.

Clustering component 405 clusters the set of samples to obtain a set of clusters. Clustering component 405 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 2.

Open-set metric component 410 selects a cluster from the set of clusters based on an open-set metric, where the set of samples includes samples from the cluster. Open-set metric component 410 selects one or more samples for annotation from the set of samples using the open-set metric based on the classification data, where the annotation includes an unknown class. In some examples, the open-set metric indicates that the set of known classes does not characterize the one or more samples to a threshold level. Open-set metric component 410 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 2.

Training component 415 trains neural network 400 to identify the unknown class based on the annotation of the set of samples. Training component 415 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 2.

Document Classification

In FIGS. 5-8, a method, apparatus, and non-transitory computer-readable medium for document classification are described. One or more embodiments of the method, apparatus, and non-transitory computer-readable medium include receiving an electronic document; classifying the electronic document using a neural network to obtain classification data, wherein the neural network is trained by iteratively selecting samples for annotation with an unknown class using an open-set metric based on predicted classification data; and displaying the electronic document via a customized user interface based on the classification data.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include generating the predicted classification data for a plurality of samples using the neural network, wherein the neural network is trained to identify a plurality of known classes. Some examples further include selecting a set of samples for annotation with the unknown class from the plurality of samples using the open-set metric based on the predicted classification data. Some examples further include training the neural network to identify the unknown class based on the annotation of the set of samples.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include identifying a document type based on the classification data. Some examples further include selecting an interface element associated with the document type, wherein the customized user interface includes the interface element.

FIG. 5 shows an example of document classification process according to aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control functional elements of an apparatus. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various substeps, or are performed in conjunction with other operations.

At operation 505, the system receives an electronic document from a user. In some cases, the operations of this step refer to, or may be performed by, a user as described with reference to FIG. 1. In some cases, the operations of this step refer to, or may be performed by, a document classification apparatus as described with reference to FIGS. 1 and 2.

At operation 510, the system generates classification data based on the electronic document. In some cases, the operations of this step refer to, or may be performed by, a document classification apparatus as described with reference to FIGS. 1 and 2.

At operation 515, the system classifies the electronic document based on the classification data. In some cases, the operations of this step refer to, or may be performed by, a document classification apparatus as described with reference to FIGS. 1 and 2.

At operation 520, the system displays the electronic document with a class label. In some embodiments, the system displays the electronic document with a class label “contract” or the phrase “this document is a contract” to the user via user interface. In some cases, the operations of this step refer to, or may be performed by, a user interface component as described with reference to FIG. 2. For example, the class label is “contract” for the document.

FIG. 6 shows an example of document classification diagram according to aspects of the present disclosure. The example shown includes document 600 and label 605. As an example shown in FIG. 6, the label space may be hierarchical. In some cases, the label space includes main class and sub-classes. For example, the labels “apartment lease” and “mortgage disclosure” are sub-classes of “contract. Similarly, labels “EMNLP”, “NeurIPS”, and “ICML” are sub-classes of “scientific publication”. In some examples, machine learning model 225 as in FIG. 2 can be used to identify a class for document 600. Machine learning model 225 can also be used to expand the existing label space (e.g., continue to refine the discovered topics or expand the label types).

Document 600 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 3 and 7. Label 605 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 3 and 7.

FIG. 7 shows an example of multi-label document classification according to aspects of the present disclosure. The example shown includes document 700 and label 705. FIG. 7 relates to multi-label document classification. For example, document 700 is associated with multiple labels, e.g., contract and apartment lease. Here, a label or class with a score of “1” indicates that the label or class is associated with document 700. On the other hand, a score of “0” indicates that the corresponding label or class is not associated with document 700.

Document 700 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 3 and 6. Label 705 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 3 and 6.

FIG. 8 shows an example of a method for document classification according to aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control functional elements of an apparatus. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various substeps, or are performed in conjunction with other operations.

At operation 805, the system receives an electronic document. In some cases, the operations of this step refer to, or may be performed by, a neural network as described with reference to FIGS. 2 and 4. In some examples, a neural network receives one or more electronic documents (e.g., PDF documents) for classification.

At operation 810, the system classifies the electronic document using a neural network to obtain classification data, where the neural network is trained by iteratively selecting samples for annotation with an unknown class using an open-set metric based on predicted classification data. In some cases, the operations of this step refer to, or may be performed by, a neural network as described with reference to FIGS. 2 and 4.

According to an embodiment of the present disclosure, a training component of document classification apparatus 200 (shown in FIG. 2) includes selecting a set of samples for annotation using an open-set metric that identifies ambiguous samples. For example, one label induction method relates to picking samples that have the least activation responses to known classes (e.g., all known classes). These samples would be identified and annotated with a previously unknown class. This enables a robust multi-label classifier that can handle a much larger pool of labels.

According to an embodiment, document classification apparatus 200 constructs a document type classification model where the labels to be predicted are unknown at the beginning of the process. These unknown labels are iteratively discovered with a human-in-the-loop driven by choosing data to be labeled based on the current state of a supervised, trained classifier. Document classification apparatus 200 uses a supervised deep multi-label classification model to discover new classes. Document classification apparatus 200 builds a classifier and expands the label space by selecting samples that are either uncertain or have low activation. Discovering hieratical or sub-classes using a top-down iterative expansion or comparative labeling will be described in greater detail in FIGS. 11 and 12.

The present disclosure includes a human-in-the-loop system and methods for efficiently building a set of classes or attributes when the space of labels is unknown. In some cases, the process may be referred to as label induction. In the present disclosure, a class represents a specific type or attribute to describe content, e.g., cat, dog, contract, house, etc. A known class is a class that exists in the known dataset. On the other hand, unknown class is a class that has yet to be defined or discovered by the user. Labeled sample represents a data sample assigned to a class (e.g., class is given). In some cases, data samples may have multiple labels. On the other hand, an unlabeled sample is a data sample that has not been assigned any labels by an annotator (e.g., a human in the active learning loop).

In some cases, a dataset has one or more “known” classes of interest. However, an initial clustering can be substituted where the user annotates a few samples as a “cold start”. For example, a cold start situation exists when a new dataset contains no classes (e.g., labeled space is null). In some cases, there is a large amount of unlabeled data with unknown classes or unknown attributes. These data may be preprocessed to remove duplicates and near duplicates using message-digest algorithm 5 (MD5) hashes or content similarity hashing (i.e., locality-sensitive hashing or LSH).

At operation 815, the system displays the electronic document via a customized user interface based on the classification data. In some cases, the operations of this step refer to, or may be performed by, a user interface component as described with reference to FIG. 2. In some examples, the electronic document is displayed in a customized interface based on the document type.

Training and Evaluation

In FIGS. 9-12, a method, apparatus, and non-transitory computer-readable medium for document classification are described. One or more embodiments of the method, apparatus, and non-transitory computer-readable medium include generating classification data for a plurality of samples using a neural network trained to identify a plurality of known classes; selecting a set of samples for annotation from the plurality of samples using an open-set metric based on the classification data, wherein the annotation includes an unknown class; and training the neural network to identify the unknown class based on the annotation of the set of samples.

Some examples of the method, apparatus, and non-transitory computer readable medium further include computing a multi-label loss based on the classification data and ground truth labels, wherein each of the ground truth labels describes a known class of the plurality of known classes. Some examples further include updating parameters of the neural network based on the multi-label loss.

Some examples of the method, apparatus, and non-transitory computer readable medium further include generating a feature embedding corresponding to each of the plurality of samples using the neural network, wherein the classification data is generated based on the feature embedding. In some examples, the classification data includes prediction logits, uncertainty measures, or both.

In some examples, the open-set metric indicates that the plurality of known classes does not characterize the set of samples to a threshold level. Some examples of the method, apparatus, and non-transitory computer-readable medium further include clustering the plurality of samples to obtain a plurality of clusters. Some examples further include selecting a cluster from the plurality of clusters based on the open-set metric, wherein the set of samples includes samples from the cluster.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include computing an activation logit value for each sample in the selected cluster. Some examples further include identifying a maximum activation logit value based on the activation logit value for each sample in the selected cluster, wherein the open-set metric is based on the maximum activation logit value. In some examples, the selected cluster minimizes the maximum activation logit value.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include identifying a class of the plurality of known classes. Some examples further include excluding the class from the plurality of known classes to obtain a reduced set of known classes. Some examples further include computing the open-set metric based on the reduced set of known classes.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include displaying the set of samples in an annotation interface. Some examples further include receiving annotation input via the annotation interface, wherein the training is based on the annotation input.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include identifying a shared label of the set of samples based on the annotation input, wherein the annotation is based on the shared label.

Some examples of the method, apparatus, and non-transitory computer-readable medium further include identifying a distinguishing label of the set of samples based on the annotation input, wherein the annotation is based on the distinguishing label.

FIG. 9 shows an example of a method for training a neural network according to aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control functional elements of an apparatus. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various substeps, or are performed in conjunction with other operations.

Supervised learning is one of three basic machine learning paradigms, alongside unsupervised learning and reinforcement learning. Supervised learning is a machine learning technique based on learning a function that maps an input to an output based on example input-output pairs. Supervised learning generates a function for predicting labeled data based on labeled training data consisting of a set of training examples. In some cases, each example is a pair consisting of an input object (typically a vector) and a desired output value (i.e., a single value, or an output vector). A supervised learning algorithm analyzes the training data and produces the inferred function, which can be used for mapping new examples. In some cases, the learning results in a function that correctly determines the class labels for unseen instances. In other words, the learning algorithm generalizes from the training data to unseen examples.

Accordingly, during the training process, the parameters and weights of the machine learning model described in the present disclosure are adjusted to increase the accuracy of the result (i.e., by attempting to minimize a loss function which corresponds in some way to the difference between the current result and the target result). The weight of an edge increases or decreases the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which a signal is not transmitted at all. In some examples, the nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is known as the input layer and the last layer is known as the output layer. In some cases, signals traverse certain layers multiple times.

The term “loss function” refers to a function that impacts how a machine learning model is trained in a supervised learning model. Specifically, during each training iteration, the output of the model is compared to the known annotation information in the training data. The loss function provides a value for how close the predicted annotation data is to the actual annotation data. After computing the loss function, the parameters of the model are updated accordingly and a new set of predictions are made during the next iteration.

At operation 905, the system generates classification data for a set of samples using a neural network trained to identify a set of known classes. In some cases, the operations of this step refer to, or may be performed by, a neural network as described with reference to FIGS. 2 and 4. According to an embodiment of the present disclosure, the training component is configured to train neural network 230 (shown in FIG. 2) using a multi-label loss (e.g., binary cross entropy “BCE”). The neural network is trained using a dataset of “known” classes. Embodiments of the present disclosure are not limited to a particular type of machine learning model as long as the neural network includes a classification head. In some examples, a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) is used for text classification. In another example, a multi-modal model is fine-tuned for document images.

In some cases, neural network 230 is run across an unlabeled dataset saving the prediction logits and feature embeddings from the classifier.

At operation 910, the system selects a set of samples for annotation from the set of samples using an open-set metric based on the classification data, where the annotation includes an unknown class. In some cases, the operations of this step refer to, or may be performed by, an open-set metric component as described with reference to FIGS. 2 and 4.

According to an embodiment, training involves the following steps. Step one is initializing simulation, e.g., randomly sample 100 documents and annotate them according to initial 10 classes. Step two is sampling data for known classes, e.g., choose 5 known classes and use active learning to label 100 documents for each class. Step three is sampling data to induce unknown classes, e.g., choose 5 clusters of 100 documents and induce new label for each cluster. Step four is training the model on labeled data and evaluate performance. Step five is repeating steps two to four for 50 cycles.

According to an embodiment, the training component selects X data samples for human labeling which increase the performance of the classifier for known classes. As a result, the most uncertain samples are selected. While doing label induction, the training component continuously improves the model for known classes to ensure stable training.

According to an embodiment, the training component selects samples x* using the following equation. However, embodiments of the present disclosure may use other uncertainty based active learning methods to select samples:

x
*

=

arg
⁢

min

x
⁢

ϵ
⁢

X

max

y
⁢

ϵ
⁢

Y

❘
"[LeftBracketingBar]"

p
⁡
(

y
❘
x

)

-
0.5

❘
"[RightBracketingBar]"

(
1
)

where X is the set of examples, Y is the set of known classes and p(y|x) is the model label predictions from the multi-label BCE loss. Details regarding computing an open-set metric will be described with regards to FIG. 10.

At operation 915, the system trains the neural network to identify the unknown class based on the annotation of the set of samples. In some cases, the operations of this step refer to, or may be performed by, a training component as described with reference to FIGS. 2 and 4. One or more embodiments of the present disclosure includes a document classification apparatus that can efficiently construct a rich document type classification model via training for a domain where the labels to be predicted are unknown at the beginning of the process.

The present disclosure includes methods that seamlessly integrate classifier development with data labeling and discovery of new labels. One or more embodiments of the present disclosure has an annotator (e.g., human-in-the-loop) that efficiently generates and organizes the rich type of label space driven by the current state of the model. Here, the label space is unknown a priori and can continue to increase the accuracy of the discovered topics to refine or expand the label types. Example experiments and evaluation demonstrate that the document classification apparatus described in the present disclosure can construct a rich document type classification model via learning. In some cases, the document classification model can be used for document cloud analytics.

One or more embodiments of the present disclosure can be used when a new classification model is needed, but its taxonomy or label space is unknown. In addition to document type classification, the present disclosure can be used in image tagging, object tagging, video tagging and actions, documents type tagging, clause tagging, and entity tagging, where there is a large but partially known label space (e.g., some labels are unknown). One or more embodiments of the present disclosure can also be used by users who share a large volume of data with an online service provider and need fine-tuned models, but where the tag space is partially unknown.

According to an embodiment, the document classification apparatus efficiently constructs a robust classifier in domains where the label space is not known and is configured to perform label induction in multi-label settings. The document classification apparatus depends on a supervised deep multi-label classification model to drive new class discovery. The document classification apparatus also depends on human-in-the-loop algorithm for jointly building a classifier while expanding the label space by selecting examples that are either uncertain or have low activation. The document classification apparatus can discover hierarchical or sub-classes using a top-down iterative expansion or comparative labeling.

FIG. 10 shows an example of a method for computing an open-set metric according to aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control functional elements of an apparatus. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various substeps, or are performed in conjunction with other operations.

At operation 1005, the system clusters the set of samples to obtain a set of clusters. In some cases, the operations of this step refer to, or may be performed by, a clustering component as described with reference to FIGS. 2 and 4.

According to an embodiment, the clustering component is used to prepare for performing the label induction to select samples that help build classifiers for unknown classes. The training component clusters the data into K clusters using the model feature embeddings. The training component may add a second set of clusters to the set using a pre-trained model (e.g., SentenceBERT, UniDoc, etc.). In some examples, K-center clustering is used to identify or determine outlier clusters.