Once the vectors have been processed by the algorithms 108A-108N, the data are then provided to the decision logic 110 of the system 100. The decision logic 110 can use various methods for engaging in document review.

In one method, as illustrated in FIG. 2, the decision logic 110 (see FIG. 1) employs an equation (V=w1*v1+w2*v2) that serves as a weighted combination of two vectors v1 and v2. These vectors could represent certain features or scores extracted from the documents. The weights w1 and w2 are responsible for determining the significance or relevance of each vector in relation to a final decision score V. Given the constraint w1+w2=1, it ensures that the combined influence of these two vectors sums up to unity.

An aspect of the decision logic process is the minimization of the squared error between the computed result score and the ground truth. The ground truth here refers to the actual categorization of the document as responsive or non-responsive, possibly determined by human reviewers/machines or a predefined benchmark. By minimizing the squared error, the model's predictions are refined to align closely with this ground truth, enhancing its accuracy and reliability.

Furthermore, result scores for both responsive and non-responsive documents are aggregated. This sum provides a holistic view of the relevance of the entire batch of documents. The weight w1 is then calculated as a ratio of these sums. By doing so, the model dynamically adjusts the significance of the vector v1 based on the collective relevance of the entire document set. This ensures that the decision logic takes into account both individual document scores and the overarching trends in the entire batch, optimizing the categorization process.

In the context of the previously discussed document review system, this decision logic method offers a sophisticated approach to automate categorization. By combining feature vectors, minimizing errors with actual categorizations, and dynamically adjusting weights based on aggregate scores, the model can effectively categorize documents as responsive or non-responsive, enhancing the efficiency and accuracy of the review process.

Another example decision logic method, as illustrated in FIG. 3, includes fusing two distinct scoring vectors, v1 and v2. The methodology relies on a weighted combination of these two vectors to arrive at a final score, V. The equation V=w1*v1+w2*v2 elucidates this, where w1 and w2 are the weights assigned to the scores in vectors v1 and v2, respectively. A critical constraint here is that the sum of weights w1 and w2 is strictly bounded to 1, ensuring that the weights act as relative importance factors and don't skew the result disproportionately.

A feature of this approach is the way the weight w1 is determined. By summing up the list positions for documents categorized as responsive or non-responsive, the method offers a dynamic way to calculate w1. This weight is further refined using a ratio of softmax-normalized sums, a technique employed in machine learning to convert raw scores into probabilities. This ensures that the weights derived are both balanced and proportionally representative of the underlying data.

However, the adaptability of this framework involves the introduction of a neural network. By training this network on the input scores from v1 and v2, along with a set ground truth y, the system learns the optimal weights ‘w1’ and ‘w2’. Neural networks can discern intricate patterns and relationships in vast datasets, making the decision logic robust and adaptive. This approach exemplifies the merging of heuristic-based decision-making with data-driven insights, catering to both structured and unstructured data scenarios.

FIG. 4 illustrates an example method of the present disclosure for enhancing document ranking in an electronic discovery system. The method begins in step 402 by obtaining a first vector of scores from a designated first algorithm. Each score within this vector is linked to a specific document found in a predefined set of documents. This vector essentially encapsulates the assessment or rating of every document as perceived by the first algorithm.

Subsequently, in step 404, a second vector of scores is acquired from a distinct second algorithm. Similar to the first vector, each score in this second vector correlates to a document within the same aforementioned set of documents. This second vector offers an alternative perspective or evaluation of the documents, as gauged by the second algorithm.

In addition to the scores from the algorithms, the method includes a step 406 of receiving what are termed “ground truth labels.” These labels distinctly indicate whether a given document is deemed “responsive” or “non-responsive.” Such labels provide a concrete, definitive classification for each document, serving as a reference or benchmark against which algorithmic assessments can be compared or validated.

With the input data in place, the method proceeds to a step 408 to compute weights specifically for the purpose of blending or merging the scores from the first and second vectors. The determination of these weights is not arbitrary; instead, they are derived from a systematic training process, ensuring that they are optimized for the most accurate and effective blending of the two score vectors.

Leveraging the previously calculated weights, the method combines the scores from the first and second vectors in step 410. This blending process yields a final ranking score for each document in the set. It's a synthesis of evaluations from both algorithms, fine-tuned by the weights, to produce a more holistic and refined assessment for each document.

To conclude the process, the method generates a sorted list of all the documents from the set, in step 412. The arrangement or order of this list is determined by the final ranking scores, with documents possessing higher scores indicating greater relevance. This list, therefore, offers a prioritized view of the documents based on their combined assessments from both algorithms and the weighting process.

In the described method, the first algorithm employed is identified as a Support Vector Machine (SVM). This algorithm plays a role in generating the initial vector of scores that evaluate the documents in the given set. Also, the second algorithm that's leveraged in this process is a Document Relation Engine (DRE). This engine offers an alternative evaluation of the same set of documents, providing a distinct vector of scores.

The training process, used for determining the weights for blending the scores from both algorithms, makes use of a supervised learning algorithm. This algorithm harnesses both the scores and the ground truth labels to estimate the optimal weights. A more specific variant of the supervised learning algorithm used in this training process is a multi-layer perceptron neural network. This neural network facilitates a deeper and more nuanced estimation of the weights.

Before delving into the weight calculation, the method involves an extra step of normalization. The scores present in both the first and second vectors undergo a normalization process, aligning them to a predetermined range of values. Ground truth labels play a role in the method, serving as benchmarks. These labels use a simple binary representation; a value of one for responsive documents and zero for those that are non-responsive.

The final ranking score for each document can be computed through multiple methods. One approach involves minimizing the squared error between the resulting score and the ground truth score, ensuring the closest approximation to the benchmark. An alternative approach for calculating the final ranking score involves summing the scores for responsive and non-responsive documents separately. Once these summations are complete, a ratio of the two sums is computed to arrive at the final score. Yet another method to determine the final ranking score takes into account the relative positions of the documents in both the first and second vectors. In this method, documents are assigned higher positions based on their assessed relevance.

For those utilizing position-based ranking, a softmax function can be employed. This function normalizes the sum of the positions, ensuring they fall within a specific range, and is particularly useful when calculating the weights.

FIGS. 5-9 collectively illustrate example schematics and flow diagrams of embodiments of the present disclosure. In enhancing e-discovery through Technology Assisted Review (TAR) with predictive coding, an approach has been introduced that significantly improves the efficiency and accuracy of document sorting by their responsiveness. As noted above, e-discovery workflows have relied on the application of a singular prediction algorithm, such as a Support Vector Machine (SVM), to calculate a responsive score for each document. This score is primarily determined based on prior human review decisions, which assess the relevance and responsiveness of documents to specific legal matters.

The embodiments disclosed involve the simultaneous application of multiple prediction algorithms. By operating in parallel, these algorithms each contribute unique perspectives on document responsiveness, thereby encompassing a broader analysis spectrum. To synthesize the outputs from these algorithms into a cohesive scoring system, a combination layer is introduced. This layer is used in integrating the various scores, deriving optimized responsive scores for each document in the dataset.

The combination layer incorporates weight parameters, which play a role in the amalgamation process. These weights are not arbitrarily assigned; instead, they are estimated based on a comprehensive analysis of available human review decisions. By examining the patterns and outcomes of these decisions, the system intelligently adjusts the weight parameters, ensuring that the combined responsive scores accurately reflect the documents' relevance. This methodology presents an advancement in TAR, offering a more nuanced and effective tool for prioritizing documents in e-discovery.

In more detail, FIG. 5 illustrates a schematic for an e-discovery Technology Assisted Review (TAR) with predictive coding aimed at re-ranking documents to determine their relevancy. The system is predicated on leveraging multiple algorithms to rank a collection of documents and employing a decision logic that assigns weights to each algorithm's output to calculate a final ranking score.

The predictive coding process starts with input from previously reviewed documents, which provide document vectors to the system. These vectors serve as training data for the predictive algorithms and are indicative of human-reviewed outcomes, labeled as ‘Responsive’ or ‘Not responsive.’ This ground truth data is essential for the model training phase, where it informs the algorithm of the characteristics that distinguish responsive documents from non-responsive ones.

Algorithm 1, designated as a Support Vector Machine (SVM), is one of the primary algorithms used in this process. The SVM is responsible for generating a vector of scores based on the model it develops during its training phase. These scores are reflective of the likelihood that each document in the set is responsive, as determined by the SVM's classification logic.

Similarly, Algorithm 2, identified as a Document Relation Engine (DRE), undergoes a parallel training and classification process. It also produces a vector of scores, which may be based on different criteria or document features compared to the SVM. The DRE's distinct approach to document classification provides a complementary perspective on the data set. The system is designed to accommodate additional algorithms, as indicated by the placeholder for Algorithm N.

The vectors of scores from each active algorithm (SVM, DRE, etc.) are then processed by the decision logic. This logic layer applies weighted voting to these inputs, considering the human review decisions as a basis for tuning the weights. The decision logic synthesizes the information, yielding a result vector of scores that reflect a consensus ranking from the combined algorithms.

The output, a result vector of scores, represents the re-ranked documents, ordered by their calculated relevance to the query or legal case. This final score vector is used to prioritize the documents for subsequent review stages, thereby enhancing the efficiency of the e-discovery process.

FIG. 6 illustrates a decision logic model for weighted sums, which is part of an e-discovery TAR system. This model integrates the output of multiple algorithms to produce a result vector of scores that ranks documents based on their predicted relevance.

Inputs to the model are two vectors of scores, labeled as v1 and v2. These vectors represent the individual rankings from separate algorithms, and they are to be combined using the model's decision logic. The decision logic employs a mathematical formula to calculate a weighted sum of the two vectors, V=w1v1+w2v2, subject to the constraint that w1+w2 equals 1. This ensures the final result vector V is a balanced combination of the two input vectors.

The decision logic determines the position of each document k within the ranked lists generated by each algorithm (sorted by v1 and v2). The number of reviewed documents is denoted by N, and the positions p1 and p2 for each document in their respective ranked lists are calculated by subtracting the document's rank from N.

The model refines the weights w1 and w2 using already reviewed documents, which serve as a reference for the weighted sum of document list positions. Two sums are computed: one for documents deemed responsive (q1{circumflex over ( )}resp) and another for non-responsive documents (q1{circumflex over ( )}neg), each sum aggregating the positions of documents as ranked by v1. Equivalent sums (q2{circumflex over ( )}resp and q2{circumflex over ( )}neg) are computed for the list sorted by v2.

These sums are then used to calculate weights w1 and w2, using a ratio of the sums for responsive and non-responsive documents, normalized by the softmax function. The softmax function here is used to convert the weighted sums into a probabilistic distribution, ensuring that the weights are bounded between 0 and 1 and sum to 1. The output of the decision logic is the result vector of scores V. This result vector represents the final ranked list of documents, ordered by their combined scores from both algorithms, adjusted by the estimated weights.

FIG. 7 illustrates a decision logic model based on voting, used within an e-discovery TAR system for predictive coding and re-ranking of documents. This model is predicated on the mathematical integration of outputs from two distinct scoring algorithms to deduce a final result vector of scores that effectively ranks documents by relevance.

The model begins by accepting two vectors of scores as inputs: ‘Vector of scores v1’ and ‘Vector of scores v2’. Each vector represents the ranking outputs from separate algorithms, based on their assessments of the documents. A weighted sum approach is used which is described by the equation V=w1v1+w2v2, constrained by the requirement that the sum of weights w1 and w2 must equal 1. This ensures the final score vector V is a normalized blend of the input vectors.

The process includes a feature for estimating the weights w1 and w2, which is based on the vote per document for the algorithm with the better matching rank. This is illustrated through the equations for Q1 and Q2, which are summations of the differences in positions for each document k between the two ranked lists, modulated by the Heaviside step function H. The function H counts the vote for an algorithm when its ranking is better than the other. The weights w1 and w2 are then calculated as a ratio of Q1 to the sum of Q1 and Q2, with w2 being the complement of w1 to maintain the total weight as 1.

The ground truth data for responsive documents is used to inform and adjust the weighting process, ensuring that the final result vector of scores V is reflective of both the algorithmic assessments and the actual document relevance as determined by human review. The output of the model is the ‘Result vector of scores V’, which represents the re-ranked list of documents according to their calculated relevance, ready for review prioritization.

FIG. 8 illustrates a decision logic model employing the least squares estimation technique within an e-discovery TAR system. This model is engineered to fuse the outputs of two scoring algorithms to generate a result vector of scores for document ranking.

Two vectors of scores, v1 and v2, are input into the decision logic model, each corresponding to the scores assigned to documents by separate algorithms. The decision logic then combines these vectors using a weighted sum formula, V=w1v1+w2v2, adhering to the constraint that the sum of the weights w1 and w2 is equal to 1. This constraint ensures that the final result is a balanced combination of both algorithms' scores.

To refine the weights w1 and w2, the model applies a least squares estimation approach, leveraging already reviewed documents as a dataset for optimization. The model seeks to minimize the sum of the squares of the differences between the predicted scores and the actual scores, as determined by the ground truth. This is achieved by taking the derivative of the sum with respect to w1 and setting it to zero, which provides the optimal value of w1 that minimizes the error. Consequently, w2 is deduced by subtracting w1 from 1.

Ground truth data, representing the actual responsiveness of documents as determined by prior review, is used to guide and validate the estimation process. The resulting vector of scores, V, is the output of the model, representing the re-ranked list of documents based on their combined scores from both algorithms, optimized by the least squares estimation.

FIG. 9 illustrates a decision logic model leveraging a neural network, specifically a Multi-Layer Perceptron (MLP), for the purpose of training weight values within an e-discovery TAR system. This decision logic model is employed to determine the optimal combination of scores from two distinct document-ranking algorithms to generate a final result vector of scores, V, for document relevancy ranking.

In this model, the system ingests two score vectors, v1 and v2, as inputs, corresponding to the assessments from two different algorithms. The decision logic then employs a weighted sum approach, articulated by the equation V=w1v1+w2v2, with the constraint that w1 and w2 must sum to 1. This ensures the final score vector is a properly normalized amalgamation of the input score vectors.

To estimate the weights w1 and w2, the model utilizes an MLP neural network. The MLP comprises two input nodes, which correspond to the input vectors v1 and v2, hidden nodes that facilitate the abstraction and interaction of features, and a single output node with a sigmoid activation function. The sigmoid activation function is chosen for its ability to map the output values between 0 and 1, making it suitable for binary classification tasks such as document responsiveness.

The neural network is trained using input vectors and ground truth data, where the ground truth represents the actual responsiveness of documents—marked with w1=1 for responsive documents and w1=0 otherwise. The training is conducted via gradient descent, a method that iteratively adjusts the weights to minimize the error between the network's predicted output and the actual output.

The result of this training is a set of optimized weights w1 and w2, which are then used to compute the final result vector of scores V. This vector represents the re-ranked list of documents, arranged according to their computed relevance, as determined by the integrated assessments of both algorithms and refined through the neural network's learned weights.

FIG. 10 illustrates another related method for an e-discovery process, the method begins with feeding an annotated seed set of documents into the algorithms 1000. This seed batch consists of documents that have already been reviewed and categorized, providing a baseline of ‘ground truth’ for the algorithms to learn from. These annotations can include labels marking each document as responsive or non-responsive to the legal matters in question. The seed set serves as the training data for the algorithms, allowing them to identify patterns and features that correlate with document relevance.

Once the seed set is ingested, the algorithms proceed to compute a ranked list of documents 1002. This ranking is based on the likelihood of document responsiveness as inferred from the training data. The algorithms apply learned weights to the features extracted from the documents, scoring each one accordingly. The outcome is a prioritized list, with documents most likely to be relevant to the case at the top.

The next phase involves picking the top N documents from this ranked list to form a batch for further review 1004. The selection of N is strategic; choosing too few may slow down the discovery process, while too many could overwhelm the review team with potentially irrelevant documents.

The correctness of the algorithms' rankings is then measured by comparing the selected batch against the ground truth 1006. Each document from the batch is reviewed to determine if it is indeed responsive or non-responsive. This step is used to validate the effectiveness of the algorithms' predictive capabilities.

Subsequent to the correctness measurement, the documents are annotated according to the established ground truth 1008. This annotation involves marking each document with its correct classification based on a thorough review. These annotated documents are then fed back into the algorithms, further refining their predictive models 1010.

This iterative process is repeated until M responsive documents are identified, where M represents a predefined threshold 1012, such as 80% of all available responsive documents for a selected issue. The test result, or the measure of efficiency, is the number of documents that have been annotated to reach this threshold of M responsive documents. This metric reflects the TAR system's effectiveness and the efficiency of the review process, highlighting the number of documents that needed to be reviewed before achieving the desired level of discovery.

In sum, the algorithm combination adds robustness to e-discovery TAR as it is able to dynamically switch to the one algorithm that better matches the past (human) review decisions. The choice of selection algorithms can be guided according the preferences of one skilled in the art. To be sure, the proposed selection methods are extendable to N algorithms.

Example Use Cases

In the complex world of legal proceedings, time is of the essence, and the need to sift through mountains of documents can be overwhelming. This technology revolutionizes legal document review and e-discovery processes. When a new case is opened, all relevant documents are scanned and fed into the system. The AI-powered system meticulously analyzes each document, applying a combination of Support Vector Machines (SVM) and Document Relation Engine (DRE) algorithms.

The system's first step involves categorizing documents as responsive or non-responsive, thereby efficiently filtering out irrelevant information. Once the initial categorization is complete, the technology goes a step further by identifying the most pertinent documents based on past human decisions and user-defined criteria.

Legal professionals are presented with a prioritized list of documents, ensuring that they focus their efforts on the most crucial pieces of evidence. This not only accelerates the e-discovery process but also minimizes the risk of missing critical information. The end result is a streamlined, cost-effective, and accurate legal document review that empowers legal teams to make informed decisions.

In the healthcare industry, the importance of quick access to accurate patient data cannot be overstated. The technology is a game-changer in the analysis of medical records, from patient histories to treatment plans. When medical records are uploaded into the system, it employs its AI capabilities to extract and categorize vital information.

The system can recognize patient demographics, diagnoses, prescribed medications, and treatment timelines, transforming unstructured data into a structured, actionable format. Healthcare professionals benefit from an intuitive platform that facilitates rapid decision-making, enabling them to provide more efficient and personalized patient care.

Doctors, nurses, and administrators no longer need to sift through voluminous paperwork. Instead, they can access patient information instantly, leading to better patient outcomes and improved hospital operations.

Financial institutions and auditing firms face the formidable task of examining countless financial documents for compliance, fraud detection, and financial analysis. This technology offers a robust solution to expedite the process and enhance accuracy.

When financial reports, invoices, and receipts are uploaded, the system employs its algorithms to extract critical data points, such as transaction amounts, dates, and payees. It can identify discrepancies or irregularities in financial statements, potentially signaling fraudulent activities.

By automating data extraction and analysis, financial professionals can focus their efforts on higher-level tasks, such as strategy development and risk management. The system ensures that financial documents are thoroughly reviewed, reducing the likelihood of human error and oversight.

FIG. 11 is a diagrammatic representation of an example machine in the form of a computer system 1, within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In various example embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a portable music player (e.g., a portable hard drive audio device such as a Moving Picture Experts Group Audio Layer 3 (MP3) player), a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The computer system 1 includes a processor or multiple processor(s) 5 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), and a main memory 10 and static memory 15, which communicate with each other via a bus 20. The computer system 1 may further include a video display 35 (e.g., a liquid crystal display (LCD)). The computer system 1 may also include an alpha-numeric input device(s) 30 (e.g., a keyboard), a cursor control device (e.g., a mouse), a voice recognition or biometric verification unit (not shown), a drive unit 37 (also referred to as disk drive unit), a signal generation device 40 (e.g., a speaker), and a network interface device 45. The computer system 1 may further include a data encryption module (not shown) to encrypt data.

The drive unit 37 includes a computer or machine-readable medium 50 on which is stored one or more sets of instructions and data structures (e.g., instructions 55) embodying or utilizing any one or more of the methodologies or functions described herein. The instructions 55 may also reside, completely or at least partially, within the main memory 10 and/or within the processor(s) 5 during execution thereof by the computer system 1. The main memory 10 and the processor(s) 5 may also constitute machine-readable media.

The instructions 55 may further be transmitted or received over a network via the network interface device 45 utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). While the machine-readable medium 50 is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals. Such media may also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like. The example embodiments described herein may be implemented in an operating environment comprising software installed on a computer, in hardware, or in a combination of software and hardware.