150 may be a wireless network, a wireline network, or may be a combination of both wireless and wireline networks.

The service provided through the health care facility interface server 130, EMR interface/CDS system module 135, criteria subset selection server 140/AI model module 145 to provide medical necessity review criteria selection support may, in some embodiments, be embodied as a cloud service. For example, health care facilities may integrate their EMR systems/order systems with the medical necessity review criteria selection support service and access the service as a Web service. In some embodiments, the medical necessity review criteria selection support service may be implemented as a Representational State Transfer Web Service (RESTful Web service).

Although FIG. 1 illustrates an example communication network including a medical necessity review criteria selection system with automatic evidence highlighting, it will be understood that embodiments of the inventive subject matter are not limited to such configurations, but are intended to encompass any configuration capable of carrying out the operations described herein.

FIG. 2 is a block diagram of an AI model used in the medical necessity review criteria selection system of FIG. 1 in accordance with some embodiments of the inventive concept. As shown in FIG. 2, a training feature set 205 for training and/or initializing the AI model may include information associated with a plurality of input variables, which include, but are not limited to, encounter diagnoses, medications, lab tests, and procedures. A Natural Language Processing (NLP) embedding module 215 may be used to create embeddings for the input variables contained in the input feature set 205 along with the criteria subsets. An embedding is a learned continuous vector representation of a discrete variable. These input variables from the training feature set 205 may be categorical or discrete variables for which a continuous vector is generated. Similarly, the multi-layer neural network 235 is initialized or trained using the embeddings provided by the NLP embedding module 215 and the one-hot encoding module 220, which may be used to create embeddings for lab test information, for example. In some embodiments, the NLP embedding module 215 may comprise a bidirectional transformer, such as a Clinical Bidirectional Encoder Representations from Transformers (ClinicalBERT) bidirectional transformer. ClinicalBERT is trained on clinical notes/electronic health records and may capture qualitative relationships among clinical concepts in a database of medical terms.

FIG. 3 is a diagram of an initialized artificial neural network system 235 according to some embodiments of the inventive concept. As shown in FIG. 3, the artificial neural network 235 includes a plurality of node layers comprising an input layer, one or more hidden layers, and an output layer. In the example shown in FIG. 3, an input layer comprises five nodes or neurons 202a, 202b, 202c, 202d, and 202e and an output layer comprises three nodes or neurons 210a, 210b, and 210c. In the example shown, three hidden layers connect the input layer to the output layer including a first hidden layer comprising five nodes or neurons 204a, 204b, 204c, 204d, and 204c, a second hidden layer comprising five nodes or neurons 206a, 206b, 206c, 206d, and 206c, and a third hidden layer comprising five nodes or neurons 208a, 208b, 208c, 208d, and 208c. Other embodiments may use more or fewer hidden layers. Each node or neuron connects to another and has an associated weight and threshold. If the output of any individual node or neuron is above the specified threshold value, that node is activated, sending data to the next layer of the network. Otherwise, no data is passed along to the next layer of the network.

The number of nodes in the input layer 202a, 202b, 202c. 202d, and 202e may correspond to the number of variables in the training feature set 205 during training and the current input feature set during inference mode. In aggregate, the embedded vectors corresponding to the plurality of input variables may be viewed as a patient health record vector. The number of nodes in the output layer 210a. 210b, and 210c may correspond to the number of different criteria subsets for use in a medical necessity review that are identified and/or scored/ranked using the neural network 235. The subsets may be, for example, subsets of the InterQual criteria in some embodiments. While only three nodes 210a, 210b, and 210c are shown in FIG. 3, the number of output layer nodes would typically be much greater due to the large number of criteria subsets that are defined for performing medical necessity reviews. In aggregate, the embedded vectors of the different criteria subsets may be viewed as an aggregated subset vector.

As described above, the multi-layer neural network 235 relies on training data to learn and improve its accuracy over time. The multi-layer neural network 235 may be trained to identify and/or score criteria subsets for use in a medical necessity review based on input information associated with a patient's medical record. Historical patient medical records may be used in the training, knowledge base, and/or vocabulary for the artificial neural network 235, the NLP embedding module 215, and the one-hot encoding module 220. Once the various parameters of the multi-layer neural network 235 are tuned and refined for accuracy, it can be used, among other applications, to generate inferences or scores for criteria subsets for use in a medical necessity review in response to input data associated with a patient, including free-text queries. Such a multi-layer neural network 235 may also be trained to perform other tasks, such as, but not limited to, classify images, recognize and interpret speech, and cluster data, amongst other uses.

Each individual node or neuron may be viewed as implementing a linear regression model, which is composed of input data, weights, a bias (or threshold), and an output. Once an input layer is determined, weights are assigned. These weights help determine the importance of any given variable, with larger ones contributing more significantly to the output compared to other inputs. All inputs are then multiplied by their respective weights and then summed, i.e., a MAC operation. In FIG. 3, node or neuron 206a, for example, receives inputs corresponding to the outputs of nodes or neurons 204a, 204b, 204c. 204d, and 204c. These inputs are multiplied by their corresponding weights and summed at node or neuron 206a. Afterward, the output is passed through an activation function, which determines the output. If that output exceeds a given threshold, it activates the node by passing data to the next layer in the network. This results in the output of one node becoming the input of the next node. This process of passing data from one layer to the next layer is an example of a feedforward artificial neural network. Some embodiments of the inventive concept may provide a rectified linear unit (ReLU) activation function for use at one or more of the multi-layer neural network 235 nodes.

Returning to FIG. 2, once multi-layer neural network 235 is trained, the multi-layer neural network 235 may operate in inference mode based on a current input feature set for a current patient. The current input feature set may include information associated with one or more of the plurality of input variables used in the training feature set 205. The current feature set are embedded into a plurality of input variable vectors using the current feature embeddings module 225, which are then aggregated to provide a patient health record vector. The multi-layer neural network 235 may then process the patient health record vector to generate confidence scores for the different criteria subsets for use in a medical necessity review, which is represented by the neural network subset confidence scores 245. These confidence scores 245 may be termed similarity scores, which may be indicative of a similarity between the patient health record vector, which comprises an aggregation of the input variable vectors, and the aggregated subset vector, which is the aggregation of the vectors associated with the criteria subsets. The individual similarity scores may be based on the similarity between the patient health record vector and the individual medical criteria subset vectors from, for example, the output layer 210a, 210b, 210c of the multi-layer neural network 235. In some embodiments, cosine similarity score(s) may be generated between the patient health record vector and the aggregated subset vector and may be used to rank 250 the various criteria subsets in terms based on their probability of being relevant or applicable to the input information from the patient's medical record, i.e., the information associated with the input variables.

FIGS. 4-7 are flowcharts that illustrate operations for selecting criteria subsets for performing a medical necessity review with automatic evidence highlighting using the AI system of FIGS. 1 and 2 in accordance with some embodiments of the inventive concept. Referring now to FIG. 4, operations begin at block 400 where input information associated with a health record of a patient is received. This information is associated with a plurality of input variable tokens. The information may be, for example, one or more input variable values for the input variables of the training feature set 205 described above, which may correspond to the current input feature set of FIG. 2. The input information associated with the plurality of input variable tokens may be embedded using the current feature embeddings module 225 to generate a plurality of input variable token vectors, respectively, at block 405. In some embodiments, the current feature embeddings module 225 may comprise a bidirectional transformer, such as a ClinicalBERT bidirectional transformer, as described above with respect to the NLP embedding module 215. The plurality of input variable token vectors may be aggregated to generate a patient health record vector at block 410. At block 415, a ranking of one or more of the plurality of subsets of criteria used for determining the appropriateness of the care plan for the patient may be generated using an AI engine, such as, for example, the multi-layer neural network 235. To assist a case manager, for example, in understanding what features in the current input data set had the greatest effect in identifying criterion subset or criteria subsets for determining the appropriateness of the care plan, a ranking of one or more of the respective ones of the plurality of input variable tokens based on how much each of the plurality of input variable tokens contributed to the identification of each of the one or more criteria may be generated at block 420.

Referring now to FIG. 5, example operations of the multi-layer neural network 235 begin at block 500 where the plurality of input variable token vectors are assigned to the input layer nodes of the neural network 235. The plurality of criteria subsets for use in a medical necessity review is assigned to nodes of the output layer of the multi-layer neural network 235 at block 505. Referring now to FIG. 6, the nodes of the input layer of the multi-layer neural network 235 are initialized with the plurality of input variable token vectors at block 600. The plurality of criteria subsets is embedded using, for example, the NLP embedding module 215, at block 605 to generate a plurality of subset vectors, respectively. The nodes of the output layer of the neural network 235 are initialized with the plurality of subset vectors, respectively, at block 610. The plurality of subset vectors is aggregated to generate an aggregated subset vector at block 615. A similarity score between the patient health record vector and the aggregated subset vector may be generated at block 620 using the multi-layer neural network 235. The similarity score may be, for example, a cosine similarity score in some embodiments of the inventive concept. A plurality of similarity scores may be generated between the patient health record vector and the individual subset vectors, which are indicative of the probability that the subset criteria are relevant or applicable to the patient's health record when performing a medical necessity review as part of forming a care plan for the patient. When multiple criteria subsets are identified and ranked, a case manager may, for example, select one or more of the higher ranked criteria subsets in performing a medical necessity review to determine the appropriateness of the care plan for the patient.

But as described above, a case manager may want to understand what features in the input information had the greatest impact on the multi-layer neural network model's 235 identification and/or ranking of the one or more criteria subsets. This evidence identification may be referred to as feature attribution, which applies generally to any AI model. Specifically, feature attributions indicate how much each feature in the AI model contributed to any prediction, inference, categorization, recommendation, or the like. In some embodiments, the evidence highlighting through feature attribution may be provided using an IG methodology.

IG can be used to assign an attribution value to each input feature, which is indicative of how much the input contributed to the final prediction, inference, categorization, recommendation, or the like. When the input feature tokens are based on text identified through NLP, the goal may be to identify which segments of the text, e.g., sentences, have the most impact on the output of the AI model. The IG methodology begins with a baseline vector, which is not representative of any information that affects the output of the AI model. When the input is text, the baseline vector may correspond to the sentence start and sentence end tokens. A number of steps are defined for getting from the baseline token vector to the AI model output. In some embodiments, each step may correspond to an input variable token. For each step or input variable token, the model may output its prediction, inference, categorization, recommendation, or the like. This may provide an attribution value for each input feature, which is indicative for how much the feature influenced the overall result of the AI model.

The size of the steps referred to above in the IG methodology may be indicative of the magnitude of the change in gradient based on the difference between an input feature token vector and the baseline vector. The IG methodology, therefore, is applicable to AI models, such as multilayer neural networks that use an activation function that is differentiable.

Operations for applying the IG methodology to the multi-layer neural network 235 used for selecting criteria subsets for performing a medical necessity review with automatic evidence highlighting will be described hereafter with reference to FIG. 7. Operations begin at block 700 where a baseline vector is defined, which is not representative of any information that affects generating identification of the criterion using the AI model, e.g., multi-layer neural network 235. The baseline vector may be, for example, a vector based on the sentence start and sentence end tokens. A baseline gradient magnitude value of the AI model is determined for the baseline vector at block 705. A plurality of token gradient magnitude values for the AI models for the plurality of input variable token vectors, respectively, is determined at block 710. A plurality of gradient change magnitude values is determined at block 715 based on the plurality of differences between the plurality of token gradient magnitude values and the base line magnitude value, respectively. At block 720, a ranking of the respective ones of the plurality of input variable tokens based on the plurality of gradient change magnitude values is generated. A mapping between the input variable tokens and segments of the input information may be generated at block 725. For example, the input variable tokens may be individual words, codes, acronyms, or groups of such information, whereas the segments of input information may be sentences, paragraphs, medications, lab tests, and/or procedures. A ranking of the segments of input information may be generated at block 730 based on the mapping. This ranking of segments of input information, such as sentences or paragraphs in the patient's health record may provide a case manager with the evidence that supports the recommendation for one or more criteria subsets that can be used to determine the appropriateness of a care plan as part of a medical necessity review.

FIG. 8 is a block diagram of a data processing system 800 that may be used to implement the criteria subset selection server 140 of FIG. 1, in accordance with some embodiments of the inventive concept. As shown in FIG. 8, the data processing system 800 may include at least one core 811, a memory 813, an Artificial Intelligence (AI) accelerator 815, and a hardware (HW) accelerator 817. The at least one core 811, the memory 813, the AI accelerator 815, and the HW accelerator 817 may communicate with each other through a bus 819.

The at least one core 811 may be configured to execute computer program instructions. For example, the at least one core 811 may execute an operating system and/or applications represented by the computer readable program code 816 stored in the memory 813. In some embodiments, the at least one core 811 may be configured to instruct the AI accelerator 815 and/or the HW accelerator 817 to perform operations by executing the instructions and obtain results of the operations from the AI accelerator 815 and/or the HW accelerator 817. In some embodiments, the at least one core 811 may be an ASIP customized for specific purposes and support a dedicated instruction set.

The memory 813 may have an arbitrary structure configured to store data. For example, the memory 813 may include a volatile memory device, such as dynamic random-access memory (DRAM) and static RAM (SRAM), or include a non-volatile memory device, such as flash memory and resistive RAM (RRAM). The at least one core 811, the AI accelerator 815, and the HW accelerator 817 may store data in the memory 813 or read data from the memory 813 through the bus 819.

The AI accelerator 815 may refer to hardware designed for AI applications. In some embodiments, the AI accelerator 815 may include medical necessity review system functionality configured to provide a service for selecting one or more subsets of criteria used in determining the appropriateness of a care plan for a patient with automatic evidence highlighting. The AI accelerator 815 may generate output data by processing input data provided from the at least one core 815 and/or the HW accelerator 817 and provide the output data to the at least one core 811 and/or the HW accelerator 817. In some embodiments, the AI accelerator 815 may be programmable and be programmed by the at least one core 811 and/or the HW accelerator 817. The HW accelerator 817 may include hardware designed to perform specific operations at high speed. The HW accelerator 817 may be programmable and be programmed by the at least one core 811.

FIG. 9 illustrates a memory 905 that may be used in embodiments of data processing systems, such as the criteria subset selection server 140 of FIG. 1 and the data processing system 800 of FIG. 8, respectively, to facilitate operation of a system for selecting criteria subsets for use in a medical necessity review with automatic evidence highlighting. The memory 905 is representative of the one or more memory devices containing the software and data used for facilitating operations of the criteria subset selection server 140 as described herein. The memory 905 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash, SRAM, and DRAM. As shown in FIG. 9, the memory 905 may contain six or more categories of software and/or data: an operating system 910, a data pre-processor module 915, an AI model 920, a similarity module 930, a feature attribution module 935, and a communication module 940. In particular, the operating system 910 may manage the data processing system's software and/or hardware resources and may coordinate execution of programs by the processor.

The data pre-processor 915 may be configured to receive and organize the information corresponding to the input variables, e.g., the training feature set 205 and the current input feature set of FIG. 1 when the neural network 235 is in training mode and when the multi-layer neural network 235 is in inference mode, respectively. The AI model module 920 may comprise a neural network module 925. The neural network module 925 may be configured to perform one or more of the operations described above with respect to the neural network 235 and FIGS. 4-7. The similarity module 935 may be configured to implement similarity evaluation functionality, such cosine similarity functionality, including one or more of the operations described above with respect to FIGS. 2-7. The feature attribution module may be configured to perform one or more of the operations described above with respect to the multi-layer neural network 235 and FIG. 7. The communication module 940 may be configured to facilitate communication between the criteria subset selection server 140 of FIG. 1 and entities, such as providers, clinical researchers, and the like.

Although