Another type of medical codes is pharmaceutical codes, which are medical classification codes used to identify medications.

Clinical data collected in medical examinations in support of applications for life insurance are typically employed to assess the applicant's health, to confirm information included in the application, and to screen for illegal drug use. Much of the collected clinical data is also obtained from other sources during the application process, and clinical test results and answers to examination questions are typically checked for consistency with the other sources.

Clinical laboratory data are a point-in-time view into an individual's health. Underwriting ties various clinical data to all-cause mortality predictions and to specific causes of mortality. Clinical assessments based on collected blood and urine samples typically test the collected fluids to screen for dozens of indicators of diseases and conditions (health indicators). Examples of clinical assessment risk factors include HIV and AIDS; sexually transmitted diseases (STD); cholesterol, (including LDL and HDL) and triglycerides (e.g., as indicators of heart disease risk factors); hemoglobin A1C, fructosamine and glucose levels (e.g., as indicators of diabetes); creatinine, hemoglobin, and proteins (e.g., as indicators of kidney disease); and urine acidity (e.g., as indicator of impaired kidney function or diabetes). Typical medical examinations also screen for nicotine and cotinine in the urinalysis in order to determine tobacco usage. Additionally, clinical assessments may include biophysical examinations such as weighing the applicant and questioning the applicant, e.g., about lifestyle.

As used in the present disclosure, a “risk factor” may refer to any variable associated with a health outcome or state, such as a risk of disease, infection, and/or health-related event, e.g., a stroke, diabetes, heart attack, cancer, or death. Risk factors may be correlated with a health outcome or state and may or may not have a causal relationship with a health outcome or state. In underwriting, risk may be stratified through unfavorable risk factors (debits) and favorable risk factors (credits). In an embodiment, one debit signifies a predetermined increase in mortality risk of an individual relative to a standard population while one credit signifies a corresponding reduction in mortality risk relative to the standard population.

FIG. 1 shows a system architecture for an algorithmic MCRS underwriting application review system 100, also herein called MCRS underwriting system, of a sponsoring enterprise. The MCRS underwriting system 100 may be hosted on one or more computers (or servers), and the one or more computers may include or be communicatively coupled to one or more databases. The application review system 100 manages predictive modeling of mortality risk factors and risk scoring of applications for life insurance or other financial products, e.g., products of a sponsoring enterprise.

In an embodiment, a sponsoring enterprise for MCRS underwriting system 100 is an insurance company or other financial services company, which may be represented by insurance agents or advisors. In some cases, an insurance agent may be associated with only a single insurance provider, sometimes referred to herein as a “captive” insurance agent. In other cases, an insurance agent may be associated with several different insurance providers, sometimes referred to herein as an “independent” insurance agent or insurance broker. In various embodiments, a user (customer or customer representative) submits an electronic application via user device 170, and the electronic application is received by MCRS underwriting application review system 100 and assigned to an agent or advisor. In an embodiment, MCRS application review system 100 extracts information from the electronic application and stores the extracted information in a database in the custody of the sponsoring enterprise, MCRS Applications database 148.

MCRS analytical module 110 includes an analytical engine 114. Analytical engine 114 can be executed by a server, one or more server computers, authorized client computing devices, smartphones, desktop computers, laptop computers, tablet computers, PDAs, and other types of processor-controlled devices that receive, process, and/or transmit digital data. Analytical engine 114 can be implemented using a single-processor system including one processor, or a multi-processor system including any number of suitable processors that may be employed to provide for parallel and/or sequential execution of one or more portions of the techniques described herein. Analytical engine 114 performs these operations as a result of central processing unit executing software instructions contained within a computer-readable medium, such as within memory. In one embodiment, the software instructions of the system are read into memory associated with the analytical engine 114 from another memory location, such as from a storage device, or from another computing device via communication interface. In this embodiment, the software instructions contained within memory instruct the analytical engine 114 to perform processes described below. Alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to implement the processes described herein. Thus, implementations described herein are not limited to any specific combinations of hardware circuitry and software.

Enterprise databases 140 consists of various databases under custody of a sponsoring enterprise, including Medical Claims Codes database 142, Underwriting Decisions database 144, Applicant Profiles database 146, and MCRS Applications database 148. Enterprise databases 140 are organized collections of data, stored in non-transitory machine-readable storage. In an embodiment, the databases may execute or may be managed by database management systems (DBMS), which may be computer software applications that interact with users, other applications, and the database itself, to capture (e.g., store data, update data) and analyze data (e.g., query data, execute data analysis algorithms). In some cases, the DBMS may execute or facilitate the definition, creation, querying, updating, and/or administration of databases. The databases may conform to a well-known structural representational model, such as relational databases, object-oriented databases, and network databases. Example database management systems include MySQL, PostgreSQL, SQLite, Microsoft SQL Server, Microsoft Access, Oracle, SAP, dBASE, FoxPro, IBM DB2, LibreOffice Base, and FileMaker Pro. Example database management systems also include NoSQL databases, e.g., non-relational or distributed databases that encompass various categories: key-value stores, document databases, wide-column databases, and graph databases.

MCRS models 150 include MC Word Embedding Model 152 and Regression Model 156. In various embodiments, MC Word Embedding Model 152 was trained against a large medical classification codes data set to learn word embedding vectors. In an example, MC Word Embedding Model 152 was trained via an MC-provider data set 182 of ICD and CPT Medical Codes, encompassing on the order of 1 Million individuals.

In an embodiment, Regression Model 156 included a random forest model ensemble 132 of risk scoring module 130. The random forest model ensemble 132 was trained to predict UW Outcomes 134 including underwriter-assigned risk ratings and to predict Relative Mortality 136. In an embodiment, Relative Mortality 136 is an index including underwriting factors specific to risk class and sex. Training data included medical claims code data 142 and underwriting decisions data 144 in enterprise databases 140. In an example, training data included medical claim codes data and underwriting decision data for around 40,000 historical underwriting applicants of the sponsoring enterprise. The risk rating to be predicted may be derived from the underwriting decision data based upon standard quantitative risk ratings of a plurality of risk classes. In an embodiment, training data for the random forest predictive model also included cohort level demographic data for historical underwriting applicants stored in applicant profiles database 146 such as applicant issue age and sex.

In various embodiments, MC Word Embedding Model 152 includes Global Vectors (GloVe). GloVe is an unsupervised learning algorithm for obtaining vector representations for words developed at Stanford University. GloVe is trained to encode semantic relationships between words as vector offsets in the learned vector space. GloVe uses insight that co-occurrence ratios, rather than raw counts, are superior conveyors of word meaning. GloVe trains only on nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. GloVe relies on the potential of ratios of word-word co-occurrence probabilities for encoding word embeddings that can be encoded as vector differences. GloVe is essentially a log-bilinear model with a weighted least-squares objective. The GloVe training objective is to learn word vectors in which their dot product equals the logarithm of the words' probability of co-occurrence. This objective associates the ratios of co-occurrence probabilities with vector differences in the word vector space. GloVe has been demonstrated to create word vectors that perform well on both word analogy tasks and on similarity tasks and named entity recognition.

MC word embedding model 152 may include other word embedding algorithms, such as Bidirectional Encoder Representations from Transformers (BERT, Devlin et al.) and Generative Pre-Training (GPT, Radford et al.). These techniques are dynamic, e.g., can learn different meanings for the same word based on context. The context of MC word embedding model 152, however, is static in that each individual medical code has the same meaning across various contexts. For that reason, MC word embedding processes of the disclosure generally prefer static word embedding techniques such as GloVe.

Applicants have assessed several other static word embedding techniques besides GloVe for use as MC word embedding processes. Techniques assessed include word2vec, fastText, and svd2vec. Word2vec and fastText each have two implementations: continuous bag-of-words (CBOW) and skip-gram. Word2vec is a shallow neural network proposed by Mikolov et al. Word2vec either uses the context to predict the target word, or uses the target word to predict the context, depending on whether CBOW implementation or skip-gram implementation is employed. There is considerable overlap between information learned by word2vec and information learned by GloVe, but GloVe takes into account global word statistics, not just local context. fastText was published in a paper by Bojanowski et al. fastText also has continuous bag of words and skip-gram implementations, but also uses information from character-level n-grams and learns information such as prefixes and root words. svd2vec is a python implementation of the findings of Levy et al. including point wise mutual information (PMI) and singular value decomposition (SVD) methods.

MC lookup groupings module 126 executes a data transformation procedure ancillary to training MC word embeddings 124. As there are over 80 thousand ICD and 26 thousand CPT and HCPCS codes, MC lookup groupings module 126 implements a procedure for grouping similar codes together to increase coverage of “scorable codes.” MC lookup groupings system 126 processes infrequently or rarely appearing medical codes to enhance MC coverage with no detrimental loss of information, taking advantage of the hierarchical structure of medical claims codes.

In various embodiments, model outputs of MCRS models 150 include risk scoring information, also herein called risk ranks. Risk ranks may include, for example, quantitative risk scores, percentiles, binary risk outcomes, and risk classes. In an embodiment, risk ranks include the user's percentile within the score distribution for a population of general users, together with the score of the particular user. In an embodiment, risk scoring is a binary outcome, such as “pass” or “fail.” In an embodiment, risk scores define one or more bins as percentile ranges in a percentile distribution for a population of general users. In an embodiment, risk scoring ranks cases by the likelihood of belonging to one risk class or the other. In an embodiment, risk scoring determines a quantitative risk score, such as net number of points, for the user and translates this risk score into one of several coarse-grained risk classes. In various embodiments, MCRS underwriting system 100 transmits an offer to user device 170 via network 160 of model outcomes 164 based on a risk score determined by the models 150. In an embodiment, upon receiving an electronic application from user device 170, MCRS underwriting system 100 determines a risk rank representative of a mortality risk for the electronic application that classifies the electronic application into one of a plurality of risk groups based upon the risk rank. MCRS system then generates a user interface that displays on the user device 170 an offer based upon the one of the plurality of risk groups.

FIG. 2 shows execution steps of a computer-implemented MCRS underwriting method for processing an electronic application for financial product to generate an offer. The illustrative method 200 shown in FIG. 2 comprises execution steps 202, 204, 206, and 208. However, it should be appreciated that other embodiments may comprise additional or alternative execution steps, or may omit one or more steps altogether. It should also be appreciated that other embodiments may perform certain execution steps in a different order. Steps may also be performed simultaneously or near-simultaneously with one another.

At step 202, upon receiving an electronic application from a user device, a server retrieves medical claims code data identified with an applicant of the electronic application. In an embodiment, the server extracts medical claims code data identified with the applicant of the electronic application from the electronic application and stores the medical claims code data information in a database in the custody of the sponsoring enterprise. In an embodiment, the server stores information extracted from the electronic application in MCRS applications database 148.

At step 204, the server executes a predictive machine learning model that receives the medical claims codes data identified with the applicant of the electronic application and determines a risk rank representative of a mortality risk for the electronic application. In an embodiment, the predictive machine learning model is trained by inputting, for each of a plurality of applicant records associated with respective historical underwriting applications, medical claims codes data for the respective applicant record into a word embedding model. In an embodiment, the word embedding model includes a GloVe (global vectors) unsupervised NLP model. In an embodiment, the predictive machine learning model is trained by inputting, for each of a plurality of applicant records associated with the respective historical underwriting applications, the medical claims codes data and underwriting decision data for the respective applicant record into the word embedding model in combination with a regression model.

At step 206, the server classifies the electronic application into one of a plurality of risk groups based upon the risk rank determined at step 204. At step 208, the server generates a user interface that displays on the user device an offer based upon the one of the plurality of risk groups. In an embodiment of step 208, in addition to displaying the offer, the user interface displays explanations of MCRS underwriting model predictions that resulted in this offer.

FIG. 3 shows execution steps of a computer-implemented method for training a word embedding model and a regression model of an MCRS underwriting system. The illustrative method 300 shown in FIG. 3 comprises execution steps 302, 304, and 306. However, it should be appreciated that other embodiments may comprise additional or alternative execution steps, or may omit one or more steps altogether. It should also be appreciated that other embodiments may perform certain execution steps in a different order. Steps may also be performed simultaneously or near-simultaneously with one another.

At step 302, a processor trains a word embedding model to generate word embedding vectors representative of semantic relationships of medical claims codes data by inputting a corpus of the medical claims code data. In an embodiment, the processor trains a GloVe (global vectors) word embedding model, an unsupervised NLP model.

At step 304, the processor executes the trained word embedding model to generate summarized embedding coordinates for each of a plurality of applicant records by aggregating the high-dimensionality embedding coordinates corresponding to the medical claims codes data for each applicant record. In an embodiment, the processor executes a GloVe word embedding model to embed MC records including high dimensional medical classification codes into a Euclidean space. In an embodiment, the Euclidian space includes 35 embedding coordinates. The 35 embedding coordinates are aggregated from the code level to an applicant level. In an embodiment, for each applicant, minimum and maximum values of each coordinate are calculated over all of the applicant's medical classification codes.

At step 306, the processor trains a regression model by inputting underwriting decisions data associated with historical underwriting applications and word embedding vectors generated by the word embedding model for medical claims codes data associated with applicant records. In an embodiment, the regression model includes a random forest model. In an embodiment, the regression model is trained to determine a risk rank representative of a mortality risk by inputting underwriting decisions data associated with a plurality of historical applicant records.

In data intake of medical claims records, these records may be received in a form similar to that shown in table 1. The records consist of ICD-9, ICD-10, CPT, and HCPCS codes. ICD-9 is the 9th edition of the International Classification of Diseases. ICD-10 (the 10th edition) replaces ICD-9 and provides more granularity. Current Procedural Terminology (CPT) and Healthcare Common Procedure Coding System (HCPCS) are code systems referencing medical procedures.

TABLE 1

Example Medical Claims History Sample

Medical
Code

Code
Type
Description
Date

87635
CPT
“Infectious agent detection
2020 Oct. 1

by nucleic acid . . . [COVID-19] . . .

probe technique”

Z20.828
ICD-10
Contact with and (suspected) exposure
2020 Oct. 1

to other viral communicable diseases.

87804
CPT
Rapid Flu Test (Influenza A)
2020 Oct. 1

878804-59
CPT
Rapid Flu Test (Influenza B)
2020 Oct. 1

71048
CPT
Radiologic examination,
2020 Oct. 2

chest; 4 or more views

J20.9
ICD-10
Acute Bronchitis
2020 Oct. 2

ICD codes are MC codes between 4 and 7 digits in length. Each ICD code relates to a specific medical diagnosis. ICD codes are hierarchical in nature. Characters 1-3 of ICD codes denote category. For example, codes starting with category E relate to endocrine, nutritional, and metabolic diseases, and codes starting with categories E00-E07 are all related to the thyroid. Characters 4-6 record etiology, anatomic site, severity, and other details. Sometimes ICD codes include a 7th character for documenting extensions. Table 2 gives an example of hierarchical structure in characters 4-7 for a subset of radius fracture ICD codes. The example codes show increasing specificity from character 5 to characters 6-7. As a result of their hierarchical structure, ICD codes can be truncated and retain meaning.

TABLE 2

Example Subset of ICD Codes: Radius Fracture ICD Codes

Code
Description

S52.50
Unspecified fracture of the lower end of radius

S52.501
Unspecified fracture of the lower end of right radius

S52.501A
Unspecified fracture of the lower end

of right radius, initial encounter for

closed fracture

S52.501B
Unspecified fracture of the lower end

of right radius, initial encounter for open

fracture type I or II

S52.501C
Unspecified fracture of the lower end of

right radius, initial encounter for open

fracture type IIIA, IIIB, or IIIC

CPT and HCPCS codes are both five digit alphanumeric codes, each corresponding to a specific procedure. CPT codes are grouped numerically. For example, codes 10021 through 69990 are all used for surgeries. A subset of HCPCS codes are identical to CPT codes, and encode the same concepts. The remainder of HCPCS codes that are not included in the CPT system are used to identify products, supplies, and services not included in CPT codes.

While there are many positive aspects of medical claims data, these data also have some limitations that pose challenges in predictive modeling of mortality. One of the most significant challenges in using medical claims data is the practice of differential diagnoses within the healthcare industry. A differential diagnosis is a list of possible conditions or diseases that could be causing a patient's symptoms. When a patient is being diagnosed or treated by a physician, that physician may assign a differential diagnosis to that patient, even before the presence of that condition is confirmed. There may be several reasons for this, including the need to justify certain procedures, medications, diagnostic tests, etc., for billing and insurance purposes.

The fact that medical claims data often reflect differential diagnoses represents a significant challenge to use of medical claims data in algorithmic underwriting, since underwriting should only consider known medical impairments. Embodiments disclosed herein include feature selection techniques for selecting medical claim data that represent known medical impairments as features with largest contributions to mortality risk.

In an example of MCRS algorithmic underwriting, the system 100 acquired a large, de-identified dataset 182 including around 1 MM individuals from MC-Provider. In an embodiment, this dataset represents a research corpus. MC Word Embeddings module 124 used this dataset to train the GloVe model 152. The dataset contained all available clinical lab data, prescription data, medical claims data, demographics, HIT/no HIT status, and date of death. Model training focused on the ~1 MM individuals with medical claims information, including a total of ~900 MM procedural or diagnostic codes.

The training dataset had the following characteristics:

- ~20% of the individuals drawn from the insurance population and ~80% drawn from the general population with an indicator that denotes which population the cases are drawn from;
- Age distribution representative across the range from 20 to 89 years old;
- Minimum five-year medical claim history;
- Includes date of birth, lab entry date, gender, and date of death (if applicable);
- At least 90% of the records had hits across all three requested sources: labs, Rx, and medical claims. ~10% was missing one of these three sources.

To address the fact there are over 80 thousand ICD and 26 thousand CPT and HCPCS codes, MCRS model training incorporated an MC Lookup Grouping system 128 for grouping similar codes together to increase coverage of “scorable codes” with no detrimental loss of information. In handling infrequently or rarely appearing codes, the lookup grouping system enhances coverage by taking advantage of the hierarchical structure of medical claims codes. In an embodiment, certain low-level codes are combined through a mapping procedure that somewhat reduces granularity, while still preserving homogenous red/green medical label (rule) information for codes with medical underwriting rule designation. In an embodiment, relatively rare codes in the same hierarchical category and same medical rule are grouped together. In an embodiment, the lookup grouping system applies a maximum occurrence threshold to ensure codes with sufficient support in the dataset 182 are not mixed.

Table 3 includes a sample of lookup grouping. The lookup is generated by stepping through ordered codes. In an embodiment, codes with 6 or 7 characters (code_t) can be combined with adjacent codes (code_t+1) if frequencies of occurrence in the dataset 182 for code; and code_t+1are less than 500. In an embodiment, these codes may be combined if there is agreement as to red/green medical label (rule) information. In the example of Table 3, combining lower-level child codes avoids detrimental loss of information. Classification with respect to ‘side of the body’ and minute details typically does not contain useful information with respect to mortality risk, in comparison with information included in the 5 character parent code.

TABLE 3

Example MCR Code Lookups

Number

Code
Description
Rule
Lookup
Observed

S52.50
Unspecified fracture
NONE
S5250_ICD
7

of the lower end of

radius

S52.501
Unspecified fracture
NONE
S5250_ICD
10

of the lower end of

right radius

S52.501A
Unspecified fracture
FALSE
S52501A_ICD
1171

of the lower end of

right radius, initial

encounter for closed

fracture

S52.501B
Unspecified fracture
FALSE
S52501_ICD
20

of the lower end of

right radius, initial

encounter for open

fracturetype I or II

S52.501C
Unspecified fracture
FALSE
S52501_ICD
1

of the lower end of

right radius, initial

encounter for open

fracturetype IIIA,

IIIB, or IIIC

In an example, frequently observed codes such as S52.501A (N=1171) can stand on their own. In comparison, conceptually similar codes that have a small number of observations (e.g., S52.501B and S52.501C, with N=20 and N=1, respectively) can be combined in a single code (e.g., S52501_ICD).

Modeling parameters of the word embedding model 152 reduce the MC codes corpus to a usable code “vocabulary.” In an example, model training of GloVe MC word embedding results in a total of ~53,000 vectors, accounting for ~74,000 distinct ICD and CPT codes.

In an example, MCRS models 150 were trained on ~40,000 historical underwriting applications with claims information and complete underwriting outcomes. In an example, medical claims records 184 were obtained from a third party source, MC-Provider-2. Production data included ~21,000 unique diagnostic codes and ~8,000 unique procedural codes. For many historical underwriting applicants, the dataset included complete underwriting information such as final underwriting path and risk class that could be used in regression risk scoring. In an embodiment, training data was stored in enterprise databases 140 including medical claims code data 142, underwriting decisions data 144, and applicant profiles data 146.

FIG. 4 is a schematic diagram of a modeling process for MCRS algorithmic underwriting. This process applied distinct methods to the large dataset 182 of medical codes records and to datasets 144, 184 associated with historical underwriting applications. In the process 400 of FIG. 4, MRCS modeling includes three discrete steps: (1) generating 410 predictors in the form of code/word embedding representations 430, (2) regressing 440 on relative mortality to produce a risk score 450, and (3) evaluating 460 that risk score in a holistic underwriting simulation 480.

There are tens of thousands of unique diagnostic and procedural codes, hence one-shot encoding of these MCs becomes intractable. In order to address the challenges of high dimensionality while still retaining information about the medical codes and their relationship to each other, MCRS algorithmic underwriting uses word embeddings generated at a code level. Word embeddings are a natural language processing technique that numerically encodes the semantic meaning within words, via information learned from a corpus of documents. Training word embeddings on a specific corpus, such as medical claims records, results in information encoded in the embeddings specific to the context of that corpus.

In an embodiment, MCRS word embeddings 124 are trained on a corpus of person-level “documents” created from a large dataset 182 of medical classification codes. This document is a medical history for each of around 1 MM de-identified individuals. For each medical code, the word embedding technique learns the meaning based on the context the code is found in, and generates a vector to represent that code.

Glove is described in a paper arising from the Stanford University NLP lab, Jeffrey Pennington, Richard Socher, Christopher D. Manning, “GloVe: Global Vectors for Word Representation.” Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014. GloVe builds a co-occurrence matrix of the target word and the surrounding words in the context window (which is determined by the hyper parameter window size), and combines information gained from this co-occurrence matrix with global statistics to inform the embedding. GloVe takes as input a corpus of “documents”, learns word-level embedding representations, and outputs one vector for each word (code).

In an embodiment applying GloVe to MC word embeddings, documents are the chronological medical histories of individuals in the large medical claims records dataset 184. In applying GloVe to this dataset, codes may been translated where appropriate (ICD-9 to ICD-10). Codes may be mapped to groupings using the lookup structure 128. Table 4 shows an example of medical codes and their respective embeddings as generated by GloVe.

TABLE 4

Example Medical Codes and Embeddings Generated by GloVe

Code
dim1
dim2
dim3
. . .
dim33
dim34
dim35

E1339_ICD
−0.0906
−0.4499
−0.4219
. . .
−0.7063
0.3585
−0.1941

Z800_ICD
0.2317
−0.3187
0.3869
. . .
0.6724
1.4283
0.9279

C7900_ICD
−0.0637
−0.0469
−1.659
. . .
0.9493
1.4252
−0.5032

In applying GloVe to MC word embeddings, the GloVe model may include customized settings. In an embodiment, the GloVe model was customized using two settings in the GloVe framework: distance weighting and model construction. Distance weighting weighs co-occurrence counts inversely by the distance between words. While this setting is turned on by default, MCRS modeling set-up turned this setting off. Often, many MCs are recorded on the same day, and this setting was turned off to avoid inferring meaning from the order in which MCs are presented within a specific date.

GloVe model construction refers to the construction of the embedding itself. The default is to save the vectors as word plus context word weightings, without biases. In an example, MCRS modeling set-up selected the default setting in order to record information about the specific MC and its context but to remove any additional noise. In other embodiments, MCRS modeling set-up may select another model construction setting, such as a setting to save just the word vector, or a setting to save word plus context word with biases.

In an example, performance tests of the trained MCRS word embeddings model 152 were conducted to confirm that the GloVe MC word embeddings are meaningful in representing medical claims relationships. The t-distributed stochastic neighbor embedding (t-SNE) algorithm is a statistical method for visualizing high-dimensional data by giving each data point a location in a two or three-dimensional map. t-SNE constructs a probability distribution over pairs of high-dimensional objects such that similar objects are assigned a higher probability while dissimilar objects are assigned a lower probability. t-SNE then defines a probability distribution over the points in the low-dimensional map and it minimizes relative entropy between the two distributions with respect to the locations of the points in the map. As such, t-SNE is a useful technique for examining N-dimensional word embedding vectors.

FIG. 5 plots a small subset of codes 500 using t-SNE-produced x and y dimensions. Select diagnostic categories 510 are chosen to highlight both clustering behavior (sickle cell) and spread (diabetes). These diagnostic categories 510 are represented by shape of glyphs. Red/green medical labels (underwriting rules) 520 are represented via solid and hollow glyphs. In the case of dental caries, forearm laceration, and sickle cell, the plot highlights close neighbor behavior among the group. Codes related to diabetes and malignant neoplasms show a somewhat different, more sprawling pattern, but this can reasonably be attributed to the greater number of codes within the topic and their respective variations in disease presentation.

FIG. 6 is a t-SNE plot 600 for visualization of GloVe word embeddings for diabetes codes. As in FIG. 5, diagnostic categories are represented by shape of glyphs, and red/green medical labels (underwriting rules) are represented via solid and hollow glyphs. When diabetes codes are split more specifically based on the context or diagnosis, more specific clusters emerge. More diffuse clusters can reflect a wide diversity of symptoms that may be associated with each version of diabetes.

In the GloVe paper, Pennington et al. employ the word analogy task: a is to b as c is to d. A good word embedding model may produce word representations that will find a proper match for d. In the GloVe paper, d is chosen using the cosine similarity of word vectors for a, b, c, and d, where d is the singular corresponding code. The MC word embedding model of the present disclosure does place a strict requirement on word analogy performance, however, since the MC corpus is built from the ‘unnatural’ language of medical codes.

- A=‘J4530’ #Mild persistent asthma,
- B=‘J4550’ #Severe persistent asthma, uncomplicated
- C=‘N182’ #Chronic kidney disease, stage 2 (mild)
- D=‘N184’ #Chronic kidney disease, stage 4 (severe)

x
=

w_B
-
w_A
+
w_C

y
=

w_D

- compute_cosine_similarity (x=x, y=w_D)

The conceptual framework of the formulas above posits that by removing A from B, the algorithm has removed “asthma” and reduced the concept to an entity related to “severity”. The algorithm adds the concept of “mild kidney disease”, and generates a result (x) that encodes “severe”+“kidney disease.” An ideal result would occur in a search of all codes in which each token is treated as Y, if surfaces N184 were the vector with highest cosine similarity. In fact, as seen in Table 5, N184 is the sixth highest in rank while the top codes are all kidney disease related. While not perfect, this result is good evidence that the learned embeddings are positioning similar codes closely within vector space.

TABLE 5

Example Cosine Similarity for Vectors Matching:

“Severe” + “Kidney Disease”

Cosine

Code
Description
similarity

N181
Chronic kidney disease, stage 1
0.884

N182
Chronic kidney disease, stage 2 (mild)
0.875

N183
Chronic kidney disease, stage 3 (moderate)
0.846

E1129
Type 2 diabetes mellitus with other
0.839

diabetic kidney complication

I129
Hypertensive chronic kidney disease with
0.836

stage 1-4/unspecified chronic kidney disease

N184
Chronic kidney disease, stage 4 (severe)
0.83

In the MCRS models 150, computing word embedding vectors functions as forms of data transformation in which individual medical codes are transformed into computer-readable, semantically meaningful representations relative to other codes in the trained MC word embedding model 152. A secondary regression model 156 serves the purpose of relating medical code histories with mortality information. Underwriting outcomes 134 in the form of discrete risk class are available for about 40,000 historical underwriting applications. In an embodiment, regression modeling defines a target quantity as a relative mortality index 136 comprised of underwriting factors specific to risk class and sex.

In preparation for regression modeling, code level predictors are aggregated for use as modeling inputs. In an embodiment, multi-code vectors for code level predictors are shown in Table 6.

TABLE 6

Example of Multi-Code Vectors for Code Level Predictors

Policy
Code
dim 1
dim 2
. . .
dim 35
Description

12345
J00
−0.1266
0.0045
. . .
0.0069
Acute

nasopharyngitis

[common cold]

J20.9
−0.4553
0.2042
. . .
−0.8988
Acute bronchitis,

unspecified

R07.8
−0.4094
−0.3873
. . .
−0.7943
Other chest pain

The predictors of Table 6 are aggregated by reducing multi-code vectors to one row for prediction, as shown in Table 7.

TABLE 7

Example Reduction of Multi-Code Vectors to One Row for Prediction

Policy
dim1 min
dim2 min
. . .
dim35 min
dim1 max
dim2 max
. . .
dim35 max

12345
−0.4553
−0.3873
. . .
−0.8988

−0.1266
. . .
0.0069

For individual i, the regression model learns relative mortality y using covariates xi, including issue age, sex, and n-dimensional word embeddings. The n-dimensional word embeddings covariate includes all possible codes j in the medical history of individual i (d1_ij, . . . , dn_ij). As shown in Table 7, these code level predictors are aggregated such that one-row-per-code becomes one-row-per-individual (d1_i, . . . , dn_i). Different aggregation techniques to go from many codes per-person to just one are considered a crucial part of the embedding use scheme. As such, model development tested different formulations. In model performance testing, between using min+max alone versus min+max+average, modeling error indicated the simpler min+max as the preferred setting for the MCRS risk scoring model.

The model takes the form:

relative
⁢

mortality

∼

age
+

aggregated
⁢

word
⁢

embeddings

The aggregated word embeddings take the form:

- dim1_min, . . . , dim_n_min, dim1_max, . . . , dim_n_max

In an embodiment, given that mortality prediction lacked longitudinal data with death status, the MCRS model constructs a modeling objection using a Relative Mortality index 136 based on underwriting risk class and sex. This index ties to relative mortality of a subpopulation. The effects of age and sex were largely marginalized out of the end ‘risk score’ through transformation of real-valued predictions to score percentiles conditioned on age+sex cohort (where insured age is binned by 5-year buckets—[17,22.5)=20, [22.5,27.5)=25 . . . , etc. In an embodiment, the MCRS risk scoring model may include other common underwriting model variables as covariates, such as BMI from the application.

As used herein, the phrase “predictive model” might refer to any class of algorithms that are used to understand relative factors contributing to an outcome, estimate unknown outcomes, discover trends, and/or make other estimations based on a data set of factors collected across prior trials. In an embodiment, the predictive model may refer to methods such as logistic regression, decision trees, neural networks, linear models, and/or Bayesian models. Random forest ensembles are well-established as standard methods for many classification and regression modeling tasks. In an example, random forest model was selected for the MCRS regression procedure based on performance comparisons between XGBoost, GLM, GLMNet, and decision tree models indicating its suitability in predicting relative mortality given a subset of ICD code probabilities as predictors.

One method of interrogating meaning encoded in the word embeddings uses the vectors to produce single-code scores from the trained regression model. Given each code's embeddings (dim 1, dim 2, . . . , dim 35) and an age arbitrarily set to 40, in which each code is repeated twice for gender=‘F’ and ‘M,’ the relative predictions might reveal what the final regression model considers low and high risk. MCRS model performance testing indicated that embeddings may allow the regression model to ‘interpret’ code relationships and associate them with risk patterns among data it has seen in training.

To assess the performance of each regression model, MCRS underwriting defines a metric that weights actual low-risk applicants and predicted low-risk applicants more heavily. Modeling is primarily concerned about individuals who receive a low-risk score from MCRS and are likely to be instant-eligible, or with individuals who truly have low relative mortality. To account for this, the MCRS analytical module 110 computes a weight, w_i, for each individual i such that a low prediction or true label is associated with a higher weight. This causes the error associated with these individuals to have a larger penalty. In an embodiment, MCRS analytical module 110 multiplies weight by 1000 to normalize the weights to a scale of roughly 1-12:

w
i

=

1

min
⁡
(

y
l

^

,

y
i

)

*
1
⁢
0
⁢
0
⁢
0

This weight is multiplied by the difference between the prediction and the label, and the total error is taken as the square root of the mean of these weighted differences. The present disclosure refers to this as the mortality-impact weighted root mean-squared error (WRMSE).

e
w

=

∑

i
⁡
(

y
l

^

-

y
i

)

*

w
i

N

Example: Given two distinct sub-models, each with various hyperparameters and structural decisions to be made regarding respective training data, model development took an iterative approach. Experimentation with corpus structure included how to build corpus (e.g., all codes, completely distinct, distinct by date) and granularity of ICD codes (e.g., truncate globally at 4 or 5 digits, retain all information, or use reflexive look-up structure). Experimentation with embedding construction included embedding types (e.g., GloVe, word2vec, fastText, svd2vec); GloVe parameters (distance weighting, model construction); and parameter search (e.g., min count, embedding dimensions, window size). Experimentation with regression modeling included granularity of regression labels, removing non-medical declines from training data, and aggregation to reduce multiple vectors per person to one tidy row. Miscellaneous experimentation included hyperparameter grid searches and multi-code per person aggregation.

Experiments as to optimal model settings for building word embeddings explored various GloVe hyperparameters. FIG. 7 is a graph of mortality-weighted RMSE for GloVe hyperparameter experiments. The upper plot features vocab count of 7, while the bottom plot features vocab count of 10. Plots are faceted by window size, and vector dimension is on the X axis.

- Minimum vocab count: (threshold to apply to occurrence frequency to ensure sufficient support for embeddings.) In final GloVe embeddings, codes appearing lessthan 7 times in the MC corpus are not used.
- Window size: (how large a window GloVe considers in building base co-occurrence statistics.) In final GloVe embeddings, the symmetric window considers 15 tokens to the left and 15 tokens to the right of the target code.
- Embedding dimension: (how many dimensions are included in a given word vector.) In the literature, 200-300 or more are standard. However, MC word embeddings experiments found there is little gain between a 35- and 200-dimensional set of vectors. Final GloVe embeddings selected 35 dimensions.

Experiments tested different aggregation techniques to go from many-codes-per-person to just one. Between using min+max alone versus min+max+average, the error consistently points to the simpler min+max as the correct setting for the MCRS model. Besides a more parsimonious model, use of min+max has added benefits in model explainability. Given embedding dimensionality of 35 and min/max scheme, the final MCRS model has 70 vector-based features that require explanations.

Example: Once GloVe settings were tested, further experiments searched for optimal random forest (RF) settings. Two main hyperparameters were considered for the RF regression model: minimum node size and number of trees.

As shown in the graph FIG. 8, a grid search for random forest parameters minimum node size (x axis) by number of trees (faceted subplots) compared top two candidate models (1, 2) with an underperforming model (x). Model error is on the Y axis. A final value of 100 was selected for minimum node size (minimum number of observations in terminal node). A final value of 1500 was selected for number of trees (individual learners in the ensemble). 1500 represented a mid-point RF setting for number of trees. FIG. 9 is a graph of model error for RF number of trees, faceted by final candidates for min node size parameter.

Explanations model: In an embodiment, MCRS algorithmic underwriting includes a post-processing step to produce explanations of model scores, as the learning algorithm is a “black box” random forest with no inherent interpretability to its structure. In various embodiments, the system and method of the present disclosure employed SHAP values, a feature attribution method that draws on game theory. Scott M. Lundberg, Su-In Lee, “A unified approach to interpreting model predictions,” NIPS'17 Proceedings of the 31st International Conference on Neural Information Processing Systems, Pages 4768-4777; Dec. 4-9, 2017. In SHAP values, the marginal contribution of a certain feature represents how much the “presence” of that feature changes the outcome of the function, given the “presence” of certain other features.

In comparison to other algorithmic life underwriting models, the MCRS model presents some unique characteristics. MCRS model covariates themselves are already an abstraction (medical claims represented through word embedding vectors), which renders them ill-suited to clustering into “interpretable groups”, or even interpreting on their own. The MCRS explanations algorithm solves this problem by producing SHAP contributions for all individual features, then labelling and summarizing them. This approach takes advantage of the min/max aggregation scheme by reappointing dimension codes such as dim1_min and dim1_max with codes that “activated” this dimension.