No aspect, component, element, structure, act, step, function, instruction, and/or the like used herein should be construed as critical or essential unless explicitly described as such. In addition, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

As used herein, the terms “communication” and “communicate” may refer to the reception, receipt, transmission, transfer, provision, and/or the like of data (e.g., information, signals, messages, instructions, commands, and/or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and/or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and/or send (e.g., transmit) information to the other unit. This may refer to a direct or indirect connection that is wired and/or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and/or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments, a message may refer to a network packet (e.g., a data packet and/or the like) that includes data.

As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and/or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and/or the like), a personal digital assistant (PDA), and/or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

As used herein, the terms “issuer,” “issuer institution,” “issuer bank,” or “payment device issuer” may refer to one or more entities that provide accounts to individuals (e.g., users, customers, and/or the like) for conducting payment transactions, such as credit payment transactions and/or debit payment transactions. For example, an issuer institution may provide an account identifier, such as a primary account number (PAN), to a customer that uniquely identifies one or more accounts associated with that customer. In some non-limiting embodiments, an issuer may be associated with a bank identification number (BIN) that uniquely identifies the issuer institution. As used herein “issuer system” may refer to one or more computer systems operated by or on behalf of an issuer, such as a server executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

As used herein, the term “merchant” may refer to one or more entities (e.g., operators of retail businesses) that provide goods and/or services, and/or access to goods and/or services, to a user (e.g., a customer, a consumer, and/or the like) based on a transaction, such as a payment transaction. As used herein “merchant system” may refer to one or more computer systems operated by or on behalf of a merchant, such as a server executing one or more software applications. As used herein, the term “product” may refer to one or more goods and/or services offered by a merchant.

As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.” Reference to “a server” or “a processor,” as used herein, may refer to a previously-recited server and/or processor that is recited as performing a previous step or function, a different server and/or processor, and/or a combination of servers and/or processors. For example, as used in the specification and the claims, a first server and/or a first processor that is recited as performing a first step or function may refer to the same or different server and/or a processor recited as performing a second step or function.

As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa®, MasterCard®, American Express®, or any other entity that processes transactions. As used herein “transaction service provider system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction service provider system executing one or more software applications. A transaction service provider system may include one or more processors and, in some non-limiting embodiments, may be operated by or on behalf of a transaction service provider.

As used herein, the term “uncertainty score” refers to a metric that represents the uncertainty (or certainty) and/or confidence (or lack of confidence) of a classification output of a model. An uncertainty score may be presented in positive or negative terms (e.g., certainty/confidence or lack thereof).

Provided are systems, methods, and computer program products for determining uncertainty from a deep learning classification model. Non-limiting embodiments result in an uncertainty score that may be input into a decision model for efficient and intelligent decision making in numerous applications. Moreover, non-limiting embodiments provide for the optimization of neurons of an artificial neural network (ANN) classification model during training, resulting in faster and more efficient computations compared to existing sampling-based training methods.

Non-limiting embodiments provide a faster and more computational efficient approach to determining and quantifying the uncertainty of a model as compared to existing techniques. Both training and run-time execution of the model are made more efficient through use of non-limiting embodiments described herein. Moreover, non-limiting embodiments provide for a pipeline that can be provided as a service that automates uncertainty determinations upon request from a client user.

Referring now to FIG. 1, illustrated is a diagram of a system 1000 for determining uncertainty from a deep learning classification model 102. As illustrated in FIG. 1, the system 1000 includes a client device 100 (e.g., a computing device) in communication with the deep learning classification model 102. The model 102 may be stored and executed by one or more server computers, as an example, or may be local to the client device 100. In non-limiting embodiments, the model 102 may be executed by a transaction service provider and offered to client computing devices as a service.

The model 102 may be trained based on input training data 104, which includes features and corresponding classifications (e.g., labels) for each data record. The model 102 may be pre-trained and/or may be trained in response to the client device 100 or another computing device inputting the training data 104. Once the model 102 is trained, the client device 100 communicates an input 101 to the model 102. In non-limiting embodiments, the input may include an account profile, an authorization request message, transaction data, and/or the like to be classified (e.g., labeled) by the model 102. The model 102 outputs a classification 105 (e.g., a label) and an uncertainty score 106. The outputs 105, 106 may be used as input to a decision engine 107, which may include a computing device and/or software application configured to output an action 112 to be performed based on the inputs 105, 106. In non-limiting embodiments, the decision engine 107 may be part of and/or executed by the client device 100. In other non-limiting embodiments, the decision engine 107 may be remote from the client device 100 and part of and/or executed by a server computer.

In some examples, with continued reference to FIG. 1, if the uncertainty score satisfies a threshold (e.g., is equal to, less than, and/or greater than a threshold value), as determined by the decision engine 107, the classification 105 and uncertainty score 106 may be input to a decision model 108. In some examples, the decision model 108 may include logic that determines to use the classification 105 output by the model 102. In other examples, the decision model 108 may include a separate model, including but not limited to a separate machine-learning model such as a deep learning model and/or any other model for dynamically determining an action 112 to be performed based on the inputs 105, 106.

It will be appreciated that an uncertainty score 106 may reflect uncertainty in a number of ways, such that a high uncertainty score 106 may indicate greater uncertainty or less uncertainty than a relatively lower uncertainty score 106. The uncertainty score may be within any defined range of values, such as 0-1, 1-100, and/or the like. If the uncertainty score fails to satisfy the threshold, the decision engine 107 may determine the action 112 to be performed based on rules data 110. For example, the rules data 110 may include merchant-specified rules, issuer-specified rules, transaction service provider-specified rules, and/or the like. The rules data 110 may include, for example, a predetermined action specified by a merchant and/or issuer for all uncertain results (e.g., an uncertainty score satisfying a threshold associated with a high degree of uncertainty).

In non-limiting embodiments, the deep learning classification model 102 may be trained to determine an uncertainty score for the classification (e.g., label) it outputs. This may be done through directly optimizing specific neurons of an ANN used in the deep learning classification model 102 during training. In some non-limiting embodiments, a Beta distribution algorithm may be utilized for training as described herein. In some non-limiting embodiments, a Gaussian logits algorithm may be utilized for training as described herein.

FIG. 4 shows a diagram of a Beta distribution algorithm according to non-limiting embodiments. In non-limiting embodiments in which the Beta distribution algorithm is utilized, two neurons of the ANN for the deep learning classification model 102 may be trained to represent two parameters (α, β) of the Beta distribution of probabilities that characterizes the predicted probability p, where p is treated as a random variable rather than a deterministic value. The probability is then used in a loss function for training. In this manner, deep learning is used to model the predicted probability as a random variable such that it has a distribution of probabilities, and the information represented by the distribution of probabilities may be used to construct the uncertainty score. This is illustrated with the following equation:

f
⁡
(

p
;
α

,
β

)

=

p

α
-
1

(

1
-
p

)

β
-
1

Beta
⁢

(

α
,
β

)

⁢

where
⁢

Beta
⁢

(

α
,
β

)

=

Γ
⁡
(
α
)

⁢

Γ
⁡
(
β
)

Γ
⁡
(

α
+
β

)

.

The predicted probability p is used to determine an uncertainty score (e.g., confidence interval) for a predicted classification. Thus, for each new input 101 to the model 102, the corresponding two parameters of the Beta distribution may be inferred, and the predicted probability and uncertainty score (e.g., confidence interval) may be determined based on these inferred parameters.

In non-limiting embodiments, the confidence interval is used to quantify the uncertainty score. For example, a predicted probability p_imay be calculated as:

p
^

i

=

α
i

α
i

+

β
i

where the variance is represented by:

v
ˆ

i

(

α
i

,

β
i

)

=

α
i

⁢

β
i

(

α
i

+

β
i

)

2

⁢

(

α
i

+

β
i

+
1

)

.

The predicted classification (e.g., predicted label) is represented by:

ŷ_i=I({circumflex over (p)}_i>0.5)

In the above representation, 0.5 is a specified threshold (e.g., cutoff) such that, in an example where the confidence interval is [0.45, 0.55], the interval contains the value of 0.5 and indicates uncertainty, and in an example where the confidence interval is [0.7, 0.9], the interval does not contain the value of 0.5 and indicates certainty.

The predicted confidence interval for the classification output may then be determined with the following equation:

p
^

i

-

q

α
2

⁢

v
^

i

,

p
^

i

+

q

α
2

⁢

v
^

i

FIG. 5 shows a diagram of a Gaussian logits algorithm according to non-limiting embodiments. In non-limiting embodiments in which the Gaussian logits algorithm is used, rather than a Beta distribution algorithm, the uncertainty score may be determined by quantifying values for the neurons on the second-to-last layer (e.g., the logit layer prior to the predicted probability layer) of the ANN for the model 102. In such embodiments, the values may be quantified as a Gaussian (e.g., normal) distribution with two parameters μ, σ. The model 102 may be trained to learn these two parameter values for each set of input training data. An uncertainty score (e.g., confidence interval) is first determined for the parameters of the logit layer. The uncertainty score for the predicted probability (e.g., the final score output of the model 102) may then be determined as a sigmoid activation function of the uncertainty score for the parameters of the logit layer. For example, the logit z_ifor each data element can be obtained by re-parametrization where

z
i

=

1
L

⁢

∑
j

u
i

+

σ
i

⁢

ε
ij

,

where ϵ_ij˜N(0, 1) and L represents a number of Monte Carlo samples. In non-limiting embodiments, for simplicity, the value L=1 may be used, although it will be appreciated that various values may be used.

In non-limiting embodiments, the confidence interval for the logit is represented as

z
i

±

q
α

2

⁢

σ
i

.

Thus, the confidence interval for the estimated probability is: (z_i±qα/2*σ_i), where α is sigmoid activation function.

In non-limiting embodiments, the loss function used for training the model 102 has two parts L=L₁+L₂, where L₁is regular cross entropy and L₂may be represented by the following equation:

-

(

∑
i

λ
1

⁢
log
⁢

σ
i

+

λ
2

(

3
-

σ
i

)

⁢

I
⁡
(

σ
i

>
3

)

)

In non-limiting embodiments, the two-part loss function penalizes the terms of a close to zero (0) (e.g., lower than a threshold), which enforces the uncertainty into the network, and penalizes larger values of a (e.g., higher than a threshold) to reduce variabilities.

In non-limiting embodiments, the confidence interval is used to quantify the uncertainty. For example, a predicted probability {circumflex over (p)}_imay be calculated as:

p
^

i

=

α
i

α
i

+

β
i

where the variance is represented by:

v
^

i

(

α
i

,

β
i

)

=

α
i

⁢

β
i

(

α
i

+

β
i

)

2

⁢

(

α
i

+

β
i

+
1

)

.

The predicted classification (e.g., predicted label) is represented by:

ŷ_i=I( [Image Omitted] >t),

where t is a threshold (e.g., cutoff) for predicted probability.

The predicted two-sided confidence interval for the classification output may then be determined with the following equation:

p
ˆ

i

(

α
i

,

β
i

)

±

q

1
-

A
2

⁢

v
^

i

(

α
i

,

β
i

)

,

where

q

1
-

A
2

is a value of

(

1
-

A
2

)

percentile of a standard normal distribution, and A is the confidence level (e.g., indicating the probability that a statistical estimate in the sample is also true for the population). In this example, the proposed uncertainty measure may be represented as:

| [Image Omitted] −t|/

Referring now to FIG. 2, a flow diagram is shown for a method for determining uncertainty from a deep learning classification model. The steps in FIG. 2 are for exemplary purposes only and it will be appreciated that additional, fewer, and/or a different order of steps may be used in non-limiting embodiments. At step 200, a deep learning classification model is trained with labeled data. As explained herein, two or more parameters of two neurons of the ANN of the deep learning classification model are trained for quantifying classification uncertainty. In non-limiting embodiments in which a service is provided to a client computer for classifying data, the training data (e.g., features and corresponding labels) may be provided by the client computer prior to step 200 being executed.

With continued reference to FIG. 2, at step 202, an input may be processed with the deep learning classification model. As an example, a service may allow for a client computer to send a request (e.g., via an Application Programming Interface (API) or other like interface) with data to be classified to a server computer. The classification may seek to classify (e.g., label) data, such as classifying transaction data and/or user data as being approved or declined for a particular process. The deep learning classification model outputs a classification (e.g., label) for the inputted data and an associated uncertainty score at step 204.

At step 206, it is determined whether the uncertainty score satisfies a threshold (e.g., is equal to, less than, and/or greater than a threshold). As an example, if the uncertainty of the classification is greater or equal to a threshold (or if a corresponding confidence score is less than a threshold), the method may proceed to step 210 in which an action is determined based on one or more rules. For example, the rules may be preconfigured by an entity (e.g., an issuer or a merchant) to specify an automated action and/or decision logic to execute if the classification lacks confidence. The action may be based on a value (e.g., a transaction value), the degree of uncertainty, and/or any other preconfigured logic. In some examples, the method may proceed from step 210 back to step 200 and the model may be trained with the result of step 210. In some non-limiting embodiments, at step 206, if the uncertainty of the classification is greater or equal to a threshold (or if a corresponding confidence score is less than a threshold), the method may proceed to step 200 for re-training of the model without executing step 210.

If the classification is determined to be certain enough at step 206 (e.g., if the uncertainty is below a threshold, if a confidence score exceeds a threshold, and/or the like), the method proceeds to step 208 and an action may be determined based on a decision model. In non-limiting examples, the decision model may be an acceptance of the output of the deep learning classification model (e.g., the label output of the deep learning classification model may be maintained and correspond to an action).

Non-limiting embodiments of the system 1000 may be employed to process transaction data (e.g., individual transactions, groups of transactions, and/or the like) in an electronic payment processing network. For example, the deep learning classification model 102 may be used for stand-in processing (STIP) in which a transaction service provider or another entity processes and authorizes transactions in place of an issuer system that may be offline or temporarily unavailable. In such cases, STIP transactions may be classified and automatically authorized or rejected in accordance with a decision model