305A-N to use features from a corresponding feature subset 210B as input (e.g., using historical training records). For example, if a feature subset 210B includes features “A,” “B,” and “C,” the security system may use training records (each indicating a set of attributes or features of the potential attack, as well as a ground truth classification or value indicating whether it was a true threat) to train a threat classifier 305 that considers only attributes “A,” “B,” and “C” (potentially excluding one or more other features).

Once the threat classifiers 305 have been thusly trained, a set of security logs 310 are evaluated using the threat classifiers 305 in order to score each classifier and, thereby, to generate a score for each feature subset 210B. In some embodiments, the security logs 310 can generally include similar data to the logs used in training the threat classifiers, but the sets of data may not overlap. For example, of a broader collection of security logs, some subset may be used as a training set, while another subset is used as a testing or evaluation set.

In some embodiments, each security log 310 is processed by each of the threat classifiers 305. The scoring component 225 can then label each security log 310 based on the threat classifier(s) 305 that accurately predicted whether the log corresponded to a true threat. This may include, for example, labeling each with the feature subset(s) 210B that resulted in a correct inference. In at least one embodiment, these labels are used to define the scores 315. For example, the scoring component 225 may rank the feature subsets 210B based on the number (or percentage) of security logs 310 that are accurately predicted by the corresponding classifier 305.

In some embodiments, prior to generating the scores 315, the security system first trains another machine learning model to act as a type classifier (e.g., type classifier 110 of FIG. 1). For example, the security system may use the security logs 310 as input, and use the feature subset(s) 210B that accurately classified the log as target output to refine the type classifier. In this way, the type classifier learns to route input logs to the proper threat classifier 305.

In some embodiments, the scoring component 225 can then process a second testing set of security logs using the type classifier, and subsequently processing each using the assigned threat classifier 305. This can allow the scoring component 225 to determine, for each threat classifier 305, how well it performs for potential threats that are within its local region of the feature space. That is, while the initial round of processing security logs 310 using the trained threat classifiers 305 does not distinguish between types of threats and processes all threats using all models, this second round targets each security log to the proper threat classifier 305, which can result in a more accurate understanding of the accuracy of each threat classifier 305.

In one embodiment, the scoring component 225 can then use these accuracies (determined using the second testing set of logs) to define the scores 315. For example, the scoring component 225 may score the threat classifiers 305 based on its accuracy (e.g., the percentage of input records that were accurately classified). For example, if a first threat classifiers 305A is more accurate, on its corresponding set of security logs 310, than a second threat classifier 305B is on its own corresponding set of security logs 310, the scoring component 225 may score the first threat classifier 305A more highly.

In at least one embodiment, the scores 315 can additionally be determined based on the size local region each threat classifier occupies in the feature space (e.g., the diversity of threats that each can accurately classify). For example, if the type classifier determines that a first threat classifier 310A should be used to process J security logs 310 and that a second threat classifier 310B should be used to process K security logs, where K is less than J, the scoring component 225 may assign the first threat classifier 305A a higher score, because it can (likely) be used to process a larger region of the space (and therefore, a larger number and diversity of potential threats).

As discussed above, these scores 315 are then assigned to the feature subsets 210B. That is, each feature subset 210B is assigned a score based on the performance of the corresponding threat classifier 305 (with respect to accuracy, generalness, and the like). These scores 315 can then be used to eliminate the worst-performing feature subsets 210B. In this way, the security system can iteratively generate better feature subsets that are better fit to classify newly-received security logs.

FIG. 4 is a flow diagram illustrating a method 400 for training machine learning models using genetic programming to improve threat detection. In some embodiments, the method 400 is performed by a security system, such as security system 105 of FIG. 1.

At block 405, the security system identifies a set of features that can be used to classify potential security threats. For example, the security system may identify the features 205 of FIG. 2. In some embodiments, these features are manually defined (e.g., by an administrator or other user). In some embodiments, some or all of the features may be automatically learned (e.g., by analyzing prior security threats). As discussed above, in an embodiment, this set of features may generally include some features that are not helpful for classifying some types of threats as real or false, even if they are helpful for classifying some other types of threats. That is, a given feature in the set of features may be useful for one more types of threats, but not useful for one or more other types of threats.

At block 410, the security system generates a set of feature subsets. In some embodiments, this may correspond to the feature subsets 210A of FIG. 2. In at least one embodiment, these initial feature subsets are defined with at least some element of randomness. For example, for each feature subset, the security system may randomly select N features from the pool of features (identified at block 405). In an embodiment, the number of feature subsets and the number of features in each subset may be specified by a user or administrator.

At block 415, the security system modifies the feature subsets using one or more genetic programming techniques. In some embodiments, as discussed above, this may include a crossover operation (also referred to as a mating operation). In some embodiments, the genetic programming techniques can include a mutation operation, as discussed above. Generally, the genetic programming techniques involve modifying one or more of the feature subsets and/or adding one or more new feature subsets with at least some element of randomness. These random changes can allow the security system to find more efficient and optimized feature subsets for particular types of threats.

At block 420, the security system selects one of the feature subsets from the modified set (generated in block 415). In embodiments, this selection can be performed using a wide variety of criteria, as all of the subsets will be similarly operated on. Additionally, although the illustrated example depicts a sequential and iterative process for conceptual clarity (where each feature subset is selected and evaluated in turn), in some embodiments, some or all of the feature subsets may be evaluated in parallel.

At block 425, the security system scores the selected feature subset. In some embodiments, as discussed above, each feature subset is scored based on how well it can be used to accurately classify potential security threats, how diverse the set of threats it accurately classifies is (e.g., how many different types), and the like. One example method for scoring the feature subsets is discussed in more detail below with reference to FIG. 5.

At block 430, the security system determines whether there is at least one additional feature subset that has not yet been scored. If so, the method 400 returns to block 420. If all of the feature subsets in the modified set have been scored, the method 400 continues to block 435.

At block 435, the security system selects one or more of the best feature subsets, based on their associated scores. In some embodiments, the security system identifies the top M feature subsets. In some embodiments, rather than identifying the top M to retain, the security system identifies the bottom L to discard. In an embodiment, the number of feature subsets to retain or discard can be defined by an administrator. In at least one embodiment, the security system discards feature subsets to ensure that the set of feature subsets is the same size as the initial set (generated at block 410).

At block 440, the security system determines whether feature selection is complete. Generally, this can include evaluation of a variety of termination criteria, such as determining whether a defined number of genetic programming and scoring iterations have been completed, determining whether a defined amount of time or computing resources have been used, determining whether the current set of (best) feature subsets are sufficiently accurate, determining whether the feature space is fully covered (e.g., whether all, or a sufficient percentage, of the testing security logs can be accurately classified), and the like.

If the feature selection process is not complete, the method 400 returns to block 415, where the security system applies one or more genetic programming techniques to the current set of feature subsets (selected in block 435). If the security system determines that the feature selection process is complete, the method 400 continues to block 445.

At block 445, the security system trains a set of threat classifiers based on the selected set of feature subset(s). In some embodiments, as discussed above, the threat classifiers are classification models that are trained to receive a specific subset of features from a security log (corresponding to a potential security threat), and output a classification indicating whether the security log is a true threat or a false alarm. In some embodiments, this training is performed using a set of training security logs, each labeled to indicate whether it is a true threat. In some embodiments, these threat classifiers are trained during the scoring operations. In at least one embodiment, therefore, the security system can retain and use these pre-trained classifiers for the feature subsets that were selected, rather than training new ones.

At block 450, the security system trains a type classifier based on the selected feature subsets (assuming multiple subsets were selected). In some embodiments, as discussed above, the type classifier is a classification model trained to receive a set of features (e.g., the features identified in block 405) for input security logs (which correspond to potential threats), and to identify which subset of features (from the set selected in block 435) should be used to classify the potential threat. As discussed above, this training can be performed using the set of training security logs which were annotated, during the scoring process, to indicate which subset of features performed best for each security log. In at least one embodiment, the security system can re-use these annotations and type classifier from the scoring. In some embodiments, the security system can re-annotate the logs (e.g., by performing the scoring operation again) to account for the fact that some of the feature subsets from the last scoring process were discarded.

After the type classifier and threat classifiers are trained, the security system can deploy them for use in analyzing received security logs to classify them appropriately. In some embodiments, these runtime security logs are generated or provided by some other threat detection system. For example, an automated threat detection system may identify potential threats, generate a security log for each, and provide these logs to the security system for classification.

FIG. 5 is a flow diagram illustrating a method 500 for scoring machine learning models and feature subsets to improve threat detection. In some embodiments, the method 500 is performed by a security system, such as security system 105 of FIG. 1. In some embodiments, the method 500 provides additional detail for block 425 of FIG. 4.

At block 505, the security system receives a set of labeled security logs. In some embodiments, as discussed above, each of these security logs can generally correspond to a potential security threat that was identified, and the label can indicate whether the potential threat was a true security threat or a false alarm. Each security log is associated with a set of features (such as the geographic origin of the potential threat, event names associated with the potential threat, signatures, IP addresses, and the like) that can be used to classify the threat.

At block 510, the security system selects one of the feature subsets that are being scored. Generally, this selection can be performed using a variety of criteria, as all of the feature subsets will be similarly processed. Additionally, though the illustrated example depicts a sequential iterative process for conceptual clarity, in some embodiments, some of all of the feature subsets may be processed in parallel.

At block 515, the security system trains a threat classifier based on the selected feature subset. As discussed above, this may generally include training a classifier to receive the selected subset of features (from the labeled security logs) as input, and to generate an output indicating whether the log represents a true threat. By using the labeled logs, the weights and parameters of the threat classifier can be iteratively refined (e.g., using batch gradient descent or stochastic gradient descent) to generate more accurate inferences. In one embodiment, the training process includes generating an output classification based on an input security log (e.g., based on the relevant subset of features). This output can then be compared against the ground truth label to compute a loss (e.g., cross-entropy loss), which is back propagated through the threat classifier to refine the internal parameters (e.g., using gradient descent).

At block 520, the security system determines whether there are additional feature subsets to be processed. If so, the method 500 returns to block 510. If not, the method 500 continues to block 525.

At block 525, once all of the feature subsets have been used to train a corresponding threat classifier, the security system classifies a set of security logs using the threat classifiers. That is, the security system processes each log, in a first set of testing security logs, using each threat classifier to generate a set of classifications for the log. The security system can then annotate each security log to indicate which feature subset(s) resulted in a correct classification, which feature subset(s) generated the highest confidence value for the correct classification, and the like.

At block 530, the security system groups the security logs based on these annotations. For example, the security system may identify, for each respective feature subset, the set of security logs with annotations indicating that the respective feature subset performed the most accurately (e.g., generating a correct classification with the highest confidence). These groupings can generally indicate the potential scope of each threat classifier. For example, based on the number of logs within each group (or the diversity of logs in each group), the system may be able to infer or determine how broadly applicable each subset of features is. As discussed above, it may be beneficial to identify feature subsets that perform accurately on a broad set of security logs.

At block 535, the security system trains a type classifier based on these identified groups of security logs. As discussed above, this may generally include training a classifier to receive a set of features (e.g., all available features) from the labeled security logs as input, and to generate an output indicating which subset of features should be used to classify the potential threat. By using the labeled logs, the weights and parameters of the type classifier can be iteratively refined (e.g., using batch gradient descent or stochastic gradient descent) to generate more accurate inferences. In one embodiment, the training process includes generating an output classification based on an input security log (e.g., based on all of the available features). This output can then be compared against the ground truth label (determined based on the groupings above) to compute a loss (e.g., cross-entropy loss), which is back propagated through the type classifier to refine the internal parameters (e.g., using gradient descent).

At block 540, the security system categorizes a set of security logs using the type classifier. In some embodiments, as discussed above, the security system uses a second testing set for this categorization. At block 545, each of the testing logs in this set are then processed using the assigned threat classifier (determined by the type classifier) to generate a classification. In some embodiments, the security system classifies each log using the threat classifiers trained at block 515. In an embodiment, as discussed above, these threat classifiers were trained on a broad set of training security logs (e.g., covering the entire feature space) as at the time of training, it was not clear which particular types of threats would be accurately classified using the feature subset.

In at least one embodiment, therefore, the security system can retrain the threat classifiers using only security logs that are within the corresponding region of the feature space. In one such embodiment, the security system may use the groupings determined at block 530 to train new threat classifiers. For example, each group of security logs from the first testing set (which are grouped based on determining that they are all best classified using the same subset of features) can be used to train a new threat classifier for this feature subset. This can result in a more targeted threat classifier that is specialized for the relevant feature space represented by the logs in the corresponding group of security logs. In on embodiment, these new threat classifiers are used in block 545. The second testing set of security logs can then be processed in blocks 540 and 545 to generate the scores.

At block 550, the security system then assigns a score to each feature subset based at least in part on the classifications generated in block 545 (based on the second testing set). For example, in one embodiment, the score is directly proportional to the accuracy of the model (e.g., the percentage of security logs, which were assigned to the model by the type classifier, that were classified correctly). In at least one embodiment, the score is also directly proportional to the size of the feature space covered by the model (e.g., determined based on the number of security logs, in the second test set, that were assigned to the model).

As discussed above, these scores can be used to select the best-performing feature subsets (and to discard lowest-performing subsets), enabling the security system to iteratively identify optimized feature subsets to cover the feature space and accurately classify input security logs.

FIG. 6 is a flow diagram illustrating a method for genetic programming and machine learning to improve threat detection. In some embodiments, the method 600 is performed by a security system, such as security system 105 of FIG. 1.

At block 605, a plurality of feature subsets, each containing a respective subset of features from a plurality of features included in a set of training security logs, are identified. For example, this may correspond to block 410 or block 435 in FIG. 4.

At block 610, the plurality of feature subsets is modified using one or more genetic programming techniques. For example, this may correspond to block 415 of FIG. 4.

At block 615, each of the plurality of feature subsets is scored using a plurality of threat classifiers, wherein the plurality of threat classifiers comprise trained machine learning models. For example, this may correspond to block 425 of FIG. 4.

At block 620, a set of feature subsets is selected from the plurality of feature subsets based on the scores. For example, this may correspond to block 435 of FIG. 4.

At block 625, a type classifier is trained based on the set of feature subsets, wherein the type classifier comprises a trained machine learning model. For example, this may correspond to block 450 of FIG. 4.

FIG. 7 is a block diagram illustrating a computing device 700 configured to use genetic programming and machine learning to improve threat detection. In some embodiments, the computing device 700 corresponds to a security system, such as security system 105 of FIG. 1.

As illustrated, the computing device 700 includes a CPU 705, memory 710, storage 715, a network interface 725, and one or more I/O interfaces 720. In the illustrated embodiment, the CPU 705 retrieves and executes programming instructions stored in memory 710, as well as stores and retrieves application data residing in storage 715. The CPU 705 is generally representative of a single CPU and/or GPU, multiple CPUs and/or GPUs, a single CPU and/or GPU having multiple processing cores, and the like. The memory