Having grouped anomalous embeddings together, an analysis is performed to identify any common patterns exhibited within an anomalous embedding group. For example, a common pattern exhibited across a subset of embedding dimensions may be identified. If so, this can be used as a basis for computationally efficient threat detection in the deployment stage, by configuring the detector 107 (or multiple detectors) to identify similar patterns in that subset of embedding dimension(s) in a log embedding received as input. In the example deployment scenario of FIG. 1, embeddings are extracted from new logs and the detector operates on the log embeddings. For example, if the detector 107 has an ML architecture, it may be trained to identify such patterns based on an identified group of log embeddings.

For embeddings corresponding to centrally-stored logs in the central log database 105, common patterns may also be identified in the underlying logs themselves, the detector 107 (or multiple detectors) may be configured to identify matching patterns in a security log received as input at deployment. In this deployment scenario, the detector 107 operates on the logs themselves. For example, if the detector 107 has an ML architecture, it may be trained to identify such patterns based on an identified group of security logs.

The two approaches described above may be combined to configure the detector 107 to operate on both log embeddings and the security logs themselves.

Reducing the size of the anomalous embedding database 102 through embedding deduplication yields a consequent improvement in computational efficiency of subsequent operation or analysis performed on the database, because fewer comparison operations need to be performed, without reducing performance as the embedding deduplication resulting in negligible information loss.

As noted, detections triggered in the deployment stage 103, in turn, trigger appropriate security mitigation actions 109B, such as those mentioned above. An incoming security log or log embedding may be associated with a target system or network (e.g., it may be associated with the local system 120, or another system in which it is deployed), triggering an action in the target system or network (such as modifying a parameter or isolating an entity).

As noted, in certain examples provided herein, a security-specific ML model, such as a security-specific transformer model, is used to extract the log embeddings. Different data formats may be accommodated by training such a model on security logs with different formats. Such a model does not depend on any specific delimiters in the log, nor does it require a log to be in a specific format.

A log embedding is a semantic representative of a word or other character string. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in vector space are expected to be similar in meaning.

In some examples, a transformer model is used to generate the log embeddings. The transformer model is trained on a security-specific training set in some examples. A transformer model trained in this way is referred to as a security transformer model (STM).

In one implementation, an encoder-only transformer architecture is used, with an STM pretrained from scratch on existing security logs using a transformer architecture described in DeBERTa (arXiv: 2006.03654). The STM comprises a series of encoder layers. It first transforms an input log into a list of tokens and then transforms the tokens into vectors of fixed length. The STM also adds positional encoding vectors to help the model understand the order of tokens. The resulting vector is fed into a sequence of encoding layers where the STM model uses an attention mechanism to learn the relation between tokens, and to create semantic meaning. A log embedding is generated at a final one of the encoding layers. The weights in the encoding layers are trained with masked language modeling task. In this task, parts of the security logs are masked out and the STM is trained to predict the masked out parts. After sufficient training, the model can predict masked out tokens with high accuracy. At this point, it is able to generate high quality log embeddings capturing security-specific semantic knowledge.

An STM or other ML model trained on a security-specific dataset is able to extract semantic embeddings capturing domain-specific security knowledge learned in training. In the following examples, an STM is implemented as an encoder-based transformer model (e.g. the encoder only transformer model described above), and its embeddings enable the central system 100 and the local system 120 to normalize input data and enable different format of input data to be utilized machine learning algorithms. This provides a flexible way to handle multiple data formats.

Log-sub-sampling is used to perform a subsampling of the STM embeddings with a focus on generating a diverse subset using either clustering or greedy distance maximization. Compared to traditional log storage technologies, this method uses the knowledge of a security specific language model, which may be a security specific language transformer model to find patterns in the data and sub sample logs for efficient storage.

When log-subsampling is done using a clustering method, an embedding is selected from each resulting and added to the database of anomalous log embeddings 102. The embedding selected from each cluster may, for example, be a cluster mean or an embedding which lies the closest to the cluster mean. may be selected (the latter is always guaranteed to correspond to an actual security log, whereas the cluster mean itself is an aggregation over multiple security logs).

When log-subsampling is done using a greedy distance maximisation method, a first log embedding is selected initially. For example, the first log embedding may be selected based on a reference point in embedding space (e.g., a centre of all embeddings). For example, the first embedding may be selected because it is closest to the reference point (e.g. center/mean of all embeddings). Next, a second embedding is selected, as the embedding furthest from the first embedding in embedding space. Next, for each remaining embedding, a minimum distance to each embedding selected so far is determined, and a third embedding having the largest minimum distance to any selected embedding is selected. This means, for each remaining embedding, determining the distance to each selected embedding, selecting the minimum of these distances (the minimum distance), and identifying the largest of these minimum distances (the maximum minimum distance). This ‘min-max’ operation ensures diversity as it ensures subsequent selections are relatively distant from all embeddings that have been selected so far. This process repeats until a predetermined number, N, of embeddings have been selected.

Log-pattern detection is used to find anomalous logs using an outlier detection method. Log-pattern-detection trains a model to predict links between high similarity data points and detect outliers in a data which have high reconstruction error. Anomalies and attacks may be found based on semantic relationships encoded by a security-specific transformer model. For example, the STM embeddings may be used to compute similarity and pairwise distances.

A threshold, representing the distance between the candidate log embedding and a non-anomalous embedding, may be defined and may be used to classify a candidate log embedding as anomalous.

FIG. 2 shows a pipeline of a method wherein anomalous logs are detected using log-pattern detection applied to a log-sub-sampled subset of STM embeddings. A batch of input logs 201, in multiple possible data formats, are fed to a STM model which generates STM log embeddings 202 from the input logs 201 in step S201. In step S202, log-pattern detection is applied to the STM log embeddings 202 to find anomalous logs 203 within the batch of logs using an outlier detection method. In step S203, log-sub-sampling is performed on the anomalous log embeddings 203, to generate a diverse subset 205 of anomalous log embeddings. It is determined in step S204 whether the anomalous log embeddings 205 are already stored in a database 206 of anomalous embeddings (such as database 102 of FIG. 1). If the log embeddings 205 are not already stored in the database 206 of anomalous log embeddings, they are then stored, in step S205, in the database 206 of anomalous log embeddings. In the case that the log embeddings 205 are found to be already stored in the database 206 in step S204, the log embeddings 205 are not stored in the database 206.

FIG. 2 broadly encompasses both the seed stage 101A and the streaming stage 101B of FIG. 1.

In some implementations of the streaming seed 101A, the comparison S204 against database 206 may be omitted. For example, if a single batch of logs is processed in the streaming stage S203, the comparison may be unnecessary (as database 202 is empty at that point), because the log subsampling step S203 ensures sufficient diversity.

In some implementations of the streaming stage 101B, log subsampling S203 may be omitted. In that case, each anomalous log embedding identifies via log pattern detection in step S202 may be compared against existing embeddings held in database 206, with the method proceeding from S202 directly to S204. Depending on the implementation specifics, it may be more efficient to perform log subsampling in the streaming stage 101B and compare only the subset of anomalous log embeddings identified via subsampling with the existing embeddings held in database 206; or, in other cases, it may be more efficient to compare each identified anomalous log embedding against database 206.

The database of anomalous embeddings 206 is an efficient way of storing logs is anomalous embeddings for detecting anomaly and attacks in large amounts of data. The storage index provides a flexible way to handle multiple data formats. The database may have a tree structure and a semantic index and can be thus queried.

In the seed stage 101A, log-pattern-detection is applied to seed logs followed by log-sub-sampling to create an initial index. In a streaming stage 101B, log-pattern-detection is applied to new data, new anomalous embeddings are compared to stored samples, new clusters are dynamically created or diversity is added to existing clusters.

FIG. 3 shows an example flow chart of the seed stage and the streaming stages of the log anomaly detection method described above. In the seed stage S301, seed data 301 is used as input to step S11 where log-pattern detection is applied to STM log embeddings of logs in the seed data 301 to detect anomaly logs. The anomalous logs 302 are used as input to step S12 where log-sub-sampling is applied to the anomaly logs 302 and duplicate anomaly logs are removed. The STM embeddings of the sub-sampled anomaly logs 303 are used to create an initial index for the semantic index database 304. In the streaming stage S302, streaming data 305 is used as input to stap S21 where log-pattern detection is applied to STM log embeddings of logs in the streaming data 305 to detect anomaly logs. The STM embeddings of the anomalous logs 306 are used to update the semantic index database 304. In a second streaming stage S303, GNN-based pattern discovery methods are applied to the semantic index database 304 to derive hierarchical patterns in the database 304.

The GNN is an anomaly classification algorithm which provides a second layer of detection. Other clustering algorithms may be used. The GNN runs on the embedding graph created from the embeddings in the database of anomalous embeddings. This unsupervised pipeline allows detected anomalies and attacks to be segmented to discover raw common entities, e.g., new or unknown anomalies and attacks.

For example, a decision tree may be generated with classification labels and cluster identifiers (ids) or other cluster labels, as described in further detail below with reference to the example of FIG. 6. For a given leaf in the decision tree, all data which corresponds to the leaf may be selected and a name entity recognition (NER) algorithm may be run on the selected data; the leaf may then be represented as the collection of all common entities associated with the classification label of the leaf. This results in a tree like structure which organizes data by pattern whose leaf nodes describe different common groupings of entities.

The anomaly classification and hierarchical patterns may be used to build a security model that, for example, assigns a particular security action for a particular common entity or dimension pattern detected at a leaf of the decision tree. A security action may, for example, be an alert to a user.

FIG. 4 shows a flowchart of a method of deriving hierarchical patterns from a database of anomalous embeddings. In step S402, a clustering method is applied to the database of anomalous embeddings 401 and cluster labels (ids) 402 are assigned to every embedding. In step S403, an embedding graph is generated based on the embeddings of security logs. For a group of embeddings, each embedding is viewed as a node. The Euclidean or cosine distance between each pair of the embeddings is computed. The embedding graph is created by assigning edges between embeddings that are closer than a threshold. In step 404, a GNN model is applied to the embedding graph generated from database 401 of anomalous embeddings. An unsupervised GNN model may be used to discover new common entities from the anomalous embeddings. Additionally, or alternatively, the GNN may be trained to classify the anomalous embeddings into known common entities. The GNN model is pre-trained on training embeddings with known classification labels. For example, the training embeddings may be extracted from security logs known to be associated with specific known security attacks. The output 404 from the GNN model is a classification label assigned to each node of the embedding graph. In step S405, hierarchical patterns in the common entities derived from the embedding graph 404, are used to build a tree-like structure for the database using the cluster ids and classification labels. In Step S406, a security model, such as an alert system, may be build based on the anomaly classification or tree-structure of the database.

Federated Storage is a system that brings together various storage resources into one single management system. Many forms of storage can be connected, such as servers, cloud accounts, etc.

The above-described method may be used with federated data. The seed stage is run in centralized storage. The streaming stage is run in each data store to create embeddings and only the sample embeddings are moved to the centralized storage, not the user data. During the streaming stage, the candidate log embedding may be generated by the user locally. The candidate log embedding is then sent to a central server. The STM is run without decrypting weights used in creating the embeddings. In the federated learning scenario, the encrypted STM model is sent to users. The users run the encrypted STM model to generate embeddings on their data. Only the embedding is passed to the central storage and the users' security data is not passed to central storage. As indicated, this federated solution stores user data in a federated way and no user data is stored centrally. Users are not required to share the log itself to the central server. The user can maintain the privacy of the log's by generating the log embedding locally and sending the log embedding (not the log itself) to the central server.

FIG. 5 shows a flowchart for detecting and storing anomalous log embeddings to a database 501 hosted on a central server 500. The seed stage S301 is run on the central server 500. Local machine 1 500A and local machine 2 500B each run streaming stages. On local machine 1 500A, log-pattern detection is run on streaming data 504 in step S505 to find anomaly logs 506. The STM semantic embeddings 507 of the anomaly logs 506 are sent to the central server 500 to update the centralised semantic database 501. In parallel, on local machine 2 500B, log-pattern detection is run on streaming data 508 in step S509 to find anomaly logs 510. The STM semantic embeddings 511 of the anomaly logs 510 are sent to the central server 500 to update the centralised semantic database 501. On the central server 500, a GNN-based pattern discovery method is run on the centralised semantic index database 501 in step S502 to find hierarchical patterns 503.

FIG. 6 shows an example decision tree index. The decision tree index supports pattern-based partitioning with a query-able semantic key enables more efficient search by combining unsupervised pattern detection and semantic representation. A tree index structure is used where branches are separated by labels. A new piece of data may be assigned to a leaf by according to its labels. For example, for data with labels A, B, a first leaf may correspond to label combination (A=0, B=0), and a second leaf might correspond to label combination (A=0, B=1). A new log embedding with labels A=0, B=1 is assigned to the second leaf in this case. For example, label A may be a classification label and label B may be a cluster label.

The tree index supports efficient data retrieval. Each leaf node stores a reference to its assigned log embeddings in the anomalous embedding database 102. In this example, a first level of the decision tree index comprises a node for each possible value of label A (0, 1 and 2 in this example). Each of these nodes, in turn, has a child node (in a second level of the decision tree index) for each possible value of label B (0, 1, 2). In this particular example, there are only two classification labels, hence the nodes at the second level are leaf nodes. Each leaf node contains a reference to each of its assigned log embeddings in the anomalous log embedding database 304. Hence, to retrieve logs with labels (A=2, B=2), a search of the decision tree index is performed to locate the corresponding leaf node, which in turn is used to locate the corresponding log embeddings in the database 304. The search of the decision tree index can be performed in a computationally efficient manner; in this example, it involves locating the A=2 node, and then locating the A=2, B=2 child node of the A=2 node. With M possible values of label A and N possible values of label B, the search can be performed with computational complexity M+N, compared with a naive search of complexity N*M.

The decision tree represents the index on the database. The decision tree leaf composition may be analysed to identify common log elements or embedding dimensions for use in configuring the detector 107. With structured logs, an element might for example correspond to specific database column or data field.

For seed data used in the seed stage, the log contents are held in the central log database 105 and, for a given leaf in the decision tree, a column in the logs and a specific column value (or subset of possible column values) might be identified as a common entity.

For streaming data used in the streaming stage, the log contents are not sent to the centralized storage. Hence, in the architecture of FIG. 1, this would only be possible for the seed embeddings E_seed.

For both the seed embeddings E_seedand streaming embeddings E_str, for a given leaf in the decision tree, a dimension (or subset of dimension) plus a range of values of that (or those) dimension(s) may be identified as a common entity.

FIG. 7 schematically shows a non-limiting example of a computing system 700, such as a computing device or system of connected computing devices, that can enact one or more of the methods or processes described above, including the filtering of data and implementation of the structured knowledge base described above. Computing system 700 is shown in simplified form. Computing system 700 includes a logic processor 702, volatile memory 704, and a non-volatile storage device 706. Computing system 700 may optionally include a display subsystem 708, input subsystem 710, communication subsystem 712, and/or other components not shown in FIG. 7. Logic processor 702 comprises one or more physical (hardware) processors configured to carry out processing operations. For example, the logic processor 702 may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. The logic processor 702 may include one or more hardware processors configured to execute software instructions based on an instruction set architecture, such as a central processing unit (CPU), graphical processing unit (GPU) or other form of accelerator processor. Additionally or alternatively, the logic processor 702 may include a hardware processor(s)) in the form of a logic circuit or firmware device configured to execute hardware-implemented logic (programmable or non-programmable) or firmware instructions. Processor(s) of the logic processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic processor optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic processor 702 may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration. In such a case, these virtualized aspects are run on different physical logic processors of various different machines. Non-volatile storage device 706 includes one or more physical devices configured to hold instructions executable by the logic processor