num="0056">Preprocessor 210 is configured to listen for ZeroMQ “advice” that a new sample has arrived in the queue. In some embodiments, preprocessor 210 is a Java or Go program. Preprocessor 210 can also be implemented using one or more python scripts or other appropriate technologies. Preprocessor 210 determines what type of file sample 202 is (e.g., its MIME filetype) and selects an appropriate helper to perform the preprocessing. In this example, sample 202 is a Windows executable, so preprocessor 210 selects a Windows PE helper to extract the PE container from the executable and verify it. Another example of a helper is a PDF helper, configured to extract executable code from PDF files. In some embodiments, if the helper is unable to parse the file (e.g., the PE container is malformed), or if an appropriate helper is not present on platform 102 for the received sample file type, platform 102 is configured to send a copy of the sample to sandbox service 106 for additional analysis (e.g., to see whether sample 202 can be executed by sandbox service 106).

Suppose sample 202 is well-formed (i.e., it can be parsed by preprocessor 210 working in conjunction with a Windows PE helper). Preprocessor 210 is configured to iterate over the data sections of the PE container (which are enumerated in the header) and create a bytestream (e.g., a vector or array of bytes) corresponding to the sample. In this example, one portion of the bytestream corresponds to the program executable code extracted from the PE, another portion of the bytestream corresponds to initialized variables, another portion of the bytestream corresponds to the uninitialized variables, and another portion of the bytestream corresponds to debug data in the linking code, etc. Bytestreams of other types of files can also be constructed by preprocessor 210 working in conjunction with an appropriate helper. In some cases, the code section in a Windows PE may be obfuscated. This obfuscation could be done for purposes of digital rights management, but could also be done by a malware author to make malware analysis more difficult. Such obfuscation does not thwart the processing performed by platform 102, as will be described in more detail below.

Once preprocessor 210 has completed generating a bytestream for sample 202, it provides the bytestream to transformation engine 212 (implemented using one or more Java programs, python scripts, or other appropriate technologies), which will transform the bytestream (i.e., from one dimension to two dimensions) using a space-filling curve. One example of a space-filling curve is a Hilbert curve, which is used in various embodiments by transformation engine 212. Other space-filling curves can also be used and the techniques described herein adapted accordingly. Examples of such other space-filling curves include Z-order curves, Peano curves, Moore curves, fractals with a Hausdorff dimension of 2, etc.

One additional task performed by preprocessor 210 (e.g., performed prior to generating a bytestream for sample 202) is to select an order for a Hilbert curve that is of an appropriate size. In particular, preprocessor 210 selects an order that results in a curve size that is larger than the bytestream. In some embodiments, the size decision is further based on a requirement that a minimum amount of negative space be present in the Hilbert curve (e.g., a minimum of fifteen percent negative space). A Hilbert order is 4″ (i.e., 2″ squared). Preprocessor 210 creates two arrays of the same size in memory-one that will hold the bytestream, and one (initialized to zero) that will hold the bytestream after the transformation using a Hilbert curve (also referred to herein as a “mapping”) has been performed.

A graphical representation of a bytestream is depicted in FIG. 3. In the example shown in FIG. 3, the bytestream has been placed in a square of appropriate size, but the Hilbert mapping has not yet been performed by transformation engine 212. Suppose the received sample that resulted in the bytestream depicted in FIG. 3 is a Windows executable where portions were packed using the UPX packer. The first portion of the bytestream representation (302) corresponds to one packed portion. The second portion of the bytestream representation (304) corresponds to a second packed portion. Note that there are visual differences between the two packed portions. The third portion of the bytestream representation (306) corresponds to a .rsrc file. The remainder of the representation (308) corresponds to empty space.

FIG. 4 illustrates an example of a graphical representation of a bytestream after a Hilbert curve mapping has been applied (e.g., by transformation engine 212). In particular, FIG. 4 depicts the representation shown in FIG. 3 after the mapping has been applied.

FIG. 5 illustrates an embodiment of a data analysis platform. The example shown is a representation of physical components that can be included in various embodiments of platform 102. Specifically, platform 102 includes commodity server-class hardware such as a high performance multi-core CPU 502 and RAM 504 (e.g., 32G of RAM) running a server-class operating system (e.g., Linux-based). Platform 102 also includes a storage 506 (such as one or more solid state disks) and one or more network interfaces 508 (e.g., Gigabit Ethernet). Platform 102 can also include one or more optional hardware accelerators. For example, platform 102 can include one or more graphics processing units (GPUs) 510 which can efficiently perform vector operations such as performing Hilbert curve mapping operations. For example, using typical commodity hardware, a given sample can be analyzed using the techniques described herein on the order of tens or hundreds of milliseconds per core. In various embodiments, platform 102 includes one or more accelerated processing units (APUs) instead of or in addition to having separate CPU(s) 502 and GPU(s) 510. Other accelerators, such as ASICs and FPGAs, can also be used and the techniques described herein adapted as applicable.

In some embodiments, the bytestream data stored in the first array (created by preprocessor 210) is mapped into the second array by transformation engine 212 as follows. The first and second arrays are copied from RAM 504 to GPU 510. In some embodiments, transformation engine 212 is implemented as kernel code, which is written in OpenCL and executed in GPU 510 (or an APU, as applicable). Two examples of kernel code for performing a Hilbert curve mapping are shown in FIGS. 6 and 7, respectively. FIG. 6 shows an example of python kernel code used in some embodiments to implement transformation engine 212. FIG. 7 shows an example of a Java implementation of transformation engine 212 that uses AMD Aparapi, which leverages Java code bytecode into OpenCL.

The mapping function that Hilbert or any other space-filling technique provides is to map an index to a coordinate. In the case of platform 102, a byte in a file is mapped to a 2D coordinate. Where platform 102 includes a GPU (or APU), the maps are calculated using a kernel and GPU (or APU). The maps can also be generated on embodiments of platform 102 that do not include a GPU (or APU), by using an approach such as the following. The maps are different for different sized files, bucketed into 4ⁿsize buckets. Fifteen different maps are pre-computed (and, e.g., included with embodiments of transformation engine 212 rather than being computed by transformation engine 212 on initialization). Calculating the size of all maps for Hilbert orders <=15 requires approximately 1.5G of RAM. Once the maps are available, embodiments of CPU 502 can look up from the map/table the coordinate instead of computing the coordinate.

Returning to the descriptions of FIGS. 2 and 5, once transformation engine 212 finishes performing the mapping to a Hilbert curve, the second array (i.e., holding the newly mapped information) is copied back to RAM 504 and the first array (i.e., holding the unmapped bytestream) is discarded. Specifically, the second array (holding the mapped information) is provided to down-sample engine 214, which can be implemented using a Java program, a set of one or more python scripts, or ‘C’ code etc., as applicable.

Down-sample engine 214 is configured to reduce the size of the second array while maintaining at least some of its entropy. A variety of approaches can be used. As one example, down-sample engine 214 can perform a blur operation on the contents of the second array. The blur operation can optionally employ anti-aliasing. An example of Hilbert curve 400 after a blur operation has been performed is depicted in FIG. 8. Examples of open source blur engines that can be used by down-sample engine 214 include the Python Imaging Library (PIL), ImageMagick, and Java Development Kit (JDK). In addition to performing a blur operation, down-sample engine 214 is further configured to perform a resize operation (resulting in what is referred to herein as a “locus”). For example, where a Hilbert curve of size 256 is used, the image can be resized to a locus that is a 16 pixel×16 pixel image (of 256 bit grayscale). Where a Hilbert curve of size 64 is used, the image can be resized to locus that is an 8 pixel×8 pixel image (also of 256 bit grayscale). Other dimensions can also be used and the techniques described herein adapted as applicable.

The result of a resize operation having been performed on blur operation output 800 is shown in FIG. 9—a locus that is a 64 byte image. Image 900 is provided to hashing engine 216 (which can be implemented as a Java program, a set of one or more python scripts, etc.) which further reduces the representation of sample 202 while preserving some of its entropy. In particular, in some embodiments, hashing engine 216 performs a locality-sensitive hashing (LSH) operation on the locus. As one example, the 64 byte image can be reduced to a 63 bit representation by performing a “less than” comparison using locality sensitive hashing. In particular, byte zero is compared to byte one of image 900. If byte zero is less than byte one, the comparison is “true” and the result is a “1.” If byte zero is not less than byte one, the comparison is “false” and the result is a “0.” Next, byte one is compared to byte two; byte two is compared to byte three; and so on, until the end of the image is reached. An example of the output of hashing engine 216 that could be generated for image 900 (i.e., using a “less than” in a locality sensitive hashing operation) is: “1000100001000100000101101111101001011000110001010100100000000000.” Other operators (or combinations of operators) can also be used in conjunction with the LSH, as applicable (e.g., greater than, removing every n bits, etc.).

In some embodiments, the binary output of hashing engine 216 is in turn provided to cluster identifier generator 218 which converts the result of hashing engine 216 into a hexadecimal string and optionally prefixes it with additional information (i.e., making the identifier more convenient for human use). As one example, the result of hashing image 900 is converted by cluster identifier generator 218 to “884416fa58c5484c.” Cluster identifier generator 218 can further refine the output to include additional information, such as the Hilbert order and/or a file format associated with the sample. As one example, the Hilbert order can be prepended as a 16 byte hexadecimal value, meaning that every sample that fits in a particular size Hilbert order has its own namespace. As another example, an indicator of the file format can instead/in addition be prepended as a character. Using “884416fa58c5484c,” an example of a refined output of cluster identifier generator 218 is “h09.884416fa58c5484c”, where “h” indicates the Microsoft Portable Executable (PE) format namespace, 09 is the Hilbert order, and “.” is a separator. In some embodiments, cluster identifier generator 218's processing is omitted, and/or is combined with the processing performed by hashing engine 216. In some embodiments, the processing performed by hashing engine 216 is omitted. Use of a locally sensitive hashing function on a locus reduces the name space that clusters occupy (e.g., in a database). Such optimization may not be necessary in various scenarios and the hashing can be omitted (or considered optional). As one example, internal accounting can be accomplished using a locus, but when displaying human readable code (e.g., in a report or web page) the locus can be substituted with a shorter identifier (e.g., a hash of the locus).

Cluster identifier generator 218 provides the output of its operation (e.g., “h09.884416fa58c5484c”) to label engine 220. Label engine 220 is configured to insert the MD5 of sample 202 into repository 206 using the identifier generated by cluster identifier generator 218. Specifically, in some embodiments, label engine 220 executes an “add to set” (hash set) command, adding the MD5 of sample 202 to the set identified by the key “h09.884416fa58c5484c.” In the event identifier “h09.884416fa58c5484c” is not already present in repository 206, the identifier will be inserted with the “add to set” operation (and the MD5 of sample 202 included). Additional processing can also be performed. For example, since the sample does not correspond to any existing samples stored in repository 206, the sample can be provided to sandbox service 106 for analysis. In some embodiments, in the event sandbox service 106 already has information about the sample (ascertainable, e.g., by platform 102 sending the MD5 hash value of the sample to sandbox service 106 before sending the sample itself), the sample is not transmitted by platform 102 to 106. Results provided by sandbox service 106 (e.g., a listing of observed malicious behaviors) can be stored in repository 206 as metadata associated with the “h09.884416fa58c5484c” family of malware. It is to be noted that hashing a locus into a cluster identifier can be optionally performed (e.g., to aid humans who will tend to prefer copying/typing smaller shorter strings). An un-hashed locus can also be used as an alternate way of identifying/managing clusters of data (e.g., where the use of 64 byte strings, 256 byte strings, etc. are also acceptable).

As another example, a copy of the sample (and/or its MD5 hash value) can be sent to scan service 108, which will return information about which (if any) of the scanning engines employed by scan service 108 detect the sample as malicious, and any names given to the sample by such engines. In some embodiments, platform 102 is configured to store (e.g., as metadata), as a human-readable name for the “h09.884416fa58c5484c” identifier, the result of performing a greatest common subsequence on any names received by scan service 108. Other approaches can also be used to select/store human-readable names for samples/clusters of samples, as applicable, such as preferentially using the name provided by one engine over another, concatenating multiple names together (e.g., where different engines use different names), etc.

If the identifier “h09.884416fa58c5484c” is already present in repository 206, this means other samples, having different MD5 hash values (but otherwise very similar to sample 202), were previously examined by platform 102 (i.e., part of a cluster of very similar samples). As one example, suppose a malicious developer changes a single pixel of an image included in a sample each time he attacks a victim. An MD5 hash value of the sample with the changed pixel will look vastly different from an MD5 hash value of the sample without the changed pixel, even though the samples are otherwise identical. As will be described in more detail below, such subtle changes will not impact the identifier ultimately generated by cluster identifier generator 218 and so both samples (with and without the changed pixel) will both be considered members of the same cluster of malware (e.g., the “h09.884416fa58c5484c” cluster). If a member of the “h09.884416fa58c5484c” cluster has already been analyzed, then resources can be conserved (e.g., by not sending the changed pixel sample to sandbox service 106, and instead only sending to sandbox service 106 those samples having an associated cluster identifier that was not previously seen). In some embodiments, the sample (or an MD5 hash of the sample, as applicable) is always sent to at least one of sandbox service 106 and/or scan service 108 (e.g., to augment information already present about the family stored in repository 206). In other embodiments, a subset of samples is sent to services 106/108. As one example, the first time a sample is received for a cluster identifier (i.e., a sample having a new cluster identifier is received), the sample can be sent to services 106/108 for analysis. As additional samples having that cluster identifier are received by platform 102, various customizable rules can be used to determine which samples should be processed. One example scenario is to send the first n samples for processing (e.g., the first ten samples for a given cluster identifier), then statistically select additional samples for processing as they arrive (e.g., send 25% of samples 11-200, then send 5% of any additional samples). Information received back from services 106/108 can be used to refine which future samples are sent for processing for a given cluster identifier. For example, if all initially sent samples are confirmed as “known good” by services 106/108, in various embodiments, platform 102 can cease sending later received samples sharing the same cluster identifier. Similarly, if services 106/108 consistently return results indicating that samples sharing a particular cluster identifier are “known malicious” with a high degree of certainty, platform 102 can reduce the number of samples sent to services 106/108 having that cluster identifier as new samples are received. Where services 106/108 are not confident in their results, platform 102 can continue to send a higher number of samples for analysis as they arrive, potentially eventually resulting in a more confident answer from services 106/108 over time.

Example Process

FIG. 10 illustrates an embodiment of a process for analyzing data. In various embodiments, process 1000 is performed by platform 102. The process begins at 1002 when a file is received. As one example, a file is received at 1002 when sample 202 is received by platform 102 (e.g., via an embodiment of interface 204). At 1004, file contents are transformed using a space-filling curve (e.g., by transformation engine 212). As explained above, additional processing can be performed after receipt of a file and prior to the transformation occurring. For example, preprocessor 210 can perform preprocessing on sample 202, resulting in a first array of bytes. Also, decisions such as what order Hilbert curve to use, and what down-sample size should be used can be made. At 1006, the result of the space-filling curve transformation is down-sampled. As one example, down-sampling can be performed by down-sample engine 214 which is configured to perform a blur operation (with anti-aliasing) and resize operation on the result of the space-filling curve transformation. Finally, at 1608, a hashing operation is performed. As one example, at 1008, hashing engine 216 performs an LSH hashing operation on the down-sampled result. As explained above, additional processing can also be performed in conjunction with process 1000, such as sending the sample to services such as 106 and 108, and associating the sample with the result of the LSH hash (or a representation thereof) in a repository. The locus can also be used (e.g., instead of/in addition to the LSH hash as a cluster identifier), for example, in namespaces with many neighbors.

Reporting

The contents of repository 206, and in particular recently updated portions thereof, can be provided to entities such as companies 110-114, services 106-108, and/or any other appropriate entities in a variety of ways. As one example, platform 102 can be configured to make available an API that such entities can use to conduct queries (and retrieve results). As another example, platform 102 can make available a daily digest (e.g., packaged as a tar.gz) of newly seen samples and associated information. As one example, suppose a few hundred thousand new samples are received and processed by platform 102 each day. The corresponding digest of results and samples, made available to customers, could comprise approximately 10G of data.

An example of a daily cluster report is depicted in FIG. 11. The daily cluster report summarizes the new samples that have been collected by platform 102 since midnight (i.e., samples not previously analyzed by platform 102). The first column (1102) lists cluster identifiers. The second column (1104) represents a probability that a member of the given cluster (i.e., a unique sample having a unique MD5 hash value from the other members) will successfully pass through an engine in VirusTotal unreported. The third column (1106) is a count of the new members added to the cluster since midnight. Finally, the last column (1108) is a label assigned to the cluster. Using the first row of the report (1110) as an example, the probability that a sample in the 03f9.1a9f2049c0000b16 cluster will successfully pass through an engine in VirusTotal is 0.55725956. The label assigned to cluster 03f9.1a9f2049c0000b16 is “kovter pemalform riskware.” Finally, since midnight (on April 26), 377 unique new samples, all members of cluster 03f9.1a9f2049c0000b16, were seen by platform 102 (which may already have had a potentially large number of members). An example of how different samples (having different MD5 hash values) will be clustered together follows.

For some of the clusters (e.g., cluster 1112 which has a cluster identifier of “03e7.11935895c2210b12”), platform 102 does not have a probability in column 1104, and does not have any labels (but has seen 120 new samples since yesterday). One reason this could be the case is that all of the member(s) of the cluster are new, and have not yet been submitted to (and/or processed by) service 108. Another reason this could be the case is that all engines in service 108 have determined that the sample is not malicious. As will be described in more detail below, in various embodiments, platform 102 determines one or more clusters that are similar to cluster 1112 (e.g., for which it does have information), and uses information about the similar cluster(s) to help gain insight about cluster 1112 (e.g., by inheriting probability scores, labels, maliciousness verdicts, etc).

FIGS. 12 and 13 depict graphical representations of bytestreams associated with two respective samples (similar to the representation shown in FIG. 3). In particular, FIGS. 12 and 13 correspond to samples that have MD5 hash values of “c6cda9e3d865c33afb4a08ff6ea53459” and “fcd40892da915d5edbf76e5c94265c11,” respectively. The two samples were authored by the same nefarious individual and perform similar malicious actions (e.g., installing a keystroke logger). The nefarious individual attempted to conceal that the samples are in essence the same, by making slight differences to the second sample.

Regions 1202 and 1302 of the respective bytestream representations comprise “text” data. Regions 1204 and 1304 of the respective bytestream representations comprise “.data” data. Regions 1206 and 1306 of the respective bytestream representations comprise “.idata” data. Regions 1208 and 1308 of the respective bytestream representations comprise “.rsrc” data. The remainder of the bytestream representations correspond to empty space.

FIGS. 14 and 15 depict graphical representations of bytestreams after a Hilbert curve mapping has been applied (e.g., by transformation engine 212) to the data shown in FIGS. 12 and 13, respectively. Samples “c6cda9e3d865c33afb4a08ff6ea53459” and “fcd40892da915d5edbf76e5c94265c11” have very different MD5 hash values, and have very similar but not identical pixel data (as seen in FIGS. 16A and 16B). However, when hashing engine 216 performs a locality-sensitive hashing operation (e.g., using “less than”) on 1602 and 1604 respectively, identical output will be obtained: “1100010011010000100110010001010101000011000000001000100000000000.” Cluster identifier generator 218 will therefore convert the result of hashing engine 216 into identical strings for both samples: “h09.c4d099154300880c.” Accordingly, both samples will be added to the same cluster, “h09.c4d099154300880c,” within repository 206, despite the very different MD5 hashes of the two samples.

FIG. 17 depicts an example of a cluster detail report. One way to arrive at the report depicted in FIG. 17 is to click on a link such as link 1112 or 1114. An alternate way to arrive at the report depicted in FIG. 17 is to perform a search (e.g., using a search box, an API call, or other appropriate technique) using a cluster identifier (e.g., searching for “m3e9.611c96cf656cc6d2” (1702)). Other ways to arrive at the report depicted in FIG. 17 include interacting with a report about a particular sample, and interacting with a neighbor of the cluster, both of which are described in more detail below.

Region 1704 of the report indicates that a total of 788 of the samples analyzed by platform 102 share a cluster identifier of “m3e9.611c96cf656cc6d2.” The MD5 hash value of each of the 788 samples is listed in column 1706. Clicking on a link in column 1710 opens the VirusTotal page associated with the corresponding sample.

FIG. 18 illustrates portions of two different sample reports, depicted side-by-side for comparison. The first report, appearing in region 1802, is an example of a report obtainable by clicking on region 1712 in report 1700 (on sample “0813715051cdd8dc91c9b5da66bb2966”). The second report, appearing in region 1804, is an example of a report obtainable by clicking on region 1714 in report 1700 (on sample “0d94aa2c8042ed79c794866575e69d7d”). As shown in regions 1806 and 1808, respectively, both samples have virtually identical Hilbert curve mapping representations (described above). As shown in regions 1810 and 1812, respectively, both samples are members of the cluster m3e9.611c96cf656cc6d2. The samples have different hash values, as shown in regions 1814 and 1816, 1818 and 1820, and 1822 and 1824, respectively. Additional information included in the reports include the time the sample was first observed by platform 102 (1826), a MIME type of the sample (1828), and a location of the sample in repository 206 (1832). Region 1830 depicts a traffic light protocol (TLP) designation (e.g., red, yellow, or green) that can be used to determine whether information about a given sample can be shared (e.g., with companies 110-114). Region 1834 indicates whether a YARA signature is available for the cluster (and if so, provides a link to the signature).

FIG. 19 illustrates portions of two different VirusTotal reports, depicted side-by-side for comparison. By clicking on the link in region 1840, a viewer of report 1802 will be taken to the VirusTotal page for sample 0813715051cdd8dc91c9b5da66bb2966. The SHA256 value is common across regions 1822 and 1906. Similarly, by clicking on the link in region 1842, a viewer of report 1804 will be taken to the VirusTotal page for sample 0d94aa2c8042ed79c794866575e69d7d. The SHA256 value is common across regions 1824 and 1908. Column 1910 of report 1902 shows a list of engines that were used to analyze the sample represented in report 1902. Column 1912 indicates a corresponding verdict provided by the engines. As indicated in region 1914, of the 61 engines that evaluated the sample, 56 of the engines considered it to be malicious. Similarly, as indicated in region 1916, of the 59 engines that evaluated the sample represented in report 1904, 56 considered it to be malicious. In various embodiments, detection ratio information, such as is shown in regions 1914 and 1916, is used by platform 102 to determine prediction information such as is shown in column 1104 of report 1100. One example way for platform 102 to determine a value for the column (e.g., “0.3766396” at 1116) is for platform 102 to subtract from 1 the average of all of the detection ratios for each sample in a cluster. Other approaches can also be used, such as by using only a subset of samples' detection ratios instead of all samples in a cluster.

Regarding column 1912, in various embodiments, the terms appearing in column 1912 are used by platform 102 to generate a label for a cluster (such as the label “kovter pemalform riskware” depicted in FIG. 11). As one example, a script (e.g., written in python) can collect all of the terms for all of the samples associated with a given cluster, remove stop words (e.g., “win32” and “worm”), remove symbols, and select as a label for the cluster, the three most common words.

Returning to FIG. 17, in various embodiments, the “Cluster Neighbors” link (1716) leads to an interface such as is shown in FIG. 20 (e.g., rendered in a web browser). Information about cluster neighbors can also be provided by platform 102 in other ways, such as through a textual list, responses to API calls, etc. Cluster m3e9.611c96cf656cc6d2 is represented in region 2002. Also present in FIG. 20 are representations of various nearest neighbors of cluster m3e9.611c96cf656cc6d2 and their sizes (i.e. number of members) relative to cluster m3e9.611c96cf656cc6d2. Each of the clusters shown in FIG. 20 as having a cluster identifier prefix of “1-bit” (e.g., cluster 2004 and cluster 2006) represents a cluster that is an edit distance of one bit away from cluster m3e9.611c96cf656cc6d2. The cluster identifiers in FIG. 17 are variously depicted as truncated to ease in visual understanding. Each of the clusters shown in FIG. 20 as having a prefix of “2-bit” (e.g., cluster 2008 and 2010) represents a cluster that is an edit distance of two bits away from cluster m3e9.611c96cf656cc6d2. Using cluster 2012 as an example, cluster 2012 has a cluster identifier of “m3e9.611c96cf656cc2d2.” There is a one bit difference between Cluster 2002 and 2012. Clusters 2002 and 2012 have approximately the same number of members.

FIG. 21 illustrates portions of two different sample reports, depicted side-by-side for comparison. Reports 2102 and 2104 pertain to samples “0f33bdbedea51ee73c3a5da111ad103c” and “02f4cc2e395abcb99d1a93efb3238a84” which are members of cluster m3e9.611c96cf656cc2d2 (depicted at 2012 in FIG. 20). As shown in regions 2106 and 2108, respectively, both samples have virtually identical Hilbert curve mapping representations (described above). Further, the images appearing in regions 2106 and 2108 are virtually to those in 1806 and 1808.

FIG. 22 illustrates portions of two different VirusTotal reports, depicted side-by-side for comparison. By clicking on the link in region 2110, a viewer of report 2102 will be taken to the VirusTotal page for sample 0f33bdbedea51ee73c3a5da111ad103c. The SHA256 value is common across regions 2114 and 2206. Similarly, by clicking on the link in region 2112, a viewer of report 2104 will be taken to the VirusTotal page for sample 02f4cc2e395abcb99d1a93efb3238a84. The SHA256 value is common across regions 2116 and 2208. Column 2210 of report 2202 shows a list of engines that were used to analyze the sample represented in report 2102. Column 2212 indicates a corresponding verdict provided by the engines. As indicated in region 2214, of the 56 engines that evaluated the sample, 53 of the engines considered it to be malicious. Similarly, as indicated in region 2216, of the 57 engines that evaluated the sample represented in report 2104, 52 considered it to be malicious. The identifiers, “virut,” and “allaple” appear frequently in each of columns 1912, 1918, 2212, 2218.

Using Clusters and Distance Relations

Suppose that benign software running on Alice's computer has recently performed an update, and as a result, a new executable file (“helperapp.exe”) has been downloaded to her computer. Agent 154, running on Alice's computer, determines that helperapp.exe is new (i.e., not previously seen by agent 154), and consults platform 102 for more information. Agent 154 transmits the MD5 hash value of helperapp.exe to platform 102.

As previously discussed, if an MD5 hash of helperapp.exe is present in dataset(s) 222, a verdict (e.g., benign or malicious) can be provided to agent 154. An MD5 hash of helperapp.exe may also be present in repository 206 (e.g., due to process 1000 having been previously performed on a copy of the sample). Information about the cluster of which the executable is a member can be used to provide instructions back to agent 154. As one example, the cluster's VirusTotal probability (examples of which are shown in column 1104) can be used as a heuristic in determining whether members of the cluster are likely to benign or malicious.

Using helperapp.exe as an example, suppose a previous version of helperapp.exe (e.g., version 1.23) was analyzed by platform 102 and assigned as a member of a cluster (having a particular cluster identifier). And, suppose that the version of herlpapp.exe installed on Alice's computer (e.g., version 1.24) is newly seen by platform 102 (i.e., platform 102 does not have an MD5 hash value corresponding to helpapp.exe version 1.24 in repository 206). Further suppose that the only changes made to helperapp.exe v1.24 from helperapp.exe v1.23 are that the software developer of helperapp.exe updated help text, and changed copyright dates from 2016 to 2017. Alice's version of helperapp.exe (v. 1.24) will have a different MD5 hash value than the previously analyzed version (v. 1.23). However, and as explained above, the two versions of helperapp.exe will likely have virtually identical Hilbert curve mapping representations, and thus ultimately have identical/near identical cluster identifiers. The following is an example of a process that can be performed on helperapp.exe (v. 1.24) by platform 102 to provide information back to agent 154.

FIG. 23 illustrates an embodiment of a process for assigning properties to a file. Process 2300 is an example of processing that can be performed by platform 102 where (e.g., with helperapp.exe (v. 1.24)), platform 102 receives a file for which it does not already have information (e.g., in dataset 222 or repository 206).

The process begins at 2302 when a file is received. As one example, a file is received at 2302 when helperapp.exe (v. 1.24) is received by platform 102 (e.g., via an embodiment of interface 204). At 2304, file contents are transformed using a space-filling curve (e.g., by transformation engine 212). As explained above, additional processing can be performed after receipt of a file and prior to the transformation occurring. For example, preprocessor 210 can perform preprocessing on sample 202, resulting in a first array of bytes. Also, decisions such as what order Hilbert curve to use, and which down-sample size should be used can be made. At 2306, the result of the space-filling curve transformation is down-sampled (resulting in a locus). As one example, down-sampling can be performed by down-sample engine 214 which is configured to perform a blur operation (with anti-aliasing) and resize operation on the result of the space-filling curve transformation. At 2308, a hashing operation is performed. As one example, at 2308, hashing engine 216 performs an LSH hashing operation on the locus (the down-sampled result). As explained above, additional processing can also be performed in conjunction with process 2300 or portions thereof, such as sending the sample to services such as 106 and 108, and associating the sample with the result of the LSH hash (or a representation thereof) in a repository.

At 2310, a cluster identifier is assigned to the sample using a result of the hashing operation. As one example, at 2310, cluster identifier generator 218 converts the result of the LSH hashing operation performed by hashing engine 216 into a hexadecimal string and prefixes it with additional information. As previously explained, an example of such a cluster identifier is “h09.884416fa58c5484c,” where “h” indicates the Microsoft Portable Executable (PE) format namespace, 09 is the Hilbert order, “.” is a separator, and “884416fa58c5484c” is the hexadecimal conversion of a hash.

As previously explained, in some cases, the cluster identifier assigned for the sample may already be present in repository 206. If so, the sample can be added to the existing cluster as an additional member. Information associated with the cluster, such as a maliciousness verdict, can be provided (e.g., to agent 154) as applicable. Suppose, however, the cluster identifier assigned to the sample at 2310 is not present in repository 206. The sample may be an entirely new piece of software, or may also be similar to samples previously seen by platform 102. The following portions of process 2300 can be used to identify whether the sample is similar to samples already present in repository 206, and if so, to assign properties to the sample using properties of similar samples.

At 2312, a set of candidate nearest neighbors is determined for the sample's cluster (i.e., that was determined at 2310) and at 2314, a determination is made as to which of the candidate nearest neighbors correspond (if any) to existing clusters within platform 102 (e.g., has a record in repository 206).

As one example of the processing performed at 2312/2314, candidate finder 224 uses the sample's cluster identifier to generate a set of candidate identifiers. In various embodiments, candidate finder 224 comprises a set of one or more scripts written in an appropriate scripting language, such as python of Java. Using the binary representation of the cluster identifier for illustrative purposes, a sample identifier of “1000100001000100000101101111101001011000110001010100100000000000” can be used to generate a set of 63 candidate identifiers where each candidate identifier has one bit flipped from the sample's cluster identifier (i.e., is one bit of edit distance away from the sample's cluster identifier). Using the first few bits of the sample's cluster identifier as an example, “10001 . . . ” is used to generate candidate identifiers that begin “00001 . . . ,” “11001 . . . ,” “10101 . . . ,” “10011 . . . ,” “10000 . . . ,” etc. In various embodiments, second order candidate identifiers are also generated, where any two given bits in the sample's cluster identifier are flipped (i.e., is two bits of edit distance away from the sample's cluster identifier, and resulting in a set of 64²candidate identifiers). Third order candidate identifiers (and beyond) can also be generated, where any three given bits in the sample's cluster identifier are flipped (resulting in a set of 64³candidate identifiers). In various embodiments, third (and/or higher) order candidate identifiers are only generated in the event no matches are found (e.g., at 2314) for lower order candidate identifiers. Loci can be used in a similar fashion (resulting in a longer name space and thus more calculations) with similar results. For example, a locus could be 64 bytes (512 bits) and neighbors would be any bit string with an edit distance of one or two bits.

Candidate finder 224 uses the set of candidate identifiers generated at 2312 as a search list and queries against information stored in repository 206. Suppose in this example that a set of candidate nearest neighbor identifiers is generated using one-bit and two-bit edit distances (i.e., a set of approximately 4096 candidate identifiers is generated). Further suppose that when candidate finder 224 queries repository 206 using those 4096 candidates as keys, three results are returned (i.e., there are three hits when performing a hash table lookup using the identifiers generated at 2312). This indicates that three clusters (having corresponding cluster identifiers included among the search list of candidate identifiers) have previously been observed/processed by platform 102 and are nearest neighbors (within one or two bits of edit distance) to the cluster identifier of the sample (i.e., that was determined at 2310).

At 2316, a set of member loci is assembled using the loci of each of the members of the respective cluster identifiers determined at 2314. And, at 2318, an edit distance between the locus of the sample and each of the loci in the set of member loci is determined. One example of processing performed at 2316/2318 is as follows. For each cluster identifier determined at 2314, collect a list of member samples for that cluster (an example of such a list is shown in column 1706 of FIG. 17). For each member sample in the list, retrieve (or, as needed, determine) the locus for that member sample. Then, determine an edit distance (e.g., Levenshtein distance) between the sample's locus, and each member locus. A threshold edit distance (e.g., an edit distance of four bits) can be used to determine whether the sample's locus and a member locus are sufficiently close to one another. If the edit distance is under the threshold, at 2320, one or more properties is assigned to the sample, as described in more detail below.

Suppose that for the three cluster identifiers obtained at 2314, the first cluster has 100 members, the second cluster has 400 members, and the third cluster has 250 members. Further suppose that, of the 100 members of the first cluster, no member locus is within an edit distance of four bits from the sample's locus. Of the 400 members of the second cluster, suppose that 20 member loci are within an edit distance of one bit from the sample's locus, 50 members are within an edit distance of two bits from the sample's locus, and 120 members are within an edit distance of three bits from the sample's locus. Finally, suppose that of the 250 members of the third cluster, 15 member loci are within an edit distance of two bits from the sample's locus, 18 member loci are within an edit distance of three bits from the sample's locus (and no member loci are within an edit distance of one bit from the sample's locus). A variety of approaches can be used to assign properties at 2320. As one example, as both the second cluster and the third cluster include member loci having an edit distance within a threshold of the sample's locus (i.e., within four bits), properties of both the second and third cluster can be inherited by the cluster identifier determined at 2310 (and thus associated with the sample). Thus, if the second and third clusters have verdicts of “not malicious,” the cluster identifier determined at 2310 can also be assigned a verdict of not malicious (and the sample received at 2302 accordingly also treated as not malicious). Other properties such as labels can also be assigned at 2320 (e.g., by using the labels of the second cluster and the labels of the third cluster as labels for the cluster identifier determined at 2310). As an alternate example, where multiple clusters have member loci within the threshold edit distance of the sample, in various embodiments, only the closest cluster's information is used at 2320. In this example, since only the second cluster has member loci within an edit distance of one of the sample's locus, in some embodiments, only the properties of the second cluster are used at 2320. In other embodiments, other schemes can be used to determine which member loci (and/or thee respective clusters of which they are members) should be used to assign properties at 2320. As one example, each member locus that is within a threshold edit distance of the sample locus can “vote” for its cluster's properties, weighted by the edit distance that the member locus is from the sample locus.

As previously mentioned, in some cases, no existing clusters (e.g., within one or two bits of edit distance) may be located at 2314, and/or at 2318, no member loci may be sufficiently close to the sample's locus (e.g., within one or two bits of edit distance). In some embodiments, in such scenarios, portions 2312-2320 of process 2300 are repeated, using a relaxed edit distance (e.g., an edit distance of 3 for either/both of portions 2312 and 2320). In various embodiments, if no loci within a threshold edit distance of the sample is found, no properties will be assigned to the file. Additional processing can be triggered by such a situation, such as by causing the sample to be transmitted to services 106/108 for further evaluation, generating an alert, etc.

Returning to the example of Alice, agent 154 is awaiting a verdict on whether helperapp.exe is malicious. As platform 102 has not previously encountered Alice's copy of helperapp.exe, providing a result after thorough analysis (e.g., subjecting the sample to analysis by services 106 and 108 and awaiting results) may be time consuming-likely too time consuming to be acceptable to Alice. Using the techniques described herein, platform 102 can provide a verdict back to agent 154 (e.g., malicious or not) by leveraging information previously ascertained from previously seen samples (that are sufficiently similar to the sample submitted by Alice). In particular, by employing an embodiment of process 2300, platform 102 can determine that helperapp.exe is either a member of an existing cluster (previously determined to be benign) or is a member of a new cluster that is sufficiently close to an existing cluster that it should inherit the properties of the existing cluster.

Additional Example: oem-drv64.sys

The file, oem-drv64.sys, is a piece of software that allows causes an installation of Microsoft WindowsNT or Windows 7 to appear to be legitimately licensed, when in fact the installation is not. The software includes two encrypted sections and copies of oem-drv64.sys often have different MD5s. As a result, suppose a total of 15,000 samples of oem-drv64.sys have been submitted to services 106/108 for analysis. The file is not malicious (and, services 106/108 have consistently returned verdicts of “not malicious” accordingly).

Using techniques described herein, various samples of oem-drv64.sys are assigned cluster identifiers of (for example): h08.928d8f3679816008, h08.960d8f3679816008, and h08.96898f3679816008, which are nearby one another using edit distance. Suppose a new sample of oem-drv64.sys (whose cluster identifier is not already present on platform 102) is provided to platform 102. Information such as the size of the nearest cluster neighbors, and the verdict information, can be used to rapidly provide a verdict on the newly received copy of oem-drv64.sys.

Additional Example: False Positive

Suppose a sample is received (e.g., at 2302) which has an MD5 value of 695aef60f6439d896ec2c8547482caf2 and is clustered (e.g., at 2310) to h10.86be1a430300c028. The software is not malicious, but has two false positive reports (e.g., as determined by services 106/108) and fifty five negative reports. A search of nearby clusters (e.g., at 2314) locates h10.86be1a430300c048 with four member hashes (2316). The locus of 695aef60f6439d896ec2c8547482caf2 is retrieved and compared to the locus of the samples in h10.86be1a430300c048 (e.g., at 2318). Suppose the four samples are each within one bit of the locus of the query sample, which is within an edit distance threshold for closeness. The four samples have MD5s, respectively, of b0fcd15501d36c9f6d6f32bcd1cd9a52, d24080d3474c38ba9c7de1715423a4e2, 8f3298572314d9a7a99472253208939b, and 8b47c1be17c139d111d1cffe5e8c1f50. Each of the four samples belongs to an open source software package and is not malicious. When platform 102 is consulted regarding the sample, the properties of the cluster (e.g., verdict good) can be inherited, and the false positives from 106/108 ignored accordingly.

Signature Generation for a Cluster of Files

Described herein are various embodiments of approaches for generating signatures for a set of files having an arbitrary file type. In various embodiments, cluster signature generation is performed by signature generator 226. An example way to implement signature generator 226 is as a set of scripts authored in one or more appropriate scripting languages (e.g., Perl). The files included in the set share some clustering commonality: the samples can be determined to be similar using techniques described above, and can also have been determined to be similar through other approaches (e.g., a third party has determined them to be similar and provided them as input). The following are examples of attributes that a given cluster can share:

- The files all have the same “type” application/object. There are several ways to define file type, some of which are: (a) the unix file(1) filemagic command; (2) TrId (a commercial application); and (3) mime-type.
- The files are all of a size lower bound of 4{circumflex over ( )}order-1 to 4{circumflex over ( )}order.
- The files all have the same locus.
- The files share a common LSH (locus) [clusterID].
- The files have nearby clusterIDs (with an edit distance of “n” where “n” is the number of bits).

One example cluster is “m9e0.425412e3c6930b14” of 17,043 MS-Word documents that have a file size between 262144 (4{circumflex over ( )}8) and 1048576 (4{circumflex over ( )}9) bytes. The order is the exponent of 4. An example file included in this cluster is md5:001ba5033f18cac00e8e8512ba865e50 with a size of 156,828 bytes.

Approach A

One example way to make a signature for the cluster is as follows. The first part of the approach is to find the commonality. The second part of the approach is to search for that commonality in all of the other samples. Example pseudocode for the approach is depicted in FIG. 24.

Find three files in the cluster that share the same loci. Recall that in various embodiments the clusterID is a locally sensitive hash of the locus (loci) and that a cluster may have more than one locus in it. Of the three files select one arbitrarily (FILE A) and load it into a suffix tree. Given that the three files all have the same order (of 9) they all are between the size of 4{circumflex over ( )}order and 4{circumflex over ( )}(order-1).

File B: Select one of the two files that were not loaded into the suffix tree. Take File B and divide it into 4{circumflex over ( )}order-1 chunks. Search the suffix tree (FILE A) for each chunk. FIG. 25 depicts example Golang code for searching the suffix array. If the search returns no result, continue to decrement the order by one. If an order of 3 is reached, quit (4{circumflex over ( )}3=64). The minimum block size is arbitrary. Larger and smaller numbers can also be used. If a match is located, exit the loop and label the matching data as CANDIDATE_SIGNATUREDATA.

Take the CANDIDATE_SIGNATUREDATA and test if it is in FILE C (the file that was not loaded into the suffix tree and not used to match). If the candidate signature data is also located in FILE C, continue. If FILE C does not have a match, exit with an error “too little commonality.”

Take the CANDIDATE_SIGNATUREDATA that matches and select the n bytes (e.g., 32 or 64 bytes) that have the highest Shannon entropy. This will become the candidate_signature. In various embodiments, the candidate_signature is located within the CANDIDATE_SIGNATUREDATA by sliding an n-byte window (e.g., 32 or 64 bytes) across the CANDIDATE_SIGNATUREDATA locating a contiguous length of n-bytes (e.g., 32 or 64) with the highest Shannon entropy.

Create a YARA format signature that defines the candidate_signature using a predefined template. Other signature formats instead of YARA signatures can also be used in accordance with the techniques described herein (e.g., Snort, Suricata, Bro, etc.).

Load the YARA (or other format) signature and iterate over the files in the cluster or locus noting how many match. Once a limit of success or failure, or a ratio has been met, exit. As applicable, signatures that meet some ratio (e.g., 99%) of matches are then subjected to an optional quality assurance process, and signatures that fall below the matching ratio are marked as “failed” signatures. One example of a quality check that can be performed is to determine whether the selected chunk (or a 32- or 64-continuous byte string after Shannon Entropy or other entropy checking) is present in all of the cluster members. In some embodiments, the selected chunk/string must be present in all cluster members, present in a percentage of cluster members (e.g., 90%), etc. In some cases, the initially selected file (e.g., A or foo.exe) may not yield any appropriate strings. In such scenario, another file in the cluster can be selected and loaded into the array for processing. When an appropriate string is obtained, a signature (e.g., Yara) can be made and then tested against a corpus of known malware (e.g., malware seen last month), known good files, etc., to see whether it results in false positives/misidentifies malware. In some embodiments, known good files are checked against prior to generating signatures. As one example, malware samples clustered together and identified as “file infector” malware can be associated with the known good file (e.g., an official Adobe DLL) that they infected. When selecting a byte sequence, a check can be made that the byte sequence is present in the cluster and not present in the known good file. In various embodiments, multiple signatures are generated from the same process (e.g., in different formats, or using slightly different Shannon entropy selections, e.g., shifted left 1 bit or right 3 bits).

The following are example ways of creating the signature:

Option 1: Choose a length for the signature (e.g., 32 or 64 bytes). Locate a contiguous bytestream with the highest Shannon Entropy (or other appropriate selection criteria) of the chosen length by iterating over a window of n bytes wide and calculating the Shannon Entropy. Choose the byte sequence with the highest entropy that is not a cryptographic token/signature/etc. within the file. Signed PE files, for example, include well-documented cryptographic sections. The offsets of these sections can be skipped from consideration. Similarly, regions of other types of files (e.g., regions of PDFs) that are unsuitable for selecting from can also be excluded. In various embodiments, pre-processing is performed when loading file A into the array for processing (e.g., to analyze file A for portions of the file that should be excluded such as cryptographic areas and designate those portions as not available for byte sequence matching). As yet another example, any offsets that occur within the first 1k of the file can be excluded from consideration (e.g., because most DOS programs share a common string in the first 1k of their files).

Option 2: Calculate a hash (e.g., MD5, SHA1, SHA256, etc.) of the chunk, and record the offset of where the chunk appears in the files. Later, malware can be found by taking a hash of the appropriate n bytes at the appropriate offset location in a sample and comparing the result to the calculated hash.

Approach B

A second approach to generating a signature for a cluster of files is as follows:

Given a file, determine a cluster and locus of the file (e.g., using techniques described above).

Given a cluster:

- find all loci in the cluster;
  - for each locus:
    - for all files in the locus:
      - locate the most entropy of length n where n>=64 bytes by iterating a window across the file (e.g., where window size is of the 4{circumflex over ( )}order-1) and decreasing the window size until a longest contiguous match between the target file and an arbitrary file from the locus is found. If the file is signed, ignore the signature portion of the file.

Once a common string has been identified, test all the rest of the files in the locus and select the most entropic string by searching for the bytes in a suffix trie.

Search nearby neighbor loci or clusters and loci for matches, and search against known good files for the string. If there are zero matches from the known good files and a minimum of n files within the cluster or locus match, publish the signature.

Graphical Example

Graphical representations of aspects of an embodiment of signature generation for a cluster of files are shown, respectively, in FIGS. 26 and 27. For a clustered set of files, arbitrarily take one file from the cluster foo.exe (2602) and put it in a (e.g., binary compatible) suffix tree. Determine which size power of four the file fits into (e.g., 4⁶), subtract one from that value, and then break a second file bar.exe (2604) into blocks of that size (e.g., 45). In this example, bar.exe shares a locus (e.g., of 64 bytes) with foo.exe and also shares a cluster identifier (e.g., of 64 bits) with foo.exe, but has a different file hash from foo.exe. The approach also works with neighbors (e.g., having respective loci that have nearby edit distances). In this approach, the biggest possible block is first used to search the suffix tree. If no results are found, bar.exe is broken into smaller blocks, and each of those blocks is used to search the suffix tree. Eventually, a hit and offset(s) of where the hit occurs are obtained, subject to applicable constraints (e.g., not occurring in cryptographic section of file, not occurring more than 10 times, etc.).

FIG. 26 illustrates an example of a chunk present in foo.exe (2606) also being present within bar.exe (at 2608). If the chunk is present in three places, the offsets of those three places will be returned when searching the suffix tree. In this example, suppose foo.exe and bar.exe are two malware samples that link against a library and the chunk is a portion of a common library call between both files. Once the chunk has been identified as present in both foo.exe and bar.exe (using suffix tree search), the same chunk can be searched for in other files in the cluster (e.g., using the Perl Compatible Regular Expression (PCRE) library to search). An example of contents of a chunk that can be found using these techniques is within a section of an application (e.g., a .bss section) where static variables are uninitialized but loaded into memory. Often, malware samples will share a segment in common and the contents of the segment can be used to generate a common signature.

Suppose padding is present in the files in the cluster being analyzed. FIG. 27 shows three files (foo.exe, bar.exe, and c.exe). Each file shares the previously described chunk (at 2702, 2704, and 2706, respectively). Each file also has a portion of padding (e.g., 1010101010101) 2708. While padding 2708 is also present in all of the files, in various embodiments, padding 2708 will not be selected for use as a signature because its entropy is low. The approaches described herein look for the biggest block shared by files in a cluster that also has sufficiently high entropy.

Amino Acid Hilbert Mapping

As mentioned above, a variety of types of data/files can be used in conjunction with techniques described herein. A further example includes biological sequences (e.g., DNA or RNA sequences), which can also be converted into images and then analyzed (and, e.g., clustered) for a variety of purposes.

The following describes an example implementation, with respect to the nucleotide “CY136712” which corresponds to a portion of Influenza A virus. This particular example is a 1,442 base pair of mRNA, a copy of which can be obtained by parsing a GenBank record (e.g., located at https://ncbi.nlm.nih.gov/nuccore/CY136712):

ORIGIN

1 agtgaaaatg aatccaaatc agaagataat aacaattggt tccgtctctc taaccattgc

1 agtgaaaatg aatccaaatc agaagataat aacaattggt tccgtctctc taaccattgc

61 aacagtatgt ttcctcatgc agattgccat cctagcaacg actgtaacac tgcatttcaa

121 gcaaaatgaa tgtagcaccc ccgcgaacaa ccaagtagtg ccatgtgaac caatcataat

181 agagaggaac ataacagaga tagtgtattt gaataatact accatagaga aagagatttg

241 tcctaaagca gtagactaca ggaattggtc aaaaccgcaa tgtcaaatta caggatttgc

301 tcctttttcc aaggacaact caattcgact ttctgctggt ggggacattt gggtaacaag

361 agaaccttat gtgtcatgtg acaccagtaa atgttaccaa tttgcacttg ggcaggggac

421 cacgctggac aacaaacatt caaatggcac aatacatgat agaatccctc atcgaaccct

481 tttgatgaat gagttgggtg ttccgtttca cttgggaacc aaacaagtgt gcatagcatg

541 gtccagctca agttgccatg atgggaaagc atggttgcac gtttgtgtca ctggggatga

601 tagaaatgcg actgctagtt tcatttatga tgggatgctt gttgacagta ttggttcatg

661 gtctcaaaat atcctcagaa ctcaggagtc agagtgcgtt tgcatcaatg ggacttgtac

721 agtagtaatg actgatggaa gtgcatcagg aaaggccgat actagaatat tattcattaa

781 agaggggaaa attgtccaca ttagcccact gtcaggaagt gctcagcata tagaggaatg

841 ttcctgttat ccccgatacc caaacgtcag atgtgtttgc agagacaact ggaagggctc

901 taataggcct gttatagata taaatatggc agattatagc attgattcca gttatgtgtg

961 ctcaggactt gttggagaca caccaaggaa cgatgatagc tctagcagca gcaactgcag

1021 ggatcctaat aatgagaggg gaaacccagg agtgaaaggg tgggcctttg acaatggaga

1081 tgatgtttgg atgggaagaa caatcagtaa agattcgcgc tcaggctatg agacctttag

1141 ggtcattggt ggttgggcca ctgctaattc caagtcacag accaataggc aagtcatagt

1201 tgacaataac aactggtctg gttattctgg tattttctct gttgaaaaca aaagctgcat

1261 caacaggtgt ttttatgtgg agttaataag aggaaggccg caggagacta gagtatggtg

1321 gacctcaaac agtattgtcg tattttgtgg cacttcaggc acctatggaa caggctcatg

1381 gcctgatggg gcgaatatca atttcatgcc tatataagct ttcgcaatgt tagaaaaaaa

1441 ct

An embodiment of data analysis platform 102 can process nucleotide “CY136712” in accordance with techniques described above as follows. Each letter (A, T, C, G, M, or N) is mapped to two bits. Some letters as special, such as “N” which can map to anything (and, is mapped to “00” in some embodiments). Example code for performing the mapping is as follows:

- var (