Optionally, generating the output indicative of whether the query document image contains an anomaly or not may be based on metadata about the query document image. Metadata may comprise information about the query document image such as image capture information, for example as can be found in any EXIF information present in obtained images, location and time of transmission, resolution, image compression, etc. That is, in addition to the features described above, metadata may also be taken into account by the method when deciding whether the query document contains an anomaly or not.

Optionally, the method may comprise aligning the training document images globally with respect to a reference document image, segmenting ROI images from each aligned training document image and further aligning the segmented ROI images locally with respect to the corresponding ROI image in the reference document image. Advantageously, this improves the alignment of ROI images in the data set beyond what is possible with a global alignment and therefore reduces unwanted noise in the data.

Optionally, the training document images may be segmented with respect to a segmentation mask, wherein the segmentation mask may be computed by obtaining the plurality of globally aligned training document images, generating a variability image by computing the pixel-level standard deviation, variance or any other suitable measure of variability of each pixel of the aligned training document images, across the aligned images, generating the segmentation mask by thresholding the variability image and defining one or more candidate ROIs for the document class by segmenting the segmentation mask. ROIs may be selected from the candidate ROIs based on the variability of pixels within the candidate ROIs between documents, for example a candidate ROI may be selected if an average of the measure of variability of its pixels or a fraction of its pixels that are above the thresholding threshold (or any similar measure) is between a lower and an upper threshold. In this way, ROI that have sufficient variability to indicate that something of interest is present while not being so variable as to indicate substantially noise rather than signal are selected for further processing.

Aspects extend to a computer program product or one or more computer-readable media encoding computer instructions that, when executed on a computing device, implement methods as described above and to a system comprising the one or more computer readable media, a memory, and a processor for executing the instructions. Further, aspects extend to a computer system comprising means for implementing the method steps described above and a computer system comprising a memory and a processor configured to carry out the method steps described above.

Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating the principles of the invention by way of example only.

BRIEF DESCRIPTION OF THE DRAWINGS

The advantages of the invention described above, together with further advantages, may be better understood by referring to the following description taken in conjunction with the accompanying drawings. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

FIG. 1 illustrates a method of training a system for detecting anomalies in images of documents in a class of documents.

FIG. 2 illustrates a method of training a system for detecting anomalies in images of document in a class of documents by generating transform-specific and region-specific anomaly scores.

FIG. 3 illustrates a schematic diagram of one embodiment comprising transform-specific and region-specific anomaly scores.

FIG. 4 illustrates a schematic diagram of another embodiment comprising transform-specific anomaly scores.

FIG. 5 illustrates a computer-implemented method for detecting anomalies in images of documents in a class of documents.

FIG. 6 shows an example result of color and background anomalies flagged by the method in a query image document.

FIG. 7 shows a) example results of anomalous Union Jack ROI images of UK driving licenses and b) a reference image of a Union Jack ROI image of a UK driving license not containing an anomaly.

FIG. 8A shows an example query document image of a UK driving license containing anomalies in the outer top left regions and in the ‘Driving License’ text region.

FIG. 8B shows results showing the anomalies detected by the method (indicated by the black arrows).

FIG. 8C shows an example of an image of a portion of a UK driving license containing anomalies (indicated by the ‘x’ symbol) and an example of a portion of a UK driving license not containing anomalies (indicated by the tick symbol).

FIG. 9 shows an example implementation of a computing device which may be used to implement the methods of the present disclosure.

FIG. 10 illustrates a schematic diagram of a plurality of ROI-specific models for particular documents.

FIG. 11 illustrates a method of sourcing fraud samples from production images to be used in training a supervised machine learning model.

FIG. 12 illustrates a schematic diagram of generating embeddings from an image of a fraudulent document and determining its reconstruction error.

DETAILED DESCRIPTION

FIG. 1 schematically depicts a high-level overview of a method 100 of training a system for detecting anomalies in images of documents in a class of documents. The documents may be government issued documents or otherwise official documents. An official document may take one of many forms such as a driving license, a passport, a utility or other bill, a birth certificate, a benefits book, a state identity card, or a residency permit. The term “official document” is therefore intended to cover any document that contains structured information that may be used to verify a person's identity or an aspect relating to a person, for example their address. The different forms of official document may be referred to herein as being a particular “class” of document, which may be further restricted by the territory of issue. For example, an official document class might be a ‘UK passport’, a ‘Finnish driving license’, etc.

With reference to FIG. 1, the method comprises three main stages: obtaining a plurality of training document images, processing each training document image to generate anomaly scores and computing thresholds to separate document images containing an anomaly from document images not containing an anomaly. A document image containing an anomaly may be indicative of fraud, and the document may thus be flagged or an alarm may be raised for the document to be reviewed by a human or rejected.

At step 102, a plurality of training document images of training documents in the class of documents is obtained. The document images may be obtained over a communications network from a client device, for example from a browser or an application. Specifically, a user on the client device may supply an image of the document, for example by taking a picture with the device or uploading a stored image.

At step 104, each training document image is segmented into a plurality of region of interest (ROI) images, each ROI image corresponding to a respective ROI of the training document. A region of interest of an image represents a region deemed to be rich in information about the image and can be a portion of or the whole image. For example, an ROI of a document can be a portion of a document that is rich in information about the document, unique and hard to reproduce. In practice, ROI images of a document could comprise images of security features, face pictures, sophisticated background patterns, digital images of holograms, etc. The ROIs of a document may be defined manually or automatically and are specific to the class of document. That is, if the class of documents is UK driving licenses, the Union Jack and the ‘Driving license’ text at the top of the document may be two of the ROIs for this class of documents. At the end of this step, a plurality of ROI images are obtained from the query document image.

At step 106, for each ROI image, a plurality of transformations is applied to the ROI image to generate respective transform-specific features for the ROI image, and transform-specific anomaly scores are generated from the transport-specific features, representing a measure of how well a feature of the ROI image matches corresponding features in ROI images of training documents of the same class of documents other than the respective training document. The number and/or type of transformations that are applied may be the same or different for each ROI image and may be advantageously chosen so they are specifically suited to the ROI image. For example, in the case of the Union Jack ROI image on a UK driving license, the transformations applied may be chosen to capture the color features of the Union Jack symbol. In some embodiments, the same number and type of transformations may be applied to all or some of the ROI images, while in other embodiments, different numbers and types of transformations may be applied to each ROI image. Steps 104 and 106 are repeated for each training document image in the plurality of training document images.

At step 108, based on the respective anomaly scores computed for the plurality of training document images, the method computes a transform-specific threshold for each transformation to separate document images containing an anomaly from document images not containing an anomaly.

With reference to FIG. 2, applying a plurality of transformations to an ROI image to generate respective features for the ROI image may comprise one or more of the following steps. A plurality of raw features may be extracted 200 from the ROI image, and then, for each raw feature extracted, a transformation may be applied 202 to generate a respective transform-specific feature for the ROI image. By way of example, applying the plurality of transformations may comprise one or more of extracting color information, frequency information, texture information (e.g., local binary patterns), shape information, or machine learnt features, such as deep learning features. Optionally, extracting, from the ROI image, a plurality of raw features may comprise pre-processing such as contrast stretching, transformation of the ROI image to another color space (e.g., YcBcR color space, grayscale image, HSV) and applying a pre-trained machine learning algorithm to the image (e.g., a pre-trained neural network). Next, respective transform-specific anomaly scores may be generated from the transform-specific features. It will be understood that the separation into raw and transform-specific transformations is optional, for example as described above with reference to FIG. 1.

At step 204, for each ROI image, a transformation may be applied to the transform-specific anomaly scores for the ROI image to generate a region-specific feature for the ROI image. The transformation applied in this step summarizes the transform-specific anomaly scores of the ROI image as the region-specific feature for the ROI image using information captured from all the transform-specific anomaly scores of the ROI image (e.g., color information, frequency information, pattern information, etc.). The transformation may, for example, simply concatenate the transform-specific anomaly scores for each ROI into a region-specific feature or may apply further operations to the transform-specific anomaly scores.

At step 206, a region-specific anomaly score may be generated from the region-specific feature for the raw image. The region-specific anomaly score may represent a measure of how well the ROI image matches corresponding ROI images of training documents of the same class of documents other than the respective training document. The region-specific anomaly score may be the region-specific feature (unity transformation), or a function of the region-specific feature with the same or different dimensions as the region-specific feature, or a scalar, etc.

After each a region-specific anomaly score is computed for each ROI image of each training document image, step 206 may compute, based on the respective region-specific anomaly scores of the plurality of training document images, a region-specific threshold for each ROI to separate document images containing an anomaly from document images not containing an anomaly.

An optional step 208 may further comprise generating, using the transform-specific and/or, as applicable, the region-specific anomaly scores for a training document image, a global anomaly score for the respective training document image. This process may then be repeated for each training document image in the plurality of training document images so that a global anomaly score is obtained for each training document image.

After the global anomaly scores are computed for each training document image, optional step 210 may compute, based on the respective global anomaly scores, a global threshold for the document class to separate document images containing an anomaly from document images not containing an anomaly.

The thresholds (i.e. transform-specific, region-specific, global) may be computed based on a false rejection rate (FRR), which represents the rate at which document images not containing an anomaly are tolerated to be incorrectly flagged as containing anomalies. The FRR may be determined a priori, and based on this the transform-specific thresholds, region-specific thresholds and global threshold may be set using holdout validation to achieve the corresponding predetermined FRR, for example as follows: a portion of the FRR is allocated to the transform-specific thresholds, a portion of the FRR is allocated to the region-specific thresholds and the remaining FRR budget is allocated to the global threshold. For example, if the FRR is set to be 6%, this may be equally allocated between the transform-specific, region-specific and global thresholds, so that T_transform=T such that flag_transform=2%, T_region=T such that flag_region=2% and T_global=T such that flag_global=2%. Alternatively, the FRR may be split unevenly between the three types of thresholds, or follow any other schedule, such as split to scale with the number of anomaly scores for each type of threshold.

The presence of an anomaly in a query document may be flagged if any of the transform-specific anomaly scores or any of the region-specific anomaly scores or the global anomaly score is above the corresponding threshold for that specific anomaly score (e.g., for the specific ROI image and for the specific transform). The presence of an anomaly in a query document image may also be flagged if the global score for the query document image is above the global threshold. In other words, a query document may be flagged to contain an anomaly if any of the transform-specific thresholds or region-specific thresholds or global thresholds are exceeded by the corresponding anomaly score of the query document.

The one or more of the transformations may comprise machine learning models and/or machine learning models may be used to produce anomaly scores as output. The machine learning models may comprise at least one supervised machine learning model or at least one unsupervised machine learning model. The machine learning models may comprise two or more class classifiers or clustering algorithms if both anomalous and non-anomalous documents are available for training a particular model at region and/or global level. The machine learning models may comprise anomaly detectors (e.g., isolation forests or one-class classifiers) if only non-anomalous documents are available for training a particular model at region and/or global level. Depending on the amount and type of training data available for each class of documents and for each ROI in the document, either a supervised or unsupervised machine learning model (anomaly detector or classifier/clustering model) may be manually or automatically selected. When sufficient document images containing an anomaly are available for a particular ROI and class of documents, a supervised machine learning model may be chosen, for example a two or more class classifier, and trained in a supervised manner using the labelled document images containing an anomaly and the labelled document images not containing an anomaly. In some embodiments, clustering models may be used in that case.

As mentioned above, training of a supervised machine learning model typically requires a reasonably large number of sample training images (i.e., document images containing an anomaly). Often, the availability of such training samples is limited, which can lead to a less robust supervised model and thus less accurate anomaly detection. To overcome this challenge, in some embodiments the system 100 can utilize a training sample generation pipeline to dynamically source a large number of training samples using production images.

The training sample generation pipeline leverages the ROI approach described herein by generating a dedicated supervised model for each of one or more ROIs in a given document, and training the dedicated models using a common data set comprising both genuine document samples and fraudulent document samples. The use of a common data set to generate and train the ROI-specific models eliminates the need to source a different training data set for each ROI. Furthermore, the common data set of genuine and fraudulent samples can be used for any ROI, even if a particular sample is not fraudulent for all ROIs in a document. As can be appreciated, the training sample generation pipeline described herein produces a flexible and scalable solution for sourcing training samples because a separate data set of genuine and fraudulent images need not be created for each ROI.

FIG. 10 illustrates a schematic diagram of a plurality of ROI-specific models for particular documents. As shown in FIG. 10, the system builds a dedicated supervised machine learning model 1006a for ROI 1008a in document 1010a, a dedicated supervised machine learning model 1006b for ROI 1008b in document 1010b, and a dedicated supervised machine learning model 1006c for ROI 1008c in document 1010b. These models 1006a, 1006b, and 1006c are built and trained using images from a production data store comprising genuine samples 1002 and fraudulent samples 1004. Advantageously, the system can train each model using one or more ROIs from genuine samples and/or fraudulent samples, regardless of whether a given sample is genuine or fraudulent for each ROI. This greatly expands the pool of available training samples for each ROI and enables generation of more accurate supervised machine learning models for the ROIs.

To accomplish this objective, the training sample generation pipeline enables an end user (e.g., an engineer, researcher, or system administrator), given one fraudulent sample, to source a large number of training samples from production data. FIG. 11 illustrates a method of sourcing fraud samples from production images to be used in training a supervised machine learning model. As shown in FIG. 11, the system trains (step 1102) one or more initial ROI model(s) using available production data (i.e., genuine samples 1002 and fraudulent samples 1004)—as described above with respect to FIG. 10. Once trained, the ROI model(s) are deployed to production (step 1104).

As the model(s) are used in production, one or more fraudulent images are identified (step 1106) using any of the anomaly detection techniques described herein. Concurrently, information relating to missed fraud and/or escalations is received (step 1108) from external sources, such as APIs and/or customer data. For example, a customer or external source using the anomaly detection system may determine that one or more fraudulent documents were not properly detected. Images and/or other data relating to the missed fraudulent documents can be provided to the system for use in generating new fraud samples.

When images of fraudulent documents are captured at steps 1106 and 1108, the system can process these images using search engine 1110 to identify similar documents from the fraudulent samples database 1104 and then use the captured images along with similar fraudulent images to train and/or retrain the ROI model(s). Specifically, the images are processed as follows:

- a) One or more ROIs in each document (or the entire document) are converted into a vector representation (or embedding);
- b) The embeddings are projected into an embedding space comprised of embeddings generated from a corresponding ROI in each of one or more images from the fraudulent samples 1104;
- c) The projected embeddings are compared to the embeddings from the sample image ROIs to determine a similarity to one or more of the embeddings (e.g., using a distance metric or other type of comparison technique);
- d) Images that have a similarity to the captured fraudulent image are identified as new fraud samples 1112; and
- e) The group of images with determined similarities are used as a training data set for subsequent training of ROI model(s).

FIG. 12 illustrates a schematic diagram of generating embeddings from an image of a fraudulent document and determining its reconstruction error. As shown in FIG. 12, the image of a fraudulent document captured during QC or via customer escalation is provided to the system, which performs ROI extraction (step 1202) to, e.g., crop one or more portions of the image that correspond to the ROI(s). For each portion, the system applies a vision transformer (ViT) model (step 1204) which converts the image portion into a series of image patches and generates embeddings (e.g., patch and positional embeddings) from these image patches. The embeddings are fed into an auto-encoder (step 1206) which creates a latent embedding space (as described above) using the labels provided (i.e., fraudulent or genuine). The system then performs similarity computation (step 1208) using the latent embedding space and decodes the embedding back to the input space to determine the reconstruction error (step 1210). The reconstruction error is used as a measure of how fraudulent a sample is, instead of using a dedicated machine learning algorithm as is commonly done.

A benefit of the training sample generation pipeline described herein is the ability to execute the pipeline process via a user interface which enables end users to provide a given image of a fraudulent document (e.g., obtained during production use of the anomaly detection processes described herein), optionally identify one or more ROIs in the image, and search the production images (i.e., databases 1102, 1104) to source a large number of additional fraudulent and/or genuine images that are similar or have similar ROIs as the given image for the purpose of training or re-training the supervised machine learning model(s) that perform the anomaly detection in a production environment. As can be appreciated, this improvement provides the ability to quickly scale a robust, accurate supervised model for use in the system without requiring time-consuming generation of training data set(s).

Alternatively, if no or insufficient number of document images containing an anomaly are available or only a very small number of such images is available, an unsupervised machine learning model or an anomaly detection model may be chosen. Such a machine learning model may be trained using document images not containing anomalies. The modularity of the method, split across different ROI regions and transformation, resulting in distinct training data sets and distinct training requirements for each machine learning model enables the training of each machine learning model to be carried out independently. Thus, during the training stage, the machine learning models of method 100 may be trained independently.

The machine learning model can, for example, be one or more artificial neural networks (ANN)—for example, deep artificial neural networks (DNN) or convolutional neural networks (CNN), support vector machines (SVM), random forests or isolation forests. Artificial neural networks—for example, recurrent neural networks—represent a specific parametrization of a non-linear function (in terms of network weights). It will be appreciated that the present disclosure is not limited by the language used to describe the non-linear function or its structure. It will be understood that an artificial neural network in the context of a computer implemented invention refers to a physical entity that exists either in terms of a physical state of a general purpose or specifically adapted computing platform or in a specifically adapted physical circuitry, for example. The machine learning model may also comprise any form of supervised or unsupervised training algorithm.

Artificial neural networks (ANN) may be trained using a class of machine learning algorithms which characteristically use a cascade of multiple layers of nonlinear processing units for extracting features, where each successive layer uses the output from the previous layer as input—that is training a deep neural network (DNN).

One or more (in some embodiments all) anomaly scores may be computed based on a distance measure that is indicative of the distance between the transform-specific or, as applicable, region-specific features of the respective ROI image and a plurality of transform-specific or, as applicable, region-specific features of corresponding ROI images of training documents in the class of documents other than the respective ROI image. For example, the distance measure may be based on a Mahalanobis distance, as presented in more detail in the ‘Example transformations’ section below. In other embodiments, the anomaly score may be generated directly from the output of a machine learning model, for example as the output of an isolation forest, a classification score or other output of a classifier, a cluster belonging score and the like.

If the plurality of training document images contains training document images that contain and anomaly and training document images that do not contain an anomaly, a distance measure may be computed as the distance between an ROI image and corresponding ROI images of training documents in the same class of documents other than the ROI image that do not contain an anomaly. Optionally, an additional distance measure may be computed as the distance between the ROI image and corresponding ROI images of training documents in the same class of documents other than the ROI image that contain an anomaly. The distance measures may then be combined to compute an anomaly score for the ROI image.

If the one or more transformations comprise a machine learning model or one or more anomaly scores are generated using a machine learning model, as explained above, each machine learning model may be trained using either supervised or unsupervised learning, according to the amount and type of training data available for the class of document and ROI to which the transformation containing the machine learning model is applied. For the anomaly scores that are computed based on a distance measure, such as a Mahalanobis distance, the anomaly score is computed for every document image in the training set, and, once the query document is received during the inference stage, the distance measure is computed between the features generated by applying the transformations to the training set and the corresponding features of the query document.

One or more (in some embodiments all) anomaly scores may be computed based on any anomaly detection algorithm, including machine learning models. The anomaly detection algorithm may, for example, be a one-class classifier or an isolation forest. One-class classifiers are trained to identify objects of a specific class (e.g., image of documents not containing anomalies) amongst all objects, by learning from a training set containing only objects of that class (e.g., a training set formed of images of training documents in the class of documents not containing an anomaly). Isolation forest is an unsupervised learning algorithm for anomaly detection that works on the principle of isolating anomalies. In order to isolate data points containing anomalies, an isolation forest recursively generates partitions on the training data set by randomly selecting an attribute of the training instances and then randomly selecting a split value for the attribute, between the minimum and maximum allowed for the attribute.

In some embodiments, the document image (i.e. training and/or query document) may be a frame of a video of the document. The video of the query document may comprise an ordered sequence of frames and each frame in the ordered sequence of frames may correspond to a document image. The plurality of transformations may in this case be applied to the sequence of frames.

Prior to training the system, the training document images may be processed as follows. First, an image quality check may be performed to ascertain whether the document image is of a sufficiently good quality. The image quality check may also comprise checks that verify whether the full document is included in the query document image or whether there is any glare or blur present in the image which would prevent the correct processing of the query input image. Second, if the query document image has passed the quality check, the query document image may further be processed by a document classifier to determine the class of document that it belongs to. This step enables the transformations determined (and, in some embodiments, trained) during the training stage for the particular class to be applied and the relevant ROI images to be segmented (e.g., if the document class determined is UK driving licenses, then one on the ROIs may be the Union Jack picture). Third, the query document image may be aligned to a reference document image (e.g., a sample from the training set), as described in more details below, and the background in the query document image may be discarded, leaving only a crop of the document itself.

The training document images may be aligned globally with respect to a reference document image, which may, for example, be a sample training document image from the plurality of training document images. Each training document image may further be segmented into one or more ROI images. To reduce the misalignment between the same ROI images across the plurality of training document images, the method may comprise further aligning the segmented ROI images locally with respect to the corresponding ROI image in the reference document image. The local alignment may be performed using any alignment algorithm. In preferred embodiments, the alignment algorithm may be enhanced correlational coefficient maximization (see G. Evangelidis and E. Psarakis, “Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization,” IEEE Transactions on Pattern Analysis and Machine Intelligence 30(10), October 2008, 1858-1865, which is incorporated herein by reference).

The training document images may be segmented with respect to a segmentation mask, wherein the segmentation mask may be computed by obtaining the plurality of globally aligned training document images. The training document images may be resized to the same dimension and may be converted to a different color space (e.g., grayscale). Next, a standard deviation image may be generated by computing the pixel-level standard deviation of the aligned training document images. Any other measure of variability, such as variance, may be used instead of standard deviation. The standard deviation image computed this way shows where the varied information in the document lays. As certain ROIs in a document may be interpreted as static regions which do not change—or are changing much less than a varying field like personal details or signatures—across a set of documents (e.g., the Union Jack across a set of UK driving licenses would be the same, whether the signature of the driver would vary from document to document), the standard deviation image enables the automatic segmentation of ROIs for the class of documents. This way, the segmentation mask may be generated by thresholding the standard deviation image to preserve the regions with relatively low standard deviation below the threshold. Thresholding may, for example, be performed using Gaussian thresholding. Finally, one or more candidate ROIs for the document class may be defined by segmenting the segmentation mask.

With reference to FIG. 3, an example embodiment of a system embodying the described methods is presented. First, the plurality of training document images 300 are received as input, and each training image is segmented into N ROI images 302 (ROI 1, ROI 2, . . . ROI N), for example as defined in a corresponding reference document for the class of documents that the training documents belong to. Next, a plurality of transformations is applied to each ROI image to generate transform-specific features for the ROI image. In this embodiment, a different number of transformations is applied to each ROI image (M₁transformations for ROI 1, M₂transformations for ROI 2, etc.). The type of transformations applied to each ROI image may be different, the same, and/or only partially overlapping between different ROIs. Applying the transformations may in some embodiments comprise extracting a plurality of raw features 304 for each ROI image, one raw feature for each transformation. Applying the plurality of transformations may comprise one or more of extracting color information, frequency information, texture information, shape information, or machine learning features, such as deep learning features.

Next, for each raw feature of an ROI image, a transformation 306 is applied to generate a transform-specific feature for the ROI and a transform-specific anomaly score is generated from the transform-specific features. By way of example, if the raw feature extracted contains texture information, the feature generated by applying a transformation may comprise information describing the global or local patterns of the texture in the ROI image to which the transformation is applied. The transform-specific features generated at this stage are specific to each transformation and each ROI. The anomaly score may be a distance measure based on the transform specific features, or the output of a machine learning model, as described above.

At the end of this stage, based on the transform-specific anomaly scores generated for all the training documents, a transform-specific threshold is computed for each transformation from the plurality of transformations. The transform-specific threshold may be used to separate document images containing an anomaly from document images not containing an anomaly.

In the following stage, for each ROI image, a region-specific transformation 308 is applied to the plurality of transform-specific anomaly scores of the ROI image to generate a region-specific feature which summarizes information from all the features to which the transformation is applied. For example, if the transform-specific features contain information about the color, texture and shape of the ROI image, the region-specific feature will summarize the information about the color, texture and shape of the ROI image. The region-specific transformation may simply concatenate the anomaly scores of the ROI image or apply further transformations. The generated region-specific feature is then used to generate a region-specific anomaly score for the ROI image. At the end of this stage, the region-specific anomaly scores are used to compute a region-specific threshold for each ROI in the document of the class of documents. The region-specific thresholds may be used, together with the transform-specific thresholds, to separate document images containing an anomaly from document images not containing an anomaly.

The transform-specific and region-specific anomaly scores are then used to compute, for each training document image, a global anomaly score. Finally, based on the global anomaly scores 310 generated for each training document, a global threshold for the class of document is computed. The global thresholds may be used, together with the transform-specific threshold and region-specific thresholds, to separate document images containing an anomaly from document images not containing an anomaly.

With reference to FIG. 4, another example embodiment of the described methods is presented. First, training document images 400 are received as input and segmented into N ROI images 402 (ROI 1, ROI 2, . . . ROI N) as defined in a corresponding reference document for the class of documents that the training documents belong to. Next, a plurality of transformations 404 is applied to each ROI image. In this embodiment, the same number and type of transformations is applied to each ROI image to generate a plurality of transform-specific features, which are then used to generate transform-specific anomaly scores. Based on the respective anomaly scores of the plurality of training document images, method 408 computes a transform-specific threshold for each transformation to separate document images containing an anomaly from document images not containing an anomaly. Method 408 further generates a global anomaly score 406 for each training document using the transform-specific anomaly scores, and based on the respective global anomaly scores, computes a global threshold for the document class to separate document images containing an anomaly from document images not containing an anomaly.

The embodiments in FIGS. 3 and 4 are presented here by way of example only and many other embodiments are also possible.

Once the system is trained, it may be used to detect anomalies in a query document image, as illustrated in FIG. 5. First, the method may comprise obtaining 502 a query document image of a query document in the class of documents. Next, the query document image may be segmented 504 into a plurality of query ROI images. Then, for each query ROI image, the plurality of transformations, may be applied 506 to the query ROI image to generate transform-specific features for the query ROI image, and transform-specific anomaly scores may then be generated from the transform-specific features. Finally, the method may generate 508, based on the computed transform-specific thresholds and transform-specific anomaly scores, an output indicative of whether the query document image contains an anomaly or not.

The method may further comprise, for each query ROI image, generating a respective region-specific anomaly score from the transform-specific anomaly scores for the ROI, wherein generating, based on the computed transform-specific thresholds and transform-specific anomaly scores, an output indicative of whether the query document image contains an anomaly or not comprises using the computed region-specific thresholds and region-specific anomaly scores. That is, the method may generate an output indicating the presence or absence of an anomaly in the query document based on the computed transform-specific thresholds and one or more transform-specific anomaly scores (e.g., generated from one transform-specific feature capturing color information in the Union Jack of a UK driving license or one transform-specific feature capturing frequency information in the hologram of the same UK driving license). Alternatively or additionally, the method may generate an output indicating the presence or absence of an anomaly in the query document based on the computed region-specific threshold and one or more region-specific anomaly scores (e.g., an anomaly score of the Union Jack of a UK driving license). In other words, the method may generate the output based on one or more particular features of corresponding one or more particular ROI images of the query document.

The method may further comprise, generating a global anomaly score from the transform-specific and/or, as applicable, region-specific anomaly scores, wherein generating, based on the computed transform-specific thresholds and transform-specific anomaly scores, an output indicative of whether the query document image contains an anomaly or not comprises using the computed global threshold and global anomaly score. That is, in situations where the transform-specific anomaly scores and transform-specific thresholds, and, as applicable, the region-specific anomaly scores and region-specific thresholds do not flag the presence of an anomaly in the query document image, the method may still generate an output indicating the presence or absence of an anomaly in the query document image based on a global anomaly score and a global threshold.

In some embodiments, the method flags an anomaly in a query document image if any of the region-specific anomaly scores is above a region-specific threshold or if any of the transform-specific anomaly score is above a transform-specific threshold or if the global anomaly score is above a global threshold. The region-specific and transform-specific thresholds may be specific not only to the transformation, but also to the ROI (e.g., a region-specific and/or transform-specific threshold may also be ROI specific: some ROIs may have higher threshold than other ROIs). These features advantageously provide a high performance control of the model. Given the large number of ROI images and transformation for a single document image, it can be difficult to control the weighing of the resulting features to generate an outcome. The use of a global as well as region-specific and transform-specific thresholds ensures that each of these can be adjusted to take into account the available amount of images available for an ROI and a transformation. For example, if only a few images of the Union Jack ROI of a UK driving license are available prior to training the machine learning, this can be reflected both by adjusting the region and/or transform specific local thresholds for this ROI as well as the global threshold for the UK driving license to reflect how much this ROI image is weighed when generating an output with respect to other features.

Optionally, generating the output indicative of whether the query document image contains an anomaly or not may be based on metadata about the query document image. Metadata may comprise information about the query document image such as image capture information, for example as can be found in any EXIF information present in obtained images, location and time of transmission, resolution etc. That is, in addition to the features described above, metadata may also be taken into account by the method when deciding whether the query document contains an anomaly or not.

Example Transformations

The one or more transformations that are applied to ROI images are chosen in such a way as to generate features that can easily be analyzed in order to establish whether the ROI image adheres to what is expected for that particular ROI image and/or ROI feature for the document type. Depending on the type of transformation applied, a different type of pre-processing may be applied, suitable for the feature to be generated. Several examples of such transformations and pre-processing are presented in the examples below.

Color information. The plurality of transformations may comprise transformations that generate features that summarize the color characteristics of a particular ROI image. Optionally, this type of transformations may be implemented based on the histograms of the RGB values. During pre-processing, contrast stretching may be used on the RGB ROI image. To fill the entire dynamic range, every channel of the RGB image is rescaled to be between the 2^ndand 98^thpercentile of that particular channel. The transformation extracts the 3D histogram over all three input channels, after which the results 3D histogram is flatted into one feature vector zhist. Then, a Gaussian Mixture Model is fitted on the feature vectors of a plurality of corresponding ROI images of documents in the same class of documents, with the number of components chosen by minimizing the Bayesian Information Criterion. After fitting the Gaussian Mixture Model, an anomaly score is computed for the ROI image of the query document by evaluating the log probability of the concatenated histograms under the trained mixture model. Every mixture component N(x; μ_k, Σ_k) is indexed by a weighting factor π k:

log
⁢

p
⁡
(

z
Hist

)

=

log
⁢

∑

i
=
0

K

π
k

⁢

N
⁡
(

z
Hist

;

μ
k

,

∑

k

)

- Where:
- N(z_Hist; μ_k, Σ_k)=Normal distribution (Gaussian) with parameters μ_kand Σ_kevaluated at point z_Hist;
- μ_k=the mean component of the k-th Gaussian;
- Σ_k=the covariance matrix of the k-th Gaussian;
- π_k=the weighting factor of the k-th Gaussian;
- z_Hist=flattened 3D color histogram (described above);
- log(x)=natural logarithm of input x.

Frequency information. The frequency content of an image provides information about the rate of change in the image, with high frequencies representing the rapidly changing parts of the image and low frequencies the more constant parts. Such information allows the system to capture anomalies due to compression and/or texture, for instance, Discrete Cosine Transforms (see N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete Cosine Transform,” IEEE Transactions on Computers, Vol. C-23, Issue 1, January 1974, pp. 90-93, which is incorporated herein by reference) are examples of commonly used transformations in digital signal processing to decompose an image in a finite sum of cosine functions oscillating on different frequencies, hence Discrete Cosine Transforms may be applied as a transformation to extract and generate features summarizing frequency information in an ROI image.

During pre-processing, the RGB image may be transformed to YcBcR color space. After this, a Discrete Cosine Transform (DCT) may be applied to obtain:

Z
DCT

(

u
,
v

)

=

(

2
N

)

0.5

⁢

(

2
M

)

0.5

⁢

∑

i
-
0

N
-
1

∑

i
=
0

M
-
1

Λ
⁡
(
i
)

·

Λ
⁡
(
j
)

·

cos
[

π
⁢
u

2
⁢
N

⁢

(

2
⁢
i

+
1

)

]

⁢

cos
[

π
⁢
v

2
⁢
M

⁢

(

2
⁢
j

+
1

)

]

·

p
⁡
(

i
,
j

)

With
⁢

Λ
⁡
(
i
)

=

1

2

⁢

if
⁢

i

=
0

,

1
⁢

if
⁢

i

>
0.

- Where:
- N=number of rows;
- M=number of columns;
- u=row element of DCT matrix;
- v=column element of DCT matrix;
- cos[x]=cosine of x;
- π=the number pi (3.14);
- p(i, j)=pixel value at location [i, j].

The feature vector obtained Z_DCTmay then be projected down to a set of principal components z_PCAto reduce its dimensionality using Principal Component Analysis (see H. Abdi and L. J. Williams, “Principal Component Analysis,” Wiley Interdisciplinary Reviews: Computational Statistics, Vol. 2, Issue 4, Jul. 15, 2010, pp. 433-459, which is incorporated herein by reference). Alternatively or optionally, an anomaly score may be calculated based on the Mahalanobis distance, which measures the distance from a point to a distribution using the equation:

MD(x)=√{square root over ((x−μ)^TΣ⁻¹(x−μ))}

- where x is the input, μ denotes the mean of the distribution, Σ⁻¹denotes the inverse of the covariance matrix and (x-μ)^Trepresents the transpose of the vector x-p. Applying this formula to the feature vector z_PCAobtained by applying the Discrete Cosine Transform and Principal Component Analysis, results in an anomaly score calculated as:
  
  MD(z_PCA)=√{square root over ((z_PCA−μ)^TΣ⁻¹(z_PCA−μ))}
- where MD(z_PCA) represents the distance from the feature vector Z_DCTof the ROI image of the query document to corresponding feature vectors of corresponding ROI images of documents in the class of documents other than the query document do not contain an anomaly.

Texture information. The plurality of transformations may comprise transformations that generate texture features, which describe repeated variations in the intensity of a surface. Two examples of transformations which may be used to model texture patterns are Gray Level Co-Occurrence Matrices (see R. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural Features for Image Classification,” IEEE Transactions on Systems, Man, and Cybernetics, Vol. SMC-3, Issue 6, November 1973, pp. 610-621, which is incorporated herein by reference) and Local Binary Patterns (see T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution Gray-scale and Rotation Invariant Texture Classification with Local Binary Patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, Issue 7, July 2002, pp. 971-987, which is incorporated herein by reference). Gray Level Co-Occurrence Matrices analyze how differing grayscale values are distributed inside an image patch and characterizes the distribution of texture across an image at a global level. Local Binary Patterns is a local texture descriptor that models the relationship between the neighboring pixels and may be applied to generate features that are descriptive of local texture in an ROI. The pre-processing for texture information transformations may comprise converting the ROI image to a grayscale image. Similarly to frequency transformations, Mahalanobis distance may be used to compute a distance measure between the query ROI image and corresponding ROI images of documents in the class of documents other than the query document do not contain an anomaly.

Shape information. The plurality of transformations may comprise transformations that generate features that model the geometry of the objects in an image, which may be done by looking at contours, key points, or any other characteristic that describes the geometry. Shape transformations may comprise Scale Invariant Feature Transforms (SIFT) (see D. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision, 60(2), November 2004, pp. 91-110, which is incorporated herein by reference), which are a type of transforms that describe the key points in an input image. The pre-processing for texture information transformations may comprise converting the ROI image to HSV color space. Similarly to frequency transformations, Mahalanobis distance may be used to compute a distance measure between the query ROI image and corresponding ROI images of documents in the class of documents other than the query document do not contain an anomaly.

Deep learning features. The plurality of transformations may comprise transformations that generate features using deep learning models. Such deep learning models may be trained on a large and varied dataset. The trained model may then be used to generate features from an ROI image by loading the trained model and then passing the ROI image through the layers of the model, and extracting the output of one of the layers, usually the output layer, optionally applying one or more transformations such as max pooling, and generating a feature vector z_deepof the ROI image as the extracted layer. By way of example, the trained deep learning architecture may be a pre-trained VGG16 on an ImageNet architecture (see K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556 [cs.CV], Apr. 10, 2015, which is incorporated herein by reference). Optionally, the dimensionality of z_deepmay be reduced using dimensionality algorithms such as Principal Component Analysis. Similarly to frequency transformations, Mahalanobis distance may be used to compute a distance measure between the query ROI image and corresponding ROI images of documents in the class of documents other than the query document do not contain an anomaly.

Example Results

With reference to FIGS. 6-8, example results are presented.

FIG. 6 shows an example result of color and background anomalies flagged by the method in a query image document. The images on the left show two ROI images of the same forest ROI of an identity document: the top image 602 contains an anomaly while the bottom image 604 does not contain an anomaly. The color of the forest in the top image 602 is of a different color than that of the forest in documents not containing an anomaly. As indicated by the ‘x’ symbol, method 100 correctly identifies this by applying color specific transformations and flags the ROI image as containing an anomaly. The images on the right show two ROI images of background details in the same type of documents. By applying a set of transformations suited to detecting fine background details, method 100 is able to correctly flag the top image 606 as containing anomalies (lack of fine background details, as can be observed by visual comparison with the bottom image 608 which does not contain any anomalies).

FIG. 7 shows a) example results of anomalous Union Jack ROI images 702 of UK driving licenses and b) a reference image 704 of a Union Jack ROI image of a UK driving license not containing an anomaly. It can be observed that method 100 is sensitive to a varied set of anomalies within the same ROI, showing that method 100 is not specific to a particular type of anomaly, but is instead able to detect multiple different fluctuations from the non-anomalous ROI of UK driving licenses (e.g., different colors, different contrasts, presence of other unexpected features).

FIG. 8A shows an example query document image 802 of a UK driving license containing anomalies in the outer top left regions and in the ‘Driving License’ text region. Taking the example of this UK driving license, with reference to FIG. 8A, ROI images that correspond to different ROIs of the license are generated. In this particular example, ROI images are generated for regions such as the Union Jack flag, the ‘Driving License’ text, the ‘UK’/EU flag region, the face picture, background patterns, different text fields, etc.

FIG. 8B shows the results 804 of processing every ROI image (as indicated by the labels of every column) with every one of M transformations (as indicated by the labels of every row). The number in every box represents the percentile of an ROI image and a transformation computed by method 100. The corresponding percentile indicates where the ROI image of the query document image falls with respect to a plurality of corresponding ROI images of documents in the class of documents not containing an anomaly. For example, a number of 99 would indicate that the ROI image of the query document is farther away from the distribution of documents not containing an anomaly (as computed, for example, using the Mahalanobis distance) than 99% of the documents not containing an anomaly. That is, the farther away the ROI image of the query document is from the distribution for a particular transformation, the higher the number. By looking at the regions highlighted by the method to potentially contain an anomaly (indicated by the black arrows), it can be seen that method