As one example the search can be a k-nearest neighbors search.

As another example, e.g., when the number of memory data items is large, the search can be an approximate k-nearest neighbors search through the key embeddings.

The system generates, from the query embedding and the respective value embeddings for the context data items that were identified using the query embedding and the respective key embeddings for the plurality of memory data items, a classifier input (step 210).

Generally, the system can combine the query embedding and the respective value embeddings for the context data items to generate the classifier input.

The system can combine the query embedding and the value embeddings for the context data items in any of a variety of ways.

As one example, the system can first generate a combined value embedding from the respective value embeddings for the context data items and then combine the combined value embedding and the query embedding to generate the classifier input. For example, the system can sum or average the combined value embedding and the query embedding to generate the classifier input.

To generate the combined value embedding, the system can generate an initial combined value embedding from the respective value embeddings for the context data items. For example, this can be done by computing a mean of the respective value embeddings for the context data items.

The system can then optionally apply a dense neural network layer to the initial combined value embedding to generate the combined value embedding. For example, the query embedding can have one dimensionality and each value embedding can have a second, different dimensionality. The system can then use the dense neural network layer to map the initial combined value embedding from the second dimensionality to the first dimensionality.

When the system uses the dense layer in any of the above examples, the dense layer is trained jointly with the classifier neural network during training on the labeled training data for the classification task (described above).

As another example, the system can combine the query embedding and the respective value embeddings for the context data items to generate the classifier input using an attention mechanism. This attention mechanism will also be referred to in this specification as a memory attention module (MAM).

In particular, the system can process the query embedding and the respective value embeddings through a sequence of one or more attention layers.

Each attention layer is configured to receive an input query embedding and compute a respective attention weight for each context data item by, for each context data item, computing an attention weight between the input query embedding and the key embedding for the context data item.

To compute the attention weights, the layer can apply one dense layer to the input query embedding to generate a query and apply another dense layer to the key embeddings to generate keys.

The layer can then perform scaled dot product attention between the queries and the keys to generate a respective attention weight for each of the context data items.

The attention layer is then configured to compute an aggregated value embedding by computing a weighted sum of the value embeddings for the context data items in accordance with the respective attention weights and then use the aggregated value embedding to update the input query embedding.

For example, to use the aggregated value embedding to update the input query embedding, the layer can generate an initial updated query embedding from the aggregated value embedding and then combine the initial updated query embedding and the query embedding to generate the updated query embedding. For example, the layer can sum the initial updated query embedding and the query embedding to generate the updated query embedding.

To generate the initial updated query embedding, the layer can apply a dense neural network layer to the aggregated value embedding to generate the initial updated query embedding. For example, as described above, the query embedding can have a first dimensionality, each value embedding can have a second, different dimensionality and the dense neural network layer can map the aggregated value embedding from the second dimensionality to the first dimensionality.

f_Lz_iM_NNψ_Qψ_KAs a particular example, the output of the L-th attention layer can be represented as:

f
L

⁢

z
i

⁢

M
NN

⁢

V
NN

⁢

ψ
Q

⁢

ψ
K

f
i
L

=

z
i

+

χ
(

σ
⁡
(

ψ
Q

(

f
i

L
-
1

)

⁢

ψ
K

(

M
NN

)

d

)

⁢

V
NN

,

f_Lz_iM_NNV_NNψ_Qψ_Kwhere is the query embedding, are the key embeddings, are the value embeddings, σ is a softmax function, d is a scaling factor, e.g., the dimensionality of the query embeddings or the value embeddings, and, and χ represent respective linear layers.

Generally, the input query embedding for the first attention layer in the sequence is the query embedding (of the query data item).

Thus, the attention layer(s) update the query embedding by incorporating context from the value embeddings of the memory context items. Because each layer generates a learned attention weight, the attention layer(s) can assign greater weights to more relevant memory context items, thereby increasing the influence of the value embeddings for those memory context items and reducing the influence of irrelevant context items on the final input to the classifier.

When the system uses the memory attention module, the attention layers of the memory attention module are trained jointly with the classifier neural network during training on the labeled training data for the classification task (described above).

The system processes the classifier input using a classifier neural network to generate a classification output for the query data item (step 212).

Optionally, after the system generates the classification output for the query data item, the system can generate a value embedding of the query data item using the embedding neural network used to generate the value embeddings and then adds the query embedding and the value embedding to the memory. That is, the system uses the query embedding of the query data item as the key embedding of a new memory data item and the value embedding of the query data item as the value embedding of the new memory data item. The system can then discard the query data item.

FIG. 3 shows an example 300 of the operation of the system when the system uses an attention module to combine the query embeddings and the value embeddings of the context data items.

As shown in the example of FIG. 3, the system receives query image.

The system then performs a kNN search over the memory data items to identify three context images as described above.

The system then processes the query embedding of the query image and respective value embeddings for each of the context images using a memory attention module to generate the final classifier input that will be processed by the classifier neural network.

As can be seen from the example of FIG. 3, the system learns the importance of each retrieved image by assigning high weights (green line in the figure) to the relevant images, and low weights (red line) to the irrelevant images.

This can therefore improve the performance of the classifier in classifying the query image.

FIG. 4 shows another example of the operation of the system.

As shown in FIG. 4, the system receives a query image.

The system also maintains a set of memory images. In the example of FIG. 4, the memory images are each associated with a respective text description.

The system generates the memory data 120 by processing the memory images using the first embedding neural network to generate the key embeddings, but generates the value embeddings by processing the text description associated with each of the memory images using a memory value encoder, e.g., a text encoder neural network, e.g., a Transformer neural network or a recurrent neural network (RNN).

The system processes the query image using the first embedding neural network to generate a query embedding of the query image.

Because the data items in the example of FIG. 4 are images, the first embedding neural network is a visual encoder neural network, e.g., a convolutional neural network or a vision Transformer neural network.

The system then performs a k-NN or approximate k-NN search over the memory to identify, as context data items, a subset of the plurality of memory data items using the query embedding and the respective key embeddings for the plurality of memory data items. In particular, the output of the search are respective key and value embeddings of a set of context data items (denoted KNN keys and values in the Figure).

The system generates, from the query embedding and the respective value embeddings for the context data items, a classifier input (a “refined embedding”).

As shown in the example 400, the system uses the memory attention module to generate the refined embedding.

The system then processes the classifier input using a classifier neural network generate a classification output for the query data item.

In particular, as can be seen from the example 400 of FIG. 4, although the query data item and the memory data items are images, the system uses embeddings of corresponding text of relevant memory images in order to provide additional information to the classifier neural network that can assist the classifier in accurately classifying the query image. Thus, the system can effectively incorporate multi-modal context in order to improve performance on an otherwise uni-modal classification task.

FIG. 5 shows an example 500 of the performance of the described techniques.

In particular, the example 500 shows the performance (in terms of accuracy) of the described techniques (“ours”) relative to three baselines and four existing, high-performing techniques on the iNaturalist2021-Mini (fine-grained classification) and Webvision (learning with noisy labels) downstream data sets.

As can be seen from FIG. 5, the described techniques significantly outperform the baselines on the fine-grained classification task and also significantly outperform the baselines and the existing techniques on the learning with noisy labels task.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.