may be trained to predict occurrence of a first type of relationship using the training data and one or more loss functions. Other types of relationships may be used in message-passing between layers of the graph encoder 122 and may be represented in the graph representation 124.

A text encoder 126 associated with the representation server(s) 114 may determine a text representation 128 based on the text data 106. In some implementations, the text representation 128 may include a vector embedding having dimensions that represent characteristics of the text data 106, such as the inclusion of words, groups of words, sub-words, groups of sub-words, characters, groups of characters, and so forth. The text representation 128 may also represent semantic information associated with the text data 106, such as the order or arrangement of words or characters, proximity of words to other words, use of capitalization and punctuation, and so forth.

An image encoder 130 associated with the representation server(s) 114 may determine an image representation 132 based on the image data 108. In some implementations, the image representation 132 may include a vector embedding having dimensions that represent characteristics of the image data 108, such as the locations and colors of pixels and so forth.

While FIG. 1 describes use of a graph encoder 122, text encoder 126, and image encoder 130, and the generation of vector embeddings as example types of representations, any type of machine learning model, network, algorithm, and so forth may be used to generate any type of representations associated with the subgraph data 118, text data 106, and image data 108. Additionally, in some implementations, one or more of the text data 106 or image data 108 may be absent or excluded from processing, and representations of only the subgraph data 118 or of only the subgraph data 118 and one of the image data 108 or text data 106 may be used. In some implementations, other types of data in addition to or in place of subgraph data 118, text data 106, or image data 108 may be associated with an entity. In such cases, encoders or other types of machine learning models may be used to determine representations based on the other type(s) of data.

An input module 134 associated with the representation server(s) 114 may determine input data 136 based on the graph representation 124, the text representation 128, and the image representation 132. In some implementations, the input module 134 may access input parameters 138, which may include one or more rules, algorithms, threshold values, dimensionalities, processing steps, tokens, or other parameters that may be used to determine the input data 136. For example, generation of the input data 136 may include use of one or more projection operations to modify one or more of the graph representation 124, text representation 128, or image representation 132 to correspond to the dimensions of an embedding space used by a large language model (LLM). In some cases, generation of the input data 136 may include the addition of one or more tokens, such as separator tokens, initial tokens, tokens indicating the end of an input, and so forth. In some implementations, multiple graph representations 124, text representations 128, or image representations 132 may be determined, and generation of the input data 136 may include performing a pooling or averaging process to determine a final graph representation 124, text representation 128, or image representation 132 for use generating input data 136.

The LLM 140 may determine an entity representation 102, that represents the characteristics of the entity indicated in the entity data 110, based on the input data 136. Because the input data 136 was determined based on the subgraph data 118, text data 106, and image data 108, the entity representation 102 may represent not only the characteristics of the entity indicated in the text data 106 and image data 108, but also characteristics of the entity indicated in the subgraph data 118, characteristics of relationships between the entity and other entities, and characteristics of related entities.

In some implementations, the LLM 140 may be trained to determine entity representations 102 based on one or more of graph data 104, text data 106, or image data 108, using training data that includes multiple pairs of values, such as seed values and target values. For example, the LLM 140 may be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens. Continuing the example, the LLM 140 may be trained using training data that includes pairs of values, a first loss function based on prediction of a first value of a pair based on a second value of the pair, and a second loss function based on contrastive loss associated with the first value and the second value of the pair. While FIG. 1 depicts generation of entity representations 102 using an LLM 140, in other implementations, other types of machine learning models may be used. In some implementations an entity representation 102 may be determined based on one or more outputs associated with the LLM 140, such as by performing an averaging or pooling operation based on multiple outputs, by projecting or otherwise modifying or formatting the output(s) to a selected dimensionality or format, and so forth.

Entity representations 102 for multiple entities may be generated and stored in a data repository 142. While FIG. 1 depicts a data repository 142 as a separate element from the representation server(s) 114 for illustrative purposes, in some implementations, the representation server(s) 114 may store the entity representations 102. The entity representations may be used by one or more downstream devices 144 for a variety of purposes.

For example, based on a distance or other metric for determining similarities between entities represented by entity representations 102, a downstream device 144 may determine one or more outputs 146 indicative of recommendations of items, recommendations of search queries or query terms, prediction of characteristics of items, and so forth.

FIG. 2 is a flow diagram 200 depicting an implementation of a method for determining entity representations 102 for an entity based on graph data 104, text data 106, and image data 108, and using entity representations 102 to determine an output 146. At 202, a first machine learning model may be trained to determine graph representations 124 using training data indicative of entity characteristics and relationships between entities. For example, the first machine learning model may include a graph encoder 122 or another type of machine learning model that may generate graph representations 124 based at least in part on subgraph data 118. The first machine learning model may be trained to predict a first type of relationship between entities, such as co-occurring user interactions, while other types of relationships may affect the determined graph representations 124. For example, a graph encoder 122 may include many convolutional layers, and based on a loss function, may be trained to predict a first type of relationship while other types of relationships are used in message-passing between layers.

At 204, entity data 110 indicative of an entity may be determined. The entity data 110 may be determined based on user input, input from a service, machine learning model, or computing device, or automatically. For example, a representation server 114 or other computing device may be configured to automatically generate entity representations 102 based on stored or accessible entity data 110. Entity data 110 associated with an entity may include an entity identifier 112 indicative of the entity and in some implementations may include one or more of text data 106 or image data 108. As described previously, in some implementations, types of entities may include items associated with an online store, search queries, item categories, or item brands or manufacturers.

At 206, subgraph data 118 may be determined based on correspondence between the entity data 110 and graph data 104 that is indicative of entity characteristics and relationships between entities. For example, in one implementation, the graph data 104 may include nodes that represent entities and may include information associated with entities, and edges that represent relationships between entities, each edge being associated with two nodes. One type of relationship represented by an edge may include co-occurrence of user interactions for two items represented by the associated nodes. Other example types of relationships represented by edges may include an indication of an association between an item and the item category to which the item belongs, or an indication of an association between an item and the brand or manufacturer of that item. Another example type of association may include a relationship between a search query that resulted in a purchase or other type of user interaction associated with an item. Other types of relationships may include an indication of items that are determined to be substitutes for one another, items that are identical or similar and offered in different regions or using different interfaces, search queries that are identical or similar and used in different regions to search different item databases, and so forth. As described with regard to FIG. 1, in some implementations, the subgraph data 118 may be determined based in part on threshold data 120, which may indicate a threshold distance value associated with a node that represents an entity. For example, based on a threshold distance value of one, subgraph data 118 that includes a node that represents the entity and each node connected to that node by an edge may be determined. Based on a threshold distance value of two, subgraph data 118 that also includes each node connected by an edge to a node that is connected to the node that represents the entity may be determined.

At 208, graph representation data may be determined based on the subgraph data 118 and the first machine learning model. For example, in one implementation, the graph representation 124 may include a vector embedding having dimensions that represent characteristics of the subgraph data 118, with the values for each dimension representing the particular characteristics of the subgraph data 118. Because the subgraph data 118 includes the node that represents the entity, one or more edges indicative of relationships between the entity and other entities, and the nodes that represent one or more other entities, the graph representation 124 may represent the characteristics of the entity, the relationships, and at least a portion of the characteristics of the other entities.

At 210, text data 106 and image data 108 associated with the entity may be determined. In some implementations, the text data 106 and image data 108 may be included in the entity data 110. In other implementations, the text data 106 and image data 108 that correspond to the entity indicated in the entity data 110 may be determined by accessing other data sources. In some cases, one or more of the text data 106 or image data 108 may be absent or excluded from use.

At 212, text representation data may be determined based on the text data 106 and a second machine learning model. For example, a text encoder 126 or other type of machine learning model may be trained to determine representations of the words, subwords (e.g., portions of words), characters, and semantic information associated with text. The text representation data may therefore represent characteristics of the text data 106, such as the words included in the text data 106, and semantic information such as the arrangement of the words, the proximity of the words relative to other words, and so forth.

At 214, image representation data may be determined based on the image data 108 and a third machine learning model. For example, the third machine learning model may include an image encoder 130 or another type of machine learning model. In some implementations, the image representation 132 may include a vector embedding having dimensions that represent characteristics of the image data 108, such as the locations and colors of pixels.

At 216, input data 136 may be determined based on the graph representation data, the text representation data, and the image representation data. In some implementations, the input data 136 may be determined by concatenating the graph representation data, text representation data, and image representation data, in some implementations with one or more initial tokens, separator tokens, or concluding tokens. Additionally, in some implementations, one or more of the graph representation data, text representation data, or image representation data may be modified, such as through use of a projection technique, to map the representation to a dimensionality that is usable with the LLM 140. Additionally, in some implementations, multiple graph, image, or text representations may be determined, and generation of the input data 136 may include use of an averaging or pooling technique to determine individual graph, image, or text representations for use in determining the input data 136.

At 218, the fourth machine learning model may be trained to determine entity representations 102 using training data that includes pairs of values, a first loss function based on prediction of a first paired value based on a second paired value, and a second loss function based on contrastive loss associated with the first and second values. For example, in some implementations the machine learning model may include a large language model (LLM) 140 that is provided with training data that includes paired seed and target values. The LLM 140 may be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens.

At 220 an entity representation 102 may be determined based on the input data 136 and the fourth machine learning model. In some implementations, the output from the fourth machine learning model may include the entity representation 102. In other implementations, the fourth machine learning model may determine multiple representations, and a final entity representation 102 may be determined using a pooling or averaging operation, or one or more other rules, algorithms, or thresholding operations. In some implementations, an output from the fourth machine learning model may be modified to determine the entity representation 102, such as through the addition or removal of data, use of a projection operation to modify the dimensionality of the output, and so forth.

At 222, one or more outputs 146 may be determined based on a relationship between the entity representation 102 and a second entity representation 102 associated with a second entity. For example, the process described with regard to FIG. 2 may be performed for multiple entities, such that each entity is associated with an entity representation 102 that represents the characteristics of that entity. Various operations, such as distance-determination operations within a common embedding space, may be used to determine similarities and differences between entities based on the entity representations 102. As described with regard to FIG. 1, entity representations 102 may be stored and accessed by downstream devices 144, which may determine outputs 146 based on the entity representations 102. For example, similarities between entity representations 102 may be used to determine identical, similar, related, or subsequent items, which may be used to determine outputs 146 that include item recommendations. As another example, similarities between entity representations 102 may be used to recommend search queries or search query terms. Similarities and differences between entity representations 102 may be used to predict characteristics of entities, classify or categorize entities, and so forth. Additionally, generation of entity representations 102 for a set of entities that may be used by a large number of downstream devices 144 may conserve time and computational resources by enabling each downstream device 144 to be configured to utilize the same types of entity representations 102, while the process performed using the representation server(s) 114 may be performed a single time, rather than by each downstream device 144.

FIGS. 3A and 3B are a diagram 300 depicting an implementation of a method for determining entity representations 102 based on graph data 104, text data 106, image data 108, and one or more machine learning models. As shown in FIG. 3A, at 302, entity data 110 may be determined. The entity data 110 may include text data 106 and image data 108 associated with an entity. For example, entity data 110 indicative of a particular entity may be received or indicated via user input, automated input associated with one or more services or computing devices, and so forth. The entity data 110 may include an entity identifier 112 indicative of a particular entity, and in some implementations may also include text data 106 and image data 108. In other implementations, one or more of the text data 106 or the image data 108 may be determined from one or more other sources in response to determination of the entity data 110, receipt of input indicative of an entity, and so forth. In still other implementations, one or more of the text data 106 or image data 108 may be omitted.

At 304, subgraph data 118 may be determined based on the entity data 110, graph data 104 indicative of entity characteristics and relationships between entities, and a threshold distance value. For example, graph data 104 may include multiple nodes, each node representing an entity and characteristics of the represented entity. The graph data 104 may also include multiple edges, each edge associated with two nodes and representing a relationship between the entities represented by the nodes. Threshold data 120 may indicate one or more threshold distance values that represent a distance from a node that represents the entity indicated in the entity data 110. Subgraph data 118 may be determined based on a set of nodes and edges that are within the threshold distance of the node that represents the entity indicated in the entity data 110.

At 306, graph representation data may be determined based on the subgraph data 118 and a first machine learning model, text representation data may be determined based on the text data 106 and a second machine learning model, and image representation data may be determined based on the image data 108 and a third machine learning model. For example, the machine learning models described with regard to 306 may include encoders that generate representation data based on input data having a particular modality. Continuing the example, as described with regard to FIG. 1, a graph encoder 122 may be used to determine a graph representation 124 based on the subgraph data 118. Because the graph representation 124 is determined based on the subgraph data 118, the graph representation 124 may represent characteristics of not only the indicated entity, but also characteristics of the relationships between that entity and other entities, and characteristics of the related entities. A text encoder 126 may be used to determine a text representation 128 based on the text data 106. The text representation 128 may represent the words and semantic information included in the text data 106. An image encoder 130 may be used to determine an image representation 132 based on the image data 108. The image representation 132 may represent characteristics of the image(s) included in the image data 108.

As shown in FIG. 3B, at 308, input data 136 may be determined based on the graph representation data, text representation data, image representation data, and input parameters 138. The input parameters 138 may include one or more rules, algorithms, threshold values, dimensionalities, processing steps, tokens, or other parameters that may be used to determine the input data 136. For example, generation of the input data 136 may include use of one or more projection operations to modify one or more of the graph representation 124, text representation 128, or image representation 132 to correspond to the dimensions of an embedding space used by a subsequent machine learning model, such as a large language model (LLM) 140. In some implementations, generation of the input data 136 may include the addition of one or more tokens, such as separator tokens, initial tokens, tokens indicating the end of an input, and so forth. In some implementations, multiple graph representations 124, text representations 128, or image representations 132 may be determined, and generation of the input data 136 may include performing a pooling or averaging process to determine a final graph representation 124, text representation 128, or image representation 132 for use generating input data 136.

At 310, an entity representation 102 may be determined based on the input data 136 and a fourth machine learning model. As described with regard to FIGS. 1 and 2, in some implementations, the fourth machine learning model may include an LLM 140. The entity representation 102 may represent the characteristics of the entity indicated in the subgraph data 118, text data 106, and image data 108, which due to the inclusion of information from the subgraph data 118 in the input data 136 may represent not only the characteristics of the indicated entity, but also of relationships between the entity and other entities, and characteristics of related entities. In some implementations an entity representation 102 may be determined based on one or more outputs associated with the LLM 140, such as by performing an averaging or pooling operation based on multiple outputs, by projecting or otherwise modifying or formatting the output(s) to a selected dimensionality or format, and so forth.

At 312, the entity representation 102 may be stored for access by one or more computing devices. For example, entity representations 102 for multiple entities may be generated and stored in a data repository 142. One or more downstream devices 144 may access the data repository 142 to perform various operations, such as recommendations of similar or related items, brands, categories, or search queries, prediction of characteristics of items, and so forth.

FIG. 4 is a block diagram 400 depicting an implementation of a computing device 402 within the present disclosure. The computing device 402 may include one or more representation servers 114, as described with regard to FIG. 1. In other implementations, one or more downstream devices 144, or other computing devices in communication with the computing device 402, may perform one or more of the functions described herein. Therefore, while FIG. 4 depicts a single block diagram 400 representative of a computing device 402, any number of computing devices 402 may be used, of similar or differing types. For example, a first computing device or set of computing devices may be used to train one or more of the subgraph module 116, graph encoder 122, text encoder 126, image encoder 130, or LLM 140, while a second computing device or set of computing devices may store and execute the LLM 140 and in some cases one or more of the subgraph module 116, graph encoder 122, text encoder 126, or image encoder 130. As another example, a first computing device or set of computing devices may determine entity representations 102, while a second computing device or set of computing devices may include one or more data repositories 142 for storing entity representations 102. In some cases, a data repository 142 may include a cache or other type(s) of memory that may enable entity representations 102 to be accessed efficiently for one or more uses by downstream devices 144.

One or more power supplies 404 may be configured to provide electrical power suitable for operating the components of the computing device 402. In some implementations, the power supply 404 may include a rechargeable battery, fuel cell, photovoltaic cell, power conditioning circuitry, and so forth.

The computing device 402 may include one or more hardware processor(s) 406 (processors) configured to execute one or more stored instructions. The processor(s) 406 may include one or more cores. One or more clock(s) 408 may provide information indicative of date, time, ticks, and so forth. For example, the processor(s) 406 may use data from the clock 408 to generate a timestamp, trigger a preprogrammed action, and so forth.

The computing device 402 may include one or more communication interfaces 410, such as input/output (I/O) interfaces 412, network interfaces 414, and so forth. The communication interfaces 410 may enable the computing device 402, or components of the computing device 402, to communicate with other computing devices 402 or components of the other computing devices 402. The I/O interfaces 412 may include interfaces such as Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s) 412 may couple to one or more I/O devices 416. The I/O devices 416 may include any manner of input devices or output devices associated with the computing device 402. For example, I/O devices 416 may include touch sensors, keyboards, mouse devices, microphones, image sensors, cameras, scanners, displays, speakers, haptic devices, printers, and so forth. In some implementations, the I/O devices 416 may be physically incorporated with the computing device 402. In other implementations, I/O devices 416 may be externally placed.

The network interfaces 414 may be configured to provide communications between the computing device 402 and other devices, such as the I/O devices 416, routers, access points, and so forth. The network interfaces 414 may include devices configured to couple to one or more networks including local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless WANs, and so forth. For example, the network interfaces 414 may include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, Z-Wave, 5G, LTE, and so forth.

The computing device 402 may include one or more buses or other internal communications hardware or software that allows for the transfer of data between the various modules and components of the computing device 402.

As shown in FIG. 4, the computing device 402 may include one or more memories 418. The memory 418 may include one or more computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory 418 may provide storage of computer-readable instructions, data structures, program modules, and other data for the operation of the computing device 402. A few example modules are shown stored in the memory 418, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

The memory 418 may include one or more operating system (OS) modules 420. The OS module 420 may be configured to manage hardware resource devices such as the I/O interfaces 412, the network interfaces 414, the I/O devices 416, and to provide various services to applications or modules executing on the processors 406. The OS module 420 may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; UNIX or a UNIX-like operating system; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Washington, USA; or other operating systems.

One or more data stores 422 and one or more of the following modules may also be associated with the memory 418. The modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store(s) 422 may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store(s) 422 or a portion of the data store(s) 422 may be distributed across one or more other devices including other computing devices 402, network attached storage devices, and so forth.

A communication module 424 may be configured to establish communications with one or more other computing devices 402. Communications may be authenticated, encrypted, and so forth.

The memory 418 may store the subgraph module