Cross-network attention for leveraging the RGB-pre-trained feature spaces. In order to better leverage the knowledge of strong pre-trained models, the system may be configured to use a novel cross-attention paradigm. Assuming two parallel networks of identical architecture, the system may be configured to replace a self-attention layer with a cross-network attention (CNA) layer. This CNA layer may take the feature maps of both networks as input, and may output identical features to both networks, who continue inference as usual. In example embodiments, the system may be configured to train the CNA layer from scratch, without significantly impacting the quality or diversity of the frozen RGB model. This effectively forces the newly trained PBR diffusion model to align to the internal feature spaces of the pre-trained network (which, in example embodiments, may be a strong anchor during training) as well as leveraging the rich internal representation for the RGB network.

Training details, model input/output, data generation. In example embodiments, the system's diffusion model is conditioned on the existing model geometry. In example embodiments, there may be various options of rasterized features, such as object-space coordinates of the geometry, camera-space normals of the geometry, and/or camera-space depth of the geometry.

In example embodiments, the system may determine that camera-space normals yield the best results. For example, camera-space normals may provide a pose- and world-invariant feature cue (much more so than object-space coordinates), while also providing a more rich representation than simple camera-space depth. In example embodiments, this may not be a huge issue for multi-view fusion; e.g., normals+object coordinates might help a lot for multi-view consistency. In example embodiments, the joint conditioning on normals and depth yields no significant improvements over normals alone, so the system may be configured to select the simpler option.

Experiments

The techniques used by the system and its configuration can be tested individual (e.g., through a slew of ablation studies) to demonstrate the advantage of generation in the PBR domain over post-hoc methods, such as those used by Fantasia3D, which generate in the RGB domain and then solve an inverse rendering problem to return to the PBR space.

In example embodiments, specific datasets are used for training and evaluation, as well as the training details for the PBR cross-modality VAE and the PBR image diffusion model. The system may be configured based on particular metrics used for the evaluations. For example, the system's results may be evaluated on one or more of (1) quality and/or diversity of the generated PBR stacks and (2) out-of-distribution generation by leveraging the expressivity of the base RGB SD model.

Preliminaries Training Dataset

A dataset for training both the VAE and the LDM may be based on a particular dataset, such as on an Objaverse data set containing 500′000+3D models associated PBR stack as we consume it, as well as annotations for what the models represent (both in terms of shape and in terms of texture). In example embodiments, each of the objects may be rendered with a tool (e.g., Blender 2.35) from a configurable number of (e.g., 12) viewpoints (e.g., using a fixed pinhole camera model and a fixed camera co-located environment map). In example embodiments, the system may store one or more of camera parameters; geometry depth, geometry normals, and a default radial geometry tangent (e.g., as used by the rendering tool); the PBR material stack: base color, metallicness, roughness; the final screen-space normals after applying the texture bump map; and/or the final rendered color.

In example embodiments, varying the camera model may not result in a meaningful difference in the trained network or its ability to generalize, so the system may be configured to simplify the data generation by keeping it fixed. In example embodiments, the environment may be configured to be (1) uniform across the dataset and/or (2) co-located with the camera. In example embodiments, the rendered color may only be used by the system to supervise the RGB diffusion model's output to make sure that the attention maps injected by the system's novel cross-network attention module are consistent with the hidden space of the RGB diffusion model. As the weights of the RGB SD model are kept fixed, there is (1) no risk of over-fitting on our dataset and (2) no reason to make the dataset more complicated than it needs to be; again, therefore, the system may be configured to keep the data generation as simple as possible.

Evaluation Datasets

In addition to re-using the above training dataset in the evaluation to verify that the system's generator well represents that dataset's distribution, the system may also be configured to leverage a higher-quality dataset, such as a dataset gathered from Polyhaven. The higher-quality dataset may contain a smaller number of (e.g., only 300) models, but these are models may be of camera-ready quality, authored by professional artists. Again, the models come with a complete PBR stack texture, as well as accurate annotation as to their shape and style. The system may be configured to render the higher quality dataset in exactly the same way as the lower-quality (e.g., Objaverse) dataset.

In example embodiments, the system may be configured to require a set of out-of-distribution prompts for any given object (e.g., to verify that that the system is able to generate qualitative content that is out-of-distribution for the original dataset. In example embodiments, the system may be configured to pick a subset of objects from both data sets (e.g., from both Objaverse and Polyhaven), and ask a large language model (e.g., ChatGPT3.5), “What are 8 very unlikely appearances for <object>”. This becomes the out-of-distribution prompt query dataset that system may use for evaluation out-of-distribution content accuracy.

Training the VAE

The system may be configured to train a VAE (e.g., with KL regularization) which downsamples a region (e.g., of 4×4 pixels) with a configurable number of (e.g., 8) channels (base color, metallicness, roughness, and/or bump map) into a single pixel with more (e.g., 14) channels. In example embodiments, the system may be configured to not use a quantized VAE (e.g., because of the high dimensionality absolute of the source space (e.g., 8 compared to existing QVAE solutions in the LDM space with, e.g., 3) as well as the strongly different characteristics of the PBR stack. For example, color, specular components, and/or bump maps may be, to a large extent, independent of each other, meaning that the practical dimensionality of the patch manifold rises even more than the (e.g., 3-to-8) ratio implies.

In example embodiments, only minor edits may be required to the VAE (e.g., for changing the input and output dimensionalities and modalities).

Training the PBR Image Diffusion Model

In example embodiments, the system may be configured to run two parallel models both for training and inference: (1) the PBR-space diffusion model and (2) the frozen SD RGB-space model. For the frozen model, the system may be configured to use the existing supervision losses in δv prediction mode to ensure that the cross-network attention module leaves the model output intact. For example, only the weights in the cross-network attention module may be trained. For the PBR-space model, the system may train the entire weight set from scratch using the same loss set on each of the PBR channels (e.g., albedo, metallicness, roughness, and/or perturbed normals) separately.

Metrics

The system may be configured to evaluate various aspects of its generative network, such as quality, diversity, and/or out-of-distribution accuracy.

In example embodiments, distribution match may be evaluated through a score, such as an inception score (IS) or Fréchet Inception Distance (FID), which may be considered a proxy to both quality and diversity. IS evaluates the classification score outputs of a pre-trained (e.g., Inceptionv3) network. The score is optimal when individual samples' classification score vectors have low entropy (e.g., Inceptionv3 would confidently predict a single class) and the per-sample winning class has high entropy over the entire dataset (e.g., the generator generates all classes). In theory, it is only directly relevant to generators over the training dataset of the (e.g., Inceptionv3) model, but in practice it may also be indicative for comparison between methods on other datasets. However, the FID, to a large extent, abstracts away the underlying dataset of the pretrained model. It compares the distributions of the last hidden state of the same (e.g., Inceptionv3) network on both generated images and a reference dataset. In example embodiments, FID may be found to much better align to perceptual quality. As the (e.g., Inceptionv3) network assumes RGB input, the system may split the PBR stack in parts, such as the following four parts: base color, metallicness and roughness (both individually as grayscale images), and the final perturbed normal maps.

Quality of the PBR maps may be more difficult to quantize without ground-truth corresponding PBR maps. Perceptually, many of existing techniques bake in illumination into the various channels, or leak details between the various channels.

Out-of-distribution accuracy indicates the level to which the system's generator can align to conditioning which it has never seen during training. In example embodiments, a score, such as CLIP alignment score, may be determined. The score may evaluate whether the text prompt's CLIP embedding is close to the generated image's CLIP embedding, (e.g., whether every aspect of the text prompt is actually present in the image without introducing frivolous unrequested content).

In example embodiments, user studies may be included as a piece of the system's evaluation, verifying that end-user artists agree with the indications provided by the above quantitative metrics. In example embodiments, the qualitative evaluation through FID and IS may not be very sensitive to low-level image details. In example embodiments, IS may even less so than FID, which are however important (perhaps even more so in PBR stacks than in RGB images). Therefore, based on a randomized blind selection query, the system may evaluate with a set of graphics artists which set of PBR maps they would prefer, and respectively which of them would require the least work to edit to their liking.

Ablation Studies

In example embodiments, the system may evaluate various design choices of its architecture. Assuming that the system builds up a model that predicts directly in the PBR space, the system determines how much do each of those choices contribute to the final performance of our model.

Joint VAE for the PBR Stack

In order to still leverage the LDM paradigm in PBR space, the system may be configured to encode the patches of the (e.g., 8) PBR channels into a single latent vector.

For example, the system may be configured to (1) train a new encoder/decoder for the same latent space as the existing VAE (practically, with minimal changes from the existing architecture), (2) split the (e.g., 8 channels into pair-wise three channels, such as base color, perturbed normals, and metallicness+roughness[+ambient occlusion=zeros]), including re-using the RGB VAE for each of the triplets, with no additional training, and re-training per-triplet VAEs from the initial checkpoint, (3) a quantized which the system retrains from scratch with a larger latent space dimensionality, and/or (4) a novel architecture which the system retrains from scratch with a larger latent space dimensionality.

In example embodiments, the system is configured to train (as relevant) each of these variants on the original (e.g., Objaverse) dataset. Quantitative and qualitative evaluations on both the lower-quality and higher-quality dataset may be analyzed. In example embodiments, the system may determine that training a larger VAE from scratch is the way to go, as it better captures both the individual channels' characteristics and the inter-dependency between the channels and/or it quantitatively and/or visually results in fewer artefacts when encoding-decoding sample patches from either dataset.

Joint Prediction of the PBR Stack

When it comes to predicting the various PBR channels (conditioned in whatever way) p(albedo, metallicness, roughness, normals|condition), the system may be configured to identify one or more options, such as one or more of the following options:

1. independent generation of each of the channels. p(albedo|condition), p(metallicness|condition), separately. This models the marginal conditional distributions and will not be able to model the interaction between the various channels that is contained in the full joint distribution, because only when all channels are independent does the relationship p(albedo|condition)p(metallicness|condition) . . . =p(albedo, metallicness, . . . |condition) hold.

2. conditional, sequential generation of the channels. E.g. p(albedo|condition), p(metallicness|sequentially. This follows from the recursive relationship p(albedo, metallicness, . . . |condition) p(albedo|condition)p(metallicness, . . . |albedo, condition). However, in practice this means that the system is configured to train and evaluate four individual networks (representing diffusion process) where the subsequent one can only start training and inferring when the preceding one has finished, which is an unpractically expensive workflow.

3. joint modeling of the direct p(albedo, metallicness, . . . |condition) as represented by the system's approach.

The system may be configured to evaluate the independent versus the joint modeling of the channels, which shows the type of performance loss when ignoring the channel interdependencies.

Cross-modality/-network attention module.

In example embodiments, the system may be configured to use one or more alternative approaches for cross-modal attention/guidance.

Such alternative approaches may include one or more of the following: (1) Similar to the VAE discussion in the previous section, creating finetuned versions of the RGB SD module to predict the channels in sets of three, using either a ControlNet or a LoRA; (2) Using minimal network intrusions to directly predict the full set of PBR channels, using either a ControlNet or a LoRA, and/or (3) a different cross-network attention paradigm.

In terms of predicting channel triplets independently (or conditioned on each other), modeling the distribution in such a way may be suboptimal, as described above. Here the discussion kind of implies that the system should try to use the conditional sequential generation of the channels using minimal intrusions on an existing pre-trained diffusion model in RGB space. For all the same reasons that we discuss in the VAE's architecture, jointly predicting the entire PBR stack using minimal intrusions on a pre-trained RGB-space diffusion model is unlikely to work well in most circumstances because the internals of the diffusion model are not able to correctly represent the higher-dimensional material property space in a disentangled way.

In example embodiments, the system may be configured to compare the full cross-network attention paradigm to one where the RGB SD model does not get new attention values injected. Here, there is only a one-way stream of attention from the RGB SD to the PBR SD. In example embodiments, the system may determine that there is a significant gain in the two-way cross-network attention paradigm. This may be attributed to forcing the PBR-based model to actively align to the RGB SD model's internal space, which is strongly structured and highly expressive—the system may be essentially forcing the PBR-based model to simultaneously leverage the RGB SD model as a rendering function proxy. This is in contrast to only pulling information directly from the RGB-space SD; a rich source of information, but it does not offer the two-fold near-differentiable-rendering-paradigm that our proposed paradigm brings.

Conditioning Cue

In example embodiments, the system may be configured to consider conditioning on either text prompts or object geometry. In example embodiments, the system may be configured explicitly to not consider conditioning on the final observations in RGB space (e.g., this would mean that our model is addressing the inverse rendering problem).

In example embodiments, the system's paradigm of cross-network attention may break down under certain circumstances. For example, the PBR-space diffusion model receives the target output of the RGBspace model as direct inputs, so it is easily able to leverage the cross-network attention to circumvent true prediction of the internal hidden space and as a result, fails to leverage the internal hidden state of the RGB network.

As far as conditioning on the geometry goes, the system may identify and perform one or more of the following options: (1) Conditioning on camera-space depth, (2) Conditioning on object-space coordinates, and/or (3) Conditioning on camera-space normals.

In example embodiments, depth and coordinates are not as rich of an input/conditioning as surface normals are.

Parametrization of the Material Properties

The system's PBR-space diffusion model may be configured to return one or more of base color, metallicness, roughness, and/or perturbed normals conditioned on object geometry. In practice, the first three can be considered “boring” channel outputs in that they are wellbehaved within a normal range, and not formulated in an “arbitrary” space. The perturbed normals, however, may be expressed in screen space, but also encode the object geometry. Instead, the system may be configured to predict a differential normal image (e.g., because the optimization problem of directly predicting screen-space perturbed normals may be a more difficult one, as well as that those features may be view-dependent so that the system cannot easily fuse them across different views in downstream applications). In example embodiments, the (e.g., Markspace) normals are (1) differential, in that they are formulated in terms of a geometry-normal-and-tangent dependent coordinate space and (2) view independent, so that they can be directly synchronized across all views in the joint latent representation. In example embodiments, they also do not depend on an arbitrary coordinate system choice stemming from the UV parametrization. The system may be configured to analyze, select, and/or use one or more of the following alternative parametrizations:

- Screen-space perturbed normals;
- World-space perturbed normals;
- Screen-space direct-differential normals (e.g., screen-space perturbed normals;
- screen-space geometry normals);
- World-space direct-differential normals (e.g., world-space perturbed normals;
- world-space geometry normals);
- Global Markspace normals (e.g., the screen-space perturbed normals expressed in the local normal-tangent-bitangent coordinate system implied by the world up axis); and/or
- Screen Markspace normals (e.g., the screen-space perturbed normals expressed in the local normal-tangent-bitangent coordinate system implied by the camera X axis).

In example embodiments, Markspace normals may be the winners in terms of performance of the generative method. Screen-based Markspace normals may perform only marginally better, so that the system—in the interest of downstream applications—may decide on Global Markspace normals for its parametrization choice.

Existing techniques focus on directly predicting only RGB-based images of objects.

Some existing methods which lift 2D models to 3D generate PBR stacks for the generated images (e.g., Fantasia3D). However, they arrive at the generated PBR stacks through variants on inverse rendering. In contrast, the disclosed techniques qualitatively compare against the output of such methods in the system's application to shape texturing.

In terms of direct PBR generation/prediction, tools such as SurfaceNet or ControlMat may generate plausible PBR stacks for an RGB observation. In example embodiments, when there are no direct generative models to compare against, the quality of the system's created PBR stacks may be evaluated against the PBR stacks created by such tools.

For example, SurfaceNet may be run as a reconstructive technique on objects in Objaverse and Polyhaven (e.g., conditioned on an observation), and one sample of the system may be generated (e.g., conditioned only on the screen-space normals, without the text prompt). The system may then calculate the FID and IS of each of these sample sets against the ground truths in the respective datasets.

In terms of geometry-conditioned (or -unconditioned) generation of PBR stacks, there are no known tools that leverage VAEs or GANs.

In example embodiments, the system's image-based PBR-space diffusion model can serve as a drop-in base model (with minimal code changes) to a series of existing text-to-3D paradigms that distill from frozen 2D image-based models. For example, the system can be used in two significantly different pipelines, such as TexFusion and SDS.

The systems and methods disclosed herein introduce a transformative method for generating PBR textures, which is a significant departure from conventional techniques. The disclosed techniques include a unique integration of a cross-domain attention module within a machine-learning framework, which facilitates the translation of RGB image data into PBR textures. This cross-domain attention module is a novel construct that enables the machine-learning model to leverage the rich feature representations of pre-trained RGB models while generating PBR textures that are photometrically accurate and consistent across various lighting conditions. The non-obviousness of this approach is underscored by its ability to resolve the inherent limitations of RGB-based texture generation methods, which often fail to capture the complex interplay of light with material surfaces in a 3D space.

In an example embodiment, a machine-learning model is trained on a dataset comprising a wide array of annotated 3D objects. This dataset includes PBR textures representing diverse material types such as metals with varying degrees of oxidation, fabrics with different weaves and thread counts, and natural elements with unique surface irregularities. In example embodiments, the dataset is meticulously curated to encompass view-independent material properties, ensuring that the generated PBR textures are optimized for multi-view fusion and yield consistent rendering results under diverse lighting scenarios. As but one example, consider generation of PBR textures for a vintage brass telescope, where the model accurately predicts the patina and specular highlights that are characteristic of aged brass under moonlight, sunlight, and artificial indoor lighting.

In example embodiments, the machine-learning model is trained using a dataset augmented with synthetic lighting conditions, which are algorithmically generated to cover a comprehensive spectrum of environmental lighting scenarios. This augmentation enhances the model's robustness and its ability to generalize the generation of PBR textures, ensuring that the textures respond accurately to real-world lighting changes.

In example embodiments, generative adversarial networks (GANs) are utilized to synthesize a vast and varied dataset of 3D objects with corresponding PBR textures. In example embodiments, the training process is further augmented by a rigorous validation step, wherein the model's output is meticulously compared against a subset of the dataset not utilized during the training phase. This comparison helps to ensure the accuracy and reliability of the model's output, thereby enabling the practical application of the disclosed techniques.

In example embodiments, to generate a 3D object with PBR textures using the disclosed system for incorporation into a game engine or similar rendering tool, a user (via a user interface) or an external system (via an application programming interface) would engage in one or more of the following steps.

Step 1: Preparing the 3D Mesh. The user or external system begins by creating or selecting a 3D mesh that will serve as the base geometry for the object. This mesh should be properly UV unwrapped to allow for accurate texturing.

Step 2: Defining Material Properties. The user or external system defines the desired material properties for the object, such as base color, metallic, roughness, and any additional features like bump maps or specular highlights. These properties can be specified directly or through descriptive text prompts that convey the intended appearance.

Step 3: Inputting Data into the Machine-Learning Model. The 3D mesh and material property prompts are input into a trained machine-learning model. The model utilizes a novel cross-domain attention module that leverages pre-trained RGB models to predict PBR textures that are photometrically accurate.

Step 4: Machine-Learning Model Processing. The machine-learning model processes the input data, generating output data that includes the PBR textures. This process involves the model's internal algorithms, which have been trained on a comprehensive dataset of annotated 3D objects with associated PBR textures.

Step 5: Reviewing Generated Textures. Once the PBR textures are generated, the user or external system reviews them for quality and accuracy. Adjustments to the text prompts or input parameters can be made if the generated textures do not meet the desired specifications.

Step 6: Applying Textures to the 3D Mesh. The generated PBR textures are applied to the 3D mesh. The system ensures that the textures respond correctly to changes in lighting and environmental conditions, providing a realistic appearance in various rendering scenarios.

Step 7: Exporting the Textured 3D Object. The textured 3D object is exported from the system in a format compatible with the user's game engine or rendering tool. This may involve the use of specialized file formats, such as .OBJ, .FBX, or .GLTF, which include the mesh and associated PBR texture data.

Step 8: Importing into a Game Engine. The user or external system imports the textured 3D object into a chosen game engine or rendering tool. The object is then integrated into the game or scene, where it can be placed, transformed, and further manipulated as required.

Step 9: Final Rendering and Adjustments. Within the game engine, the user or external system performs a final rendering of the object. At this stage, additional adjustments can be made to the object's materials and textures to fine-tune its appearance under the specific lighting and environmental conditions of the game or scene.

Step 10: Iteration and Optimization. The user or external system iterates on the texturing process as necessary, optimizing the 3D object for performance and visual quality within the game engine. This may involve adjusting texture resolutions, LODs (levels of detail), or shader parameters to achieve the best balance between realism and performance.

In summary, users or external systems can leverage the disclosed system to generate high-quality PBR textures for 3D objects, enhancing the realism and visual fidelity of their digital content within game engines and rendering tools.

The system for generating 3D objects with PBR textures is architecturally designed to integrate seamlessly with modern game engines and rendering tools. In example embodiments, the architecture of the system comprises one or more components that work in concert to facilitate the generation, application, and/or export of PBR textures for use in various digital content creation environments.

Graphical User Interface (GUI). The system includes a GUI that allows users to, for example, input 3D meshes, define material properties through text prompts or graphical selectors, and review the generated PBR textures.

Machine-Learning Model. The system includes a trained machine-learning model equipped with a cross-domain attention module, as described herein. This model processes the input data and generates PBR textures based on the defined material properties and learned patterns from a comprehensive dataset.

Data Processing Unit. The system may include a dedicated processing unit that handles the computational tasks associated with the machine-learning model, including data ingestion, model inference, and/or texture synthesis.

Texture Application Module. The system may include a module responsible for applying the generated PBR textures to the input 3D mesh, ensuring that the textures align correctly with the UV mapping of the mesh.

Export Module. The system may include a module facilitates the export of the textured 3D object in various file formats compatible with different game engines and rendering tools.

Storage System. The system may include a storage solution for saving the input 3D meshes, generated PBR textures, and/or the final textured 3D objects. This storage system may include local storage, network-attached storage, or cloud-based storage services.

The system is designed to be communicatively coupled to one or more additional systems to enhance its functionality and interoperability.

For example, the system can be directly integrated with game engines (e.g., Unity) and rendering tools (e.g., Blender, Maya) through APIs or plugin frameworks. This allows for the seamless import and export of 3D objects and textures. For larger production environments, the system can be coupled with digital asset management systems to organize and track the various assets being created and used throughout the content creation process. The system can be configured to interface with collaboration platforms and version control systems (e.g., Perforce, Git) to enable multiple users to work on the same 3D objects and textures simultaneously, ensuring consistency and synchronization of assets. To leverage scalable computing resources, the system can be connected to cloud computing services (e.g., AWS, Azure) for processing-intensive tasks such as training the machine-learning model or generating textures for complex objects. The system architecture allows for the integration of data analytics and user feedback systems to monitor usage patterns, gather user feedback, and continuously improve the machine-learning model based on real-world data.

The system offers various configurable aspects to accommodate different user needs and integration requirements. Configuration can be performed through an administrative GUI or programmatically via an application programming interface (API).

The administrative GUI is designed for ease of use, allowing users to configure settings through a graphical interface. Users can log in to the administrative panel to access one or more configuration settings, examples of which are provided herein and may include one or more of the following.

Dataset Selection. Users can select and manage datasets used for training the machine-learning model. For example, a user might choose a dataset focused on automotive materials for generating textures specific to vehicle models.

Model Training Parameters. The GUI allows users to set parameters for training the machine-learning model, such as the number of training epochs, learning rate, and batch size. A user might increase the number of epochs for a more thorough training process.

Texture Resolution. Users can define the resolution for generated textures, adjusting for different levels of detail required by their project. For instance, a user might set a high resolution for close-up renders and a lower one for background objects.

Export Formats. The system supports various export formats, and users can select their preferred format (e.g., .OBJ, .FBX, .GLTF) based on the target game engine or rendering tool.

Rendering Settings. Users can configure rendering settings such as lighting models, shadow quality, and reflection intensity to preview how textures will appear in the final render.

For programmatic access and integration with external systems, the system provides an API that allows for automated configuration. The API can be used to set configurations (e.g., by sending HTTP requests with the desired parameters). Examples are described herein and may include one or more of the following.

Automated Dataset Management. An external system can use the API to upload new datasets or switch between existing ones by sending a request with the dataset identifier.

Dynamic Texture Resolution Adjustment. Depending on the computational resources available, an external system might dynamically adjust the texture resolution via the API to balance quality and performance.

Batch Processing Configuration. For bulk texture generation, the API can be used to configure batch processing settings, such as queue size and priority levels.

Integration with Asset Pipelines. The API can be used to integrate the system with existing asset pipelines, allowing for the automated application of textures to 3D models as they move through the production stages.

Custom Export Workflows. External systems can configure custom export workflows using the API, specifying file naming conventions, directory structures, and post-export actions such as triggering additional processing or notifications.

For example, in example embodiments, a game development studio might use the administrative GUI to configure the system for high-resolution texture generation, selecting a specific dataset that includes a variety of natural environment materials for their open-world game.

An animation studio could use the API to integrate the system into their rendering pipeline, setting the texture resolution and export formats to match their in-house tools and ensuring seamless asset transitions.

A mobile game developer might configure lower texture resolutions and simplified rendering settings via the UI to optimize for performance on mobile devices.

Both the administrative GUI and the API provide robust and flexible methods for configuring the system, ensuring that it can be tailored to meet the specific requirements of various projects and integrated seamlessly into different production environments.

The one or more machine-learned models in the system take one or more inputs that may include, for example, one or more of the following.

3D Mesh Input, which may include the geometric data of the 3D object, which may include vertex positions, edges, faces, and UV mapping information necessary for texturing.

Text Prompts, which may include descriptive phrases or keywords that specify the desired material properties such as “rusty metal,” “polished wood,” or “rough stone.” These prompts guide the model in generating textures that match the user's intent.

Additional Contextual Data, if applicable), which may include lighting conditions, environmental settings, or any other relevant information that can influence the appearance of the material under different scenarios.

The outputs of the machine-learned model may consist of one or more of the following.

Material Parameters, which may include a set of PBR material parameters such as base color, metallic, roughness, and possibly others like specular, emissive, and normal maps. These parameters define how the material interacts with light and are essential for achieving photorealistic rendering.

Texture Maps, which may include one or more image files corresponding to the material parameters, which can be directly applied to the 3D mesh. These may include diffuse/albedo maps, metallic maps, roughness maps, and normal maps.

Training of the one or more machine-learned model may include one or more of the following steps.

Dataset Compilation. A comprehensive dataset of 3D objects with associated PBR textures may be compiled. Each object in the dataset may be annotated with metadata that describes its material properties and/or other relevant attributes.

Preprocessing. The dataset may be preprocessed to normalize the data, which may include scaling the texture images, converting them to a consistent format, and/or encoding the text prompts.

Feature Extraction. The model may learn to extract features from both the geometric data of the 3D meshes and the descriptive text prompts. This may involve, for example, using convolutional neural networks (CNNs) for image-based features and natural language processing (NLP) techniques for text-based features.

Model Architecture. A neural network architecture suitable for the task may be selected or designed. This architecture could include a generative model, such as a Generative Adversarial Network (GAN), or a Variational Autoencoder (VAE), which are capable of generating high-quality image data.

In example embodiments, the model is trained using the prepared dataset. During training, the model learns the correlation between the input features and the corresponding PBR textures. The training involves adjusting the model's parameters to minimize the difference between the generated textures and the ground truth textures in the dataset.

If the model includes a cross-domain attention module, it may be trained to link the PBR material properties with a pre-trained RGB model. This module helps the system to leverage the rich feature representations of pre-trained models while ensuring the generated textures are consistent with PBR principles.

In example embodiments, the model's performance is validated using a separate set of data not seen during training. Based on the validation results, the model may be fine-tuned to improve its accuracy and generalization capabilities.

The model may be updated continuously with new data to refine its ability to generate PBR textures for a wider variety of materials and conditions.

In this way, the machine-learned model becomes capable of generating PBR textures that can be applied to 3D objects, resulting in realistic materials that respond appropriately to changes in lighting and environmental conditions (e.g., within a rendering engine or game engine).

In example embodiments, the system employs diffusion models, which are a type of generative model that can produce high-quality images. These models work by gradually transforming a random noise distribution into a structured image over a series of steps, akin to reversing a diffusion process.

The diffusion model may be trained on a dataset of images that represent PBR textures. During training, the model learns the distribution of the dataset and how to generate new textures that are similar to those in the training set.

In example embodiments, the system uses a novel cross-domain attention module that allows the diffusion model to utilize features from a pre-trained RGB model. This helps in generating textures that are not only visually appealing but also consistent with the underlying PBR material properties.

In example embodiments, to generate a new PBR texture, the system starts with a noise image and progressively refines it using the trained diffusion model. The model applies learned transformations at each step, guided by the input 3D mesh and text prompts, until a detailed and realistic texture is formed.

In example embodiments, the system predicts a set of PBR material parameters, such as base color, metallic, roughness, and normal maps, as part of the diffusion process. These parameters are essential for rendering the material accurately under various lighting conditions.

In example embodiments, the image-based diffusion process is specifically tailored to generate textures that adhere to PBR principles. This means that the textures not only look realistic but also behave correctly when rendered, reflecting light and casting shadows as expected based on their material properties.

In example embodiments, by using image-based diffusion, the system can generate textures that are well-parametrized for 3D rendering, avoiding issues such as photometric inconsistency and ambiguities in inverse rendering that are common with traditional RGB image generation methods.

The image-based diffusion in this system is a sophisticated technique that synthesizes PBR textures by learning from a dataset of existing materials and applying that knowledge to generate new textures that are both visually detailed and physically accurate. This approach is particularly useful for automating the texture generation process in 3D content creation, saving time and resources while ensuring high-quality results.

In example embodiments, the cross-network or cross-domain attention mechanism is a sophisticated component within a machine-learning architecture that enables the system to integrate and leverage information from different domains or feature spaces. In the context of generating PBR textures, this mechanism allows the system to utilize the rich feature representations of pre-trained RGB models to enhance the generation of PBR textures that are consistent with the underlying material properties.

The cross-domain attention mechanism may be configured to align the feature spaces of two parallel networks: one trained on RGB images (RGB-space) and the other focused on PBR textures (PBR-space). This alignment enables the PBR-space network to access and utilize the internal representations of the RGB-space network.

During the diffusion process, the mechanism may facilitate an exchange of information between the RGB and PBR networks. It allows the PBR network to query the RGB network's feature maps and vice versa, effectively creating a bidirectional flow of information.

The mechanism may replace traditional self-attention layers with cross-network attention layers. These layers take the feature maps of both networks as input and output modified features to both networks, which continue processing with these enhanced features.

The cross-network attention layers may be trained from scratch, allowing them to learn the most effective way to integrate features from both domains. The pre-trained RGB model's weights may remain frozen to preserve its rich internal representation, which serves as a strong anchor during training.

The attention mechanism may operate on the principle of queries, keys, and values. In example embodiments, the PBR network generates queries that are matched against keys from the RGB network to retrieve relevant values (features) that are then used to enhance the PBR network's hidden state.

The cross-network attention may involve sharing weights between the RGB and PBR networks' attention layers, ensuring that the learned attention is consistent and applicable across both domains.

During backpropagation, gradients may flow through the attention mechanism, allowing the PBR network to adjust its parameters not only based on its own output but also considering the influence of the RGB network's features.

The attention layers may consist of multi-head attention units that allow the system to focus on different parts of the feature space simultaneously, capturing a more comprehensive representation of the cross-domain features.

The cross-domain attention mechanism may be designed to be computationally efficient while providing significant performance gains. It may leverage the pre-existing knowledge embedded in the RGB network to improve the PBR texture generation without the need for extensive retraining.

By leveraging features from a pre-trained RGB model, the system can generate PBR textures with higher fidelity and photorealistic quality.

The attention mechanism may help ensure that the generated PBR textures are consistent across different lighting conditions, which is crucial for their use in 3D rendering environments.

The mechanism allows for efficient training of the PBR network by utilizing the already learned features of the RGB network, reducing the need for large PBR-specific datasets.

The cross-domain attention enables the PBR network to access a broader and more complex feature set than it could learn on its own, leading to better generalization and creativity in texture generation.

The cross-domain attention significantly enhances the generation of PBR textures by intelligently leveraging and integrating the knowledge from pre-trained RGB models. This mechanism helps the system to produce high-quality, consistent, and realistic textures for use in various 3D content creation workflows.

While illustrated in the block diagrams as groups of discrete components communicating with each other via distinct data signal connections, it will be understood by those skilled in the art that the various embodiments may be provided by a combination of hardware and software components, with some components being implemented by a given function or operation of a hardware or software system, and many of the data paths illustrated being implemented by data communication within a computer application or operating system. The structure illustrated is thus provided for efficiency of teaching the present various embodiments.

It should be noted that the present disclosure can be carried out as a method, can be embodied in a system, a computer readable medium or an electrical or electro-magnetic signal. The embodiments described above and illustrated in the accompanying drawings are intended to be exemplary only. It will be evident to those skilled in the art that modifications may be made without departing from this disclosure. Such modifications are considered as possible variants and lie within the scope of the disclosure.

Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A “hardware module” is a tangible unit capable of performing certain operations and may be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

In some embodiments, a hardware module may be implemented mechanically, electronically, or with any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware module may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an Application Specific Integrated Circuit (ASIC). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware module may include software encompassed within a general-purpose processor or other programmable processor. Such software may at least temporarily transform the general-purpose processor into a special-purpose processor. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

Accordingly, the phrase “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where a hardware module comprises a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as respectively different special-purpose processors (e.g., comprising different hardware modules) at different times. Software may accordingly configure a particular processor or processors, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented module” refers to a hardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface (API)).

The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented modules may be distributed across a number of geographic locations.

FIG. 5 is a block diagram 500 illustrating an example software architecture 502, which may be used in conjunction with various hardware architectures herein described to provide a gaming engine 501 and/or components of the rendering engine. FIG. 5 is a non-limiting example of a software architecture and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 502 may execute on hardware such as a machine 600 of FIG. 6 that includes, among other things, processors 610, memory 630, and input/output (I/O) components 1050. A representative hardware layer 504 is illustrated and can represent, for example, the machine 600 of FIG. 6. The representative hardware layer 504 includes a processing unit 506 having associated executable instructions 508. The executable instructions 508 represent the executable instructions of the software architecture 502, including implementation of the methods, modules and so forth described herein. The hardware layer 504 also includes memory/storage 510, which also includes the executable instructions 508. The hardware layer 504 may also comprise other hardware 512.

In the example architecture of FIG. 5, the software architecture 502 may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture 502 may include layers such as an operating system 514, libraries 516, frameworks or middleware 518, applications 520 and a presentation layer 544. Operationally, the applications 520 and/or other components within the layers may invoke application programming interface (API) calls 524 through the software stack and receive a response as messages 526. The layers illustrated are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide the frameworks/middleware 518, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system 514 may manage hardware resources and provide common services. The operating system 514 may include, for example, a kernel 528, services 530, and drivers 532. The kernel 528 may act as an abstraction layer between the hardware and the other software layers. For example, the kernel 528 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The services 530 may provide other common services for the other software layers. The drivers 532 may be responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 532 may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth depending on the hardware configuration.

The libraries 516 may provide a common infrastructure that may be used by the applications 520 and/or other components and/or layers. The libraries 516 typically provide functionality that allows other software modules to perform tasks in an easier fashion than to interface directly with the underlying operating system 514 functionality (e.g., kernel 528, services 530 and/or drivers 532). The libraries 616 may include system libraries 534 (e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries 516 may include API libraries 536 such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries 516 may also include a wide variety of other libraries 538 to provide many other APIs to the applications 520 and other software components/modules.

The frameworks 518 (also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications 520 and/or other software components/modules. For example, the frameworks/middleware 518 may provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware 518 may provide a broad spectrum of other APIs that may be utilized by the applications 520 and/or other software components/modules, some of which may be specific to a particular operating system or platform.

The applications 520 include built-in applications 540 and/or third-party applications 542. Examples of representative built-in applications 540 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a game application. Third-party applications 542 may include any an application developed using the Android™ or iOS™ software development kit (SDK) by an entity other than the vendor of the particular platform, and may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. The third-party applications 542 may invoke the API calls 524 provided by the mobile operating system such as operating system 514 to facilitate functionality described herein. In example embodiments, the applications 520 may include a rendering module 543. In example embodiments, any of the operations described herein, such as the operations described with respect to FIGS. 1-4, may be implemented by the rendering module 543.

The applications 520 may use built-in operating system functions (e.g., kernel 528, services 530 and/or drivers 532), libraries 516, or frameworks/middleware 518 to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interactions with a user may occur through a presentation layer, such as the presentation layer 544. In these systems, the application/module “logic” can be separated from the aspects of the application/module that interact with a user.

Some software architectures use virtual machines. In the example of FIG. 5, this is illustrated by a virtual machine 548. The virtual machine 548 creates a software environment where applications/modules can execute as if they were executing on a hardware machine (such as the machine 600 of FIG. 6, for example). The virtual machine 548 is hosted by a host operating system (e.g., operating system 514) and typically, although not always, has a virtual machine monitor 546, which manages the operation of the virtual machine 548 as well as the interface with the host operating system (i.e., operating system 514). A software architecture executes within the virtual machine 548 such as an operating system (OS) 550, libraries 552, frameworks 554, applications 556, and/or a presentation layer 558. These layers of software architecture executing within the virtual machine 548 can be the same as corresponding layers previously described or may be different.

FIG. 6 is a block diagram illustrating components of a machine 600, according to some example embodiments, configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, FIG. 6 shows a diagrammatic representation of the machine 600 in the example form of a computer system, within which instructions 616 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 600 to perform any one or more of the methodologies discussed herein may be executed. As such, the instructions 616 may be used to implement modules or components described herein. The instructions transform the general, non-programmed machine into a particular machine programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine 600 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine