260 is below a predetermined threshold. During training, training engine 122 iteratively adjusts the parameters of coefficient matrix C of spherical codebook 240, NeRF model 220, and OLAT model 230 based on the rendering loss. Training engine 122 transmits training data 200 to relightable NeRF model 210. As described above, training data includes a plurality of 2D representations of a scene, with each 2D representation including ground truth color values for each pixel in the 2D representation. Each 2D representation further includes ground truth data for the location of a camera used to capture the 2D representation and the location of one or more light sources illuminating the scene during capture.

For each 3D location x_iin the scene, training engine 122 transmits 3D location x_ito NeRF model 220. NeRF model 220 generates neural feature vector f_iand density value a based on the 3D location x_i. NeRF model 220 transmits neural feature vector f_ito OLAT model 230 and transmits density value a to volume rendering engine 250.

Training engine 122 generates an OLAT basis based on a light direction ω_ifrom a given illuminated light source p in training data 200 to the 3D location in the scene and transmits the OLAT basis to spherical codebook 240. Training engine 122 queries spherical codebook 240 on a K-order spherical harmonic parameterization SH(ω_i) of the OLAT basis. Spherical codebook 240 multiplies the spherical harmonic parameterization by a coefficient matrix C to generate OLAT code o_ωi. Spherical codebook 240 transmits OLAT code o_ωito OLAT model 230.

Based on OLAT code o_ωiand the neural feature vector f_ireceived from NeRF model 220, OLAT model 230 generates a radiance field function including a diffuse color value c_idand a specular color value c_isfor the given 3D location x_iin the scene.

If a 2D representation from training data 200 includes ground truth data for more than one illuminated light source p, training engine 122 repeatedly generates an OLAT basis associated with a light direction from each illuminated light source p and queries spherical codebook 240 for each illuminated light source p. For the 3D location x_i, spherical codebook 240 generates an OLAT code o_ωifor each additional light source and OLAT model 230 generates diffuse color values c_idand specular color values c_isassociated with each additional light source p. Training engine 122 generates combined diffuse and specular color values based on a weighted combination of the diffuse color values and specular color values associated with the each of the plurality of illuminated light sources.

The neural feature vector f_iand density value σ_igenerated by NeRF model 220 are only generated once for each 3D location x_iin the scene, regardless of the number of light sources. The neural feature vector f_itransmitted from NeRF model 220 to OLAT model 230 is held constant for each 3D location x_iin the scene.

Training engine 122 continues processing 3D locations in the scene until NeRF model 220 and OLAT model 230 have generated density, diffuse color, and specular color values for all 3D locations in the scene.

Volume rendering engine 250 analyzes the diffuse color and specular color values generated for 3D locations in the scene and generates a 2D rendering of the scene as described above based on the density, diffuse color and specular color values for 3D locations in the scene generated by NeRF model 220 and OLAT model 230. Volume rendering engine 250 transmits the generated 2D rendering to rendering loss engine 260.

Rendering loss engine 260 generates a rendering loss for the generated 2D rendering that represents a difference between the generated 2D rendering and the ground truth data associated with the 2D rendering in training data 200. As described above in various embodiments, the rendering loss may include one or more terms representing pixelwise differences in color values between the generated 2D rendering and the ground truth data. Training engine 122 analyzes the rendering loss and if the rendering loss is below a predetermined threshold, training engine 122 may terminate training. Training engine 122 may also terminate training if a predetermined training time has elapsed or if a predetermined number of training iterations have been completed. If the rendering loss is at or above the predetermined threshold and neither the predetermined training time nor predetermined number of iterations have been exceeded, training engine 122 continues training relightable NeRF model 210.

FIG. 3 is a more detailed illustration of inference engine 124 of FIG. 1, according to some embodiments. Given a trained relightable NeRF model 310, inference engine 124 generates a 2D rendering of a scene that represents the scene as captured from a given camera position and as illuminated by a given plurality of light sources. As shown, inference engine 124 further includes input data 300, NeRF model 320, OLAT model 330, spherical codebook 340, and volume rendering engine 350.

Input data 300 includes a given camera position and a lighting map that includes position and intensity values for a plurality of light sources. Inference engine 124 transmits input data 300 to trained relightable NeRF model 310.

For each pixel in the generated 2D rendering, volume rendering engine 350 calculates a directional viewing ray ω_ofrom the given camera position through the pixel into the scene. Volume rendering engine calculates a plurality of 3D positions x_iin the scene that intersect directional viewing ray ω_o. For each of the plurality of 3D positions x_iin the scene that intersect directional viewing ray ω_o, inference engine 124 transmits the 3D position x_ito NeRF model 320.

NeRF model 320 receives the 3D position x_iin the scene and calculates a density value a associated with the 3D position x_i. NeRF model 320 further generates a neural feature vector f_iand transmits the neural feature vector to OLAT model 330.

Inference engine 124 generates an OLAT basis based on a direction ω_ifrom a given illuminated light source p in input data 300 to the 3D location x_iin the scene and transmits the OLAT basis to spherical codebook 340. Spherical codebook 340 is indexed on a K-order spherical harmonic parameterization SH(ω_i) for the light direction ω_i. In some embodiments, spherical codebook 340 is indexed on a 5^th-order spherical harmonic parameterization SH(ω_i) for the lighting direction ω_i. Spherical harmonic parameterization SH(ω_i) is a representation of the spatial relationship between the given illuminated light source and the 3D location in the scene. Spherical codebook 340 includes a coefficient matrix C of learnable parameters. Inference engine 124 queries spherical codebook 340 on the spherical harmonic parameterization SH(ω_i) and spherical codebook 340 multiplies the spherical harmonic parameterization SH(ω_i) by coefficient matrix C to generate OLAT code o_ωi. OLAT code o_ωiis a vector, for example a 256-dimensional vector. Spherical codebook 340 transmits OLAT code o_ωito OLAT model 330.

Based on OLAT code o_ωifrom spherical codebook 340 and the neural feature vector f_ireceived from NeRF model 320, OLAT model 330 generates a diffuse color value c_idand a specular color value c_isfor the given 3D location x_iin the scene.

If input data 300 includes position data for more than one illuminated light source p, inference engine 124 repeatedly generates an OLAT basis associated with each light source p and transmits the OLAT basis to spherical codebook 340. Spherical codebook 340 and OLAT model 330 process the OLAT basis as described above to generate additional diffuse color values c_idand specular color values c_isfor the 3D locations x_iassociated with each additional light source p. Inference engine 124 generates combined diffuse and specular color values for the 3D location x_ibased on a weighted combination of the diffuse color values and specular color values associated with each additional light source.

The neural feature vector f_iand density value a generated by NeRF model 320 are only generated once for each 3D location x_iin the scene, regardless of the number of light sources in input data 300. The neural feature vector f_itransmitted from NeRF model 320 to OLAT model 330 is held constant for the 3D location x_iin the scene.

Volume rendering engine 350 repeats the above process for each of the plurality of 3D locations x_iin the scene that intersect directional viewing ray ω_o. Volume rendering engine 350 generates a color value for the pixel in the 2D rendering based on the color and density values for each of the plurality of 3D positions x_iin the scene that intersect directional viewing ray ω_o. Volume rendering engine 350 repeats the above process for each of the pixels in the 2D rendering and generates a completed 2D rendering of the scene.

FIG. 4 is a flow diagram of method steps for training a machine learning model to generate 2D renderings of a 3D scene, according to various embodiments. Although the method steps are described in conjunction with the systems of FIGS. 1-3, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, in operation 402, training engine 122 receives training data 200. Training data 200 includes multiple 2D representations of a 3D scene and ground truth data associated with each 2D representation describing the location of a camera used to capture the scene and the locations of one or more light sources illuminating the captured scene.

In operation 404, training engine 122 trains a NeRF model 220 and an OLAT model 230 by iteratively adjusting parameters of one or more of the NeRF and OLAT models. For each 3D position in the scene, NeRF model 220 generates a density value for the 3D location and a neural feature vector. NeRF model 220 transmits the neural feature vector to OLAT model 230.

Training engine 122 generates an OLAT basis based on a direction from a light source in the training data to the 3D location in the scene. Training engine 122 transmits the OLAT basis to spherical codebook 240. Spherical codebook 240 generates an OLAT code and transmits the OLAT code to OLAT model 230.

OLAT model 230 generates diffuse and specular color values for the 3D location in the scene, based on the neural feature vector received from the NeRF model and the OLAT code received from the spherical codebook. If the training data includes multiple light sources, training engine 122 generates an OLAT basis for each light source and OLAT model 230 generates separate diffuse and specular color values for the 3D location based on each of the plurality of light sources. Training engine 122 combines the separate diffuse and specular color values for the 3D location in the scene.

In operation 406, training engine 122 performs volume rendering based on the density and color values generated by NeRF model 220 and OLAT model 230. Based on the camera location data from training data 200, volume rendering engine 250 generates a directional viewing ray from the camera through a pixel of a 2D rendering and into the 3D scene modeled by the NeRF and OLAT models. For each of a plurality of 3D locations in the scene that intersect the directional viewing ray, volume rendering engine 250 calculates a density value and color values based on the outputs of the NeRF and OLAT models. Volume rendering engine 250 generates a color value for the pixel based on the density and color values of the 3D locations that intersect the directional viewing ray. Volume rendering engine repeats the above process for each pixel in the 2D rendering and transmits the completed 2D rendering to rendering loss engine 260.

In operation 408, training engine 122 generates a rendering loss based on a difference between the 2D rendering generated by volume rendering engine 250 and ground truth data for the 2D representation in training data 200. In various embodiments, the rendering loss may be a pixelwise mean-squared error loss based on the difference between a predicted color value for a pixel in the 2D rendering and ground truth color data for the corresponding pixel in the ground truth data for the 2D representation in training data 200. In various embodiments, each 2D representation in training data 200 may include a background mask that defines portions of the 2D representation that represent a background of a scene rather than an object in the scene. The rendering loss may include a term that penalizes density values calculated by NeRF model 220 for 3D locations in the scene that represent empty space based on the directional viewing ray coo and the background mask. Training data 200 may also include depth maps representing ground truth distances from one or more cameras to locations within the scene. The rendering loss may include a term that penalizes differences between calculated and ground truth depth values.

To enforce smoothness and avoid overfitting, rendering loss engine may apply regularization to coefficient matrix C. The regularization is based on the order K of the spherical harmonic parameterization described above and penalizes high-frequency specular color coefficients generated by OLAT model 230.

In operation 410, training engine 122 compares the rendering loss to a predetermined threshold. If the rendering loss is less than the predetermined threshold, training engine 122 terminates training of relightable NeRF model 210. Training engine 122 may also terminate training if a predetermined training time has elapsed or if a predetermined number of training iterations have been completed. If the rendering loss is at or above the predetermined threshold and neither the predetermined training time nor predetermined number of iterations have been exceeded, training engine 122 returns to operation 404 and continues training relightable NeRF model 210 with additional training data from training data 200.

FIG. 5 is a flow diagram of method steps for generating a 2D rendering of a scene, according to various embodiments. Although the method steps are described in conjunction with the systems of FIGS. 1-3, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, in operation 502, inference engine 124 receives a trained relightable NeRF model 310 and input data 300 including a given camera position and a lighting map that includes position and intensity data for a plurality of illuminated light sources.

In operation 504, inference engine 124 performs volume rendering to generate a 2D rendering of a scene modeled by NeRF model 320 and OLAT model 330 of trained relightable NeRF model 310. Based on the camera location data from input data 300, volume rendering engine 350 generates a directional viewing ray from the camera through a pixel of a 2D rendering and into the 3D scene modeled by the trained NeRF and OLAT models.

In operation 506, for each of a plurality of 3D locations in the scene that intersect the directional viewing ray, inference engine 124 calculates a density value and a neural feature vector based on the outputs of NeRF model 320.

In operation 508, inference engine 124 calculates diffuse and specular color values for each of the 3D locations that intersect the directional viewing ray. Inference engine 124 generates an OLAT basis based on a direction from a light source in input data 300 to the 3D location in the scene. Inference engine 124 transmits the OLAT basis to spherical codebook 340. Spherical codebook 340 generates an OLAT code and transmits the OLAT code to OLAT model 330.

OLAT model 330 generates diffuse and specular color values for the 3D location in the scene, based on the neural feature vector received from the NeRF model and the OLAT code received from the spherical codebook. If input data 300 includes multiple light sources, inference engine 124 generates an OLAT basis for each light source and OLAT model 330 generates separate diffuse and specular color values for the 3D location based on each of the plurality of light sources. Inference engine 124 combines the separate diffuse and specular color values for the 3D location in the scene.

In operation 510, inference engine 124 analyzes the diffuse and specular color values for each of the 3D locations that intersect the directional viewing ray and generates a color value for the pixel of the 2D rendering.

In operation 512, inference engine 124 determines whether a color value has been generated for every pixel in the 2D rendering. If a color value has been generated for every pixel in the 2D rendering, the method terminates. If there are remaining pixels in the 2D rendering for which color values have not been generated, the method returns to operation 504.

In sum, a training engine processes multiple 2D representations of a 3D scene from varying viewpoints and under varying illumination conditions to train a relightable neural radiance field (NeRF) model including a NeRF model and a “one light at a time” (OLAT) model. The output of the trained relightable NeRF model is a combined radiance field function representing the scene. The combined radiance field function maps a 3D location within the scene, an associated viewing direction, and given illumination information for the scene to a diffuse color value, a specular color value, and a density value. The relightable NeRF model may be used to generate 2D representations of the scene from arbitrary viewpoints and under arbitrary illumination conditions. The relightable NeRF model includes a NeRF model that generates density values for 3D locations in the scene and an OLAT model that generates color values for 3D locations in the scene. The relightable NeRF model further includes a spherical codebook that provides an OLAT code as an input feature vector to the OLAT model.

In operation, the training engine trains the relightable NeRF model on a data set of 2D representations of the 3D scene. Each 2D representation in the data set further includes ground truth data including a viewing direction for the 2D representation and a lighting map including illumination information for the scene. The combined output of the trained relightable NeRF model is a radiance field function representing the 3D scene. The training goal for the training engine is to produce a radiance field function such that 2D renderings of the 3D scene from different viewpoints generated based on the radiance field function match the ground truth 2D representations in the training data set. The training engine achieves the training goal by minimizing a loss function that represents at least a difference between a 2D rendering generated by the relightable NeRF model and ground truth data associated with the corresponding 2D representation in the training data set. At the conclusion of training, the relightable NeRF model may be used to render novel 2D representations of the scene focused on arbitrary locations within the scene, viewed from arbitrary viewpoints under arbitrary illumination conditions.

One technical advantage of the disclosed technique relative to the prior art is that the disclosed technique may generate 2D representations of a scene not only from arbitrary viewpoints but also under arbitrary lighting conditions. Unlike existing techniques that are limited to producing 2D representations with fixed illumination characteristics, the disclosed technique allows for realistically rendering 2D representations of a scene into a variety of environments with different illumination conditions.

- 1. In various embodiments, a computer-implemented method comprises determining, for each of a plurality of three-dimensional (3D) locations in a 3D scene, a density value associated with the 3D location based on an output of a first trained machine learning model, determining, for each of the plurality of 3D locations, a diffuse color value and a specular color value associated with the 3D location based on a given camera location, a given lighting map, and an output of a second trained machine learning model, determining a pixel color value for each of a plurality of pixels in a two-dimensional (2D) representation of the scene based on the density values, the diffuse color values, and the specular color values associated with the plurality of 3D locations, and generating, based on the pixel color values associated with the plurality of pixels, a 2D rendering of the scene.
- 2. The computer-implemented method of clause 1, further comprising generating, using the first trained machine learning model, a neural feature vector based on the 3D location, and transmitting the neural feature vector to the second trained machine learning model.
- 3. The computer-implemented method of clause 1 or 2, wherein the first trained machine learning model is a neural radiance field (NeRF) model and the second trained machine learning model is a “one light at a time” (OLAT) model.