In another example, the action selection output 212 may directly define the action to be performed by the agent, e.g., by defining the values of torques that should be applied to the joints of a robotic agent.

In another example, the action selection output 212 may include a respective Q-value for each action in the set of possible actions that can be performed by the agent. If being used to directly control the agent, the Q-values could be processed (e.g., using a soft-max function) to generate a respective probability value for each possible action, which can be used to select the action to be performed by the agent (as described earlier). The action with the highest Q-value could also be selected as the action to be performed by the agent.

The Q value for an action is an estimate of a “return” that would result from the agent performing the action in response to the current observation 210 and thereafter selecting future actions performed by the agent 206 in accordance with current values of the policy network parameters.

A return refers to a cumulative measure of “rewards” received by the agent, for example, a time-discounted sum of rewards. The agent can receive a respective reward at each time step, where the reward is specified by a scalar numerical value and characterizes, e.g., a progress of the agent towards completing an assigned task.

At each time step, the state of the environment 208 at the time step (as characterized by the latent representation 222) depends on the state of the environment 208 at the previous time step and the action 204 performed by the agent 206 at the previous time step.

In some implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment.

In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent as it interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator.

For example in the case of a robot the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot.

In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations.

The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

In the case of an electronic agent the observations may include data from one or more sensors monitoring part of a plant or service facility such as current, voltage, power, temperature and other sensors and/or electronic signals representing the functioning of electronic and/or mechanical items of equipment.

In these implementations, the actions may be control inputs to control the robot, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements of the vehicle or higher-level control commands.

In other words, the actions can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. Action data may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the actions may include actions to control navigation, e.g., steering, and movement, e.g., braking and/or acceleration of the vehicle.

In some implementations the environment is a simulated environment and the agent is implemented as one or more computers interacting with the simulated environment. Training an agent in a simulated environment may enable the agent to learn from large amounts of simulated training data while avoiding risks associated with training the agent in a real world environment, e.g., damage to the agent due to performing poorly chosen actions. An agent trained in a simulated environment may thereafter be deployed in a real-world environment.

The simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle.

In another example, the simulated environment may be a video game and the agent may be a simulated user playing the video game. Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.

In a further example the environment may be a chemical synthesis or a protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals/intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that performs or controls the protein folding actions selected by the system automatically without human interaction. The observations may include direct or indirect observations of a state of the protein and/or may be derived from simulation.

In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharma chemical drug and the agent is a computer system for determining elements of the pharma chemical drug and/or a synthetic pathway for the pharma chemical drug. The drug/synthesis may be designed based on a reward derived from a target for the drug, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the drug.

In some applications the agent may be a static or mobile software agent, i.e., a computer programs configured to operate autonomously and/or with other software agents or people to perform a task. For example the environment may be an integrated circuit routing environment and the system may be configured to learn to perform a routing task for routing interconnection lines of an integrated circuit such as an ASIC. The rewards (or costs) may then be dependent on one or more routing metrics such as an interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters such as width, thickness or geometry, and design rules. The observations may be observations of component positions and interconnections; the actions may comprise component placing actions, e.g., to define a component position or orientation and/or interconnect routing actions, e.g., interconnect selection and/or placement actions. The routing task may thus comprise placing components, i.e., determining positions and/or orientations of components of the integrated circuit, and/or determining a routing of interconnections between the components. Once the routing task has been completed an integrated circuit, e.g., ASIC, may be fabricated according to the determined placement and/or routing. Or the environment may be a data packet communications network environment, and the agent be a router to route packets of data over the communications network based on observations of the network.

Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.

In some other applications the agent may control actions in a real-world environment including items of equipment, for example in a data center, in a power/water distribution system, or in a manufacturing plant or service facility. The observations may then relate to operation of the plant or facility. For example the observations may include observations of power or water usage by equipment, or observations of power generation or distribution control, or observations of usage of a resource or of waste production. The actions may include actions controlling or imposing operating conditions on items of equipment of the plant/facility, and/or actions that result in changes to settings in the operation of the plant/facility, e.g., to adjust or turn on/off components of the plant/facility.

In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources, e.g., on a mobile device and/or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources.

Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, and so on.

At each time step, the agent 206 can receive a reward (represented as a numerical value) based on the current state of the environment 208 (as characterized by the latent representation 222), the action 204 performed by the agent 206 at the time step, and the subsequent state of the environment after the agent performs the action.

FIG. 4 is a flow diagram of an example process 400 for training a policy neural network and a value neural network. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

In general, the system can repeatedly perform the process 400 on different training tuples selected, e.g., through random sampling, from the set of training tuples which can be derived from the experience information generated as a consequence of the interaction of the agent or another agent with the environment or with another instance of the environment. In some implementations, each training tuple can include data specifying (i) a respective current observation characterizing a respective current state of the environment, (ii) a respective preceding action performed by the agent, and (iii) a respective preceding latent representation characterizing a respective preceding state of the environment.

The system receives a latent representation in a latent space and characterizing a current state of the environment (402). That latent representation can be generated by a representation neural network based on processing a representation network input including an observation (e.g., an image or a video frame) of the current state of the environment. The latent representation of the observation can be represented as an ordered collection of numerical values, e.g., a vector or matrix of numerical values, and generally has a lower dimensionality than the observation itself.

The system generates a trajectory of multiple latent representations that starts with the received latent representation (404). In brief, and as will be described in more detail below with reference to FIG. 5, generating each trajectory of latent representations can include repeatedly generating an action selection output by using the policy neural network, selecting an action that would be performed by the agent by using the action selection output, and then processing the current latent representation and the selected action using a transition neural network to generate an updated latent representation.

For each latent representation in the trajectory, the system determines a predicted reward (406) and determines a predicted state value (408) that represents a predicted value of the environment being in the state characterized by the latent representation in terms of a cumulative measure of predicted rewards to be received by the agent. For example, the predicted state value may specify a long-term time-adjusted, e.g., time-discounted, sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs starting from the state characterized by the latent representation.

In more detail, for each latent representation in the trajectory, the system can determine the predicted reward by processing the latent representation using a reward neural network and in accordance with current values of a set of reward network parameters to generate a reward network output that specifies the predicted reward.

The system can determine the predicted state value by processing the latent representation using a value neural network and in accordance with current values of a set of value network parameters to generate a value network output that specifies the predicted state value.

The system determines a corresponding target state value for each latent representation in the trajectory (410) from the predicted rewards and the predicted state values for the latent representations in the trajectory. For each latent representation in the trajectory (referred to as a “current latent representation” below), the system can make this determination in any of a variety of ways.

In some implementations, the system can compute a sum of all predicted rewards received over H latent representations starting from a current latent representation and use the computed sum as the target state value, where H is a given (e.g., user-specified) positive integer. The sum can be an unweighted sum or a time-adjusted sum.

In some other implementations, the system can make this determination by factoring in predicted rewards to be received by the agent starting from an arbitrary step, which can be any step that is within or after the H steps. This allows the system to better account for the long-term rewards that would be received by the agent, e.g., distant future rewards to be received by the agent at states beyond the states characterized by the latent representations included in generated trajectories.

In such implementations, the system can determine one or more different values for a positive integer K and thereafter determine a respective candidate target state value for each of the values of K. The system can do this by, for each of the values of K: determining a time-adjusted (e.g., time-discounted) sum of the predicted rewards over H latent representations starting from a current latent representation, and then determining, by using the value neural network, an estimated state value for a latent representation that is H latent representations after the current latent representation. The respective H for any given latent representation can be a minimum of (i) a sum between a numeric position of the given latent representation in the trajectory and K, and (ii) a total number of latent representations in the trajectory.

In some such implementations, the system can determine the target state value by computing an exponentially-weighted average of the respective candidate target state values for the one or more values of K. This enables the system to balance the bias and variance that likely occur during the computation of the target state values.

In general, the system can repeatedly perform the steps 402-410 to generate multiple different trajectories including determining target state value information associated with the trajectories, e.g., until a predetermined number of trajectories have been generated. In some implementations, the system can do this in parallel, asynchronously, and/or in a decentralized manner, e.g., at different computing units that can operate independently of each other. A computing unit may be, for example, a computer, a core within a computer having multiple cores, or other hardware or software within a computer capable of independently performing the required operations for generating trajectory information.

The system then proceeds to use the target state values that have been determined for each trajectory of latent representations to determine updates to respective parameter values of the neural networks, as described in more detail below.

The system determines, based on the target state values, an update to the current values of the policy neural network parameters (412) that encourages the policy neural network to generate action selection outputs that result in actions being selected that improve the predicted state values. In other words, for each trajectory of latent representations, the system trains the policy neural network to generate action selection outputs that can be used to select actions that maximize, with respect to the corresponding target state values associated with the latent representations, a cumulative measure of predicted rewards to be received by the agent. In particular, the system can do this by computing analytic gradients of the target state values with respect to the policy network parameters, e.g., through stochastic backpropagation.

The system determines an update to the current values of the value neural network (414) parameters that encourages the value neural network to regress the corresponding target state values. The system can do this by evaluating, for each trajectory of latent representations, a value objective function that measures a difference between the predicted state values and the target state values, and thereafter computing a gradient of value objective function with respect to the value network parameters. For example, the difference can be computed as a mean squared error (MSE) difference.

In some implementations, the assigned task may be a complex task which requires selecting actions to be performed by the agent from a large discrete action space, a continuous action space, or a hybrid action space, i.e., with some sub-actions being discrete and others being continuous, and the action selection outputs of the policy neural network may parametrize a probability distribution over the entire action space. In such implementation, the system can determine the updates to network parameters by estimating the gradients through reparameterized sampling techniques, which view sampled actions as deterministically dependent on the action selection output of the policy neural network and allow for backpropagation of analytic gradients through the sampling operation.

In some implementations, the action space may be a discrete space, and the policy neural network and representation neural network may both be configured to generate discrete value outputs, i.e., discrete value action selection outputs and discrete value latent representations, respectively. In such implementations, the system can determine the updates to network parameters through straight-through gradient estimation techniques. This results in a biased gradient estimate with low variance, and thereby accelerates early stage training of the system.

The system then proceeds to apply the determined updates to policy and value network parameters by using an appropriate gradient descent optimization methods, e.g., stochastic gradient descent, RMSprop or Adam. Alternatively, the system only proceeds to update the current values of the network parameters once the process 400 have been performed for an entire mini-batch of selected training tuples. A mini-batch generally includes a fixed number of training tuples, e.g., 10, 20, or 50. In other words, the system combines, e.g., by computing a weighted or unweighted average of, respective gradients that are determined during the fixed number of iterations of the process 400 and proceeds to update the current network parameter values based on the combined gradient.

In general, the system can repeatedly perform the process 400 until a termination criterion is reached, e.g., after the process 400 have been performed a predetermined number of times or after the gradient has converged to a specified value.

FIG. 5 is a flow diagram of an example process 500 for generating a trajectory of latent representations. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

In general, the system can repeatedly perform the process 500 to generate a trajectory including multiple latent representations that starts with the received latent representation.

The system processes a current latent representation using the policy neural network and in accordance with current values of the policy neural network parameters to generate an action selection output (502). During training noise is added to the action selection output to facilitate action exploration. For example, the noise can be Gaussian distributed noise with an exponentially decaying magnitude.

The system selects, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current latent representation (504). Selecting the action by using the action selection output is similarly described above with reference to FIG. 2.

The system processes the current latent representation and the selected action using a transition neural network and in accordance with the current values of transition network parameters to generate an updated latent representation (506). The updated latent representation characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation.

For example, the transition neural network may be configured as a recurrent neural network, and the system may use the transition neural network to receive a current transition network input including the current latent representation and the selected action and to update a current hidden state of the transition neural network by processing the received input, i.e., to modify the current hidden state of the transition neural network that has been generated by processing previous inputs by processing the current received input. The system may then use the updated hidden state to generate a current output which specifies the updated latent representation.

Prior to or during the training of the policy neural network and value neural network by performing processes 400 and 500, the system trains the representation neural network to encourage the generation of high-quality latent representations, e.g., latent representations that can capture salient (e.g., task-specific) information contained in observation of the environment, or latent representations that are predictive of subsequent states of the environment. The system also trains the transition neural network to more accurately simulate latent dynamics of the environment, i.e., state transitions of the environment that would be caused by the agent performing the proposed actions, and trains the reward neural network to more accurately estimate rewards that would be received by the agent from the environment in response to performing the proposed actions.

To this end, the system can determine trained parameter values of the neural networks by applying the plurality of training tuples as training data for these neural network and by using representation learning techniques, as described in more detail below.

FIG. 6 is an example illustration of training a representation neural network, a reward neural network, and a transition neural network.

For each training tuple, the system processes, by using the representation neural network 610 and in accordance with current values of the representation network parameters, the training tuple, e.g., a training tuple including observation 606B, to generate a latent representation, e.g., latent representation 612B. The latent representation characterizes a current state of the environment into which the environment would transition, i.e., from the preceding state, if the preceding action, e.g., action 642A, specified in the training tuple were to be performed by the agent. When the training tuple is a first training tuple in each sequence of training tuples, the system can generate the current latent representation that characterizes an initial state of the environment by processing a current observation that is specified by the training tuple using the representation neural network.

Additionally, for each training tuple, the system processes the current latent representation, e.g., latent representation 612B, generated by the representation neural network to determine an estimation of a current reward, e.g., estimated reward 632B, to be received by the agent at the current state of the environment and in response to the agent performing the preceding action, e.g., action 642A. The system can do so by using the reward neural network and in accordance with current values of the reward network parameters.

For each training tuple, the system processes the preceding action, e.g., action 642A, performed by the agent and the preceding latent representation, e.g., latent representation 612A, that are specified by the training tuple to generate a current latent representation, e.g., latent representation 612B. The system can do so by using the transition neural network and in accordance with current values of the transition network parameters.

To provide richer training signals, in some implementations, the system also makes use of an observation neural network that is configured to process the latent representations, e.g., latent representation 612B, generated by the representation neural network to generate reconstructed observations, e.g., reconstructed observation 608B, emulating (or approximating) the original observations that characterize different states of the environment from which the latent representations are generated. For example, the observation neural network can have an architecture that performs operations that approximately invert the operations performed by the representation neural network described above with reference to FIG. 1.

The system then evaluates a representation learning objective function which measures a respective quality of each of the aforementioned network outputs, including the latent representation generated by the representation neural network, the reward estimation generated by the reward neural network, the latent representation generated by the transition neural network, and, in some implementations, the reconstructed observation generated by the observation neural network.

In general, the representation learning objective function can include any of a variety of components or terms that can be used to evaluate any of a variety of aspects of the quality of the network outputs. For example, the quality can be evaluated with respect to reward prediction, observation reconstruction, contrastive estimation, or a combination thereof. Reward prediction is described in more detail at J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019, and at K. Gregor, D. J. Rezende, F. Besse, Y. Wu, H. Merzic, and A. v. d. Oord. Shaping belief states with generative environment models for rl. arXiv preprint arXiv:1906.09237, 2019. Image reconstruction is described in more detail at D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018. Contrastive estimation is described in more detail at A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

The system determines, based on computing a gradient of the representation learning objective function with respect to the representation network parameters, an update to current values of the representation network parameters. The system also determines, based on computing respective gradients of the representation learning objective function with respect to the reward network parameters and the transition network parameters, respective updates to current values of the reward neural network parameters and the transition neural network parameters. In implementations where the observation neural network is used, the system also computes a gradient of the representation learning objective function with respect to the observation network parameters and thereafter backpropagates the gradient into the parameters of other neural networks.

The system then proceeds to apply the determined updates to respective parameters of the representation neural network, reward neural network, transition neural network, and, optionally, the observation neural network, as similarly described above with reference to FIG. 4.

In some implementations, the system can interleave the training of the neural networks with the control of the agent in performing one or more actual tasks while interacting with the environment. This allows the system to generate and add new experience data to the training tuple set that can thereafter be used in the subsequent training of the system. For example, at certain points during the training process, e.g., upon determining that a predetermined number of iterations of processes 400 and 500 have been performed, the system can use the policy neural network and the representation neural network and in accordance with current values of their parameters (as of the iteration) to control the agent to perform a sequence of actions when interacting with the environment, i.e., by generating a latent representation from each new observation and then selecting an action to be performed by the agent in response to the new observation using the action selection output generated from the latent representation.

An example algorithm for training the policy neural network together with other trainable components of the system is shown below.

Algorithm 1: Dreamer

Initialize dataset [Image Omitted]

with S random seed episodes.

Initialize neural network parameters θ, ϕ, ψ randomly.

while not converged do

for update step c = 1 .. C do

// Dynamics learning

Draw
⁢

⁢
B
⁢

sequences
⁢

⁢

{

(

a
t

,

o
t

,

r
t

)

}

t
=
k

k
+
L

∼

𝒟
.

Compute model states s_t~p_θ (s_t| s_t−1, a_t−1, o_t).

Update θ using representation learning.

// Behavior learning

Imagine
⁢

trajectories
⁢

{

(

s
τ

,

a
τ

)

}

τ
=
t

t
+
H

⁢

from
⁢

each
⁢

s
t

.

Predict rewards E(q_θ (r_τ | s_τ)) and values v_ψ ( s_τ).

Compute value estimates V_λ ( s_τ)

Update
⁢

ϕ

←

ϕ
+

α
⁢

∇
ϕ

∑

τ
=
t

t
+
H

⁢

V
λ

(

s
τ

)

.

Update
⁢

ψ

←

ψ
-

α
⁢

∇
ψ

∑

τ
=
t

t
+
H

⁢

1
2

⁢



v
ψ

(

s
T

)

-

V
λ

(

s
T

)



2

.

// Environment interaction

o₁← env.reset( )

for time step t = 1 .. T do

Compute s_t~p_θ (s_t| s_t−1, a_t−1, o_t) from history.

Compute a_t~q_ϕ(a_t|s_t) with the action model.

Add exploration noise to action.

Add
⁢

experience
⁢

to
⁢

dataset
⁢

⁢
𝒟

←

D
⋃

{

(

o
t

,

a
t

,

r
t