13 or the environment sensor 2 among the information acquired from the robot hand 1 and the environment sensor 2 to the RND 351 and calculates the internal reward relating to the contact state. The network of the RND 351 calculates the internal reward by using the method of Reference Document 1 described above. The calculation of the internal reward may be performed by the kernel method or the like. The RND 351 outputs the internal reward when a contact state is input. For example, when an action at is taken in a state s_t, only the contact state of the next state s_t+1that is output by an environment 354 (Env) is input to the RND 351, and the RND 351 calculates the internal reward.

The first strategy 352 is a strategy considering only the internal reward. The first strategy 352 is trained and updated by the internal reward. Information of the acquired state quantity (a joint angle or the like) of the robot, an object position and attitude, a detection value of the haptic sensor 11, a detection value of the tactile sensor 12, a command performed previously, a target contact position, or the like is input to the first strategy 352. The output of the first strategy 352 is an action.

The second strategy 353 is a strategy considering only the external reward. The second strategy 353 is trained and updated by the external reward. Information of the acquired state quantity (a joint angle or the like) of the robot, an object position and attitude, a detection value of the haptic sensor 11, a detection value of the tactile sensor 12, a command performed previously, a target contact position, or the like is input to the second strategy 353. The command performed previously is stored by the storage portion 307. Further, the target contact position is generated, for example, by the learning process portion 308. The output of the second strategy 353 is an action. The calculation of the external reward differs depending on a set task. For example, in the case of a task that changes the position of an object, the external reward is calculated only from some information (the position of the object) of the state quantity. Further, in the case of another task, the external reward is calculated from all information of the state quantity.

The number of each strategy is not limited to one. For example, when there are a plurality of goals, a strategy may be provided for each goal. Alternatively, a plurality of goals may be used and input to one strategy. That is, the strategy may be parallelized or may be used with a goal condition.

The external reward can be stored, and the value of the external reward can be used as is at the time of training. In the case of an on-policy, the internal reward is also temporarily stored, and the value of the internal reward is used as is at the time of training but is discarded after the training. It is determined whether or not the internal reward in the case of an off-policy is rare in consideration of the data experienced so far. Therefore, after the number of data is increased, recalculation is required.

Then, in the training using the external reward, a state s_t, an action a_t, an external reward r^{{circumflex over ( )}Ext}, and a state s_t+1after the action are taken out from the storage portion 307 for a plurality of t, and the parameter of a strategy network is updated based on these values.

In the training using the internal reward, a state s_t, an action a_t, and a state s_t+1after the action are taken out from the storage portion 307 for a plurality of t, an internal reward r^{{circumflex over ( )}Int}is recalculated with respect to s_t+1, and the parameter of a strategy network is updated based on these values.

The environment 354 (Env) probabilistically selects the outputs of the first strategy 352 and the second strategy 353 and performs the action output by the strategy by an environment model. The probability of selection is, for example, 40% for the first strategy 352 and 60% for the second strategy 353. The output of the environment 354 (Env) is only the external reward calculated based on the state quantity, object information, and contact information of the robot.

For example, when the state s_tand the action at are input, the environment 354 (Env) outputs the next state s_t+1and the external reward r^{{circumflex over ( )}Ext}. When the action output by the first strategy 352 is input or when the action output by the second strategy 353 is input, the environment 354 (Env) outputs the next state s_t+1and the external reward r^{{circumflex over ( )}Ext}. Further, when the scales output by the first strategy 352 and the second strategy 353 are different from each other, for example, normalization is performed such that the scale ranges become the same. The stored information is the state s_t, the action at, the external reward r^{{circumflex over ( )}Ext}, and the next state s_t+1. Further, the internal reward r^{{circumflex over ( )}Int}is recalculated at the time of training.

In this way, the environment 354 (Env) corresponds to the external reward calculation portion 305 that calculates the external reward. Further, the RND 351 corresponds to the internal reward calculation portion 304 that calculates the internal reward.

(Learning Procedure of Manipulation Method Learning Apparatus)

Next, an example of a process procedure of a manipulation method learning apparatus is described.

FIG. 4 is a flowchart showing a learning procedure of a manipulation method learning apparatus according to the present embodiment.

The process of FIG. 4 is an example of the case of an on-policy. The on-policy is a process that accumulates data and does not reuse the data for the update of the policy.

(Step S1) The learning process portion 308 initializes the frequency of occurrence of the contact state stored by the storage portion 307, that is, the parameter of the RND or the like. The contact state includes, for example, information such as a contact state between a target object and a finger portion or a position of a target object touched by a finger portion.

(Step S2) The learning process portion 308 initializes (set to a predetermined value or the like) parameters of networks of the first strategy and the second strategy.

(Step S3) The first acquisition portion 301 to the third acquisition portion 303 acquire a present state quantity (the position of the target object, the attitude of the target object, the detection value of the haptic sensor, the detection value of the tactile sensor, the previous command, and the target contact position) including a present contact state.

(Step S4) The internal reward calculation portion 304 obtains the frequency of occurrence of the contact state and calculates the internal reward. That is, the internal reward calculation portion 304 calculates the internal reward by using some of the state quantity relating to the contact state and by using the RND. Further, the external reward calculation portion 305 calculates the external reward by using all or some of the state quantity depending on the task. The learning process portion