>
e
k

⁢
ϵ
⁢

Neigh
⁡
(

e
u

)

m

k
,
u

.

Step 404, obtaining the environment state at the moment t according to the user context feature vector, the user feature vector, and the item feature vector. In other words, the user feature vector, the document feature vector, and the user context feature vector are merged to become the final environment state at the moment t, i.e., st=(the user feature vector, the document feature vector, the user context feature vector), also noted as, the state space s=(E(e_u), E_D, E(e_u)).

It should be noted that the rank in which the user feature vector, the item feature vector, and the user context feature vector are obtained is arbitrary and the above flowchart does not result in a limitation on the steps in which the user feature vector, the item feature vector, and the user context feature vector are obtained.

Step 302, obtaining an initial score vector of an item at the moment t based on the state at the moment t and the policy network.

Exemplarily, a Deep Q network (DQN) algorithm is used to perform real-time optimization on the ranks of documents. In this algorithm, the output of the Actor is a deterministic policy, i.e. a certain action, the state space s, i.e. the state environment at the moment t, is input to the Actor, and finally the initial rank score vector a of the document is output, the formula is as following;

$μ_{θ^{μ}} (s) = a$

$a = (a_{1}, a_{2}, \dots, a_{k})$

$θ^{μ} = (θ_{1}^{u}, θ_{2}^{u}, \dots, θ_{m}^{a});$

where a is a K-dimensional vector representing an output of the initial rank score at of the K documents, and θ^uis a parameter for the training of the policy network.

Step 303, obtaining a weight of the item based on a matching degree between the user and the item.

Exemplarily, the cosine similarity between the user label and the document label is used as the rank weight of the document, and the greater the similarity between the user label and the document label, the higher the rank weight. The set of label feature vectors of the user u is

${l_{u_{1}}, l_{u_{2}}, \dots l_{u_{m_{u}}}},$

l_u_irepresents the ith label vector of the user u, and m_urepresents the number of labels of the user u. The set of label feature vectors of the document D_dis

${l_{D_{d, 1}}, l_{D_{d, 2}}, \dots l_{D_{d, n_{d}}}},$

l_D_d,jrepresents the jth label vector of the document D_d, n_drepresents the number of labels of the document D_d, C_u,drepresents the matching degree, i.e., the similarity, between the user u and the document, the embodiment uses the cosine similarity to calculate, the formula is as following:

$C_{u, d} = \sum_{i = 1}^{m_{u}} \sum_{j = 1}^{n_{d}} \frac{l_{u_{i}} \cdot l_{D_{d, j}}}{ l_{u_{i}}   l_{D_{d, j}} } .$

The above matching degree obtained by using the cosine similarity is used as the weight of the item.

Step 304, generating the item recommendation rank based on the initial score of the item at the moment t and the weight of the item. The item recommendation rank is obtained by combining the initial score and weight obtained from the policy network to make the item recommendation rank more relevant to the actual needs of the user.

Carrying on from the above embodiments, λ_u,drepresents the final score of the document D_dof the user u, the λ_u,dis expressed with the following formula:

$λ_{u, d} = \frac{C_{u, d} \cdot a_{d}}{\sum_{i = 1}^{k} C_{u, i}};$

The rank results of final k documents are generated according to {λ_u,1, λ_u,2, . . . , λ_u,k}.

Step 202, obtaining an experience at the moment t based on the feedback result from the user on the item recommendation rank.

In an embodiment, the experience at the moment t includes: a state at the moment t, an initial score of an item at the moment t, a reward at the moment t, and a state at moment t+1. The initial score of the item at the moment t is obtained based on the state at the moment t and the policy network; the reward at the moment t is obtained based on the feedback result from the user on the item recommendation rank.

In an embodiment, the feedback result includes: a primary click, a secondary click and a page number in a recommendation page where the item is located; obtaining the reward at the moment t based on the primary click, the secondary click and a preset function. An independent variable of the preset function is the page number in the recommendation page where the item is located, and the preset function represents a variation tendency of the reward at the moment t with a variation of the page number in the recommendation page. The generation of the reward at the moment t takes into account not only whether the user clicks on the document, but also the secondary click of the user, such that the generated reward is more accurate. In addition, the designed preset function indicates the variation tendency of the reward at the moment t with a variation of the page number where the item is located, which simulates the variation of the actual user's operation according to the page number, to further improve the accuracy of the generated reward.

Exemplarily, C_irepresents the ith document click, C_iis 1 when the click occurs, otherwise C_iis 0, D_irepresents whether the ith document is secondly clicked, D_iis 1 when the secondary click occurs, otherwise D_iis 0, X_irepresents the page number in the recommendation page where the document is located, the designed H(x) function represents the variable tendency of the reward with a variation of the page number, the variable tendency of H(x) varies from fast to slow, H(x) meets: H(x)>0, H′(x)<0, H″(x)>0 (x>0), i.e., the first order derivative of H(x) is less than zero, the second order derivative of H(x) is greater than zero. The embodiment sets

$H (x) = \frac{1}{1 + β e^{x}} .$

α, β are coefficients (α>0, β>0). Then the sum of the rewards of the K documents is as follows:

$R = \sum_{i = 1}^{k} (C_{i} + α D_{i}) (1 + \frac{1}{1 + β e^{x,}});$

The sum of the above k documents is noted as rt, and the rt is the reward at the moment t, and the state of moment t+1 noted as st+1 is obtained after the user operation is completed.

From the above, the state st at the moment t, the initial score at of the item at the moment t, the reward rt at the moment t, and the state st+1 at the moment t+1 are obtained, and the experience (st, at, rt, st+1) at the moment t is obtained.

Step 203, aggregating the experience of each user at the moment t to a global experience pool.

Following the above embodiment, when multiple users query documents online at the same time, each user is assigned an actor, and the actor is responsible for generating document rank for the user and collecting feedback from the user to store (st, at, rt, st+1) as experience in the local experience pool. The experience pool of each actor is aggregated into the global experience pool.

Step 204, obtaining an updated policy network based on experiences in the global experience pool.

In an embodiment, the experience set is sampled from the global experience pool and the updated policy network is obtained based on the experience set, which can enhance the training speed of the policy network.

The recommendation method of the embodiment combines ReinforcementLearning (RL). ReinforcementLearning, also known as augmented learning, is used to describe and solve the problem of an intelligent agent learning a policy to maximize the reward or achieve a specific goal during its interaction with the environment. RL, compared with a traditional machine learning, enables online incremental learning with the following main operational processes: generating decisions based on the current environment state and translating them into actions; changing the current environment and collecting the reward, and performing a decision of the next action according to the itself state, by parity of reasoning, this model is referred to as the “state-action-feedback-state-update” cycle, in which the decision-making capability of the intelligent agent is continuously improved to maximize the reward on a specific goal. The embodiment can continuously improve the fit between the item recommendation rank and the user's own needs in the policy network update.