For example, suppose there is a paragraph with 6 sentences and the first three are related to one topic whereas the last three are related to a new topic. Smart chunking will automatically split the paragraph into two chunks, the first consisting of the first three sentences and the second consisting of the latter three sentences. To do this, first sentence embeddings (d-dimensional vectors) are generated for each sentence that represent their semantic content (sentences with vectors that are closer together in Euclidean space are more similar in meaning).

In an example, a paragraph consists of three identical sentences, followed by three identical sentences that are different from the first three (e.g., “I love reindeer.” “I love reindeer”. “I love reindeer”. “What's for dinner?” “What's for dinner?” “What's for dinner?”). In this case the first three sentences will have the same sentence embedding and the cosine similarity between any pair of them will be equal to 1. Likewise for the last three sentences. However, the cosine similarity between any sentence in the first set and any sentence in the second set will be much lower (e.g., cosine similarity near 0). Then a sliding window of size 2 may be used, and weights of 0.8 and 0.2 respectively (higher weights for sentences that are closer in position within the paragraph). For the first sentence, we calculate its similarity with subsequent sentences as 0.8×1+0.2×1=1. For the second sentence, the similarity with subsequent sentences is 0.8×1+0.2×0=0.8. For the third sentence, the similarity is 0.8×0+0.8×0=0. For the fourth sentence, it is 0.8×1+0.8×1=1. Following this example illustrates that the weighted similarity hits a local minimum at the third sentence, which is where the paragraph is split. As a final step, if any of the resulting chunks exceeds some threshold, it may be split again in order to satisfy maximum context length requirements.

FIG. 5 is a simplified diagram illustrating the neural network structure, according to some embodiments. In some embodiments, the RAG module 230 may be implemented at least partially via an artificial neural network structure shown in FIG. 5. The neural network comprises a computing system that is built on a collection of connected units or nodes, referred to as neurons (e.g., 544, 545, 546). Neurons are often connected by edges, and an adjustable weight (e.g., 551, 552) is often associated with the edge. The neurons are often aggregated into layers such that different layers may perform different transformations on the respective input and output transformed input data onto the next layer.

For example, the neural network architecture may comprise an input layer 541, one or more hidden layers 542 and an output layer 543. Each layer may comprise a plurality of neurons, and neurons between layers are interconnected according to a specific topology of the neural network topology. The input layer 541 receives the input data such as training data, user input data, vectors representing latent features, etc. The number of nodes (neurons) in the input layer 541 may be determined by the dimensionality of the input data (e.g., the length of a vector of the input). Each node in the input layer represents a feature or attribute of the input.

The hidden layers 542 are intermediate layers between the input and output layers of a neural network. It is noted that two hidden layers 542 are shown in FIG. 5 for illustrative purpose only, and any number of hidden layers may be utilized in a neural network structure. Hidden layers 542 may extract and transform the input data through a series of weighted computations and activation functions.

For example, as discussed in FIG. 2, the RAG module 230 may receive input 240 such as text from a document and generate an output 250 which may be a sentence embedding. A neural network such as the one illustrated in FIG. 5 may be utilized to perform, at least in part, the embedding. Each neuron receives input signals, performs a weighted sum of the inputs according to weights assigned to each connection (e.g., 551, 552), and then applies an activation function (e.g., 561, 562, etc.) associated with the respective neuron to the result. The output of the activation function is passed to the next layer of neurons or serves as the final output of the network. The activation function may be the same or different across different layers. Example activation functions include but not limited to Sigmoid, hyperbolic tangent, Rectified Linear Unit (ReLU), Leaky ReLU, Softmax, and/or the like. In this way, after a number of hidden layers, input data received at the input layer 541 is transformed into rather different values indicative data characteristics corresponding to a task that the neural network structure has been designed to perform.

The output layer 543 is the final layer of the neural network structure. It produces the network's output or prediction based on the computations performed in the preceding layers (e.g., 541, 542). The number of nodes in the output layer depends on the nature of the task being addressed. For example, for sentence embedding, the output layer may consist of a d nodes representing a vector of length d for the sentence embedding.

Therefore, the RAG module 230 may comprise the transformative neural network structure of layers of neurons, and weights and activation functions describing the non-linear transformation at each neuron. Such a neural network structure is often implemented on one or more hardware processors 410, such as a graphics processing unit (GPU).

In one embodiment, the RAG module 230 may be implemented by hardware, software and/or a combination thereof. For example, the RAG module 230 may comprise a specific neural network structure implemented and run on various hardware platforms 560, such as but not limited to CPUs (central processing units), GPUs (graphics processing units), FPGAs (field-programmable gate arrays), Application-Specific Integrated Circuits (ASICs), dedicated AI accelerators like TPUs (tensor processing units), and specialized hardware accelerators designed specifically for the neural network computations described herein, and/or the like. Example specific hardware for neural network structures may include, but not limited to Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA AI-focused GPUs, and/or the like. The hardware 560 used to implement the neural network structure is specifically configured based on factors such as the complexity of the neural network, the scale of the tasks (e.g., training time, input data scale, size of training dataset, etc.), and the desired performance.

In one embodiment, the neural network based RAG module 230 may be trained by iteratively updating the underlying parameters (e.g., weights 551, 552, etc., bias parameters and/or coefficients in the activation functions 561, 562 associated with neurons) of the neural network based on a loss function. For example, during forward propagation, the training data such as document text. The data flows through the network's layers 541, 542, with each layer performing computations based on its weights, biases, and activation functions until the output layer 543 produces the network's output 550. In some embodiments, output layer