206 may include an antenna, a modem, LAN port, WiFi card, WiMAX card, mobile communications hardware, near-field communication hardware, satellite communication hardware, and/or any wired or wireless hardware for communicating with other networks and/or devices. In one embodiment, the network interface hardware 206 includes hardware configured to operate in accordance with the Bluetooth® wireless communication protocol. For example, the network interface hardware 206 of the trajectory and collision prediction system 100 may transmit predicted trajectory 103, predicted collision 105, ML model confidence, collision confidence with a server or the vehicles 101.

Referring back to FIG. 1, the controller 201 may generate predicted trajectory 103 for the vehicles 101, such as autonomous vehicles, and further predicted collision 105 to entities (such as the neighboring vehicle 127 and the pedestrian 117) and obstacles (such as the curb 125) on the road 137. The vehicles 101 may generate and transmit sensory data, such as the vehicle kinematic data, the vehicle control data, and the road condition data to the controller 201. In embodiments, the vehicle kinematic data may be a vehicle position, a vehicle velocity, and a vehicle acceleration of the vehicle 101. The vehicle control data may be vehicle steering, vehicle throttle, and brake inputs of the vehicle 101. The road condition data may include road surface conditions, road geometry, and traffic conditions of the road 137.

Upon receiving the sensory data and other input data, such as local weather conditions, the controller 201 may generate, using the ML module 222, the ML predicted trajectory 103 of the vehicle 101 and the ML model confidence of the predicted trajectory 103. The ML model confidence may a number between 0 and 1, with 0 indicating the lowest confidence value and 1 indicating the highest confidence value. In embodiments, the ML module 222 may generate the model confidence based on a probabilistic model or a distance-based model. In some embodiments, the ML module 222 may include one or more probability-based models. the ML module 222 may include a Softmax activation function in the output layer to determine the probability of each output of the predicted trajectory and/or collision. The ML module 222 may include an uncertainty estimation function, such as Bayesian Neural Networks, to provide uncertainty estimates of the predicted trajectory and collision. In some embodiments, the ML module 222 may include one or more distance-based models. For example, the ML module 222 may include a distance-to-decision-boundary function. The confidence may be inferred by measuring the distance of any output as point features to the decision boundary of the sample data and the historical data. Output points far from the boundary are denoted with higher confidence, while output points near the boundary may have lower confidence. Similarly, the ML module 222 may include a Margin-based Confidence function. The margin between the decision boundary of the sample data and the historical data, and the output data points are used as a measure of confidence. Larger margins indicate higher confidence.

In embodiments, the ML module 222 may include one or more neural networks. The neural networks may include an encoder, one or more layers of hidden layers, and a decoder. The neural networks may feed training data 237 during the pre-training process into the encoder to generate a lower-dimensional representation of the target input-output pairs. For example, the lower-dimensional representation may include input of sensory data collected using the vision sensor 208 and vehicle sensors 212 pairing with predicted trajectories and collisions as well as detected trajectories and collisions of the vehicles 101 relative to the entities and objects on the road, such as obstacles (e.g. curbs 125), pedestrians 117, and the neighboring vehicles 127. The neural networks may output ML predicted trajectory 103 and predicted collisions 105.

In embodiments, the controller 201 further generates collision confidence associated with the predicted trajectory 103 using the ML module 222, the heuristic module 232, or both. In some embodiments, the controller 201 may determine whether the collision confidence is beyond a collision threshold. In some embodiments, the controller 201 may generate the collision confidence as a ML collision confidence based on the ML models in response to determining that the ML model confidence satisfies the metric. The metrics may include a model confidence threshold, a probability or certainty threshold, a prediction interval or confidence interval, a cross-validation performance, consistency over time, feature inputs, or a combination thereof. For example, the ML module 222 may include a ML model confidence threshold, a probability or certainty threshold, such as 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, 100%, or any number between 0 and 1. The ML predicted trajectory 103 is used when the ML model confidence is more than the model confidence threshold or the probability or certainty threshold. The ML module 222 may further use prediction intervals or confidence intervals for a quantifiable range within which the actual trajectory is likely to fall. The ML module 222 may use the cross-validation performance metrics, such as accuracy or precision, to assess overall reliability, and monitor consistency over time to identify fluctuations in ML model confidence. The integration of ML model confidence metrics into the decision-making process for trajectory prediction introduces a dynamic mechanism where the system adapts based on the model's perceived reliability. In some embodiments, in response to determining that the ML model confidence satisfies the metric, the controller 201 may apply the ML predicted trajectory 103 as the predicted trajectory 103 for further determinations, such as collision determination. In some embodiments, in response to determining that the ML learning model confidence fails the metric, the controller 201 may apply the heuristic predicted trajectory 103 as the predicted trajectory 103 for further procedures.

In embodiments, the controller 201 may generate the collision confidence as a heuristic collision confidence based on the heuristic model in response to determining that the ML model confidence fails the metric. In some embodiments, the controller 201 may generate the collision confidence as a blending collision confidence generated by blending a ML collision confidence and a heuristic collision confidence using weights based on the ML model confidence. For example, the blending collision confidence may be calculated with the following equation:

Blending collision confidence=ML model confidence×ML collision confidence+(1−ML model confidence)×heuristic collision confidence

In embodiments, after the controller 201 determines that the collision confidence is beyond a collision threshold, the controller 201 may inform the vehicle 101 of a predicted collision. In some embodiments, after the controller 201 determines that the collision confidence is beyond a collision threshold, the controller 201 may generate a trajectory planning to avoid the predicted collision. The controller 201 may instruct and/or operate the vehicle 101 to follow the trajectory planning.

For example, the ML module 222 may generate a predicted trajectory 103 with a ML model confidence of 0.6. the ML module 222 may further determine the collision confidence, based on the predicted trajectory 103, the current motion data of the vehicle 101, and the geometry of the road 137. The collision confidence may represent the possibility for the vehicle 101 to collide with an obstacle (e.g. curb 125) or an entity (a pedestrian 117 or a neighboring vehicle 127) on the road 137. For example, as illustrated in FIG. 1, the ML module 222 may determine that the vehicle 101 in lane 123 may collide with the curb 125 with a collision confidence 0.4. Similarly, the heuristic module 232 may determine that the vehicle 101 in lane 123 may collide with the curb 125 with the collision confidence 0.7. The controller 201 may determine that the collision confidence 0.4 generated by the ML module 222 is not beyond the collision threshold of 0.45, after determining that a ML model confidence of 0.6 satisfy the metric. The controller 201 may not inform the vehicle 101 of the predicted collision 105 at the curb 125. In another example, the controller 201 may use the blending approach to generate the blending collision confidence of 5.2 (i.e. 0.6×0.4+ (1−0.6)×0.7) using the weights based on the ML model confidence 0.6 and determine the blending collision confidence is beyond the collision threshold of 0.45. The controller 201 may then inform the vehicle 101 at lane 123 of a predicted collision 105 at the curb 125. The controller 201 may further generate a trajectory planning (e.g. brake the vehicle 101 to full stop) to avoid the predicted collision, and instruct and/or operate the vehicle 101 to follow the trajectory planning (e.g., decelerate the vehicle 101 to a full stop).

In another example, as illustrated in FIG. 1, the ML module 222 and heuristic module 232 may generate a predicted trajectory 103 of the vehicle 101 in lane 122 in moving in the lane 122 to the north. The vehicle 101 or the controller 201 may detect a pedestrian 117 at the right end of the crosswalk 151 and a neighboring vehicle 127 in the lane 132 is moving toward the intersection 135. Based on the ML models, the controller 201 may determine that the pedestrian 117 may walk onto the crosswalk 151 to move to the other side of the road 137 and the collision confidence of the predicted collision 105 with the pedestrian 117 may be beyond a collision threshold. Based on the heuristic models, the controller 201 may determine that the pedestrian 117 may be about to stay still based on the current motion of the pedestrian 117 with a collision confidence less than the collision threshold. In one case, the controller 201 may inform the vehicle 101 in the lane 122 that the predicted collision 105 with the pedestrian 117 based on the ML model confidence of the predicted trajectory 103 satisfying the metric. In another case, the controller 201 may balance the predicted collision based on ML models and the heuristic models by blending a ML collision confidence and a heuristic collision confidence using weights based on the ML model confidence. Similarly, the controller 201 may determine, based on the ML models, that the neighboring vehicle 127 may stop at the intersection 135 and may not collide the vehicle 101 in the lane 122 at the intersection 135 such that the collision confidence is below the collision threshold. The controller 201 may determine, based on the heuristic models, that the neighboring vehicle 127 may continue moving in the lane 132 based on the current motion data of the neighboring vehicle 127, and predict a collision 105 with the vehicle 101 in the lane 122 at the intersection that has a collision confidence greater than the collision threshold. The controller 201, after determining that the ML model confidence satisfies a metric, may determine a low possibility of the predicted collision 105 with the neighboring vehicle 127 at the intersection. Alternatively, the controller 201 may balance the predicted collision based on ML models and the heuristic models by blending a ML collision confidence and a heuristic collision confidence using weights based on the ML model confidence.

Referring back to FIG. 2, in some embodiments, the ML module may include one or more neural network having been trained with the training data 237 and the historical data 217. The neural networks may include an encoder or/and a decoder conjunct with a layer normalization operation or/and an activation function operation. The encoded input data may be normalized and weighted through the activation function before being fed to the hidden layers. The hidden layers may generate a representation of the input data at a bottleneck layer. After delivering neural-network processed data to the final layer of the neural network, a global layer normalization may be conducted to normalize the ML model confidence, collision confidence, predicted trajectories, and predicted collision. The outputs may be normalized and converted using an activation function for training and verification purposes, as described in detail further below. The activation function may be linear or nonlinear. The activation function may be, without limitations, a Sigmoid function, a Softmax function, a hyperbolic tangent function (Tan h), or a rectified linear unit (ReLU). The neural networks may feed the encoder with historical data 217, such as historical vehicle kinematic data, historical vehicle control data, historical road condition data, historical predicted trajectory, historical detected trajectory, historical predicted collision, historical detected collision, and other historical vehicle operation data for continuation training.

In embodiments, the one or more vehicle modules may be pre-trained using training data 237, including ground-truth examples and scenarios where multiple entities (e.g. the vehicles 101 and the neighboring vehicles 127) driving on a shared surface, such as road 137 including multiple lanes, intersection, curbs, and crosswalks. The pre-training may include labeling the entities and desirable trajectory and collisions based on the entities and the share surface in the examples and scenarios and using one or more neural networks to learn to predict the desirable and undesirable trajectory and collisions based on the training data 237. The pre-training may further include fine tuning, evaluation, and testing steps. The modules may be continuously trained using the real-world collected data as the historical data 217 to adapt to changing conditions and factors and improve the performance over time. The neural network may be trained based on the activation functions mentioned further above. The encoder may generate encoded input data h=(Wx+b) that is transformed from the input data of one or more input channels. The encoded input data of one of the input channels may be represented as h_ij=g(Wx_ij+b) from the raw input data x_ij, which is then used to reconstruct output {tilde over (x)}_ij=f(W^Th_ij+b′). The neural networks may reconstruct outputs, such as ML model confidence, collision confidence, predicted trajectories 103, and predicted collision 105, into x′=(WTh+b′), where W is weight, b is bias, WT and b′ are transverse values of W and b and are learned through back propagation. In this operation, the neural networks may calculate, for each input data, a distance between an input data x and a reconstructed input data x′, to yield a distance vector |x−x′|. The neural networks may minimize the loss function which is a utility function as the sum of all distance vectors. The training process may enable the neural network to learn linear or non-linear representations of the input data. The accuracy of the predicted output may be evaluated by satisfying a preset value, such as a preset accuracy and area under the curve (AUC) value computed using an output score from the activation function (e.g. the Softmax function or the Sigmoid function). For example, the trajectory and collision prediction system