18, speaker, etc.). Example operation and uses of these modules are disclosed in International Application No. PCT/US2023/083416 filed on Dec. 11, 2023, entitled “CAMERA MONITOR SYSTEM WITH TRAILER CURB STRIKE ALERT AND TRAILER STRIKING AREA,” which is incorporated herein by reference in its entirety.

The lane detection module 100 also uses image processing of the captured images to identify markings on the roadway, such as lane markers that visually divide adjacent lanes. One example algorithm is described in United States Publication No. US2023/117,719, entitled “CAMERA MIRROR SYSTEM DISPLAY FOR COMMERCIAL VEHICLES INCLUDING SYSTEM FOR IDENTIFYING ROAD MARKINGS”, which is incorporated by reference in its entirely. In that publication, a lane detection module is described in which an object detection algorithm identifies a lane marking in a roadway by filtering a color of the lane marking from a surrounding portion of the captured image. Other techniques based upon deep learning technology or another computer vision method may be used, if desired.

The object detection module 102 includes one or more image processing algorithms configured to identify objects in the captured images. The algorithms may be used to identify VRU's (e.g., pedestrians or cyclists), attributes of the tractor 12 and/or trailer 14, other vehicles, signs, curbs, trees, buildings and/or other inanimate objects.

The trailer end detection module 104 is another image processing module that extracts one or more trailer features from the captured images to determine the location of the end of the trailer in 3D space. These extracted attributes can be used to detect objects such as tractor wheels, trailer edges and other features. Example wheel detection algorithm techniques are disclosed in United States Publication No. US2023/202,394 entitled “CAMERA MONITOR SYSTEM FOR COMMERCIAL VEHICLES INCLUDING WHEEL POSITION ESTIMATION”, which is incorporated herein by reference in its entirety. Example trailer edge detection algorithm techniques are disclosed in United States Publication No. US2023/125,045 entitled “TRAILER END TRACKING IN CAMERA MONITORING SYSTEM”, which is incorporated herein by reference in its entirety. Other techniques may be used, if desired.

In one example operation, the CMS 15 utilizes the kinematics module 106 to predict a striking zone of the trailer 14 during a turn operation and generates a two dimensional overlay to digitally impose over at least one of the displayed Class II/IV images thereby showing the vehicle operator an expected striking zone of the trailer 14 and allowing the vehicle operator adjust the vehicle operations accordingly. The CMS 15 uses the received captured images from the cameras 20A, 20B, as well as any other cameras and vehicle operation data received from a general vehicle controller through a data connection, such as a CAN or LIN bus, to estimate a predicted position of the tractor and/or trailer side at each of multiple side positions and multiple points in time. These positions are converted to a geometric area encompassing all the positions. In this way, the shape and size of the geometric area is not fixed, but rather reflects an actual predicted striking area of the trailer (e.g., see, FIGS. 7 and 8).

In order to avoid accidental strikes, the striking area prediction system uses the vehicle data (e.g. steering angle, steering rate, trailer angle, vehicle speed, trailer wheelbase, tractor wheelbase, hitch point location, yaw rate and the like) to generate a predicted striking zone over time. The predicted striking zone is a prediction of the path the trailer will take over the course of the turn and is re-calculated continuously as the turn progresses. The trailer striking area is also useful in a potential “curve cut” scenario when the vehicle 10 is traveling down a curved roadway. In a curvy road, it becomes more likely for the trailer end to cross the lane markers. indicating boundaries to adjacent lanes, creating a potentially dangerous situation.

The CMS 15 includes a Decision Manager or Collision Alert Module 112 that communicates with the modules 100-110 to evaluate the proximity between the predicted tractor and/or trailer paths (i.e., the tractor and trailer striking areas) and one or more objects (e.g., predicting an imminent curb strike, curve cut, object collision etc.). The decision manager considers the estimated time to the event, severity (what the object is), closing rate between objects, etc. and may provide an overlay and/or alert.

While various overlays and alerts may useful in increasing operator awareness and enhancing safety, it is desirable to provide the operator with information to more easily and proactively manage the vehicle to safely navigate through its surroundings. One such approach is to provide a Picture-in-Picture (PIP) window on at least one of the displays during various operational states in anticipation of what display the operator is likely to be viewing while navigating that operational state. This enables the operator to assimilate more, useful information provided by the CMS while also limiting distractions.

As will be discussed below in greater detail, the CMS 15 includes functionality for providing operator selectable predetermined PIP configurations to display CMS images based on touchscreen commands received through a touchscreen interface. The touchscreen interface may be provided on one of the CMS displays, or through a separate touchscreen interface. For example, a “bring your own device” configuration may be supported, in which a vehicle occupant can use their own personal device (e.g., a tablet) as a CMS display and/or a touchscreen interface (e.g., a tablet).

The touchscreen of at least one of the displays 18A-18D and 18G-18H, if so provided, can be used to enable the (PIP) option to support providing the right information to the operator at the right time. In one example, the driver side display 18A or secondary information display 18D are used as the primary display for managing desired PIP configurations across all displays (e.g., for turning ON or OFF preset PIP window options), as these displays are within easy reach of the operator. It is also possible that a display within reach of a passenger may be used to enable the PIP functionality. As one example, the left-hand display 18A servers as the primary display, and display area 25A provides the touchscreen.

FIG. 6 is a flowchart of an example method 200 for the CMS 15. During operation of the vehicle 10, multiple fields of view (e.g., FOV_EX1, FOV_EX2, FOV_EX3, FOV_EX4) are captured with the cameras (e.g., cameras 20A-20E; block 202) and displayed on the displays (e.g., displays 18A-18H; block 204)). At least one of the displays serves as primary display (e.g., display 18A) with which the operator may interface to select a PIP option. In one example, the operator touches the touchscreen on the primary display, which brings up a picture-in-picture (PIP) indicium 40 (FIG. 7), which the operator can then touch (block 206) to bring up a PIP options menu 42 (FIG. 8). From this PIP options menu, the operator can select a predetermined PIP configuration that will display a streaming image from one of the multiple fields of view on at least one of the multiple displays 18