The imaging system 300 is configured for capturing real-time images and/or video for the entire operating width of the agricultural machinery in a more efficient and reliable way compared to standard RGB cameras by using multispectral imaging. Multispectral imaging involves capturing images of a scene or object over multiple discrete wavelength bands and extracting spectral content from that data. Multispectral imaging captures image data within wavelength ranges across the electromagnetic spectrum. The wavelengths may be separated by filters or detected with the use of components that are sensitive to particular wavelengths, including light from frequencies beyond the visible light range, i.e., IR and ultra-violet light.

The stereoscopic multispectral imaging sensor 302 enables detailed measurements of the morphology of the field to be acquired and/or calculated, as well as the position and orientation with respect to the part of the field scanned. The stereoscopic multispectral imaging sensor 302 is configured to provide distance and/or depth information for objects in the captured images. The stereoscopic multispectral imaging sensor 302 includes a wide-angle lens. The wide-angle lens (for example, an angle of view of about 90° to about 150°) is configured to encompass the entire operating width of the agricultural machinery (typically a width of about 20 to about 46 meters).

The imaging system 300 can use measurements acquired from the IMU 306 to improve the accuracy of measurements and calculations. The IMU 306 is configured to generate a signal indicating an acceleration, an angular rate, and/or orientation of the stereoscopic multispectral imaging sensor 302. In aspects, the stereoscopic multispectral imaging sensor 302 may include a gyroscope, a magnetometer, and/or an accelerometer. The IMU measurements may be used to improve the accuracy of the imaging system 300 measurements and calculations.

The GPS receiver 304 is configured to generate real-time location information for the captured images to increase the accuracy of the location of the field elements. The outcome of the above measurements and calculations provides an accurate determination of the location of the field elements 1004, either relative to the vehicle 1002 or positioned on an absolute scale, using the GPS receiver 304.

Referring to FIG. 4, a top view of the imaging system 300 mounted to the agricultural vehicle 1002 is shown. By using a wide lens (e.g., about 120 degrees), the imaging system 300 has a field of view that encompasses the entire working width of the agricultural vehicle 1002.

Referring to FIG. 5, a side view of a field 1008 with a change in ground incline is shown. The IMU 306 of the imaging system 300 enables accurate detection of the field elements 1004 even when there is a change in ground incline by providing the angle and direction of the imaging system 300 relative to the field elements 1004.

Referring to FIG. 6, there is shown a flow chart of an exemplary processor-implemented approach 600 for detecting, identifying, localizing, and/or determining characteristics of field elements 1004, and detecting, identifying, localizing, and/or determining field morphology in agricultural fields 1008 in real-time using a stereoscopic multispectral imaging sensor, mounted on an agricultural vehicle, as the agricultural vehicle moves through a field in accordance with aspects of the present disclosure. Although the blocks of FIG. 6 are shown in a particular order, the blocks need not all be performed in the specified order, and certain blocks can be performed in another order. For simplicity, FIG. 6 will be described below, with the controller 200 performing the operations. However, in various aspects, the operations of FIG. 6 may be performed in part by the controller 200 of FIG. 2 and in part by another device, such as a remote server. These variations are contemplated to be within the scope of the present disclosure.

Initially, at block 602, the controller 200 causes the stereoscopic multispectral imaging sensor 302 of the imaging system 300 to capture real-time images of the field as the agricultural vehicle 1002 moves through the field 1006. The imaging system 300 is mounted on the agricultural vehicle 1002. The captured images may be 2D and/or 3D. The captured images may include depth information.

In aspects, each pixel of the captured images may be assigned a pixel value representing a light intensity measurement in a light spectrum. The stereoscopic multispectral imaging sensor 302 is configured to acquire real-time images containing an entire operational working width of the agricultural vehicle using wide-angle lenses. In aspects, the controller 200 may perform real-time image registration and/or alignment between the multispectral images, for example, aligning an IR image and a red light image.

In aspects, the controller 200, using information from the stereoscopic multispectral imaging sensor 302, can calculate distances of all elements in the acquired images in order to create a depth map. The controller 200 may determine, for example, the position and orientation of the imaging system 300 with respect to the field 1006, the morphology of the field itself, and/or the detection of terrain changes (FIG. 5) based on the depth map. These measurements and calculations are used to increase the accuracy of the estimated field element 1004 location. More specifically, the estimated location based on the images is affected by the relative location and orientation between the camera and the sensed field element and the estimation of the distance between the imaging system 300 and the field element 1004. Both factors are accurately calculated using a combination of image processing from the imaging system 300 and the depth map.

At block 604, the controller 200 identifies, in real-time, field elements based on analyzing the captured real-time images. The field elements may be determined based on light-intensity images in different light spectrums.

At block 606, the controller 200 determines, in real-time, characteristics, including a vegetation index, of the identified field elements, by analyzing the captured real-time image as a standalone image or in combination with each other. A vegetation index is a single value calculated by transforming the observations from multiple spectral band images. The vegetation index can be generated, for example, by using two or more spectral image bands. The vegetation index is used to enhance the presence of green vegetation features and thus help to distinguish them from the other objects present in the image. The vegetation index may be used as an input to a machine-learning model for field element 1004 identification.

At block 608, the controller 200 determines, in real-time, a field morphology as the agricultural vehicle moves through the field based on the captured real-time images.

At block 610, the controller 200 accesses sensor data from the IMU 306 (