PRIORITY/RELATED DOCUMENTS

This patent application incorporates by reference in their entireties and claims priority to the co-pending patent application filed on Sep. 11, 2017, entitled “Vanishing Point Computation and Online Alignment System and Method for Image Guided Stereo Camera Optical Axes Alignment,” and both with the same inventor(s).

FIELD OF THE DISCLOSURE

The field of the disclosure is in general related to autonomous vehicles and, in particular, to a system and a method for corner point extraction system and method for image guided stereo camera optical axes alignment.

BACKGROUND OF THE DISCLOSURE

In calibration of a vision system which consists of two or more cameras, an essential step is the evaluation of the geometry of the cameras. In particular, the geometrical properties of an imaging process within a single camera (intrinsic parameters), of the rotation matrix R, and of the translation vector T between pairs of cameras (extrinsic parameters). The accuracy of the measurements of these quantities is essential for a full exploitation of the epipolar constraint and for an accurate reconstruction of 3D data. The intrinsic and extrinsic parameters of a stereo system may be evaluated by using optical calibration without using direct mechanical measurements. Optical calibration allows the reconstruction of camera geometry by simply viewing a suitable calibrating pattern without using direct measurements, which are impractical in many real environments.

All referenced patents, applications and literatures throughout this disclosure, for example, including the following references, are incorporated herein by reference in their entirety:

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

BRIEF SUMMARY OF THE DISCLOSURE

Various objects, features, aspects and advantages of the present embodiment will become more apparent from the following detailed description of embodiments of the embodiment, along with the accompanying drawings in which like numerals represent like components.

Embodiments of the present disclosure provide a method of aligning optical axes of cameras for a non-transitory computer readable storage medium storing one or more programs. The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform by one or more autonomous vehicle driving modules execution of processing of images from a camera and data from a LiDAR using the following steps comprising: calibrating intrinsic parameters of a set of cameras; extracting corner points associated with a pattern; and computing a vanishing point based on information on the extracted corner points.

In an embodiment, the method further comprises: adjusting the set of cameras to align their optical axes in parallel with each other based on the vanishing point.

In another embodiment, the pattern includes a checkerboard pattern.

In yet another embodiment, calibrating a camera comprises: disposing multiple patterns at different orientations in a scene; and obtaining a single image of the multiple patterns.

In still another embodiment, calibrating a camera comprises: disposing a single pattern at different orientations in a scene; and obtaining an image of the single pattern at each of the orientations.

In yet still another embodiment, calibrating a camera comprises: obtaining an image of a planar pattern including corners; recovering from the image the corners of the planar pattern and assigning 3D positions to the recovered corners; and calibrating the set of cameras based on correspondences of corner locations between a 2D pixel coordinate and the 3D positions.

In still yet another embodiment, extracting corner points comprises: disposing the pattern in a scene viewable to the set of cameras; and recovering boundary corner points of the pattern from a pair of images.

In a further embodiment, computing a vanishing point comprises: constructing a pair of parallel lines based on 2D positions of the recovered boundary corner points.

In another further embodiment, the method further comprises: constructing the pair of parallel lines by connecting a pair of recovered boundary corner points on a same side of the pattern.

Embodiments of the present disclosure also provide a system for aligning optical axes of cameras. The system includes an Internet server that further includes an I/O port, configured to transmit and receive electrical signals to and from a client device; a memory; one or more processing units; and one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions by one or more autonomous vehicle driving modules execution of processing of images for: calibrating intrinsic parameters of a set of cameras; extracting corner points associated with a pattern; and computing a vanishing point based on information on the extracted corner points.

BRIEF DESCRIPTION OF THE DRAWINGS

It should be noted that the drawing figures may be in simplified form and might not be to precise scale. In reference to the disclosure herein, for purposes of convenience and clarity only, directional terms such as top, bottom, left, right, up, down, over, above, below, beneath, rear, front, distal, and proximal are used with respect to the accompanying drawings. Such directional terms should not be construed to limit the scope of the embodiment in any manner.

FIG. 1 is a flow diagram showing a method of aligning optical axes of cameras, in accordance with an embodiment;

FIG. 2 is a flow diagram showing a method of calibrating cameras, in accordance with an embodiment;

FIG. 3 is a flow diagram showing a method of aligning optical axes of cameras, in accordance with another embodiment;

FIG. 4 is a block diagram of a processor in a system for aligning optical axes of cameras, in accordance with an embodiment; and

FIG. 5 is a block diagram of a system for aligning optical axes of cameras, in accordance with an embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The embodiment and its various embodiments can now be better understood by turning to the following detailed description of the embodiments, which are presented as illustrated examples of the embodiment defined in the claims. It is expressly understood that the embodiment as defined by the claims may be broader than the illustrated embodiments described below.

Any alterations and modifications in the described embodiments, and any further applications of principles described in this document are contemplated as would normally occur to one of ordinary skill in the art to which the disclosure relates. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. For example, when an element is referred to as being “connected to” or “coupled to” another element, it may be directly connected to or coupled to the other element, or intervening elements may be present.

In the drawings, the shape and thickness may be exaggerated for clarity and convenience. This description will be directed in particular to elements forming part of, or cooperating more directly with, an apparatus in accordance with the present disclosure. It is to be understood that elements not specifically shown or described may take various forms. Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment.

In the drawings, the figures are not necessarily drawn to scale, and in some instances the drawings have been exaggerated and/or simplified in places for illustrative purposes. One of ordinary skill in the art will appreciate the many possible applications and variations of the present disclosure based on the following illustrative embodiments of the present disclosure.

The appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be appreciated that the following figures are not drawn to scale; rather, these figures are merely intended for illustration.

It will be understood that singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, relative terms, such as “bottom” and “top,” may be used herein to describe one element's relationship to other elements as illustrated in the Figures.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Many alterations and modifications may be made by those having ordinary skill in the art without departing from the spirit and scope of the embodiment. Therefore, it must be understood that the illustrated embodiment has been set forth only for the purposes of example and that it should not be taken as limiting the embodiment as defined by the following claims. For example, notwithstanding the fact that the elements of a claim are set forth below in a certain combination, it must be expressly understood that the embodiment includes other combinations of fewer, more, or different elements, which are disclosed herein even when not initially claimed in such combinations.

The words used in this specification to describe the embodiment and its various embodiments are to be understood not only in the sense of their commonly defined meanings, but to include by special definition in this specification structure, material or acts beyond the scope of the commonly defined meanings. Thus if an element can be understood in the context of this specification as including more than one meaning, then its use in a claim must be understood as being generic to all possible meanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claims therefore include not only the combination of elements which are literally set forth, but all equivalent structure, material or acts for performing substantially the same function in substantially the same way to obtain substantially the same result.

In this sense it is therefore contemplated that an equivalent substitution of two or more elements may be made for any one of the elements in the claims below or that a single element may be substituted for two or more elements in a claim. Although elements may be described above as acting in certain combinations and even initially claimed as such, it is to be expressly understood that one or more elements from a claimed combination can in some cases be excised from the combination and that the claimed combination may be directed to a subcombination or variation of a subcombination.

Reference is now made to the drawings wherein like numerals refer to like parts throughout.

As used herein, the term “wireless” refers to wireless communication to a device or between multiple devices. Wireless devices may be anchored to a location and/or hardwired to a power system, depending on the needs of the business, venue, event or museum. In one embodiment, wireless devices may be enabled to connect to Internet, but do not need to transfer data to and from Internet in order to communicate within the wireless information communication and delivery system.

As used herein, the term “Smart Phone” or “smart phone” or “mobile device(s)” or “cellular phone” or “cellular” or “mobile phone” or the like refers to a wireless communication device, that includes, but not is limited to, an integrated circuit (IC), chip set, chip, system-on-a-chip including low noise amplifier, power amplifier, Application Specific Integrated Circuit (ASIC), digital integrated circuits, a transceiver, receiver, or transmitter, dynamic, static or non-transitory memory device(s), one or more computer processor(s) to process received and transmitted signals, for example, to and from the Internet, other wireless devices, and to provide communication within the wireless information communication and delivery system including send, broadcast, and receive information, signal data, location data, a bus line, an antenna to transmit and receive signals, and power supply such as a rechargeable battery or power storage unit. The chip or IC may be constructed (“fabricated”) on a “die” cut from, for example, a Silicon, Sapphire, Indium Phosphide, or Gallium Arsenide wafer. The IC may be, for example, analogue or digital on a chip or hybrid combination thereof. Furthermore, digital integrated circuits may contain anything from one to thousands or millions of signal invertors, and logic gates, e.g., “and”, “or”, “nand” and “nor gates”, flipflops, multiplexors, etc., on a square area that occupies only a few millimeters. The small size of, for instance, IC's allows these circuits to provide high speed operation, low power dissipation, and reduced manufacturing cost compared with more complicated board-level integration.

As used herein, the terms “wireless”, “wireless data transfer,” “wireless tracking and location system,” “positioning system” and “wireless positioning system” refer without limitation to any wireless system that transfers data or communicates or broadcasts a message, which communication may include location coordinates or other information using one or more devices, e.g., wireless communication devices.

As used herein, the terms “module” or “modules” refer without limitation to any software, software program(s), firmware, or actual hardware or combination thereof that has been added on, downloaded, updated, transferred or originally part of a larger computation or transceiver system that assists in or provides computational ability including, but not limited to, logic functionality to assist in or provide communication broadcasts of commands or messages, which communication may include location coordinates or communications between, among, or to one or more devices, e.g., wireless communication devices.

FIG. 1 is a flow diagram showing a method 10 of aligning optical axes of cameras, in accordance with an embodiment.

In some embodiments in accordance with the present disclosure, a non-transitory, i.e., non-volatile, computer readable storage medium such as memory 59 illustrated in FIG. 5, is provided. The non-transitory computer readable storage medium is stored with one or more programs. When the program is executed by the processing unit of a computing device, i.e., that are part of a vehicle, the computing device is caused to conduct specific operations set forth below in accordance with some embodiments of the present disclosure.

Examples of non-transitory storage computer readable storage medium may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In certain embodiments, the term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In some embodiments, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

In some embodiments in accordance with the present disclosure, in operation, a client application is transmitted to the computing device upon a request of a user, for example, by a user device 54 (see FIG. 5). For example, the user device 54 may be a smart phone downloading the application from a computer server. In operation, the application is installed at the vehicle. Accordingly, specific functions may be executed by the user through a computing device, such as calibrating sensors and time synchronization, and, for example, sending and receiving calibration files for data alignment purposes.

In particular, referring to FIG. 1, in operation 12, a set of cameras is calibrated to obtain their intrinsic parameters. Intrinsic parameters are necessary to link pixel coordinates of an image point with corresponding coordinates in a camera reference frame. Intrinsic parameters, depending on camera characteristics, may involve estimation of focal length, skew parameter and image center. In contrast, extrinsic parameters define the location and orientation of a camera reference frame with respect to a known world reference frame. Extrinsic parameters, depending on the position of a camera, may involve estimation of rigid body transformation between the sensors.

Next, in operation 14, corner points associated with a planar pattern are extracted. In an embodiment, multiple checkerboard patterns viewable to the set of cameras are disposed in a scene. In another embodiment, the set of cameras observe a planar pattern shown at a few different orientations. Each of the checkboard patterns may be a black and white checkerboard of a size greater than, for example, 4-by-4 squares in a 2-D true color or grayscale image. Alternatively, the checkerboard may include a rectangular board with four notable corner patterns. The checkboard patterns may be attached at walls in the scene. Moreover, the checkboard patterns are presented either in a single shot or multiple shots with respect to the set of cameras. In an embodiment, the checkboard patterns are presented in a single range or camera image per sensor, which is termed “single shot.” The distance between the inner checkerboard corners may be determined for resolving the scale ambiguity. In another embodiment, multiple images of a single calibration target are presented at different orientations, which is termed “multiple shots.” An image of a single checkerboard disposed in front of, for example, a pair of cameras is taken. The four boundary corner points are recovered in the pair of images.

In operation 16, a vanishing point based on information on the extracted corner points is determined. With recovered 2D positions of the corner points, two parallel lines are constructed by connecting corner points in pairs on the same side of the checkerboard pattern. An intersection of the two parallel lines is determined to be the vanishing point in pixel coordinate. Further, the pixel position is transformed into image coordinate using camera intrinsic parameters and sensor properties.

Subsequently, in operation 18, the set of cameras is adjusted to align their optical axes in parallel with each other based on the vanishing point. In an embodiment, the relative position of each vanishing point is used for adjustment of the set of cameras to be parallel. Difference between the x coordinate and y coordinate in an image plane is used to adjust a pan and a tilt angle of a motor head, respectively. In some embodiments, instead of using a motor head or a motorized pan/tilt head to adjust the camera, a manual adjustment is used.

The method 10 utilizes the relative position of a specific vanishing point for alignment of optical axes of stereo cameras. For cameras with parallel optical axes, the intersection point of parallel lines in a 3D space appears to meet at a common vanishing point in an image plane coordinate.

In some existing approaches, cameras are calibrated and then transformation between the cameras is obtained. The transformation serves as guidance for alignment. Unlike the existing approaches, in the exemplary method 10, the optical axes of a set of stereo cameras are parallel-calibrated by aligning the vanishing point of parallel lines in image pairs. A planar pattern such as a checkerboard pattern is positioned to be viewable for all cameras. Two parallel lines are constructed by detecting the four boundary corner points on the checkerboard. In addition, the location of the intersection point is found in the image. To obtain the corresponding vanishing point, the intersection point in each image is reprojected back to image coordinate in real time using camera intrinsic parameters, which are calculated through automatic camera calibration with multiple checkerboards. The relative position of each vanishing point is used for adjustment of the set of stereo cameras to be parallel.

The method 10 provides a solution which only relies on a checkerboard and cameras. Since no other equipment is required, the method 10 effectively simplifies the alignment process. In addition, the method 10 gives real-time guidance on how to align multiple optical axes. The method 10 is more convenient and efficient than existing approaches that would iteratively adjust two stereo cameras after obtaining a calibration result. In addition, an image area needs to crop during rectification is reduced and thus more information can be retained. Depth information of a target in the image can thereby be recovered more accurately.

FIG. 2 is a flow diagram showing a method 20 of calibrating cameras, in accordance with an embodiment.

Referring to FIG. 2, in operation 22, an image of a planar pattern including corners is obtained.

Next, in operation 24, the corners of the planar pattern are recovered from the image and then 3D positions are assigned to the recovered corners.

Subsequently, in operation 26, a set of cameras is calibrated based on correspondences of corner locations between a 2D pixel coordinate and the 3D positions.

FIG. 3 is a flow diagram showing a method 30 of aligning optical axes of cameras, in accordance with another embodiment.

Referring to FIG. 3, in operation 31, a planar pattern that is viewable to a set of cameras is disposed in a scene.

In operation 32, boundary corner points of the planar pattern are recovered from a pair of images.

Next, in operation 33, a pair of parallel lines is constructed based on 2D positions of the recovered boundary corner points. Then in operation 34, an intersection point of the pair of parallel lines is determined to be a vanishing point in a pixel coordinate.

In operation 35, pixel points in the pixel coordinate are transformed into feature points in an image coordinate.

Subsequently, in operation 36, aligning the feature points together by camera images. The features may include structured features and unstructured features. The structured features may include, for example, planes, straight lines and curved lines, and the unstructured features may include sparse 3D points.

FIG. 4 is a block diagram of a processor 51 in a system for aligning optical axes of cameras, in accordance with an embodiment.

Referring to FIG. 4, the processor 51 includes a calibrating module 41, an extracting module 43, a computing module 45 and an adjusting module 47. Also referring to FIG. 1, the calibrating module 41 is configured to calibrate intrinsic parameters of a set of cameras. The extracting module 43 is configured to extract corner points associated with a planar pattern. The computing module 45 is configured to determine a vanishing point based on information on the extracted corner points. The adjusting module 47 is configured to adjust the set of cameras to align their optical axes in parallel with each other based on the vanishing point.

In addition, also referring to FIG. 2, the calibrating module 41 may be further configured to, in response to an image of a planar pattern including corners, recover from the image the corners of the planar pattern and assign 3D positions to the recovered corners, and calibrate a set of cameras based on correspondences of corner locations between a 2D pixel coordinate and the 3D positions.

FIG. 5 is a block diagram of a system 50 for aligning optical axes of cameras, in accordance with an embodiment.

Referring to FIG. 5, the system 50 includes a processor 51, a computer server 52, a network interface 53, an input and output (I/O) device 55, a storage device 57, a memory 59, and a bus or network 58. The bus 58 couples the network interface 53, the I/O device 55, the storage device 57 and the memory 59 to the processor 51.

Accordingly, the processor 51 is configured to enable the computer server 52, e.g., Internet server, to perform specific operations disclosed herein. It is to be noted that the operations and techniques described herein may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described embodiments, e.g., the processor 51, the computer server 52, or the like, may be implemented within one or more processing units, including one or more microprocessing units, digital signal processing units (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components.

The term “processing unit” or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of the present disclosure.

In some embodiments in accordance with the present disclosure, the computer server 52 is configured to utilize the I/O port 55 communicate with external devices via a network 58, such as a wireless network. In certain embodiments, the I/O port 55 is a network interface component, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive data from the Internet. Examples of network interfaces may include Bluetooth®, 3G and WiFi® radios in mobile computing devices as well as USB. Examples of wireless networks may include WiFi®, Bluetooth®, and 3G. In some embodiments, the internet server 52 is configured to utilize the I/O port 55 to wirelessly communicate with a client device 54, such as a mobile phone, a tablet PC, a portable laptop or any other computing device with internet connectivity. Accordingly, electrical signals are transmitted between the computer server 52 and the client device 54.

In some embodiments in accordance with the present disclosure, the computer server 52 is a virtual server capable of performing any function a regular server has. In certain embodiments, the computer server 52 is another client device of the system 50. In other words, there may not be a centralized host for the system 50, and the client devices 54 in the system are configured to communicate with each other directly. In certain embodiments, such client devices 54 communicate with each other on a peer-to-peer (P2P) basis.

The processor 51 is configured to execute program instructions that include a tool module configured to perform a method as described and illustrated with reference to FIGS. 1 to 3. Accordingly, in an embodiment in accordance with the method 10 illustrated in FIG. 1, the tool module is configured to execute the operations including: calibrating a set of cameras, extracting corner points associated with a planar pattern, computing a vanishing point based on information on the extracted corner points, and adjusting the set of cameras to align their optical axes in parallel with each other based on the vanishing point.

In an embodiment in accordance with the method 20 illustrated in FIG. 2, the tool module is configured to execute the operations including: obtaining an image of a planar pattern including corners, recovering from the image the corners of the planar pattern and assigning 3D positions to the recovered corners, and calibrating a set of cameras based on correspondences of corner locations between a 2D pixel coordinate and the 3D positions.

In an embodiment in accordance with the method 30 illustrated in FIG. 3, the tool module is configured to execute the operations including: disposing a planar pattern that is viewable to a set of cameras, recovering boundary corner points of the planar pattern from a pair of images, constructing a pair of parallel lines based on 2D positions of the recovered boundary corner points, determining an intersection point of the pair of parallel lines to be a vanishing point in a pixel coordinate, transforming pixel points in the pixel coordinate into feature points in an image coordinate, and aligning the feature points together by camera images.

The network interface 53 is configured to access program instructions and data accessed by the program instructions stored remotely through a network (not shown).

The I/O device 55 includes an input device and an output device configured for enabling user interaction with the system 50. In some embodiments, the input device comprises, for example, a keyboard, a mouse, and other devices. Moreover, the output device comprises, for example, a display, a printer, and other devices.

The storage device 57 is configured for storing program instructions and data accessed by the program instructions. In some embodiments, the storage device 57 comprises, for example, a magnetic disk and an optical disk.

The memory 59 is configured to store program instructions to be executed by the processor 51 and data accessed by the program instructions. In some embodiments, the memory 59 comprises a random access memory (RAM) and/or some other volatile storage device and/or read only memory (ROM) and/or some other non-volatile storage device including other programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a solid state drive (SSD), a compact disc ROM (CD-ROM), a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. In certain embodiments, the memory 59 is incorporated into the processor 51.

Thus, specific embodiments and applications have been disclosed. It should be apparent, however, to those skilled in the art that many more modifications besides those already described are possible without departing from the disclosed concepts herein. The embodiment, therefore, is not to be restricted except in the spirit of the appended claims. Moreover, in interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms “comprises” and “comprising” should be interpreted as referring to elements, components, or steps in a non-exclusive manner, indicating that the referenced elements, components, or steps may be present, or utilized, or combined with other elements, components, or steps that are not expressly referenced. Insubstantial changes from the claimed subject matter as viewed by a person with ordinary skill in the art, now known or later devised, are expressly contemplated as being equivalent within the scope of the claims. Therefore, obvious substitutions now or later known to one with ordinary skill in the art are defined to be within the scope of the defined elements. The claims are thus to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what can be obviously substituted and also what essentially incorporates the essential idea of the embodiment.