202 and camera 204 are communicatively connected (i.e., communicatively coupled) together such that the camera 204 can send data (e.g., images) to the processing system 202, and the processing system 202 can send data (e.g., commands) to the camera 204. According to one or more embodiments described herein, the processing system 202 includes a processor 222 that provides for the operation of the system 200. In an embodiment, the processor 222 includes one or more processors that are responsive to executable computer instructions when executed on the one or more processors. It should be appreciated that one or more of the processors may be located remotely from the processing system 202. In an embodiment, the processor 222 uses distributed computing with some of the processing being performed by one or more nodes in a cloud-based computing environment and/or another suitable system/device. The processor 222 may accept instructions through a user interface (i.e., an input device), such as but not limited to a keyboard, a mouse, or a touch screen for example.

The processor 222 is capable of converting signals representative of system data received from the camera 204 and/or one or more sensors 230 of the processing system 202. The system data may include distance measurements and encoder signals that may be combined to determine three-dimensional coordinates on surfaces in the environment. Other system data may include images or pixel voltages from the camera 204. In general the processor 222 receives system data and is given certain instructions, which can cause one or more of generating a 3D coordinate, registering a plurality of coordinate systems, applying color to points in the point cloud, identifying retroreflective or reflective targets, identifying gestures, simultaneously localizing and generating a map of the environment, determining the trajectory of a measurement device, generating a digital twin representation of an object or environment, using a digital twin representation of an environment to perform image localization, and/or the like, including combinations and/or multiples thereof.

The processor 222 also provides operating signals to the camera 204. For example, the signals may initiate control methods that adapt the operation of the processing system 202 and/or the camera 204, such as causing the camera 204 to capture one or more images.

The processor 222 is coupled to one or more system components by data transmission media (e.g., twisted pair wiring, coaxial cable, fiber optical cable, wireless protocols, and/or the like). Data transmission media includes, but is not limited to, wireless, radio, and infrared signal transmission systems. In the embodiment of FIG. 2, data transmission media couples to the processor 222 to the camera 204, a communications circuit 224, a storage device 226 (e.g., nonvolatile memory), a memory 228 (e.g., random access memory or read-only memory), and one or more sensors 230.

The communications circuit 224 is operable to transmit and receive signals between the camera 204 and the processing system 202 and/or from external sources, including but not limited to nodes in a distributed or cloud-based computing environment. The communications circuit 224 may be configured to transmit and receive signals wirelessly (e.g. WiFi or Bluetooth), via a wired connection (e.g. Ethernet, Universal Serial Bus), or a combination thereof.

The storage device 226 is any form of non-volatile memory such as an EPROM (erasable programmable read only memory) chip, a disk drive, and/or the like, including combinations and/or multiples thereof. Stored in storage device 226 are various operational parameters for the application code. According to one or more embodiments described herein, the storage device 226 can store position data associated with each image captured by the camera 204. According to one or more embodiments described herein the storage device 226 can store images captured by a camera (not shown) of the processing system 202, position data associated with the images captured by the camera of the processing system 202, and/or position data of annotations made by a user to the images captured by the camera of the processing system 202.

In an embodiment, the sensors 230 may include a LIDAR sensor, an inertial measurement unit, an integral camera or cameras, and/or the like including combinations and/or multiples thereof. For example, the processing system 202 can also include a LIDAR sensor. As discussed in more detail herein, the LIDAR sensor (e.g., the sensor 230) can be configured to emit light from a light source, which is reflected off a surface in the environment, and the reflected light is received by a light receiver, such as a photosensitive array.

The processor 222 includes operation control methods embodied in application code, such as the methods described herein. These methods are embodied in computer instructions written to be executed by the one or more processors, typically in the form of software. The software can be encoded in any programming language. The processor 222 may further be electrically coupled to a power supply 232. The power supply 232 receives electrical power from a power source (e.g., a battery) and adapts the characteristics of the electrical power for use by the system 200.

In an embodiment, the system 200 may include a mobile platform 234. The mobile platform 234 may be any movable assembly capable of supporting the processing system 202 and/or the camera 204 during operation. As such, the mobile platform 234 can have wheels or articulated legs. In one or more embodiments, the mobile platform 234 may be, but is not limited to, a cart or a trolley for example. In other embodiments, the mobile platform 234 may be an airborne device, such as an unmanned aerial vehicle (UAV) or a drone for example. The mobile platform 234 may include a handle positioned for an operator to push or pull the mobile platform 234 through the environment where coordinates are to be acquired. In some embodiments, the mobile platform 234 may be autonomously or semi-autonomously operated. In this embodiment, the mobile platform 234 may include a power source/battery 236, a power supply 238, and a motor controller 240, although other configurations are also possible. In some examples, the mobile platform 234 is a tripod that can be positioned at and moved between different locations throughout an environment.

In an embodiment, the processor 222 is configured to execute one or more engines 242. In an embodiment, the engines 242 may be in the form of executable computer instructions that perform certain operational methods when executed on one or more processors. The engines 242 may be stored on the storage device 226 or the memory 228 for example. The engines 242, when executed on the processor 222, may receive inputs, such as from the one or more sensors 230 of the processing system 202 and/or from the camera 204, and transform data, generate data, and/or cause the processing system 202 and/or the camera 204 to perform an action. In an embodiment, the engines 242 include one or more of, but not limited to, a determine 3D coordinates engine 244, a photogrammetry engine 246, a register point cloud engine 248, a colorize point cloud engine 250, a digital twin engine 252, an identify gestures engine 254, a tracking engine 256, and a trajectory determination engine 258. It should be appreciated that, in examples, other engines can be utilized. For example, one or more of the engines 242 can be eliminated and/or one or more other engines can be added.

In an embodiment, the colorize point cloud engine 250 aligns the images acquired by the camera 204 with either the point cloud (from the register point cloud engine 248) or with the 3D points from individual scans. In either case, once aligned, the color values from the images may be mapped to the points and the color value assigned to the point. In this way, when the point cloud is displayed in color, the image will appear realistic.

In an embodiment, the photogrammetry engine 246 and the determine 3D coordinates engine 244 may cooperate to determine 3D coordinates of points on surfaces in the environment using the image(s) captured by the camera 204. In an embodiment, the register point cloud engine 248 may receive 3D coordinates from the engine 244 and register them into the same coordinate frame of reference based at least in part on image(s) acquired by the camera 204.

In an embodiment, the identify gestures engine 254 may receive an image from the omnidirectional camera 204. In respond to receiving the image, the engine 254 may perform image analysis to identify an operator within the image. Based at least in part on identifying the operator, the engine 254 may determine the operator is performing a gesture, such as by positioning their hands or their arms in a predetermined position (e.g. using a skeletal model). This predetermined position is compared with a table of operator positions and an associated control method is performed (e.g., measure 3D coordinates). In an embodiment, the identify gestures engine 254 operates in the manner described in commonly owned U.S. Pat. No. 8,537,371 entitled “Method and Apparatus for Using Gestures to Control a Laser Tracker”, the contents of which are incorporated by reference herein.

In an embodiment, the processing system 202 and the omnidirectional camera 204 are moved through the environment, such as on the mobile platform 234 or by an operator in hand. In an embodiment, a plurality of images are acquired by the camera 204 while the mobile platform 234 is moved through the environment. These plurality of images may be used to generate a two-dimensional (2D) map of the environment using a method such as simultaneous localization and mapping (SLAM) for example. According to one or more embodiments described herein, the tracking engine 256 using SLAM techniques to track the processing system 202. In other examples, the tracking engine 256 tracks the processing system 202 based on data from the sensor 230 and/or based on images captured by a camera (not shown) integrated into the processing system 202.

The tracking engine 256 may cooperate with trajectory determination engine 258 to determine the trajectory (e.g., the 3D path) that the processing system 202 follows through the environment. In an embodiment, the determined trajectory is used by the register point cloud engine 248 to register the 3D coordinates in a common frame of reference.

According to one or more embodiments described herein, the processing system 202 can trigger the camera 204 to capture images. The triggering can be, for example, a manual triggering event (e.g., a user pressing a button on a touch screen of the processing system 202) and/or an automatic triggering event (e.g., every “X” seconds, every “X” distance, based on a predefined grid or predefined location, and/or the like, including combinations and/or multiples thereof).

According to one or more embodiments described herein, the processing system 202 can cause to be displayed, on a display (not shown) a trajectory (e.g., the trajectory from the trajectory determination engine 258, recorded 3D coordinate data (e.g., point cloud data), a confidence/completeness of the 3D coordinate data along with the 3D coordinate data, a mesh generation of the 3D coordinate data, an image trigger to cause multiple images to be captured, and/or the like including combinations and/or multiples thereof.

It should be appreciated that the camera 204 provides advantages to the engines 242 in allowing the control methods to be executed faster (e.g., less images are used) or perform methods that are not possible with traditional cameras with a narrower field of view.

The digital twin engine 252 uses images captured by the camera 204 and data captured by the processing system 202 (e.g., from a LIDAR sensor) to generate a digital twin representation of the environment through with the camera 204 and the processing system 202 are moved. The digital twin engine 252 uses the digital twin representation of the environment to perform image localization as described herein.

According to one or more embodiments described herein, the system 200 includes, or is in communication with, a cloud computing system 260. Cloud computing can supplement, support or replace some or all of the functionality of the elements of the system 200. For example, some or all of the functionality of the elements of system 200 can be implemented as a node of the cloud computing system 260. The cloud computing system 260, in examples, can perform additional functions not described with reference to the system 200.

Further features and functionality of the system 100 and/or the system 200 are now described with reference to FIGS. 3, 4A, 4B, 5A, and 5B.

Particularly, FIG. 3 depicts a flow diagram of a method 300 for image localization using a digital twin representation of an environment according to one or more embodiments described herein. The method 300 can be performed by any suitable system and/or device, including combinations thereof. For example, the method 300 can be performed by the system 100 (including the processing system 102 and the camera 104), by the system 200 (including the processing system 202 and the camera 204), and/or the like. According to one or more embodiments described herein, the method 300 is performed, in whole or in part, by a processing system (e.g., the processing system 102, the processing system 202) that is located remotely from the camera (e.g., the camera 104, the camera 204). For example, the processing system can be one or more nodes of the cloud computing system 260. According to one or more embodiments described herein, the method 300 is performed by a combination of a processing system (e.g., the processing system 102, the processing system 202), a camera (e.g., the camera 104, the camera 204), and a cloud computing system (e.g., the cloud computing system 260).

In the example of FIG. 3, the method 300 is generally segmented into three phases: a preparation phase 302, a capture phase 304, and an optimization phase 306. According to one or more embodiments described herein, the preparation phase 302 and the optimization phase 306 are performed by the cloud computing system 260 and the capture phase 304 is performed by one or more of the systems 100, 200 (e.g., (e.g., the processing system 102 and the camera 104, the processing system 202 and the camera 204). According to one or more embodiments described herein, each of the preparation phase 302, the capture phase 304, and the optimization phase 306 are performed by one or more of the systems 100, 200 (e.g., the processing system 102 and the camera 104, the processing system 202 and the camera 204). The method is now described in more detail with reference to the elements of FIG. 2 but is not so limited.

The preparation phase 302 begins at block 310, where a project is created. For example, a project is created using the cloud computing system 260. The project contains information about the environment to be captured. As an example, the information about the environment to be captured can include a model, such as a computer aided design (CAD) model, a building information modeling (BIM) model, and/or the like, including combinations and/or multiples thereof. FIG. 4A depicts an example of a layout plan 400 according to one or more embodiments described herein.

With continued reference to FIG. 3, at block 312, environment information is extracted from the project and/or from the model. For example, room dimensions, window size and position, door size and position, and/or the like, including combinations and/or multiples thereof is extracted from the CAD model or BIM model.

At block 316, the extracted information is transmitted, such as from the cloud computing system 260 to the processing system 202. The extracted information can be transmitted via any suitable wired and/or wireless link(s) using any suitable protocol(s). In examples, the extracted information is transmitted via a network (not shown), which represents any one or a combination of different types of suitable communications networks such as, for example, cable networks, public networks (e.g., the Internet), private networks, wireless networks, cellular networks, or any other suitable private and/or public networks. Further, the network can have any suitable communication range associated therewith and may include, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), or personal area networks (PANs). In addition, the network can include any type of medium over which network traffic may be carried including, but not limited to, coaxial cable, twisted-pair wire, optical fiber, a hybrid fiber coaxial (HFC) medium, microwave terrestrial transceivers, radio frequency communication mediums, satellite communication mediums, or any combination thereof.

The capture phase 304 begins at block 316, where a first image is captured. For example, the camera 204 captures a first image of an environment corresponding to the project created at block 310. The camera 204 can capture the first image based, for example, on an input received from a user, automatically based on a determined location of the camera 204, automatically based on a command from the processing system 202, and/or the like, including combinations and/or multiples thereof. As an example, the processing system 202 executes a software application (referred to as an “application”) that displays, on a display, a user interface that, among other things, provides for a user to cause the camera 204 to capture the first image. For example, FIG. 4B depicts an image 410, which is an example of a first image according to one or more embodiments described herein.

With continued reference to FIG. 3, at block 318, the first image is positioned in the layout plan of the environment. For example, a user can select a location on the layout plan 400, and the first image 410 is then associated with (i.e., “positioned at”) the selected location on the layout plan 400. For example, with reference to FIG. 4A, the user can select a location 401 on the layout plan 400, and the first image 410 is associated with the location 401. The location 401 represents the approximate location where the first image 410 was captured. According to one or more embodiments described herein, the first image 410 is located automatically on the layout plan 400, such as based on location metadata associated with the first image or other suitable information.

With continued reference to FIG. 3, at block 320, the processing system 202 detects features of the environment in the first image and checks position and/or orientation of the first image with respect to the layout plan. Examples of features of the environment can include, but are not limited to, windows, window openings, walls, doors, door openings, and/or the like, including combinations and/or multiples thereof. In the case of a window or other similar opening, the processing system 202 can detect the presence of the feature using brightness. For example, where the feature is a window, the processing system 202 can detect the feature (i.e., window) based on a first brightness value of a first pixel of the first image associated with the window. The processing system 202 can then compare the first brightness value of the first pixel of the first image to a second brightness value of a second pixel of the first image associated with other than the window. This enables the processing system 202 to detect the window. Different types of object recognition can be used to detect the window, for example, or any other suitable type of object. For example, object detection can be performed using a trained artificial intelligence model. Labeled training data showing objects to be detected (e.g., doors, windows, etc.) and corresponding labels can be used to train a neural network to detect and classify the objects, for example. This is referred to as supervised learning. As another example, an object such as a window can be detected based on a property within an image, such as a brightness level of a group of pixels in the case of a window and a shape of the group of pixels. In the case of a window, the pixels may have a substantially rectangular shape and a relatively brighter brightness level than neighboring or nearby pixels. The processing system 202 can compare the location of detected features with the environment information that extracted from the project and/or from the model at block 312.

Where features are detected in the first image and the layout plan, the features can be used to refine/correct the position of the first image in the layout plan at block 322. For example, the processing system 202 performs a second positioning at block 322 to correct the position and/or orientation of the first image based at least in part on the detected feature (from block 320) as compared to the extracted feature (from block 312) to refine the position of the first image in the digital layout plan of the environment.

At block 324, a second image of the environment is captured. For example, the camera 204 captures the second image of the environment corresponding to the project created at block 310.

At block 326