14 in accordance with this disclosure is a trained convolutional neural network (CNN), thereby applying computer vision-oriented machine learning technology to detect a misapplication of tape 14 as applied to substrate 16. Aspects of CNNs described in International Patent Applications with Publication Numbers WO2021/033061A1 and WO2020/003150A2, the entire disclosure of each of which is incorporated herein by reference. One example of a trained CNN that the processing circuitry of drone 4 or computing system 8 may execute to perform the defect detection aspects of this disclosure is a Mask R-CNN.

FIGS. 3A & 3B are conceptual diagrams illustrating further details of misapplications of tape 14 to substrate 16 that aspects of system 10 may detect using the techniques of this disclosure. FIG. 3A shows one example of a tape application defect that the processing circuitry of drone 4 or computing system 8 may detect using the trained models of this disclosure. Some of the defects shown in FIG. 3A correspond to what is referred to herein as “fishmouth creasing.” As used herein, fishmouth creasing refers to a tape misapplication that involves a non-adhesion of an edge of tape 14 from substrate 16, with an adjacent non-adhesion or non-flushness of interior portions of tape 14 with substrate 16. The opening caused by the edge non-adhesion and the adjacent protruding crease of tape 14 as applied to substrate 16 creates an opening for fluids (air, water, etc.) to ingress and compromise the functionality of tape 14 as applied to substrate 16.

Another type of defect that the processing circuitry of drone 4 or computing system 8 may detect by executing the trained models include “tenting,” which refers to a non-adhesion and protrusion of an internal protrusion of tape 14 with respect to substrate 16 (e.g., without the edge ingress point of the fishmouth crease shown in FIG. 3A). Tenting of tape 14 may be caused by faulty application procedure, or by applying tape 14 on top of an “underdriven fastener.” An underdriven fastener refers to a nail, screw, bolt, or other type of fastener that is partially driven into substrate 16, but with the head of the fastener protruding out of substrate 16 to a magnitude that causes tape 14 to tent when applied above the fastener's head.

FIG. 3B shows another example of a tape application defect that the processing circuitry of drone 4 or computing system 8 may detect using the trained models of this disclosure. The defect shown in FIG. 3A is referred to herein as a “missing tape segment.” Missing tape segment 18 illustrates a misapplication of tape 14 that leaves a portion of substrate 16 exposed when it should not be exposed, for indoor conditioning concerns or otherwise. For example, missing tape segment 18 may expose a substrate seam or substrate patch that should not be exposed to the elements. The processing circuitry of drone 4 or computing system 8 may also execute the trained models of this disclosure to detect discontinuities in tape 14 that are different from missing tape segment 18. As an example, the processing circuitry of drone 4 or computing system 8 may execute the trained models of this disclosure to detect tears that do not span the entire breadth of tape 14, or scratches that do not expose substrate 16 under tape 14, but instead, compromise or diminish the efficacy of tape 14. The processing circuitry of drone 4 or computing system 8 may also execute the trained models of this disclosure to analyze images received from image capture device 12 to detect other types of misapplications of tape 14 as applied to substrate 16, such as insufficient adhesions (other than the examples discussed above), insufficient tension (e.g., as may cause “slack” with respect to tape 14 if tape 14 is not rolled down with enough force when being applied to substrate 16), etc. In some examples, the processing of drone 4 or computing system 8 (whether locally or by leveraging cloud computing resources) may analyze images of tape 14 to determine branding information, standards-compliance information, etc. with respect to tape 14, whether or the information is represented in a manner that is visible to the human eye without image processing.

FIGS. 4A-4D are diagrams illustrating various deep learning-generated image labels that the trained models of this disclosure may generate. Each of FIGS. 4A-4D illustrates a model output that the processing circuitry of drone 4 or computing device 8 may generate by executing trained models of this disclosure at varying levels of computational complexity. FIGS. 4A-4D, in sequence, illustrate an ascending order of computational complexity with respect to generating the image labels of this disclosure.

In the example of FIG. 4A, a trained classification model of this disclosure implements full image classification on an image of tape 14 as applied to substrate 16. In this example, the processing circuitry of drone 4 or computing system 8 returns a “fail” classification in the form of image label 19. The “fail” result of image label 19 is the result or model output generated by the trained classification model based on detecting at least one defect at any location in the image received from image capture hardware 12. In the example of FIG. 4B, a trained detection model of this disclosure implements sub-image classification on the image of tape 14 as applied to substrate 16. In this example, the trained detection model of this disclosure breaks the image into multiple sub-images 20, and classifies each respective sub-image 20 with a “pass” or “fail” label as was done for the entire image as a whole in the example of FIG. 4A by way of image label 19.

In the example of FIG. 4C, the trained detection model of this disclosure implements object detection on the image of tape 14 as applied to substrate 16. According to the object detection-based techniques shown in FIG. 4C, the trained detection model of this disclosure returns rectangular bounding boxes (bounding boxes 22) around areas of the image that show defects in tape 14 as applied to substrate 16, and for which the trained detection model of this disclosure returns respective “fail” results as model outputs. As shown in the non-limiting example of FIG. 4C, the trained detection model may implement object detection to return multiple (and potentially, as in the case of FIG. 4C, overlapping) bounding boxes 22.

In the example of FIG. 4D, a trained segmentation model of this disclosure implements image segmentation on the image of tape 14 as applied to substrate 16. According to the image segmentation-based techniques shown in FIG. 4D, the trained segmentation model of this disclosure returns a pixel-by-pixel labeling of defects with respect to tape 14 as applied to substrate 16, as represented in the image obtained from image capture hardware 12. FIG. 4D shows defect segments 24, which the trained segmentation model of this disclosure identifies based on a pixel-by-pixel analysis of the image received from image capture hardware 12 showing tape 14 as applied to substrate 16.

FIGS. 3A-4D, in various examples, represent images expressed in various color spaces, such as a red-green-blue (RGB) color space, a grayscale color space, a black and white color space, or various other chromaticity spaces that are partially or wholly discernible to the human eye. In these examples, image capture device 12 represents digital camera hardware configured to produce digital image data in any of these color spaces. In other examples, image capture hardware 12 may represent a so-called “polarization camera.” A polarization camera may produce image data in a variety of formats by performing calculations on data output by polarization sensors of the polarization camera.

FIG. 5 is a conceptual diagram illustrating a polarization image that system 10 may analyze to detect defects with respect to tape 14 as applied to substrate 16, in accordance with aspects of this disclosure. In examples in which image capture hardware 12 represents a polarization camera, the processing circuitry of drone 4 or computing system 8 may analyze various data output by image capture hardware 12 to detect defects with respect to tape 14 as applied to substrate 16. As examples, the processing circuitry of drone 4 or computing system 8 may execute trained classification models of this disclosure to analyze the following two values: unpolarized light in the polarization image, and the degree of linear polarization (DoLP) exhibited in the polarization image. FIG. 5 shows a preprocessed or processed polarization image containing computed DoLP data for each pixel in each color channel, which trained models of this disclosure may utilize to detect the illustrated creasing-based application defects of tape 14 with an enhanced level of confidence.

In some non-limiting examples in which image capture hardware represents a polarization camera, image sensor hardware of image capture hardware 12 includes four polarization filters oriented at 0 degrees, 45 degrees, 90 degrees, and 135 degrees, respectively. Using a notation in which the four images obtained via the four polarization filters are denoted by I₀, I₄₅, I₉₀, and I₁₃₅, the polarization camera of image capture hardware 12 may compute Stokes vectors (S₀, S₁, S₂, and S₃) for each color channel according to the following calculations:

$\begin{matrix} \begin{matrix} S_{0} = I_{0} + I_{90} \\ S_{1} = I_{0} - I_{90} \\ S_{2} = I_{45} - I_{435} \\ S_{3} = 1 - \sqrt{S_{1}^{2} + S_{2}^{2}} \end{matrix} & (1) \end{matrix}$

The Stokes vectors calculated according to equations (1) above represent, respectively, intensity images of unpolarized light (S₀), intensity images of linearly or horizontally polarized light (S₁), intensity images of light polarized at 45 degrees or 135 (S₂), and light that is circularly polarized (S₃). The polarization camera may calculate the DoLP using the above-described Stokes vectors according to equation (2) below:

$\begin{matrix} DoLP = \frac{\sqrt{S_{1}^{2} + S_{2}^{2} + S_{3}^{2}}}{S_{0}} & (2) \end{matrix}$

The processing circuitry of drone 4 or computing system 8 may execute the trained models of this disclosure to use the degree of linear polarization to detect creasing-based defects, such as crease 26 and/or fishmouth crease 28, with respect to tape 14 as applied to substrate 16. Because the surface of tape 14 reflects light differently depending on the angle of the incident light being reflected, the presence and potentially the magnitude (in terms of angle outward from substrate 16) of crease 26 and/or fishmouth crease 28 cause the polarization-based measurements described above to vary.

The trained models of this disclosure, when executed, may use the DoLP calculated according to equation (2) to measure the consistency of the light reflections while remaining agnostic to the directionality of the light reflections. In various experiments, tape 14 had a black and glossy appearance. Regardless of the color of tape 14, the trained models of this disclosure may leverage glossiness traits of tape 14 to use DoLP to detect shadowing and other effects of creasing to detect defects in the application of tape 14 to substrate 16. Darker colors (such as black, used in the experiments described above) of tape 14 may further enhance the ability of the trained models of this disclosure to use DoLP to detect crease 26 and/or fishmouth crease 28 in tape 14 as applied to substrate 16.

FIGS. 6A-6D are diagrams illustrating various deep learning-generated image labels that the trained models of this disclosure may generate using the polarization image shown in FIG. 5. Each of FIGS. 6A-6D illustrates an image label that the processing circuitry of drone 4 or computing device 8 may generate by executing trained models of this disclosure on the polarization image of FIG. 5 at varying levels of computational complexity. FIGS. 6A-6D, in sequence, illustrate an ascending order of computational complexity with respect to generating the image labels using the polarization image of FIG. 5.

In the example of FIG. 6A, the trained classification model of this disclosure implements full image classification on the polarization image of tape 14 as applied to substrate 16. In this example, the processing circuitry of drone 4 or computing system 8 returns a “fail” result based on a full-image analysis of the polarization image, a condition satisfied upon detecting the first-detected of crease 26 or fishmouth crease 28, or both in the case of substantially simultaneous detection. The “fail” result of the model output is shown by way of image label 29 in FIG. 6A.

In the example of FIG. 6B, a trained detection model of this disclosure implements sub-image classification on the polarization image of tape 14 as applied to substrate 16. In this example, the trained detection model of this disclosure breaks the polarization image into multiple sub-images 30, and classifies each respective sub-image 30 as a “pass” or “fail” result, as was done for the entire polarization image as a whole in the example of FIG. 6A.

In the example of FIG. 6C, the trained detection model of this disclosure implements object detection on the polarization image of tape 14 as applied to substrate 16. According to the object detection-based techniques shown in FIG. 6C, the trained detection model of this disclosure returns rectangular bounding boxes (bounding boxes 32) around areas of the polarization image that show defects in tape 14 as applied to substrate 16. As shown in the non-limiting example of FIG. 6C, the trained detection model may implement object detection to return multiple (and potentially overlapping) bounding boxes 32.

In the example of FIG. 6D, a trained segmentation model of this disclosure implements image segmentation on the polarization image of tape 14 as applied to substrate 16. According to the image segmentation-based techniques shown in FIG. 6D, the trained segmentation model of this disclosure returns a pixel-by-pixel labeling of crease 26 and fishmouth crease 28 with respect to tape 14 as applied to substrate 16, as represented in the image obtained from the polarization camera of image capture hardware 12. FIG. 6D shows defect segments 24, which the trained segmentation model of this disclosure identifies based on a pixel-by-pixel analysis of the light polarization exhibited in the polarization image of FIG. 5 showing tape 14 as applied to substrate 16.

FIG. 7 is a graph 36 illustrating aspects of polarization image analysis the trained models of this disclosure may perform to detect one or more defects with respect to tape 14 as applied to substrate 16. Plot lines 38, 40, and 42 show the change in the true positive rate (for image classification) as a function of the corresponding false positive rate under different polarization image analysis scenarios. As a whole, graph 36 represents a receiver operator characteristic (ROC) curve generated from a test data set classified by trained models of this disclosure with respect to DoLP images and unpolarized (S₀) images.

As shown in graph 36, the area under the curve (AUC) is largest for DoLP plot line 38 (which corresponds to polarized images) when compared to the AUC for S₀plot line 40 (which corresponds to unpolarized images) and random plot line 42 (which is provided as a baseline ground truth). The AUCs shown in FIG. 7 provide an overall measure of how discriminable the corresponding classes of images are in the datasets provided to the trained models of this disclosure. The AUC for DoLP plot line 38 is the greatest of the plot lines shown in graph 38, indicating that the trained models of this disclosure can discriminate crease 26 and/or fishmouth crease 28 more easily from DoLP images than using other images.

In this way, the techniques of this disclosure improve the accuracy of defect detection with respect to tape 14 as applied to substrate 16, by using trained models (e.g., one or more of a classification, detection, or segmentation model) of this disclosure. Whether image capture hardware 12 provides images in an RGB color space, a grayscale color space, or as a polarization image (or DoLP image), the trained models of this disclosure detect various types of defects with respect to tape 14 as applied to substrate 16 while improving data precision (e.g., by mitigating human error arising out of different eyesight or perception capabilities). While primarily described as being coupled to or integrated into drone 4 as a non-limiting use case example, it will be understood that the trained model-based image analysis techniques of this disclosure also provide these data precision enhancements in non-drone-based implementations as well.

For example, the trained models of this disclosure may use images captured by image capture hardware 12 in examples in which image capture hardware 12 is integrated into a mobile computing device, such as a smartphone, a tablet computer, a wearable computing device, etc. As another example, the trained models of this disclosure may use images captured by image capture device 12 if image capture device 12 is a dedicated digital camera or a dedicated polarization camera. In any of the non-drone-based examples listed above, the trained models of this disclosure may use images of tape 14 as applied to substrate 16 based on a manual capture of the images, such as by way of user input provided via an actuator button of a digital camera or a touch input provided at a touchscreen of the mobile computing device.

According to the drone-hosted implementations described above, the systems of this disclosure improve safety and also improve the ability to capture and analyze images from difficult-to access areas of substrate 16. For instance, by using drone 4 to transport image capture hardware 12 to potentially hazardous locations and capture images at these locations, system 10 alleviates or potentially eliminates the need to endanger human workers by requiring the workers to access these locations for manual image capture. Drone 4 may also provide maneuverability capabilities not otherwise available to equipment used by workers to survey substrate 16, thereby improving accessibility and tape inspection capabilities with respect to these areas of substrate 16.

FIG. 8 is a conceptual diagram illustrating drone-hosted substrate inspection aspects of this disclosure. Based on navigation instructions received from drone controller 6 or computing system 8, the control logic of drone 4 may cause the motion guide to navigate drone 4 to areas that are proximate to particular areas of substrate 16. Upon detecting that drone 4 is positioned and oriented sufficiently to enable image capture by image capture hardware 12, the control logic may activate image capture hardware 12 to capture one or more images of portions of substrate 16 that are in view of the lens hardware of image capture hardware 12.

The processing circuitry of drone 4 or computing system 8 may analyze the image(s) received from image capture hardware 12 by executing any of the trained models described above (e.g., one or more of classification models, detection models, or segmentation models) to detect a defect in substrate 16. As described above, in various examples the trained model may be a trained deep neural network, such as a trained CNN. In these and other examples, the trained models of this disclosure may apply computer vision-oriented machine learning technology to detect a defect in substrate 16.

FIGS. 9A-9C are conceptual diagrams illustrating examples of an underdriven fastener in substrate 16 that the trained models of this disclosure may detect as a substrate defect. As used herein, the term “underdriven fastener” may refer to any of a nail, screw, bolt, tack, or other penetrative fastener that is not driven into substrate 16 to a sufficient depth or a sufficiently even depth to put fastener head 44 at a substantially flush position with the surface of substrate 16. Underdriven fasteners compromise the structural integrity of building envelopes or other structures represented by substrate 16. In instances in which tape 14 is applied over portions of substrate 16 surrounding and including the underdriven fastener shown in FIG. 9A, the protrusion of fastener head 44 may result in a tenting-based misapplication of tape 14 with respect to substrate 16.

Aspects of system 10 may capture the image of substrate 16 shown in FIG. 9A based on positioning of image capture hardware 12 within sufficient proximity to substrate 16 (e.g., by using drone 4 as shown in FIG. 8 or via manual positioning as discussed above with respect to other examples) and activating image capture hardware 12 to capture the image. The processing circuitry of drone 4 or computing system 8 may execute a trained model of this disclosure using the image of FIG. 9A to detect the underdriven fastener based on image data representing the position and/or orientation of fastener head 44. In this way, the trained models of this disclosure may, in its execution phase, provide a model output that indicates the underdriven status of the fastener of FIG. 9A, thereby enabling remediation of the underdriven fastener in a timely way.

For example, the underdriven fastener that a trained model of this disclosure detects based on the position and/or orientation of fastener head 44 may be remediated, based on the model output provided by the trained models of this disclosure, before further construction-related tasks are performed on top of substrate 16. In this example, the trained models of this disclosure reduce or potentially eliminate the need for additional dismantling or deconstruction purely to access the underdriven fastener before remediation. Instead, by detecting the underdriven fastener based on analyzing image data representing fastener head 44 during envelope layer inspection, the trained models of this disclosure enable remediation of the underdriven fastener in a timely and efficient manner.

The image shown in FIG. 9B illustrates defects in the effectiveness of tape 14 when applied over the underdriven fastener defect of substrate 16 shown by way of the protrusion of fastener head 44 in FIG. 9A. In examples in which a trained model of this disclosure is executed using the image of FIG. 9B, the model output enables various types of remediation, such as a sequence of removal of tape 14, remediation of the underdriven fastener evinced by the position and/or orientation of fastener head 44, and a reapplication of a new segment of tape 14 to the remediated substrate 16. In some examples, such as examples in which the trained model is also trained to detect defects in the application of tape 14 to substrate 16 (as described above with respect to FIGS. 1-6), the trained model may also communicate a model output indicating the misapplication (in this particular example, a tearing) of tape 14 at the location of fastener head 44.

The image shown in FIG. 9C illustrates defects in the effectiveness of tape 14 if applied over the underdriven fastener defect of substrate 16 shown by way of tenting 47 in tape 14 as applied to substrate 16. Tenting 47 occurs due to tape 14 being applied over an underdriven fastener embedded improperly in substrate 16, but without tension that causes the underdriven fastener to break or penetrate tape 14. In examples in which a trained model of this disclosure is executed using the image of FIG. 9C, the model output enables various types of remediation, such as a sequence of removal of tape 14, remediation of the tenting 47, such as by removing the underdriven fastener or driving the underdriven fastener to be flush with substrate 16, and a reapplication of a new segment of tape 14 to the remediated substrate 16 such that the new segment of tape 14 is flush with substrate 16.

FIGS. 10A and 10B are conceptual diagrams illustrating examples of board disjointedness in substrate 16 that the trained models of this disclosure may detect as a substrate defect. As used herein, the term “disjointedness” refers to a non-flush junction (such as non-flush junction 45) between two adjacent boards of substrate 16. Non-flush junction 45 may represent a gap between boards that are not butted together tightly enough, or may represent a grade difference between adjacent boards positioned at different depths, or a combination of these defects. Board disjointedness arising out of conditions such as non-flush junction 45 compromise the structural integrity of building envelopes or other structures represented by substrate 16.

Aspects of system 10 may capture the image of substrate 16 shown in FIG. 10A based on positioning of image capture hardware 12 within sufficient proximity to substrate 16 (e.g., by using drone 4 as shown in FIG. 8 or via manual positioning as discussed above with respect to other examples) and activating image capture hardware 12 to capture the image. The processing circuitry of drone 4 or computing system 8 may execute one or more of the trained models of this disclosure using the image of FIG. 10A to detect the presence of non-flush junction 45. In this way, the trained models of this disclosure may, in its execution phase, provide a model output that indicates the presence of non-flush junction 45, enabling remediation of the resulting board disjointedness in a timely way. In various examples, the model output of the trained models of this disclosure may enable various types of remediation, such as manual repair, automated remediation (e.g., using drones or other equipment), or any other suitable remediation scheme or mechanism.

For example, the board disjointedness caused by non-flush junction 45 may be remediated, based on the model output provided by the trained model(s) of this disclosure, before further construction-related tasks are performed on top of substrate 16. In this example, a trained model of this disclosure reduces or potentially eliminates the need for additional dismantling or deconstruction purely to access non-flush junction 45 before remediation. Instead, by detecting the board disjointedness caused by non-flush junction 45 during envelope layer inspection, the trained model of this disclosure enables remediation of the board disjointedness in a timely and efficient manner.

The image shown in FIG. 10B illustrates defects in the effectiveness of tape 14 when applied over the board disjointedness defect of substrate 16 shown by way of non-flush junction 45 in FIG. 10A. In examples in which the trained model of this disclosure is executed using the image of FIG. 10B, the model output enables various types of remediation, such as a sequence of removal of tape 14, remediation of the board disjointedness stemming from non-flush junction 45, and a reapplication of a new segment of tape 14 to the remediated substrate 16. In some examples, such as examples in which the model is also trained to detect defects in the application of tape 14 to substrate 16 (as described above with respect to FIGS. 1-7), the trained model may also communicate a model output indicating the non-adhesion of tape 14 at non-flush junction 45.

FIG. 11 is a conceptual diagram illustrating an example of a defect in substrate 16, which is caused by an overdriven fastener, and detected using the trained models of this disclosure. As used herein, the term “overdriven fastener” refers to a nail, screw, tack, bolt, or other fastener that is driven to an excessive depth such that the head or other type of proximal end of the fastener has penetrated substrate 16 and is currently positioned below the substrate. The overdriving of the fastener causes hole 50 to form in the surface (and to some depth below the surface) of substrate 16. Hole 50 represents a defect that may compromise the structural integrity of substrate 16, as well as the integrity of substrate 16 with respect to shielding the interior of building 2 from weather conditions such as temperature, water, and other elements. Hole 50 may cause heat transfer, water ingress, or other diminishment of function with respect to substrate 16. While hole 50 is described herein as being caused by an overdriven fastener as an example, hole 50 may also be caused by other factors, such as windborne debris, a removed fastener, etc.

The processing circuitry of drone 4 or computing system 8 may execute a trained model of this disclosure using the image of FIG. 11 to detect the presence of hole 50. In this way, the trained model(s) of this disclosure may, in the respective execution phase(s), provide a model output that indicates the presence of hole 50, enabling remediation of the resulting board disjointedness in a timely way. The trained model(s) of this disclosure may also provide a documentation trail for construction site administrators, inspectors, contractors, etc., thereby aiding in construction management, for providing information related to insurance, and potentially for clarifying disputed items in future disputes. In this example, the trained model of this disclosure reduces or potentially eliminates the need for additional dismantling or deconstruction purely to access hole 50 before remediation. Instead, by detecting the structural defect in substrate 16 represented by hole 50 during envelope layer inspection, the trained model of this disclosure enables remediation of hole 50 in a timely and efficient manner.

FIGS. 12A and 12B are conceptual diagrams illustrating examples of impact-related damage in substrate 16 that the trained models of this disclosure may detect as a substrate defect. As used herein, the term “impact-related damage” refers to any type of damage to substrate 16 that might result from striking (inward pressure) or gouging (outward pressure). While substrate 16 may exhibit impact-related damage due to a number of different causes, the examples of FIGS. 12A and 12B are described herein with respect to damage caused by hammer claw-end gouging and hammer strikes as non-limiting examples. The impact-related damage illustrated in FIGS. 12A and 12B compromise the structural integrity of building envelopes or other structures represented by substrate 16.

Aspects of system 10 may capture the image of substrate 16 shown in FIG. 12A based on positioning of image capture hardware 12 within sufficient proximity to substrate 16 (e.g., by using drone 4 as shown in FIG. 8 or via manual positioning as discussed above with respect to other examples) and activating image capture hardware 12 to capture the image. The processing circuitry of drone 4 or computing system 8 may execute the trained model of this disclosure using the image of FIG. 12A to detect the presence of surface tears 46. The trained model of this disclosure may, in its execution phase, detect tears of various breadths and severity. As shown in FIG. 12A, the trained model of this disclosure detects two relatively large tears, as well as a number of smaller tears or “dings” in substrate 16. In this way, the trained model of this disclosure may, in its execution phase, provide a model output that indicates the presence of surface tears 46, enabling remediation of surface tears 46 in a timely way.

For example, the board disjointedness caused by non-flush junction 45 may be remediated, based on the model output provided by the trained model of this disclosure, before further construction-related tasks are performed on top of substrate 16. In this example, the trained model of this disclosure reduces or potentially eliminates the need for additional dismantling or deconstruction purely to access non-flush junction 45 before remediation. Instead, by detecting the board disjointedness caused by non-flush junction 45 during envelope layer inspection, the trained model of this disclosure enables remediation of the board disjointedness in a timely and efficient manner.

The image shown in FIG. 12B illustrates surface indentations 48 in substrate 16. Surface indentations 48 may be caused by excessive force and/or improper angling applied when striking the surface of substrate 16 with a hammer, or due to other factors. In examples in which the trained models of this disclosure are executed using the image of FIG. 12B, the respective model output identifies surface indentations 48, which show instances of hammer strikes that exposed material (e.g. wood) positioned underneath a weather-proof coating applied to substrate 16.

FIGS. 13A and 13B illustrate examples in which drone 4 is equipped and configured to mark objects of potential survey interest (OoPSIs) on substrate 16 or tape 14, in accordance with aspects of this disclosure. In some examples, aspects of system 10 may navigate drone 4 to areas near OoPSIs that are identified using the trained models described above with respect to FIGS. 1-12B, or that are identified in other ways. In the example of FIG. 13A, drone 4 is equipped with a top mount 52. Top mount 52 may represent any hardware or combination of hardware components that, when physically coupled to an upper surface of (in-flight oriented) drone 4, enables further coupling of drone 4 to additional attachments and components. In other embodiments, drone 4 may be equipped with a bottom mount that enables coupling of additional attachments and/or components via a ground-facing surface of drone 4 when in-flight.

Drone 4 is equipped with shock absorption sub-assembly 54. In the example of FIG. 13A, top mount 52 couples drone 4 to shock absorption sub-assembly 54. In some examples, shock absorption sub-assembly 54 represents a compression spring set, which may include a single compression spring, or multiple compression springs. In other examples, shock absorption sub-assembly 54 may represent other types of shock-absorption technology, such as a hydraulic device, a compression bladder, struts, magnetorheological fluid, etc. In any event, shock absorption sub-assembly 54 is configured to absorb and/or damp shock impulses by converting impact-related shock into another form of energy that can be dissipated, such as heat energy.

Drone 4 is also equipped with marking device 56. Marking device 56 may represent various types of equipment configured to mark areas of substrate 16, or areas of tape 14 as applied to substrate 16. In one example, marking device 54 represents an ink-dispensing system, such as a pen, felt pen, marker, bingo dauber, etc. that is configured to dispense ink upon contact between a distal tip of marking device 56 and a receptacle, such as substrate 16 or tape 14 as applied to substrate 16. In another example, marking device 56 is configured to dispense a self-adhesive paper strips onto a receptacle (e.g., substrate 16 or tape 14 as applied to substrate 16) with which the distal tip of marking device 56 comes into contact. In other examples, marking device 56 is configured to mark a receptacle (such as substrate 16 or tape 14 as applied to substrate 16) in other ways.

FIG. 13B shows further details of certain aspects of drone 4 as configured in the example of FIG. 143. FIG. 13B is a side view of various components that are coupled to drone 4 via top mount 52. FIG. 13B shows compression range 58 of shock absorption sub-assembly 54. Compression range 58 represents a length to which shock absorption sub-assembly 54 enables temporary reduction of the overall length of the combination of components that are coupled to drone 4 via top mount 52.

It will be appreciated that compression range 58 does not represent the length to which shock absorption sub-assembly 54 compresses at every instance of marking device 56 impacting a body, such as substrate 16. Rather, compression range 58 represents the maximum compression afforded by shock absorption sub-assembly 54 upon a distal tip of marking device 56 making contact with a rigid or semi-rigid body (e.g., substrate 16 or tape 14 as applied to substrate 16). In the example of the orientation shown in FIG. 13B, the right-side end of marking device 56 includes the distal tip that that comes into contact with substrate 16 as part of the OoPSI marking functionalities described herein.

Depending on the force of the impact, shock absorption sub-assembly 54 may compress to either the full magnitude of compression range 58, or to a magnitude that is less than compression range 58. In the configuration shown in FIG. 13B, shock absorption sub-assembly 54 is positioned between marking mount 60 and rear stop 64. Marking mount 60 represents a component configured to receive marking device 56. In some examples, marking mount 60 has an expandable or configurable diameter and/or shape, thereby enabling marking mount 60 to receive marking devices or other peripherals of varying shapes, sizes, form factors, etc.

In this way, marking mount 60 enables the use of various types of marking peripherals in accordance with the systems and techniques of this disclosure. Rear stop 64 represents a rigid component with a fixed position. Rear stop 64 enables drone 4 to provide a counterforce to the impact of the distal tip marking device 56 with substrate 16 or tape 14, while accommodating the compression provided by shock absorption sub-assembly 54 up to a maximum length represented by the full length of compression range 58.

According to the configuration shown in FIG. 13B, drone 4 is also equipped with motion guide 66. In various examples, motion guide 66 is a linear motion guide that provides a sliding framework for reciprocating movement of marking mount 60 (which holds marking device 56) in response to the distal tip of marking device 56 impacting substrate 16. Motion guide 66 is coupled to drone 4 via top mount 52 and holds shock absorption sub-assembly 54 in place between motion guide 66 and marking mount 60 using one or more fasteners (e.g., in a slotted channel or another type of channel).

Control circuitry of drone 4 is configured to navigate drone 4 to an area associated with (e.g., at, including, or proximate to) an identified OoPSI. The control circuitry may use a local position tracker and other hardware of drone 4 to effectuate these movements of drone 4. For example, the control circuitry may navigate drone 4 to the area associated with the identified OoPSI based on instructions received from control logic of drone 4. The control logic of drone 4 may, in turn, navigate drone 4 to the area associated with the OoPSI based on navigation instructions that the control logic receives from the processing circuitry of drone 4 or computing system 8.

In some examples, drone 4, as configured in the examples of FIGS. 13A & 13B, may navigate to and mark OoPSIs that are associated with defects in substrate 16, such as the examples shown in and described with respect to FIGS. 8, 9A, 10A, and 11-12B. Examples of substrate defect OoPSIs that drone 4 may navigate to and mark in accordance with aspects of FIGS. 13A & 13B include surface tears, underdriven fasteners, overdriven fasteners, surface gouging, excess sealant, board disjointedness, gaps, etc.

In some examples, drone 4, as configured in the examples of FIGS. 13A & 13B, may navigate to and mark OoPSIs that are associated with tape misapplication(s) with respect to tape 14 as applied to substrate 16, such as the examples shown in and described with respect to FIGS. 2-7, 9B, and 10B. Examples of tape misapplication-related OoPSIs that drone 4 may navigate to and mark in accordance with aspects of FIGS. 13A & 13B include fishmouth creasing, tenting of tape 14 as applied to substrate 16, missing segment(s), various types of insufficient adhesion, insufficient tension, etc.

FIGS. 14A-14C are conceptual diagrams illustrating examples in which drone 4 is equipped and configured to remediate OoPSIs on substrate 16 or tape 14 as applied to substrate 16, in accordance with aspects of this disclosure. In some examples, aspects of system 10 may navigate drone 4 to areas near OoPSIs that are identified using the trained models described above with respect to FIGS. 1-12B, or that are identified in other ways. In the example of FIG. 14A, drone 4 is equipped with top mount 52 and a lower mount 68. Lower mount 68 may represent any hardware or combination of hardware components that, when physically coupled to a lower surface or ground-facing surface of (in-flight oriented) drone 4, enables further coupling of drone 4 to additional attachments and components.

By way of lower mount 68, drone 4 is equipped with dispenser sub-assembly 72. Dispenser sub-assembly 72 includes a housing 75 that receives syringe 76. As shown, housing 75 is configured to receive syringe 76 in a position and orientation such that an applicator of syringe 76 is positioned distally from housing 75. As such, dispenser sub-assembly 72 is configured to house syringe 76 in a position and orientation that enables extrusion of any contents of syringe 76 in a distal direction from an airframe of drone 4.

As described with respect to FIG. 13B, control circuitry of drone 4 is configured to navigate drone 4 to an area associated with (e.g., at, including, or proximate to) an identified OoPSI based on instructions that control logic of drone 4 generates based on navigation instructions received from the processing circuitry of drone 4 or computing system 8. The control logic of drone 4 may also receive extruding instructions from the processing circuitry of drone 4 or computing system 8.

FIG. 14B shows a top view of dispenser sub-assembly 72. Based on the extruding instructions received from the processing circuitry, the control logic may activate actuator motor 77. For instance, based on the received extruding instructions, the control logic of drone 4 may cause actuator motor 77 to move actuator arm 80 in an extension phase of a reciprocating motion. The extension phase of the reciprocating motion represents a phase in which actuator arm 80 moves on a linear path distally from the airframe of drone 4. An appointed distance 82 signifies the a distance the actuator can move within the dispenser sub-assembly 72, which may correlate to the depletion of material in this form factor.

By moving actuator arm 80 in the extension phase of the reciprocating motion, actuator motor 77 causes actuator arm 80 to extrude a portion of the contents of syringe 76. Based on drone 4 being positioned at an area associated with an identified OoPSI, actuator motor 77 causes actuator arm 80 to extrude the contents of syringe 76 at the area associated with the OoPSI. In some examples, based on the navigation instructions and/or the extruding instructions, the control logic of drone 4 is configured to move drone 4 in parallel, or substantially in parallel, with the surface of substrate 16 while actuator arm 80 is in the extension phase of the reciprocating motion to extrude the contents of syringe 76.

As used herein, movement of drone 4 substantially in parallel with the surface of substrate 16 refers to movement in any pattern that is substantially parallel to the X-Y plane of substrate 16. By moving drone 4 substantially in parallel with the X-Y plane of substrate 16 while actuator arm 80 is in the extension phase, the control logic of drone 4 processes the navigation instructions and the extruding instructions to extrude the contents of syringe 76 over some, most, or all of the identified OoPSI.

That is, the navigation instructions may correspond to a movement pattern that, upon completion, covers some, most, or all of the identified OoPSI. In some examples, based on an extrusion increment associated with the extruding instructions, the control logic of drone 4 may cause actuator motor 77 to move actuator arm 80 in a retraction phase of the reciprocating motion to cease extruding the contents of syringe 76. That is, the extrusion increment may define an amount of the contents of syringe 76 to be extruded in order to rectify the OoPSI, assuming movement of drone 4 to cover a sufficient area of the OoPSI while the contents of syringe 76 are being extruded.

Actuator coupler 74 physically couples the distal end of actuator arm 80 (with respect to the airframe of drone 4) to the proximal end of syringe 76 (with respect to the airframe of drone 4), causing the proximal end of syringe 76 to track both extension and retraction phases of the reciprocating motion of actuator arm 80.

FIG. 14C shows further details of slotted channel 70 shown in FIG. 14A. As shown in FIG. 14A, slotted channel 70 is configured to couple dispenser sub-assembly 72 to the airframe of drone 4. Slotted channel 70 provides a self-weighted uncontrolled degree of freedom (DOF) 88 for radial movement of dispenser sub-assembly 72 with respect to the reference point of the point of fixture to the airframe of drone 4. By providing uncontrolled DOF 88 for the movement of dispenser sub-assembly 72, slotted channel 70 provides an error buffer (e.g., against wind gusts, rotor wash, etc.) with respect to the radial movement of dispenser sub-assembly 72.

Uncontrolled DOF 88 provided by slotted channel 70 reduces the need for additional motors and onboard component infrastructure that would be required in the case of controlled DOF implementations, which in turn would add weight to a potentially weight-sensitive system. However, it will be appreciated that controlled-DOF implementations and/or rigidly affixed implementations are also consistent with the adhesive-dispensing drone implementations of this disclosure. FIG. 14C shows pivot hub 84 and radial fasteners 86A & 86B. Radial fasteners 86A & 86B are positioned equidistantly from pivot hub 84, providing an arc included in uncontrolled DOF 88.

While only an arc represented by the circumferential movement between radial fasteners 86A & 86B is shown in FIG. 14C purely for ease of illustration, it will be appreciated that slotted channel 70 may include varying numbers of radial fasteners to provide uncontrolled DOF 88. In various examples, syringe 76 may be loaded with various types of adhesive content, such as caulk, general purpose silicone adhesives, nitrocellulose adhesives, paste sealant, epoxy acrylic, or other adhesive suitable to be dispensed using dispenser sub-assembly 72. In some examples, drone 4 may be equipped with swappable syringes, with syringe 76 representing a currently in-use syringe, with other backup and/or used-up syringes also on board.

The embodiments of drone 4 shown in FIGS. 14A-14C may dispense the adhesive contents of syringe 76 to remediate various types of OoPSIs, including, but not limited to, defects in substrate 16 such as surface tears, underdriven fasteners, overdriven fasteners, surface gouging, gaps or other discontinuities between boards, impact-related damage, etc. and/or misapplications of tape 14 such as fishmouth creasing, tears or scrapes, creasing, tenting, missing tape segments, an insufficient adhesion, insufficient tension, etc.

FIG. 15 is a conceptual diagram illustrating another example in which drone 4 is equipped and configured to remediate OoPSIs on substrate 16 or tape 14 as applied to substrate 16, in accordance with aspects of this disclosure. In some examples, aspects of system 10 may navigate drone 4 to areas near OoPSIs that are identified using the trained models described above with respect to FIGS. 1-12B, or that are identified in other ways. In the example of FIG. 15, drone 4 is equipped with dispenser sub-assembly 90. Dispenser sub-assembly 90 includes a housing 94 that receives aerosol dispensing system 102. While dispenser sub-assembly 90 is shown in FIG. 15 as being mounted on the top surface of drone 4 as a non-limiting example, it will be appreciated that, in another examples in accordance with this disclosure, dispenser sub-assembly 90 may be coupled to drone 4 in other ways.

Aerosol dispensing system 102 may represent one or more types of cans or storage devices configured to release compressed contents upon open of a pressure valve, such as by depressing nozzle 104. As described with respect to FIG. 13B, control circuitry of drone 4 is configured to navigate drone 4 to an area associated with (e.g., at, including, or proximate to) an identified OoPSI based on instructions that control logic of drone 4 generates based on navigation instructions received from the processing circuitry of drone 4 or computing system 8. The control logic of drone 4 may also receive dispensing instructions from the processing circuitry of drone 4 or computing system 8.

Based on the dispensing instructions received from the processing circuitry, the control logic may activate motor 92. For instance, based on the received dispensing instructions, the control logic of drone 4 may cause motor 92 to move trigger 98 in a retracting phase of a reciprocating motion. The retraction phase of the reciprocating motion represents a phase in which trigger 98 moves proximally towards the airframe of drone 4. For example, when activated in this way by the control logic of drone 4, motor 92 may retract link wire 96 towards the airframe of drone 4, thereby retracting trigger 98, which is coupled to link wire 96.

By moving trigger 98 in the retraction phase of the reciprocating motion, motor 92 causes trigger 98 to depress nozzle 104, thereby releasing a portion of the contents of aerosol dispensing system 102. Based on drone 4 being positioned at an area associated with an identified OoPSI, motor 92 causes trigger 98 to depress nozzle 104 and dispense the contents of aerosol dispensing system 102 at the area associated with the OoPSI. In some examples, based on the navigation instructions and/or the dispensing instructions, the control logic of drone 4 is configured to move drone 4 in parallel, or substantially in parallel, with the surface of substrate 16 while trigger 98 is in the retraction phase of the reciprocating motion to keep nozzle 98 depressed and to thereby dispense the contents of aerosol dispensing system 102.

As used herein, movement of drone 4 substantially in parallel with the surface of substrate 16 refers to movement in any pattern that is substantially parallel to the X-Y plane of substrate 16. By moving drone 4 substantially in parallel with the X-Y plane of substrate 16 while trigger 98 is in the retraction phase of the reciprocating motion, the control logic of drone 4 processes the navigation instructions and the extruding instructions to dispense the contents of aerosol dispensing system 102 over some, most, or all of the identified OoPSI.

That is, the navigation instructions may correspond to a movement pattern that, upon completion, covers some, most, or all of the identified OoPSI. In some examples, based on a dispensing increment associated with the extruding instructions, the control logic of drone 4 may cause motor 92 to release at least part of the tension applied to link wire 96 to move trigger 98 in an extension phase of the reciprocating motion to cease dispensing the contents of aerosol dispensing system 102. That is, the dispensing increment may define an amount of the contents of aerosol dispensing system 102 to be sprayed in order to rectify the OoPSI, assuming movement of drone 4 to cover a sufficient area of the OoPSI while the contents of aerosol dispensing system 102 are being sprayed.

The contents of aerosol dispensing system 102 may include any aerosol-propelled sealant or any other material suitable to be sprayed over an identified OoPSI for sealing or molding purposes, such as a rubber sealant, a weatherproof spray paint, pressurized foam sealant, etc. The embodiment of drone 4 shown in FIG. 15 may dispense the contents of aerosol dispensing system 102 to remediate various types of OoPSIs, including, but not limited to, defects in substrate 16 such as surface tears, overdriven fasteners, surface gouging, gaps or other discontinuities between boards, impact-related damage, etc. and/or misapplications of tape 14 such as tears or scrapes, missing tape segments, an insufficient adhesion, etc.

In some implementations consistent with FIGS. 14A-15, drone 4 may be equipped with a light source, a light sensor, and an optical fiber link coupling the light source to the light sensor. In these implementations, the control logic of drone 4 may activate the light source based on the dispensing/extruding instructions, and motor 92 or actuator motor 77 (as the case may be) is configured to move trigger 98 in the retraction phase or actuator arm 80 in the extension phase of the respective reciprocating motion. In these examples, the control logic of drone 4 uses these light-based techniques to depress nozzle 104 or to extrude the contents of syringe 76 to dispense the contents of aerosol dispensing system 102 or syringe 76 at the area associated with the OoPSI in response to the light sensor detecting the activation of the light source via the optical fiber link.

In other implementations consistent with FIGS. 14A-15, drone 4 may be equipped with a microcontroller, a Bluetooth® or other near-field, low power, wireless transceiver, and a power source, such as a battery or battery pack. The microcontroller may continuously run a script, which, at appropriate function calls, may initiate a connection with the wireless transceiver, and send signals corresponding to the dispensing increment or extrusion increment. In these examples, the microcontroller-transceiver based subsystem is separate and independent from the firmware of drone 4, and is therefore portable between and agnostic to different underlying UAV platforms, pending certain mechanical adjustments to suit the underlying UAV platform.

FIG. 16 is a flowchart illustrating an example process 110 of this disclosure. Process 110 may begin with a survey of building 2 (106). For instance, control logic of drone 4 may navigate drone 4 and activate image capture hardware 12 to capture one or more images of building 2. In turn, processing circuitry of drone 4 or computing system 8 may analyze the one or more images (108). In various examples, the processing circuitry of drone 4 or computing system 8 may analyze the image(s) received from image capture hardware by executing one or more of a trained classification model, a trained detection model, or a trained segmentation model of this disclosure to generate a model output.

The processing circuitry may report the model output (112). For instance, the processing circuitry may be communicatively coupled to output hardware communicatively coupled to the processing circuitry. In these examples, the processing circuitry may be configured to output model output via the output hardware, which may be a monitor, a speaker, a communications interface configured to relay the model input to another device, etc. As described above with respect to FIGS. 4A-4D and FIGS. 6A-6D, the model output may be indicative of defective condition and/or of specific OoPSI(s) shown in the image(s).

Process 110 includes a determination of whether or not to mark a detected OoPSI using drone 4 (decision block 114). If the determination is to mark the detected OoPSI using drone 4 (‘YES’ branch of decision block 114), control logic of drone 4 may cause drone 4 to mark the OoPSI (116), such as by using techniques described above with respect to 13A & 13B. If the determination is to not mark the detected OoPSI using drone 4 (‘NO’ branch of decision block 114), then site administrators may optionally mark the detected OoPSI manually (118). The optional nature of manual marking of a detected OoPSI is shown by way of the dashed-lined border of step 118 in FIG. 16.

Process 110 also includes a determination of whether or not to remediate a detected OoPSI using drone 4 (decision block 120). If the determination is to remediate the detected OoPSI using drone 4 (‘YES’ branch of decision block 120), control logic of drone 4 may cause drone 4 to remediate the OoPSI (122), such as by using techniques described above with respect to 14A-15. If the determination is to not remediate the detected OoPSI using drone 4 (‘NO’ branch of decision block 120), then site administrators may optionally remediate the detected OoPSI manually (124). The optional nature of manual remediation of a detected OoPSI is shown by way of the dashed-lined border of step 124 in FIG. 16. In various examples, the control logic of drone 4 may be configured to navigate drone 4 to the area surrounding the OoPSI and effectuate the remediation measure in response to the processing circuitry detecting a mark placed manually or by drone 4 by analyzing image(s) received from image capture hardware 12.

In some implementations, a software application executing on computing system 8 (which in these implementations is communicatively coupled to controller 6) autonomously identifies one or more targets on substrate 16 to be remediated via spraying by aerosol dispensing system 102. The application may process video data of a video feed received from drone 4 (e.g., via image capture hardware 12 or other video capture hardware with which drone 4 may be equipped). For example, the application may identify a crack between two plywood boards, cause the control logic of drone 4 to align drone 4 with an edge or end of the crack, to activate aerosol dispenser system 90 to begin spraying, and to move drone 4 along the crack until drone 4 reaches the opposite end of the crack, at which point the control logic may deactivate aerosol dispensing system 102, causing the spraying to stop.

In another example, the application may identify a gap that circumscribes the junction of a pipe with substrate 16, cause the control logic of drone 4 to align drone 4 with the edge of the crack, to activate aerosol dispenser system 90 to begin spraying, and to move drone 4 along a circular path that tracks the junction of the pipe with substrate 16 until drone 4 fully circumnavigates the junction, at which point the control logic may deactivate aerosol dispensing system 102, causing the spraying to stop. In either of these examples, the application may identify the crack, the pipe, or the pipe junction by executing a computer vision-oriented machine learning model trained using a dataset of numerous images of substrate 16 at different distances, angles, lighting conditions, etc. While described as “circular” it will be appreciated that the path of drone 4 to remediate a gap at a pipe junction may be oblong, elliptical, or any other type of closed shape.

Computer vision processing may be performed areas within labeled bounding boxes around areas of interest. In one example of a computer vision processing workflow of this disclosure, the application running on computing system 8 may execute a trained machine learning algorithm to read a video frame received from image capture hardware