130 z^{k} can be enhanced into a domain tensor 132, Z^{k}, and Z^{k} can be concatenated with the input tensor to input z^{k} into the ROI and mask head computations. The (l, m_v, m_h) element of Z^{k} is defined as

Z

l
,

m
ν

,

m
h

{
k
}

≡

z
l

{
k
}

(
5
)

The elements of the tensor whose first indexes are l and have the same values over all their second and third indexes (m_v, m_h), and their values are the same as that of the l-th element of the original z^{k}. The size of the domain tensor is L×M_v×M_h, where L is the same as the number of elements of z^{k}, M_vand M_hare the same as the size of the input tensors. Z{k} can be concatenated with the input tensor along the channel dimension, where the size of the resultant input tensor becomes (C+L)×M_v×M_h. This operation corresponds to considering each element in z^{k} as additional channels in the input tensor.

The pipeline for inferring the domain vector z can be modeled with Deep sets 134, which is the deep neural network for sets, not single instances, as

z

{
k
}

(

X

{
k
}

)

≡

ρ
⁢

θ
v

(

1

N
k

⁢

∑

x
∈

X

{
k
}

η

θ
η

,

(

B
⁡
(
x
)

)

)

,

(
6
)

- where X^{k} represents a set of images 136 in ^{k}, each of ρ and η is modeled by a fully-connected layer, and B is the backbone network for η, which is also the pre-trained efficient net for the feature extraction from images. This specific architecture including ρ and η provides a permutation invariant due to the average. First, each image with η is converted, and then the average is taken over the set, and finally the function ρ is applied. The pipeline for domain adaptation 140 is shown in FIG. 1. From the definition of the domain vector 130 in Equation (6), above, the domain vector 130 is computed once for a set. The domain vector 130 does not need to be computed for each image.

It should be noted that θ_ρand θ_η respectively are the parameters for ρ and η. The same objective function can be used with Mask R-CNN in Equation (2), above, for training θ_ρand θ_η, and the training is done accompanied with that for Mask R-CNN. Mask R-CNN and Deep sets are trained to compute Y (x|z^{k}) tailored to the k-th domain in accordance with z^{k} while generalizing the inference for z among domains. Since the parameters for all functions are shared among all domains, retraining is not required, even for θ_ρ and θ_η, on the target domain. The domain vector 130 can be inferred from [Image Omitted] ^{T}for the target domain and can be input into Mask R-CNN through the architecture with the domain adaptation pipeline according to aspects of the present disclosure.

Referring to FIG. 2, a domain-vector based approach for an autoencoder 200 has a similar procedure with the architecture described above in terms of domain adaptation. However, the domain vector-based approach cannot readily be applied to state-of-the-art visual inspection methods, such as Mask R-CNN. This is because convolutional output layers need to be handled. This limitation, however, is overcome with the tensor enhancement using the domain tensor 132 as described above.

The effectiveness of the above described approach in the task of visual inspection on real-world civil infrastructure data is demonstrated. Civil infrastructure data was used, which contains images collected with drones flying around structures and the corresponding labels provided by human engineers. Each defect in an image is detected, with six categories: algae, crack, crack (net-crack), crack with precipitation, rust, and spalling. Two different datasets were used, so, sequentially, one dataset is considered as the source domain and the other as the target domain. Because there is only a single source domain, the data augmentation technique is used to produce multiple domains synthetically to maintain the generalization capability of inferring z. An image was enhanced in the source domain into multiple augmented images and each one was considered as an image from a different domain. The augmentation methods such as GaussianSmoothing with the parameters 5 and 15, GrayScale, ChangeBrightness with the parameters 0.8 and 1.2, and ChangeContrast with the parameters 0.8 and 1.2 were applied. Thus, a single domain was enhanced into eight domains. Twenty percent of the training set was randomly sampled for use as a validation set to determine the best hyperparameters.

The model described above, according to aspects of the present disclosure, was compared with two baselines: Mask R-CNN and Fine-tuning. Mask R-CNN is a state-of-the-art method on instance segmentation, but it does not use domain adaptation, where the model is trained only on the target domain and the model is tested on the test set on the target domain. Fine-tuning is Mask R-CNN with domain adaptation, where the model is trained on the source domains, the model is tuned on the target domain, and finally, the model is tested on the test set on the target domain.

The impact of inputting the domain vector into multiple layers in Mask R-CNN was initially investigated. The proposed model, as described above, was evaluated with different configurations, where the domain vector is input into (1) the class and box, (2) the ROI head, (3) the mask head, and (4) all heads. In this experiment, it is shown how the performance in unsupervised domain adaptation, where the model was trained on the source domain without requiring any prior knowledge of the target domain, such as the need to use any data from the target domain or the need for any labelled images from the target domain. The average performance of the most important visual inspection metric was reported as the mean of intersection over union (mIoU).

Referring to FIG. 3, it can be seen that inputting the domain vector into multiple layers, especially the convolutional layers (the mask and ROI heads), improved the domain adaptation performance. The method “All” performed the best, where the domain vector and domain tensor are input into the mask head, the ROI head and the class and box.

Performances in multiple scenarios with different numbers of the labeled target samples were investigated. The number of labeled samples was gradually increased from the target domain for training the model and show each performance. Labeled target data for training the model was partially used. Thus, this example is a semi-supervised domain adaptation. Here, the mIoU and the average precision with 50% IoU threshold (AP50) were reported.

In FIGS. 4 through 7, the charts show the instance segmentation performance over the different numbers of labeled target samples for the baselines and the model according to the present disclosure. It can be seen that the proposed model, according to the present disclosure, showed a higher performance than the baselines in each case. This shows that the proposed model, according to the present disclosure, can accelerate learning regarding sample complexity/efficiency.

FIG. 8 illustrates a schematic representation of supervised learning on a target domain 800. In this figure, no samples are used from the source domain. A number of samples (Np) from the target domain 800 are used as a set 802 for training to generate a prediction model 804. The prediction model 804 is tested to provide evaluation results 806 that are compared with a training set 808 on the target domain.

FIG. 9 illustrates a schematic representation of supervised learning on both a target domain and a source domain. A number of samples (Np) from the target domain 800 and all the samples from the source domain 900 are used as a set 902 for training to generate a prediction model 904. The prediction model 904 is tested to provide evaluation results 906 that are compared with a training set 808 on the target domain.

FIG. 10 illustrates a schematic representation of supervised learning on both a target domain and a source domain, with Deep sets, according to an illustrative embodiment of the present disclosure. Similar to FIG. 9, a number of samples (Np) from the target domain 800 and all the samples from the source domain 900 are used as the set 902 for training to generate the prediction model 904. The prediction model 904 is tested, with Deep sets 134 (see FIG. 1) to provide evaluation results 1000 that are compared with the training set 808 on the target domain.

Example Computing Platform

Various aspects of the present disclosure are described by narrative text, flowcharts, block diagrams of computer systems and/or block diagrams of the machine logic included in computer program product (CPP) embodiments. With respect to any flowcharts, depending upon the technology involved, the operations can be performed in a different order than what is shown in a given flowchart. For example, again depending upon the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, concurrently, or in a manner at least partially overlapping in time.

A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media (also called “mediums”) collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device that can retain and store instructions for use by a computer processor. Without limitation, the computer readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these mediums include: diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as punch cards or pits/lands formed in a major surface of a disc) or any suitable combination of the foregoing. A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals communicated through a wire, and/or other transmission media. As will be understood by those of skill in the art, data is typically moved at some occasional points in time during normal operations of a storage device, such as during access, de-fragmentation or garbage collection, but this does not render the storage device as transitory because the data is not transitory while it is stored.

Referring to FIG. 11, computing environment 1100 includes an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods, such as a domain adaptation of an object detection model block 1200. In addition to block 1200, computing environment 1100 includes, for example, computer 1101, wide area network (WAN) 1102, end user device (EUD) 1103, remote server 1104, public cloud 1105, and private cloud 1106. In this embodiment, computer 1101 includes processor set 1110 (including processing circuitry 1120 and cache 1121), communication fabric 1111, volatile memory 1112, persistent storage 1113 (including operating system 1122 and block 1200, as identified above), peripheral device set 1114 (including user interface (UI) device set 1123, storage 1124, and Internet of Things (IoT) sensor set 1125), and network module 1115. Remote server 1104 includes remote database 1130. Public cloud 1105 includes gateway 1140, cloud orchestration module 1141, host physical machine set 1142, virtual machine set 1143, and container set 1144.

COMPUTER 1101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer or any other form of computer or mobile device now known or to be developed in the future that is capable of running a program, accessing a network or querying a database, such as remote database 1130. As is well understood in the art of computer technology, and depending upon the technology, performance of a computer-implemented method may be distributed among multiple computers and/or between multiple locations. On the other hand, in this presentation of computing environment 1100, detailed discussion is focused on a single computer, specifically computer 1101, to keep the presentation as simple as possible. Computer 1101 may be located in a cloud, even though it is not shown in a cloud in FIG. 11. On the other hand, computer 1101 is not required to be in a cloud except to any extent as may be affirmatively indicated.

PROCESSOR SET 1110 includes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitry 1120 may be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitry 1120 may implement multiple processor threads and/or multiple processor cores. Cache 1121 is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set 1110. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located “off chip.” In some computing environments, processor set 1110 may be designed for working with qubits and performing quantum computing.

Computer readable program instructions are typically loaded onto computer 1101 to cause a series of operational steps to be performed by processor set 1110 of computer 1101 and thereby effect a computer-implemented method, such that the instructions thus executed will instantiate the methods specified in flowcharts and/or narrative descriptions of computer-implemented methods included in this document (collectively referred to as “the inventive methods”). These computer readable program instructions are stored in various types of computer readable storage media, such as cache 1121 and the other storage media discussed below. The program instructions, and associated data, are accessed by processor set 1110 to control and direct performance of the inventive methods. In computing environment 1100, at least some of the instructions for performing the inventive methods may be stored in block 1200 in persistent storage 1113.

COMMUNICATION FABRIC 1111 is the signal conduction path that allows the various components of computer 1101 to communicate with each other. Typically, this fabric is made of switches and electrically conductive paths, such as the switches and electrically conductive paths that make up busses, bridges, physical input/output ports and the like. Other types of signal communication paths may be used, such as fiber optic communication paths and/or wireless communication paths.

VOLATILE MEMORY 1112 is any type of volatile memory now known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory 1112 is characterized by random access, but this is not required unless affirmatively indicated. In computer 1101, the volatile memory 1112 is located in a single package and is internal to computer 1101, but, alternatively or additionally, the volatile memory may be distributed over multiple packages and/or located externally with respect to computer 1101.

PERSISTENT STORAGE 1113 is any form of non-volatile storage for computers that is now known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is being supplied to computer 1101 and/or directly to persistent storage 1113. Persistent storage 1113 may be a read only memory (ROM), but typically at least a portion of the persistent storage allows writing of data, deletion of data and re-writing of data. Some familiar forms of persistent storage include magnetic disks and solid state storage devices. Operating system 1122 may take several forms, such as various known proprietary operating systems or open source Portable Operating System Interface-type operating systems that employ a kernel. The code included in block 1200 typically includes at least some of the computer code involved in performing the inventive methods.

PERIPHERAL DEVICE SET 1114 includes the set of peripheral devices of computer 1101. Data communication connections between the peripheral devices and the other components of computer 1101 may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insertion-type connections (for example, secure digital (SD) card), connections made through local area communication networks and even connections made through wide area networks such as the internet. In various embodiments, UI device set 1123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smart watches), keyboard, mouse, printer, touchpad, game controllers, and haptic devices. Storage 1124 is external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage 1124 may be persistent and/or volatile. In some embodiments, storage 1124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 1101 is required to have a large amount of storage (for example, where computer 1101 locally stores and manages a large database) then this storage may be provided by peripheral storage devices designed for storing very large amounts of data, such as a storage area network (SAN) that is shared by multiple, geographically distributed computers. IoT sensor set 1125 is made up of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

NETWORK MODULE 1115 is the collection of computer software, hardware, and firmware that allows computer 1101 to communicate with other computers through WAN 1102. Network module 1115 may include hardware, such as modems or Wi-Fi signal transceivers, software for packetizing and/or de-packetizing data for communication network transmission, and/or web browser software for communicating data over the internet. In some embodiments, network control functions and network forwarding functions of network module