CROSS REFRENCE

The present application claims the benefit under 35 U.S.C. § 102020200340.8 filed on Jan. 14, 2020, which is expressly incorporated herein by reference in its entirety.

BACKGROUND INFORMATION

Monitored learning is used as the basis for many applications, including the classification of camera images, the detection of objects in sensor measurements, or for voice-to-text conversions.

Monitored learning usually requires a large set of so-called training data that include labels, i.e., annotations. Obtaining these training data may be very complicated.

SUMMARY

In accordance with an example embodiment of the present invention, a computer-implemented method for active machine learning provides that annotated data are provided, a model being trained for a classification of the data as a function of the annotated data, the model trained in this way being calibrated, as a function of the annotated data, with regard to confidence for a correctness of the classification of the annotated data by the model, for unannotated data, the confidence for the correctness of the classification of the unannotated data being determined, using the model calibrated in this way, and the unannotated data for the active machine learning whose confidence satisfies a criterion being acquired from the unannotated data. The overall required annotation effort is thus reduced.

A set of unannotated data is preferably provided, a subset being selected from the set of unannotated data, the annotated data being determined from the subset by in particular manual, semi-automatic, or automatic annotation of unannotated data. Given an initially unlabeled, i.e., unannotated, random sample, only a small fraction of the data is labeled, i.e., annotated, in the first step. This subset may be randomly selected, for example.

The subset preferably includes the acquired unannotated data for the active machine learning. The next fraction to be annotated is thus selected with the aid of a suitable acquisition function.

The confidence may define the confidence as a function of at least one probability that the classification by the model is correct. In this way, particularly well-suited data are selected for the training.

For a sample from the unannotated data, the criterion preferably defines two classes having the highest probability compared to the other classes in which the sample is classifiable by the model, the samples for which a difference between the probabilities of the two classes exceeds a threshold value being acquired from the unannotated data. This represents a quicker and better acquisition function for deep neural networks in particular. For example, the trained deep neural network is initially calibrated with the aid of temperature scaling, and a confidence difference is subsequently used as an acquisition function.

The model is preferably iteratively trained and calibrated, a check being made as to whether an abort criterion is satisfied, and the active machine learning being ended when the abort criterion is satisfied. This progressively improves the accuracy.

The abort criterion may define a reference for an accuracy of the classification of annotated or unannotated data by the model, the abort criterion being satisfied when the accuracy of the classification reaches or exceeds the reference. Iterative training is thus carried out until the desired accuracy is achieved.

At least one parameter of the model is preferably calibrated as a function of an expected proportion of correct classifications for the annotated data and an empirical proportion of correct classifications of the annotated data.

Unannotated data are preferably randomly selected in a first iteration of the method for a determination of the annotated data.

Preferably only data that are not already acquired for the subset are selected from the unannotated data for the subset.

A device for active machine learning is designed to carry out the method, in accordance with an example embodiment of the present invention.

Further advantageous specific embodiments result from the following description and the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic illustration of a device for active machine learning, in accordance with an example embodiment of the present invention.

FIG. 2 shows steps in a method for active machine learning, in accordance with an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Device 100 for active machine learning in accordance with an example embodiment of the present invention, illustrated in FIG. 1 is designed to carry out a method described below.

Device 100 includes a processing device 102, in particular one or multiple processors, and at least one memory for data. Device 100 includes a model 104.

Device 100 may be designed for digital image processing in which digital images are classified by model 104. In the following discussion, an acquisition function that is based on model 104 is used. In the example, model 104 is a single deep neural network. Model 104 may also be formed from multiple in particular deep neural networks, or may have some other architecture.

A system 106 for machine learning may include device 100, a detection device 108 for detecting digital images, and a control device 110 for a machine. System 106 may be designed to control the machine into a class of multiple classes as a function of a classification of a detected digital image. The machine may be a vehicle or a robot. Instead of a classification of digital images, classification for a detection of objects in sensor measurements or for voice-to-text conversions may be provided. Instead of detecting digital images, general sensor measurements, such as 3D sensor measurements by radar or LIDAR sensors, for example, may be used.

A computer-implemented method for active machine learning is described with reference to FIG. 2. Model 104 is trained using the method, and may be subsequently used in system 106. The method assumes that the architecture of model 104 is fixed and parameters of model 104 are already initialized. In the example, model 104 is a deep neural network whose hyperparameter defines an input layer, an output layer, and a plurality of concealed layers in between. The parameters of the deep neural network are determined as follows.

Deep neural networks in particular may be poorly calibrated. To determine a suitable calibration, the expected proportion of correct predictions is compared to the empirical proportion of correct predictions, based on the confidence. The confidence that is output by the deep neural network does not match the empirical accuracy on a test random sample.

In the example, the temperature scaling method is used for the calibration. This is described in Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (August 2017), “On calibration of modern neural networks,” Proceedings of the 34th International

Conference on Machine Learning, Volume 70 (pp. 1321-1330), JMLR.org.

This means that the calibration is carried out after a training of model 104. After the calibration, the confidence of the network matches the empirical accuracy better than prior to the calibration.

In the example, a difference of the two classes having the highest probability is formed as described in Luo, Tong, et al., “Active learning to recognize multiple types of plankton,” Journal of Machine Learning Research 6 (April 2005): 589-613.

The difference between the two classes is used in the method as an acquisition function. This acquisition function is better and quicker than the formation of an ensemble average value, for example. The calibrated difference therefore yields better results.

The method is based on monitored learning. Training data including labels, i.e., annotations, are necessary for monitored learning. These annotations are used as target outputs to allow application of an optimization algorithm. Depending on the application, the creation of such a training data set is very complicated. The labeling of 3D sensor measurements, for example of point clouds that are deliverable by a radar or LIDAR sensor, is very complicated and requires expert knowledge. In addition, in the field of medical imaging it may be very complicated to obtain training data from digital images.

The parameters of model 104, i.e., the deep artificial neural network in the example, may be randomly initialized.

A set of unannotated data is provided in a step 202. The unannotated data include patterns, in the example digital images or their representation. A step 204 is subsequently carried out.

A subset for determining the annotated data is selected from the set of unannotated data in a step 204. The subset includes the acquired unannotated data for the active machine learning.

In a first iteration of the method, the subset is randomly selected from the set of unannotated data in step 204.

Preferably only data that are not already acquired for the subset in a previous iteration are selected from the unannotated data for the subset in step 204.

A step 206 is subsequently carried out.

Annotated data are provided in a step 206. The annotated data may be determined from the subset by manual, semi-automatic, or automatic annotation of unannotated data. For this purpose, for example the digital images are displayed and provided with a label by a human. Automated labeling methods are also usable.

A step 208 is subsequently carried out.

Model 104, the deep neural network in the example, is trained for a classification of the data in step 208 as a function of the annotated data. The training takes place, for example, with the aid of gradient descent methods, for example ADAM, by monitored learning using the annotated data from the subset.

A step 210 is subsequently carried out.

Model 104 trained in this way is calibrated in step 210, as a function of the annotated data, with regard to confidence for a correctness of the classification of the annotated data by the model. In the example, the confidence is defined as a function of at least one probability that the classification by the model is correct.

For example, at least one parameter of model 104 is calibrated as a function of an expected proportion of correct classifications for the annotated data and an empirical proportion of correct classifications of the annotated data.

In the example, the temperature scaling is used, a perfect calibration for an input X and label Y being indicated as

P(Ŷ=Y|{circumflex over (P)}=p)=p,∀p∈[0.1]

where Ŷ is a classification and {circumflex over (P)} is the confidence associated with same.

The expected proportion of correct classifications is indicated by an expected value thereof. The empirical proportion of correct classifications is determined as a function of the annotated data and their classification by model 104. For the calibration, the parameters of model 104, i.e., of the deep neural network, are determined which minimize an expected calibration error indicated below:

E(|P(Ŷ=Y|{circumflex over (P)}=p)−p|)

A step 212 is subsequently carried out.

The confidence for the correctness of the classification of the unannotated data is determined for unannotated data in step 212, using the model calibrated in this way.

The unannotated data for the active machine learning whose confidence satisfies a criterion are acquired from the unannotated data.

In the example, active learning is used according to the “breaking ties” approach in order to improve the confidence of the multiclass classification. Accordingly, an argmax_pP(p) is associated with a class designation x in such a way that P(a) indicates the highest probability and P(b) indicates the second-highest probability for a class a or a class b, respectively. In this case, the data for which the difference between P(a) and P(b) is smallest are eliminated from the unannotated data.

For a sample from the unannotated data, the criterion defines two classes, in the present case class a and class b, which have the highest probability, P(a) and P(b) in the present case, compared to the other classes into which the sample is classifiable by the model. The samples for which a difference between the probabilities of two classes, difference P(a)−P(b) in the present case, exceeds a threshold value are acquired from the unannotated data. The threshold value may be defined as a function of the probabilities for the other of the classes. For example, the threshold value is defined by a difference between the probabilities of two classes, which are compared to difference P(a)−P(b) between highest probability P(a) at that moment and second-highest probability P(b) at that moment.

Model 104 is preferably iteratively trained and calibrated.

A check may be made in a step 214 as to whether an abort criterion is satisfied, and the active machine learning is ended when the abort criterion is satisfied.

This means that in the example, steps 202 through 214 are carried out repeatedly in this order or in some other order.

The abort criterion may define a reference for an accuracy of the classification of annotated or unannotated data by the model. The abort criterion is satisfied, for example, when the accuracy of the classification reaches or exceeds the reference.

The calibration of the deep neural network may be checked, for example, when high-quality annotated data are present. For example, the confidence of the deep neural network, for example 0.8, is compared to the empirical accuracy. For a correct calibration, this would be 80% correct predictions, for example.

Different acquisition functions may be used for a given network architecture. These acquisition functions may be compared by determining which set of training data is necessary for achieving a certain accuracy.