In some embodiments, the determination of the selections, in particular via the activation differences, and the pruning is restricted to selected layers of the neural network. As a result, layers used for feature extraction, for example, may be excluded from the pruning. As such, pruning may be reserved for particular subtasks, above all in the case of neural networks which perform various tasks (multi-task learning) or whose task fulfillment is broken down into various sub steps. An example of this is region proposal networks, which identify image sections that are initially relevant for object recognition and then classify and assess said image sections. In this connection, it may be expedient to focus the pruning on classification in order to prevent relevant image regions from being overlooked as a result of the pruning.

In some embodiments, the neural network is retrained after pruning. As a result, a functional quality of the pruned or compressed neural network may be improved. However, this does not have to involve complete training. Instead, the pruned or compressed neural network may be retrained with only a portion of the training data originally used for the training. The retraining in particular takes place by means of the central server.

In some embodiments, the elements are deactivated for pruning at least temporarily. For this purpose, parameters of the element to be pruned are, for example, set to zero, such that the element in the neural network no longer influences a result in the subsequent layer or output layer of the neural network. This has the benefit that the deactivation is easier to reverse compared to removal of an element of the neural network. As such, if, for example, it turns out that a functional quality of the neural network is impaired too much after an element has been deactivated, said element may be reactivated. In particular, the reactivated element may subsequently be marked such that it is not deactivated and/or pruned again during subsequent runs of the method. In the other case, the deactivated element may be removed in a later step, i.e., a structure of the neural network is adapted. This is done, in particular, if a functional quality or other target variables are achieved in spite of the deactivated element.

In some embodiments, it is provided for the pruning to only take place if at least one trigger criterion is met. As a result, the pruning may always take place if a particular state is achieved. In particular, a constant or frequent pruning and/or a selection of the elements of the neural network to be pruned that varies too strongly may be prevented as a result. The trigger criterion may, for example, be a predefined number of elements in the ranking. Furthermore, the trigger criterion may additionally or alternatively be a convergence of the ranking. In this case, “convergence of the ranking” means that the elements in the determined ranking no longer change over a predefined number of runs of the method or a predefined period of time at least for a predefined number of ranks. The presence of the trigger criterion is checked, in particular, by means of the central server.

In some embodiments, the pruned neural network is subsequently transmitted to at least one member of the fleet. This takes place, for example, in the form of a digital data packet that is transmitted via the communication interfaces and that comprises a structure and weightings as well as parameters of the pruned neural network. The members of the fleet receive the pruned neural network and may then replace the neural network stored in a respective memory apparatus with the received pruned neural network. Subsequently, the method may be carried out again on the pruned neural network.

In some embodiments, the determination of the ranking is designed as an iterative process. In this case, the transmission and merging (or aggregation step) on the central server is followed by a redistribution of the ranking, for example as a table, to the members of the vehicle fleet. The members of the fleet then continue or update the ranking until a relevant deviation from the previously distributed ranking occurs and the next aggregation step is triggered. This process ends when the ranking does not change or when the ranking only changes marginally.

Other features of the embodiment of the member of the fleet and of the central server as well as of the system will become apparent from the description of further embodiments of the method. The benefits of the member of the fleet and of the central server as well as of the system are in each case the same as in the embodiments of the method.

In the following, the invention is explained in greater detail based on further exemplary embodiments and with reference to the drawings.

Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS.

FIG. 1 is a schematic representation of an embodiment of the system 1. The system 1 comprises multiple members 2 of a fleet in the form of motor vehicles 50 as well as a central server 20.

Each member 2 of the fleet comprises a computing apparatus 3, a memory apparatus 4, and a communication interface 5. Furthermore, each member 2 of the fleet comprises a sensor system 6, for example in the form of a camera 51, which provides camera data as input data 7 via an input interface 8, which data are fed from the input interface 8 to the computing apparatus 3. A neural network 9, i.e. a structure, weightings, and parameters that uniquely describe the neural network 9, is stored in each memory apparatus 4. The computing apparatus 3 may perform computing operations in the memory apparatus 4 and, in particular, operates the neural network 9.

The central server 20 comprises a computing apparatus 21, a memory apparatus 22, and a communication interface 23.

The computing apparatus 3 of the members 2 of the fleet is designed to locally execute the neural network 9 stored in the memory apparatus 4 on the input data 7 and during at least one inference phase each determine a selection 10 of elements of the neural network 9 that should be pruned in each case. Once the selection has been made, the computing apparatus 3 transmits the determined selection 10 via the communication interface 5 to a central server 20.

The computing apparatus 21 of the central server 20 is designed to receive selections 10 of elements of the neural network 9 transmitted by the respective members 2 of the fleet via the communication interfaces 5, 23, to merge said selections, and to generate a merged selection 11. The neural network 9 is subsequently pruned by the computing apparatus 21 on the basis of the merged selection 11, such that a pruned neural network 12 is generated.

The pruned neural network 12 may then be transmitted to the members of the fleet via the communication interfaces 5, 23. The computing apparatuses 3 of the members 2 of the fleet may then replace the neural network 9 in the memory apparatus 4 with the pruned neural network 12.

It may be provided that the selections 10 be transmitted to the central server 20 if at least one transmission criterion 13 is met. A transmission criterion 13 may, for example, be a specific number of locally collected or selected elements of the neural network 9. A transmission criterion 13 may also require that a selection no longer change after multiple local runs or inference phases.

It is possible for the members 2 of the fleet to each produce a ranking 14 of the selected elements and for the selection 10 to be transmitted to the central server 20 in the form of the produced ranking 14, wherein the central server 20 produces a merged ranking 15 on the basis of the transmitted rankings 14 in order to carry out the merging. The neural network 9 is then pruned on the basis of the merged ranking 15.

It is possible for the neural network 9 to be fed input data 7 that vary with time in order to determine the respective selection in each case, wherein temporal activation differences of elements of the neural network 9 are determined for temporally adjacent input data 7, and wherein the elements of the neural network 9 are selected depending on the determined temporal activation differences. In particular, for the selection, a ranking 14 of the elements of the neural network 9 may be produced on the basis of the respective temporal activation differences determined for the elements. The selection 10 then comprises the ranking 14 produced in this manner.

It is possible for the neural network 9 to be retrained in the central server 2 after pruning.

It is possible for the elements to be deactivated for pruning at least temporarily. For this purpose, parameters of the element are, for example, set to zero, such that the element in the neural network 9 no longer influences a result in the subsequent layer or output layer of the neural network 9. As such, if, for example, it turns out that a functional quality of the neural network 9 is impaired too much after an element has been deactivated, said element may be reactivated. In particular, the reactivated element may subsequently be marked such that it is not deactivated and/or pruned again during subsequent runs of the method. In a later step, in particular after the neural network 9 has been retrained, the deactivated element may be removed, i.e. a structure of the neural network 9 is adapted such that the pruned neural network 12 is generated. This is done, in particular, if a functional quality or other target variables are achieved in spite of the deactivated element.

Furthermore, it is possible for the pruning to only take place if at least one trigger criterion 30 is met. The trigger criterion 30 may, for example, consist in a predefined number of elements in the merged ranking 15 being reached. Furthermore, the trigger criterion 30 may additionally or alternatively be a convergence of the merged ranking 15. In this case, “convergence of the merged ranking 15” means that the elements in the merged ranking 15 no longer change over a predefined number of runs of the method or a predefined period of time at least for a predefined number of ranks. The presence of the trigger criterion 30 is checked, in particular, by means of the central server 20.

FIG. 2 is a schematic flow diagram for illustrating an embodiment of the method for compressing a neural network 9. The method is carried out by means of a system 1 that is designed in the same way as the embodiment shown in FIG. 1, for example. Part of the method is carried out in each of the members 2 of the fleet, and another part in the central server 20. The flow diagram is explained based on the example of input data 7 that are provided in the form of a video 40 consisting of video frames 41. The video frames 41 each correspond to a point in time t_i.

In a method step 100, the video frames 41 are fed to the neural network 9. This is shown for two adjacent time increments, i.e. for a video frame 41 that corresponds to the point in time t_iand for a video frame 41 that corresponds to the following point in time t_i+1. The neural network 9 is applied to the video frames 41 and infers a result at outputs of an output layer of the neural network 9. The result may, for example, comprise object recognition or classification and/or the creation of bounding boxes for the recognized objects. During the inference, values for activations 43 of elements of the neural network 9 are acquired or read out. If the neural network 9 is designed as a convolutional neural network, the activations 43 correspond to the respective values at outputs of filters of the convolutional neural network. By way of example, the result is in each case provided as a list, in which the activation 43 associated with each element of the neural network 9 is stored for the points in time t_iand t_i+1in question.

In a method step 101, a temporal activation difference 44 is determined for the two points in time t_iand t_i+1in question from the values for the activations 43 of the individual elements. This is done element by element for all elements of the neural network 9. In particular, an amount of a difference between the values of the activation 43 of the individual elements is considered. Subsequently, a temporal activation difference 44 in relation to the two points in time t_iand t_i+1is available for each of the elements of the neural network 9. The result is provided, for example, as a list, in which the temporal activation difference 44 for each element of the neural network 9 is stored for the points in time t_iand t_i+1in question.

It is possible for the temporal activation differences 44 to be averaged and for the subsequent method steps 102-106 to be carried out proceeding from the averaged activation differences 44.

It is possible for at least some of the determined activation differences 44 to be determined depending on at least one influencing parameter 45. For example, a speed (provided, e.g., by a GPS sensor), current weather (provided, e.g., by a rain sensor), and/or a steering angle (provided, e.g., by a steering angle sensor) may be used to amplify or attenuate a determined activation difference 44 depending on the situation.

In a method step 102, the determined temporal activation differences 44 are sorted according to size. This produces a ranking 14 in which the elements of the neural network 9 with the largest temporal activation differences 44 occupy the upper ranks.

In a method step 103, the produced rankings 14 are transmitted by the members 2 of the fleet via communication interfaces to the central server 20.

In the central server 20, in a method step 104, a merged ranking 15 is generated on the basis of the transmitted rankings 14. The merging, which may also be referred to as aggregation, may in particular comprise the formation of an arithmetic average value, geometric averaging methods (e.g. centroids of vectors formed from the individual rankings 14 may be formed), moment-based averaging methods, swarm-based averaging methods, geographic averaging methods, and/or safety-focused averaging methods.

It is possible for the merged ranking 15 to be produced taking into account at least one target variable 46. The at least one target variable 46 may, for example, relate to a robustness of the neural network 9. Moreover, elements or filters that have a greater influence within the neural network 9 may be retained in spite of a large temporal activation difference 44. Furthermore, elements or filters that resemble a functionality according to other elements or filters (e.g. filters that filter camera data by way of a convolution with a filter function) may be shifted further up in the merged ranking 15, i.e. they may take preference during pruning or removal. Conversely, elements or filters that do not resemble other elements or filters may be shifted further down in the merged ranking 15, i.e. less preferentially deleted. Other target variables 46 may be plurality-oriented, i.e. ensure that many different features are recognized or filtered out in the input data 7. Accordingly, a merged ranking 15 may be adapted in such a way that many different features or a minimum number of different features are recognized or filtered out. Another target variable 46 may also be a performance of the neural network 9.

It is possible for the elements to be deactivated for pruning at least temporarily. In a subsequent method step, for example after successful retraining, the neural network 9 may then be pruned.

In a method step 105, the neural network 9 is pruned in that elements of the neural network 9 with the largest temporal activation differences 44 according to the merged ranking 15 are removed from the structure of the neural network 9. As a result, a pruned neural network 12 is provided.

It is possible, in a method step 200, to carry out a preliminary check as to whether at least one trigger criterion 30 is met. The neural network 9 is only pruned if the trigger criterion 30 is met.

In a method step 106, the pruned neural network 12 may be retrained in order to increase a functional quality of the pruned neural network 12.

The method steps 100-106 described are carried out for additional points in time t_i+x. It is in particular provided for the pruning to be carried out in method step 105 on the basis of an merged ranking 15 that was produced for average values of the temporal activation differences 44. In this connection, an average is in particular formed over multiple points in time t_i.

In particular, the method steps 100-106 are repeated cyclically, wherein current input data 7 are used in each case. In particular, the method steps 100-106 are repeated using the (retrained) pruned neural network 12, wherein the neural network 9 is exchanged for the respective (retrained) pruned neural network 12 for this purpose.

The neural network 9 may be compressed by means of the above-described embodiment of the method, wherein a stability of the neural network 9 increases at the same time, since unstable elements of the neural network 9 are removed or deactivated.

In the following, the determination of a ranking proceeding from temporal activation differences will be explained with the aid of a mathematical example. It is assumed that the input data consist of a video sequence consisting of video frames. Depending on the number of video frames per unit of time (e.g. per second), there are only minor changes in adjacent video frames. The method takes benefit of this and uses it to prune the neural network based on stability. The method is applied, in particular, to a well trained neural network, in particular a convolutional neural network (CNN). Filters of the CNN, in particular, are considered as elements of the neural network. In this connection, filters (also referred to as filter kernels) whose activations exhibit a significant change in the activation (i.e. for which a temporal activation difference is large) for adjacent video frames, i.e. for input data that vary with time, are considered unstable. In the embodiment of the method, filters of this kind are placed higher in a ranking. Only unmarked (unlabeled) input data are required as input data, for example video frames of a video sequence detected by means of a camera for detecting the surroundings of a motor vehicle.

A (convolutional) layer in particular comprises multiple filters (also referred to as filter kernels), wherein each filter in particular receives an entire output of a preceding layer and each filter provides an associated feature map as the output.

For a video frame in a sequential data set (video sequence) with a height H, a width W, channels C, and a point in time t, with t∈ [Image Omitted] :[1 . . . T], x_t∈G^H×W×Cis defined as a video frame of a data set x, wherein G={0≤z≤255|z∈N}. The video frame x_tis the input (i.e. corresponds to the input data at the input) of a neural network (⋅,θ), wherein θ are parameters of the neural network. Said neural network [Image Omitted] (⋅,θ) comprises L∈ (convolutional) layers, the outputs of which are Ψ_l(⋅)∈^H^l^×W^l^×k^l, with a height H_l, a width W_l, and a number of feature maps k_lfor l∈{1, . . . , L}.

Ψ_l,jdenotes the jth feature map of the output of the layer l, i.e. the feature map pertaining to the filter j of the layer l, wherein j∈{1, . . . , k_l}. [Image Omitted] _t={Ψ_l,j}∀l={1, . . . , L}, j={1, . . . , k_l} may then be defined as a set of all feature maps in the neural network at a point in time t.

A stability is in particular defined as a change in the output Ψ_l,j(activation difference) in relation to a change in the associated input x_t, i.e. a ranking function rank results for a filter j in the layer l and for a point in time t:

rank
(

l
,
j
,
t

)

=

❘
"[LeftBracketingBar]"

∂

Ψ

l
,
j

(

x
t

)

∂

x
t

❘
"[RightBracketingBar]"

(
1
)

with

❘
"[LeftBracketingBar]"

·

❘
"[RightBracketingBar]"

:

ℝ

H
l

×

W
l

→

ℝ

≥
0

In simple terms, an instability is greater the higher a resulting value of the ranking function for a particular filter. The ranking function is in particular determined for each element of the neural network, in the present example for each filter.

The changes in the outputs Ψ_l,j(x_t), i.e. in the activations of the filters, may be caused by two things. Firstly, a change at the output may be brought about by a change in the video frame x_tat the input. Secondly, a change may also be brought about by a change in the activations at the output of a preceding layer ψ_l-1(x_t).

The structural similarity index measure (SSIM), which is known per se, may for example be used to calculate a difference between successive video frames (i.e. temporally adjacent input data). The SSIM is used to measure a similarity between two images.

Δ
⁢

x
t

=

1
-

SSIM
⁡
(

x
t

,

x

t
-
1

)

(
2
)

Since the stability of Ψ_l,jalso depends on a stability of the outputs (activations) of ψ_l-1, a filter in the layer l should be prevented from being pruned on account of an instability in the layer l−1. Therefore, the contribution of the preceding layer is taken into account in the ranking function. [Image Omitted] _l,tis the normalized output of the lth layer to the subsequent layer at the point in time t. This is calculated in the following equation. In order to be able to compare the changes in the outputs (activations) of the filters in the entire neural network, the outputs (activations) are normalized with a height H_l, a width W_l, and a number k_lof channels (feature maps) of the layer l.

𝒟

l
,
t

=

❘
"[LeftBracketingBar]"

ψ
l

(

x
t

)

-

ψ
l

(

x

t
-
1

)

H
l

×

W
l

×

k
l

❘
"[RightBracketingBar]"

(
3
)

The equations (2) and (3) are combined with the equation (1) in order to calculate the stability-based rank for each filter. The rank is defined here as

R

l
,
j

a

,

wherein a∈ [Image Omitted] [1, . . . , A] is a video sequence. Equation (4) defines the rank as a (temporal) activation difference, averaged over all points in time t (i.e. over a number T of video frames), with respect to a change at the inputs of the filters, i.e. with respect to a (temporal) change of the input data:

R

l
,
j

a

=

∑

t
=
1

T

⁢

1
T

⁢

❘
"[LeftBracketingBar]"

Ψ

l
,
j