^past(l,k)) that precede the current audio frame X₀(l,k) in time, such that:

X
⁡
(

l
,
k

)

=

[

X
0

(

l
,
k

)

,

X
future

(

l
,
k

)

,

X
past

(

l
,
k

)

]

X
future

(

l
,
k

)

=

[

X

+
Δ

(

l
,
k

)

,

X

+
2

⁢
Δ

(

l
,
k

)

,
…

,

X

+
c

⁢
Δ

(

l
,
k

)

]

X
past

(

l
,
k

)

=

[

X

-
Δ

(

l
,
k

)

,

X

-
2

⁢
Δ

(

l
,
k

)

,
…

,

X

-
d

⁢
Δ

(

l
,
k

)

]

where c+d≥1 and Δ is a delay parameter that determines a delay between successive frames in the series of input audio frames X(l,k). In some implementations, the delay parameter Δ may be set to a value less than a frame hop associated with the speech enhancement system 200 (such as 1 sample) to ensure temporal speech correlation across the input audio frames X(l,k).

For example, given a fast Fourier transform (FFT) size of K (where K is the number of frequency bins associated with the FFT), the current audio frame X₀(l,k) can be expressed as:

FFT
⁡
(

x
[
n
]

,

x
[

n
+
1

]

,
…

,

x
[

n
+
K
-
1

]

)

=

[

X
⁡
(

l
,
0

)

,

X
⁡
(

l
,
1

)

,
…

,

X
⁡
(

l
,

K
-
1

)

]

≡

X
0

(

l
,
k

)

and the past audio frames X^past(l,k) can be expressed as:

FFT
⁡
(

x
[

n
-
Δ

]

,

x
[

n
-
Δ
+
1

]

,
…

,

x
[

n
-
Δ
+
K
-
1

]

)

≡

X

-
Δ

(

l
,
k

)

FFT
⁡
(

x
[

n
-

d
⁢
Δ

]

,

x
[

n
-

d
⁢
Δ

+
1

]

,
…

,

x
[

n
-

d
⁢
Δ

+
K
-
1

]

)

≡

X

-
d

⁢
Δ

(

l
,
k

)

and the future audio frames X^future(l,k) can be expressed as:

FFT
⁡
(

x
[

n
+
Δ

]

,

x
[

n
+
Δ
+
1

]

,
…

,

x
[

n
+
Δ
+
K
-
1

]

)

≡

X

+
Δ

(

l
,
k

)

FFT
⁡
(

x
[

n
+

c
⁢
Δ

]

,

x
[

n
+

c
⁢
Δ

+
1

]

,
…

,

x
[

n
+

c
⁢
Δ

+
K
-
1

]

)

≡

X

+
c

⁢
Δ

(

l
,
k

)

In some implementations, the speech enhancement system 220 may include a linear filter 210, DNN 220, and a nonlinear post-filter 230. The linear filter 210 is configured to produce a denoised audio frame Y(l,k) based on the series of input audio frames X(l,k). More specifically, the linear filter 210 may suppress or attenuate a noise component (N(l,k)) of the current audio frame X₀(l,k) based on a temporal correlation associated with the series of input audio frames X(l,k). In some implementations, the linear filter 210 may include a multi-frame beamformer. Example suitable multi-frame beamformers include, but are not limited to, multi-frame minimum variance distortionless response (MF-MVDR) beamformers.

Multi-frame beamformers exploit the temporal characteristics of single-channel audio signals to enhance speech. More specifically, multi-frame beamforming relies on accurate predictions or estimations of the temporal correlation of speech between consecutive audio frames (also referred to as the “interframe correlation of speech”). With reference for example to Equation 1, the speech component S(l,k) of an audio signal can be decomposed into a correlated part (a(l,k) s (l,k)) and an uncorrelated part (s′(l,k)):

S
⁡
(

l
,
k

)

=

a
⁡
(

l
,
k

)

⁢

s
⁡
(

l
,
k

)

+

s

′

(

l
,
k

)

a
⁡
(

l
,
k

)

=

Φ
SS

(

l
,
k

)

⁢
e

E
[

❘
"[LeftBracketingBar]"

s
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

2

]

=

Φ
SS

(

l
,
k

)

⁢
e

e

T

⁢

Φ
SS

(

l
,
k

)

⁢
e

where a(l,k) is an interframe correlation (IFC) vector associated with the speech component of the audio frames X(l,k), Φ_SS(l,k) is a matrix representing the covariance of the speech component, and e is a vector selecting the first column of Φ_SS(l,k). Accordingly, the multi-frame signal model can be expressed as:

X
⁡
(

l
,
k

)

=

a
⁡
(

l
,
k

)

⁢

s
⁡
(

l
,
k

)

+

s

′

(

l
,
k

)

+

N
⁡
(

l
,
k

)

where the uncorrelated speech component s′(l,k) is treated as interference.

A multi-frame beamformer may use the IFC vector a(l,k) to align the series of input frames X(l,k), for example, so that the speech component S(l,k) combines in a constructive manner (or the noise component N(l,k) combines in a destructive manner) when the input frames X(l,k) are summed together. For example, an MF-MVDR beamformer may apply a vector of weights w=[w₀, . . . , w_Q]^Tto the series of audio frames X(l,k) to produce the denoised audio frame Y(l,k):

Y
⁡
(

l
,
k

)

=

w
H

(

l
,
k

)

⁢

X
⁡
(

l
,
k

)

In some aspects, the linear filter 210 may determine a vector of weights w(l,k) that optimizes the denoised audio frame Y(l,k) with respect to one or more conditions. For example, the linear filter 210 may determine a vector of weights w(l,k) that reduces or minimizes the variance of the noise component of the audio frame Y(l,k) without distorting the speech component of the audio frame Y(l,k). In other words, the vector of weights w(l,k) may satisfy the following condition:

arg

min
w

w
⁡
(

l
,
k

)

⁢

Φ
NN

(

l
,
k

)

⁢

w
⁡
(

l
,
k

)

⁢

s
.
t
.

w
H

(

l
,
k

)

⁢

a
⁡
(

l
,
k

)

=
1

where Φ_NN(l,k) is a matrix representing the covariance of the noise component of the audio frames X(l,k). The resulting vector of weights w(l,k) represents an MF-MVDR beamforming filter (w_MVDR(l,k)), which can be expressed as:

w
MVDR

(

l
,
k

)

=

ϕ
NN

-
1

(

l
,
k

)

⁢

a
⁡
(

l
,
k

)

a

H

(

l
,
k

)

⁢

ϕ
NN

-
1

(

l
,
k

)

⁢

a
⁡
(

l
,
k

)

(
2
)

In some implementations, the linear filter 210 may estimate or track the IFC vector a(l,k) and the noise covariance matrix Φ_NN(l,k), over time, as a function of X(l,k)X^H(l,k). More specifically, the linear filter 210 may update the IFC vector a(l,k) when speech is present or otherwise detect in the input audio signal and may refrain from updating the IFC vector a(l,k) when speech is absent or otherwise not detected in the input audio signal. On the other hand, the linear filter 210 may update the noise covariance matrix Φ_NN(l,k) when speech is absent or otherwise not detected in the input audio signal and may refrain from updating the noise covariance matrix Φ_NN(l,k) when speech is present or otherwise detected in the input audio signal.

The DNN 220 is configured to infer a probability of speech p_DNN(l,k) in the current audio frame X₀(l,k) based on a neural network model, where 0≤p_DNN(l,k)≤1. In some implementations, the DNN 220 may be one example of the DNN 122 of FIG. 1. In some aspects, the linear filter 210 may update the IFC vector a(l,k) or the noise covariance matrix Φ_NN(l,k) based, at least in part, on the probability of speech p_DNN(l,k) inferred by the DNN 220. For example, the linear filter 210 may determine whether speech is present or absent in the audio signal (also referred to as voice activity detection (VAD)), and thus whether to update the IFC vector a(l,k) or the noise covariance matrix Φ_NN(l,k), based on the probability of speech p_DNN(l,k). Accordingly, the linear filter 210 may determine the vector of weights w MVDR(l,k) to be applied to the current audio frame X₀(l,k) based on the probability of speech in the previous audio frame p_DNN^(l−1,k).

In some aspects, the DNN 220 may further produce a speech signal (Z(l,k)) and a noise signal (N(l,k)) based on the probability of speech p_DNN(l,k), where the speech signal Z(l,k) represents a speech component of the denoised audio frame Y(l,k) and the noise signal N(l,k) represents a noise component of the denoised audio frame Y(l,k). For example, the DNN 220 may compute a spectral suppression gain (G_DNN(l,k)) based on the probability of speech p_DNN(l,k) and may apply the spectral suppression gain G_DNN(l,k) to the denoised audio frame Y(l,k) to produce the speech signal Z(l,k), where Z(l,k)=G_DNN(l,k) Y(l,k). The noise signal N(l,k) may be computed as a difference between the denoised audio frame Y(l,k) and the speech signal Z(l,k), where N(l,k)=Y(l,k)−Z(l,k)

In some aspects, the DNN 220 may be biased towards minimizing speech distortion, rather than maximizing noise suppression. In other words, the spectral suppression gain G_DNN(l,k) may be tuned to ensure that the speech component of the denoised audio frame Y(l,k) is not distorted in the resulting speech signal Z(l,k). For example, the DNN 220 may calculate the speech signal Z(l,k) as a function of the denoised audio frame Y(l,k), the probability of speech p_DNN(l,k), and a tuning parameter (σ) that controls the amount of noise reduction by the DNN 220:

❘
"[LeftBracketingBar]"

Z
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

=

min
⁡
(

p
DNN

(

l
,
k

)

+
σ

,
1

)

⁢

❘
"[LeftBracketingBar]"

Y
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

phase
(

Z
⁡
(

l
,
k

)

)

=

phase
(

Y
⁡
(

l
,
k

)

)

where |Z(l,k)| is the magnitude of the speech signal Z(l,k), phase (Z(l,k)) is the phase of the speech signal Z(l,k), and 0≤σ<1. In some implementations, the tuning parameter σ may be configured so that the speech signal Z(l,k) contains no speech distortion (and may thus contain some residual noise).

In the example of FIG. 2, the neural network model is trained to infer a probability of speech p_DNN(l,k). In some other implementations, the neural network model may be trained to infer the speech component Ž(l,k) of the denoised audio frame Y(l,k). In such implementations, the probability of speech p_DNN(l,k) and the speech signal Z(l,k) may be computed as a function of the denoised audio frame Y(l,k) and the inferred speech component Ž(l,k):

p
DNN

(

l
,
k

)

=

❘
"[LeftBracketingBar]"

Z
~

(

l
,
k

)

❘
"[RightBracketingBar]"

❘
"[LeftBracketingBar]"

Y
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

Z
⁡
(

l
,
k

)

=

Z
~

(

l
,
k

)

+

σ
⁢

❘
"[LeftBracketingBar]"

Y
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

The nonlinear post-filter 230 is configured to produce the enhanced audio frame S(l,k) based on the speech signal Z(l,k) and the noise signal N(l,k). More specifically, the nonlinear post-filter 230 may attenuate the residual noise in the speech signal Z(l,k) based on one or more voice activity detection (VAD) features associated with the speech signal Z(l,k) and the noise signal N(l,k). As used herein, the term “VAD feature” refers to any characteristics of the speech signal Z(l,k), the noise signal N(l,k), or any combination thereof, that reflects the presence of speech in the current audio frame X₀(l,k). In some implementations, the one or more VAD features may include a normalized difference (e (l,k)) between the speech signal Z(l,k) and the noise signal N(l,k).

Aspects of the present disclosure recognize that the speech signal Z(l,k) contains mostly target speech and the noise signal N(l,k) contains mostly background noise. As such, the normalized difference e (l,k) between the signals Z(l,k) and N(l,k) may be closer to +1 when the target speech is present in the speech signal Z(l,k) and closer to −1 when the target speech is absent from the speech signal Z(l,k):

e
⁡
(

l
,
k

)

=

Z
⁡
(

l
,
k

)

-

N
⁡
(

l
,
k

)

Z
⁡
(

l
,
k

)

+

N
⁡
(

l
,
k

)

(
3
)

In some implementations, the nonlinear post-filter 320 may use an online Gaussian mixture model (GMM) to determine a probability of speech (p_GMM(l,k)) associated with the speech signal Z(l,k) based on the normalized difference e (l,k) between the speech signal Z(l,k) and the noise signal N(l,k). For example, the normalized difference e (l,k) can be used to create a bimodal model with two Gaussian probability density functions (PDFs), including a Gaussian PDF for which the target speech is dominant and a Gaussian PDF for which the noise is dominant. The online GMM can be used to calculate a weight (w_c), mean (μ_c), and variance (σ_c) for each Gaussian PDF, where c=1 represents the Gaussian PDF for which target speech is dominant and c=2 represents the Gaussian PDF for which noise is dominant:

w
c

(

l
,
k

)

=

(

1
-

η
c

)

⁢

w
c

(

l
-
1

,
k

)

+

η
c

⁢

p
c

[

c
⁢

❘
"[LeftBracketingBar]"

e
⁡
(

l
,
k

)

,

λ
⁡
(

l
-
1

,
k

)

]

(
4
)

μ
c

(

l
,
k

)

=

(

1
-

η
c

)

⁢

μ
c

(

l
-
1

,
k

)

+

η
c

⁢

p
c

[

c
⁢

❘
"[LeftBracketingBar]"

e
⁡
(

l
,
k

)

,

λ
⁡
(

l
-
1

,
k

)

]

⁢

e
⁡
(

l
,
k

)

w
c

(

l
,
k

)

σ
c

(

l
,
k

)

=

(

1
-

η
c

)

⁢

σ
c

(

l
-
1

,
k

)

+

η
c

⁢

p
c

[

c
⁢

❘
"[LeftBracketingBar]"

e
⁡
(

l
,
k

)

,

λ
⁡
(

l
-
1

,
k

)

]

⁢

(

e
⁡
(

l
,
k

)

-

μ
c

(

l
,
k

)

)

2

w
c

(

l
,
k

)

p
c

[

c
⁢

❘
"[LeftBracketingBar]"

e
⁡
(

l
,
k

)

,

λ
⁡
(

l
,
k

)

]

=

w
c

(

l
,
k

)

⁢

p
[

e
⁡
(

l
,
k

)

⁢

❘
"[LeftBracketingBar]"

c
,

λ
⁡
(

l
,
k

)

]

∑

c
=
1

2

⁢

w
c

(

l
,
k

)

⁢

p
[

e
⁡
(

l
,
k

)

⁢

❘
"[LeftBracketingBar]"

c
,

λ
⁡
(

l
,
k

)

]

p
[

e
⁡
(

l
,
k

)

⁢

❘
"[LeftBracketingBar]"

c
,

λ
⁡
(

l
,
k

)

]

=

1

σ
c

(

l
,
k

)

⁢

√

2

⁢
π

⁢

e

-

1
2

⁢

(

e
⁡
(

l
,
k

)

-

μ
c

(

l
,
k

)

σ
c

(

l
,
k

)

)

2

p
GMM

(

l
,
k

)

=

p

c
=
1

[

c
=

1
⁢

❘
"[LeftBracketingBar]"

e
⁡
(

l
,
k

)

,

λ
⁡
(

l
,
k

)

]

where p_GMM(l,k) is a soft probability of speech at each time-frequency index, λ(l,k)={w₁(l,k), μ₁(l,k), σ1(l,k), w₂(l,k), μ₂(l,k), σ₂(l,k)}, and η_cis a learning rate step size.

In some implementations, the step size η_ccan be adaptively determined based on the denoised audio frame Y(l,k) and the probability of speech p_DNN(l,k) inferred by the DNN 220:

VAD
DNN

(
l
)

=

∑

k
=

k
min

k
max

⁢

Y
⁡
(

l
,
k

)

⁢

p
DNN

(

l
,
k

)

∑

k
=

k
min

k
max

⁢

Y
⁡
(

l
,
k

)

η
1

=

η
⁢

VAD
DNN

(
l
)

η
2

=

η
⁡
(

1
-

VAD
DNN

(
l
)

)

where η is a maximum step size that can be used for sub-band parameter tracking (and may be a tunable hyperparameter associated with the speech enhancement system 200), and k_minto k_maxrepresents a frequency range for which speech is dominant (such as 0-2 kHz).

In some implementations, a VAD based on multi-frame temporal processing (VAD_t(l)) can be expressed as a function of the current audio frame X₀(l,k) and the denoised audio frame Y(l,k):

p
t

(

l
,
k

)

=

Y
⁡
(

l
,
k

)

X
0

(

l
,
k

)

VAD
t

(
l
)

=

∑

k
=

k
min

k
max

⁢

X
0

(

l
,
k

)

⁢

p
t

(

l
,
k

)

∑

k
=

k
min

k
max

⁢

X
0

(

l
,
k

)

VAD
update

(
l
)

=

{

1

if
⁢

VAD
t

(
l
)

>

c
1

⁢

and
⁢

VAD
DNN

(
l
)

>

c
2

0

if
⁢

VAD
t

(
l
)

<

c
3

⁢

and
⁢

VAD
DNN

(
l
)

<

c
4

-
1

otherwise

where c₁, c₂, c₃, and c₄are tuning thresholds (between 0 and 1). In some implementations, VAD_update(l) may be used to indicate which sets of GMM parameters (such as for speech or noise) should be updated:

VAD
update

(
l
)

=

{

1

update
⁢

w
1

(

l
,
k

)

,

μ
1

(

l
,
k

)

,

σ
1

(

l
,
k

)

0

update
⁢

w
2

(

l
,
k

)

,

μ
2

(

l
,
k

)

,

σ
2

(

l
,
k

)

In some implementations, the nonlinear post-filter 230 may estimate the magnitude (or power) of noise (p_n(l,k)) in the speech signal Z(l,k) based on the probability of speech p_GMM(l,k) and, using spectral subtraction, determine a spectral suppression gain (G_GMM(l,k)) that can be applied to the speech signal Z(l,k) to produce the enhanced audio frame S(l,k):

P
n

(

l
,
k

)

=

p
GMM

(

l
,
k

)

⁢

P
n

(

l
-
1

,
k

)

+

(

1
-

p
GMM

(

l
,
k

)

)

⁢

❘
"[LeftBracketingBar]"

Z
⁡
(

l
,
f

)

❘
"[RightBracketingBar]"

(
5
)

G
GMM

(

l
,
k

)

=

max
⁡
(

max
⁡
(

❘
"[LeftBracketingBar]"

Z
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

-

P
n

(

l
,
k

)

,
0

)

❘
"[LeftBracketingBar]"

Z
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

,
g

)

(
6
)

S
_

(

l
,
k

)

=

G
GMM

(

l
,
k

)

⁢

Z
⁡
(

l
,
k

)

where g is a tuning parameter which represents a floor gain associated with the spectral subtraction.

Aspects of the present disclosure recognize that multiple VAD features (including the normalized difference e (l,k) between the speech signal Z(l,k) and the noise signal N(l,k)) can be combined to produce a spectral suppression gain G_GMM(l,k) that is more robust to different noisy environments. Other example suitable VAD features may include a cepstral peak, a spectral entropy, and a harmonic product spectrum (HPS) of the speech signal Z(l,k), among other examples. In some implementations, the nonlinear post-filter 230 may determine a respective probability of speech

(

p
GMM
i

(

l
,
k

)

)

associated with each VAD feature (eⁱ(l,k)), where i≥1, and calculate the spectral suppression gain G_GMM(l,k) based on the lowest probability among the probabilities of speech

p
GMM
i

(

l
,
k

)

associated with any of the VAD features eⁱ(l,k).

FIG. 3 shows a block diagram of an example nonlinear filter 300 for single-channel audio signals, according to some implementations. In some implementations, the nonlinear filter 300 may be one example of the nonlinear filter 230 of FIG. 2. More specifically, the nonlinear filter 300 may produce an enhanced audio frame S(l,k) based on a speech signal Z(l,k) and a noise signal N(l,k). As described with reference to FIG. 2, the speech signal Z(l,k) and the noise signal N(l,k) may represent a speech component and a noise component, respectively, of an audio frame Y(l,k), where Z(l,k)+N(l,k)=Y (l,k). In some implementations, the signals Z(l,k) and N(l,k) may be inferred or otherwise produced by a neural network (such as the DNN 220 of FIG. 2).

In some aspects, the nonlinear filter 300 may suppress residual noise in the speech signal Z(l,k) based on a number (M) of VAD features associated with the speech signal Z(l,k) and the noise signal N(l,k). In some implementations, the nonlinear filter 300 may suppress the residual noise in the speech signal Z(l,k) based on multiple VAD features (M>1). In the example of FIG. 3, the nonlinear filter 300 is shown to include M feature extractors 310(1)-310(M), M GMMs 320(1)-320(M), and a noise suppressor 330. Each of the feature extractors 310(1)-310(M) is configured to compute a respective VAD feature e_i(l,k), where i ∈ {1, . . . , M}, based on the speech signal Z(l,k), the noise signal N(l,k), or any combination thereof.

In some implementations, at least one of the feature extractors 310(1)-310(M) may compute a normalized difference between the speech signal Z(l,k) and the noise signal N(l,k). For example, a first VAD feature (e₁(l,k)) may be calculated according to Equation 3. Although each of the feature extractors 310(1)-310(M) is shown to receive the speech signal Z(l,k) and the noise signal N(l,k), some feature extractors may calculate a respective VAD feature e_i(l,k) based on the speech signal Z(l,k) alone. For example, one or more of the VAD features e_i(l,k) may be computed based on a power spectrum φ_ZZ(l,k) of the speech signal Z(l,k):

φ
ZZ

(

l
,
k

)

=

β
⁢

φ
NN

(

l
,
k

)

+

(

1
-
β

)

⁢

❘
"[LeftBracketingBar]"

Z
⁡
(

l
,
k

)

❘
"[RightBracketingBar]"

2

where β is a smoothing factor (between 0 and 1) and φ_NN(l,k) is a noise power spectrum, which can be estimated by averaging the power spectrum φ_ZZ(l,k) during pauses in speech (such as when speech is not detected in the speech signal Z(l,k)).

In some implementations, at least one of the feature extractors 310(1)-310(M) may compute a cepstral peak associated with the speech signal Z(l,k). For example, a second VAD feature (e₂(l,k)) may be computed as:

e
2

(

l
,
k

)

=

max
τ

{

∑

k
=
0

K
-
1

log
⁡
(

φ
ZZ

(

l
,
k

)

)

⁢

cos
⁡
(

τ
⁡
(

k
+
0.5

)

⁢
π

K

)

}

where τ represents a lag associated with the n^thsample of the l^thframe and K is the total number of frequency bins associated with the speech signal Z(l,k).

In some implementations, at least one of the feature extractors 310(1)-310(M) may compute a spectral entropy associated with the speech signal Z(l,k). For example, a third VAD feature (e₃(l,k)) may be computed as:

e
3

(

l
,
k

)

=

-

∑

k
=
0

K
-
1

φ
ZZ

(

l
,
k

)

⁢

log
⁡
(

φ
ZZ

(

l
,
k

)

)

where K is the total number of frequency bins associated with the speech signal Z(l,k).

In some implementations, at least one of the feature extractors 310(1)-310(M) may compute a harmonic product spectrum (HPS) associated with the speech signal Z(l,k). For example, a fourth VAD feature (e₄(l,k)) may be computed as:

e
4

(

l
,
k

)

=

max
τ

{

∑

r
=
1

R

log
⁡
(

φ
ZZ

(

l
,
rk

)

)

⁢

cos
⁡
(

τ
⁡
(

k
+
0.5

)

⁢
π

K

)

}

where R is a number of harmonic components corresponding to k, τ represents a lag associated with the n^thsample of the l^thframe, and K is the total number of frequency bins associated with the speech signal Z(l,k).

Each of the GMMs 320(1)-320(M) is configured to determine a probability of speech

p
GMM
i

(

l
,
k

)

,

associated with the speech signal Z(l,k), based on a respective VAD feature e_i(l,k). For example, each of the VAD features e_i(l,k) may be used to create a respective bimodal model with two Gaussian PDFs, including a Gaussian PDF for which the target speech is dominant and a Gaussian PDF for which the noise is dominant. In some implementations, each of the GMMs 320(1)-320(M) may compute the respective probability of speech

p
GMM
i

(

l
,
k

)

based on a weight w_c, mean μ_c, and variance σ_cof each Gaussian PDF associated with the corresponding VAD feature e