in∈

^T×F×2, where T represents the number of time periods in time domain and F represents the number of frequency bands in frequency domain. As shown in FIG. 2, the time-frequency representation 201 is inputted into the classification network 250.

In some implementations, the classification network 250 may include one or more convolution layers 210, for example, 1×7 convolution layers and 7×1 convolution layers. “m×n convolution layers” refer to convolution layers with a convolution kernel size of m×n herein. The time-frequency representation 201 is fed into the one or more convolution layers 210 to generate a feature representation S⁰∈ [Image Omitted] ^T×F×C, where C represents the number of channels. In some other implementations, the classification network 250 may also not include one or more convolution layers 210. The time-frequency representation 201 may be directly inputted to the correlation unit 220-1.

The correlation unit 220-1 may generate a feature representation S¹based on the time-frequency representation 201 or the feature representation S⁰. Other correlation units 220-2 to 220-N each may generate a new feature representation based on the feature representation of the preceding correlation unit. Generally, the feature representation generated by the i^thcorrelation unit 220 may be represented as Sⁱ, where i∈{1, 2, . . . , N}. In some implementations, for example, in offline implementations, the feature representation Sⁱmay have the same dimensions as the feature representation S⁰, that is, S^N∈ [Image Omitted] ^T×F×C. In other implementations, for example, in online implementations, the feature representation Sⁱmay be less than the feature representation S⁰in time dimension, which will be described with reference to FIG. 5 below.

The one or more correlation units 220 may use the frequency correlation and the time correlation of the audio signal to generate the feature representation Sⁱ. In some implementations, both the frequency correlation and the time correlation may be simultaneously used in each correlation unit 220. In some implementations, both the frequency correlation and the time correlation may be simultaneously used in at least one correlation unit, and only the frequency correlation is used in other correlation units. For example, both the frequency correlation and the time correlation may be simultaneously used in the first correlation unit 220-1, and only the frequency correlation is used in other correlation units 220-2 to 220-N.

In this specification, feature representations S¹, S². . . S^N-1generated by the first (N−1) correlation units are also referred to as “intermediate feature representations,” and the feature representation S^Ngenerated by the last correlation unit 220-N is also referred to as the “target feature representation.” The intermediate feature representation and the target feature representation are used to distinguish the speech component and the noise component.

The feature representation S^Ngenerated by the last correlation unit 220-N is fed to a mask generation layer 230. The mask generation layer 230 generates a mask M∈T×F×C_mbased on the feature representation S^N, where C_mis the number of channels. As one example, the mask generation layer may be a head layer, which may be implemented as a 1×1 convolution layer with C_m=2.

As shown in FIG. 2, the classification network 250 generates the mask M as an output. Then, the mask M is applied to at least one part of the time-frequency representation 201 to determine a masked time-frequency representation S^out. It should be appreciated that in the masked time-frequency representation S^out, the speech component is enhanced relative to the noise component. Therefore, the masked time-frequency representation S^outmay also be referred to as an enhanced time-frequency representation.

In some implementations, for example, in offline implementations, the mask M may be applied to the complete time-frequency representation 201. As one example, if the mask generation layer 230 is implemented as a 1×1 convolution layer with C_m=2, then the masked time-frequency representation S^outmay be determined as below:

S^out=f(M)⊙g(Sⁱⁿ) (1)

where f(M)=tanh(|M|)*M/|M|, g(Sⁱⁿ)=Sⁱⁿand ⊙ denotes complex multiplication.

In other implementations, for example, in online implementations, the time-frequency representation 201 may include the first time-frequency representation of the processed audio signal and the second time-frequency representation of the input audio signal 170 as described above. Therefore, in such implementations, the mask M may only be applied to a part of the time-frequency representation 201, which corresponds to the second time-frequency.

The masked time-frequency representation S^outis fed to a decoder 240 and converted into an output audio signal 180 by the decoder 240. In some implementations, the decoder 240 may be a decoder based on fixed transform. For example, the decoder 240 may implement an ISTFT corresponding to an STFT which is used to generate the time-frequency representation of the input audio signal 170.

In some implementations, the decoder 240 may be a trained decoder. Such a decoder 240 may be trained to transform intensities of an audio signal at different frequencies over time into a waveform over time. Although a decoder does not need to perform exactly opposite operations of an encoder, the following method may be used to ensure the convergence of an encoder network in some implementations. For example, the network structure of the decoder 240 may be configured as the same convolution implementation as ISTFT. Specifically, the encoder 240 may be implemented as a one-dimensional transposed convolution layer whose kernel size and stride are the window length and hop size used in the STFT.

In such implementations, the decoder 240 may be trained together with the classification network 250. The system 200 may be implemented as a fully learnable end-to-end system. In this way, the learnability of the system 200 is increased, thereby helping to improve the performance of speech enhancement.

Attention Mechanism

The overall architecture of the system 200 for speech enhancement has been described above. Description is presented below to some implementations of the correlation unit 220. FIG. 3 shows a block diagram of the correlation unit 220 in accordance with some implementations of the subject matter described herein. The correlation unit 220 may determine an output feature representation Sⁱbased on an input feature representation S^i-1.

Generally, the correlation unit 220 may include two parts. The first part includes a group of convolution layers 311, 312, 313 and 314, and the second part includes an attention block 320. In the training process of the system 200, the convolution layers 311, 312, 313 and 314 may learn the partial correlation of an audio signal. Therefore, when processing the input audio signal 170, the partial correlation of the input audio signal 170 may be used via the convolution layers 311, 312, 313 and 314. As one example, the convolution layers 311, 312, 313 and 314 may be implemented as 3×3 convolution layers.

As shown in FIG. 3, in some implementations, a skip connection, also referred to as a residual connection may be employed between every two convolution layers. With the skip connection, the performance of speech enhancement can be improved by increasing the network depth.

The above described number of convolution layers, connection mode and convolution kernel size are merely schematic and not intended to limit the scope of the subject matter described herein. In accordance with implementations of the subject matter described herein, any proper number of convolution layers and any proper convolution kernel size may be used. In some implementations, different correlation units 220 may have different numbers of convolution layers.

The second part of the correlation unit 220 may include the attention block 320. During the training process of the system 200, the attention block 320 may learn the long-range correlation (also referred to as global correlation) of an audio signal based on the attention mechanism. Therefore, when processing the input audio signal 170, the long-range correlation of the input audio signal 170 may be used via the attention block 320. Specifically, the attention block 320 receives from the convolution layer 314 a feature representation S^conwhich is generated using the partial correlation, and determines the feature representation Sⁱbased on the feature representation S^conand the long-range correlation with the attention mechanism.

In the time-frequency representation (e.g., spectrogram), the long-range correlation exists both in time domain (i.e., along a time axis) and frequency domain (i.e., along a frequency axis). It should be appreciated that as time series, the audio signal contains the global correlation along the time axis and contains the harmonic correlation along the frequency axis. In this specification, the correlation along the time axis is also referred to as time correlation, and the correlation along the frequency axis is also referred to as frequency correlation.

The attention block 320 may use the time correlation and the frequency correlation in parallel. Such attention is also referred to as “dual-path attention block (DAB).” A light-weight DAB solution is provided herein in order to make full use of the time correlation and the frequency correlation without causing too much computational consumption.

FIG. 4 shows a block diagram of the attention block 320 in accordance with some implementations of the subject matter described herein. In the example of FIG. 4, a feature representation 401 (also denoted as S^con) from the convolution layer 314 has T×F×C dimensions. As described above, T means that the time-frequency representation 201 is associated with T frequency bands in time domain, F means that the time-frequency representation 201 is associated with F frequency bands in frequency domain, and C represents the number of channels. It should be appreciated that for the correlation unit 220-1, the feature representation 401 is the convolved time-frequency representation 201; for the other correlation units 220-2 to 220-N, the feature representation 401 is the feature representation generated by a preceding correlation unit, that is, the intermediate feature representation mentioned in this specification.

On the frequency path, the two-dimensional feature representation 401 with C channels is decomposed into TC (i.e., T×C) frequency feature representations 410-1 to 410-8, which are collectively or separately referred to as frequency-domain feature representations 410. Each of the frequency-domain feature representations 410 corresponds to one of T time periods and includes multiple features in the F frequency bands. Each of the frequency-domain feature representations 410 may be implemented as a vector with dimension 1×F.

As shown by an arrow 420, the attention mechanism is applied to each frequency-domain feature representation 410 along the frequency axis to obtain an updated frequency-domain feature representation 430. In this specification, the set of frequency-domain feature representations updated by applying the attention mechanism is referred to as frequency correlation information 430. If there are multiple correlation units 220, the frequency correlation information in the attention blocks 320 of the second to Nth correlation units 220 may also be referred to as the updated frequency correlation information.

In some implementations, a self-attention mechanism may be used to learn the attention map for each frequency-domain feature representation (may be considered as a sample) 410. Each element in the attention map indicates the correlation of an audio signal in corresponding frequency bands.

Inventors of the present disclosure have realized that the harmonic correlation of an audio signal in frequency domain is sample-independent, and in some classical speech enhancement solutions, usually a uniform non-linear function is applied along the frequency axis to reproduce harmonics. Therefore, in some implementations, the sample-independent attention mechanism may be applied along the frequency axis. Specifically, in such implementations, applying the attention mechanism shown by the arrow 420 may be implemented through a fully connected layer. In the process of training the system 200 with a reference audio signal, a parameter of the fully connected layer may be learned. In other words, the parameter of the fully connected layer indicates the correlation of the reference audio signal in the F frequency bands. The process of applying the sample-independent attention mechanism to a certain frequency-domain feature representation 410 corresponds to the process of multiplying a 1×F vector by an F×F weight matrix. In this specification, the parameter of the fully connected layer indicating the frequency correlation is also referred to as frequency-domain weighting information.

Likewise, on the time path, the two-dimensional feature representation 401 with C channels is decomposed into FC (i.e., F×C) time-domain feature representations 440-1 to 440-8, which may also be collectively or separately referred to as time-domain feature representation 440. Each time-domain feature representation 440 corresponds to one of F frequency bands and includes multiple features in T time periods. Each time-domain feature representation 440 may be implemented as a vector with dimension T×1.

As shown by an arrow 450, the attention mechanism is applied to each time-domain feature representation 440 along the time axis to obtain an updated time-domain feature representation 440. In this specification, the set of time-domain feature representations updated by applying the attention mechanism is referred to as time correlation information 460. If there are multiple correlation units 220, the time correlation information in the attention blocks 320 of the second to Nth correlation units 220 may also be referred to as the updated time correlation information.

In some implementations, a self-attention mechanism may be used to learn the attention map for each time-domain feature representation (may be considered as a sample) 440. Each element in the attention map indicates the correlation of an audio signal in corresponding time periods.

Like sample-independent harmonic correlation, inventors of the present disclosure have realized that the time correlation is time-invariant. Therefore, in some implementations, the sample-independent attention mechanism may also be applied along the time axis. Specifically, in such implementations, applying the attention mechanism shown by the arrow 450 may be implemented through a fully connected layer. In the process of training the system 200 with a reference audio signal, a parameter of the fully connected layer may be learned. In other words, the parameter of the fully connected layer indicates the correlation of the reference audio signal in T time periods. The process of applying the sample-independent attention mechanism to a certain time-domain feature representation 440 corresponds to the process of multiplying a T×1 vector by a T×T weight matrix. In this specification, the parameter of the fully connected layer indicating the time correlation is also referred to time-domain weighting information.

In some implementations, the sample-independent attention mechanism may be applied on both the frequency and time paths. In such implementations, the attention block 320 may also be referred to a sample-independent dual-path block (SDAB). By applying the sample-independent attention mechanism, the computational complexity of the network of the system 200 may be reduced without degrading the performance of speech enhancement.

Still with reference to FIG. 4, after obtaining the time correlation information 430 and the frequency correlation information 460, the feature representation Sⁱis determined based on the frequency correlation information 430, the time correlation information 460 and the feature representation 401. Specifically, the frequency correlation information 430 including TC frequency-domain feature representations may be rebuilt as a feature representation with dimension T×F×C, and the time correlation information 460 including FC frequency-domain feature representations may be rebuilt as a feature representation with dimension T×F×C. The two rebuilt feature representations and the feature representation 401 are combined into the feature Sⁱ.

As one example, in FIG. 4, the two rebuilt feature representations and the feature representation 401 may be concatenated by channel. Then, a convolution layer 470 may be applied to the concatenated feature representation, so that the feature representation Sⁱwith dimension T×F×C is obtained. As another example, features in the same positions of the two rebuilt feature representations and the feature representation 401 may be directly added up to obtain the feature representation S with dimension T×F×C. Implementations of the subject matter described herein are not limited in this regard.

Description has been presented to DAB with reference to FIG. 4. Now reference is made back to FIG. 2. In some implementations, each of the correlation units 220-1 to 220-N may include DAB. In other words, both the time correlation and the frequency correlation may be used in each correlation unit 220. In other implementations, at least one of the correlation units 220-1 to 220-N may include DAB, and other correlation units may only use the attention mechanism on the frequency path.

As one example, the dual-path attention mechanism may only be used in the first correlation unit 220-1, and at the same time the attention mechanism may only be used on the frequency path in the following correlation units 220-2 to 220-N. Using the time correlation in the front part of the classification network 250 helps to improve the speech enhancement performance.

As mentioned above, in some implementations, the dimension of the target feature representation in the time domain may be below the dimension of the time-frequency representation 201 in the time domain. For example, in an online implementation, in order to meet real-time requirements, the duration of the input audio signal 170 is usually short. In order to apply the long-range correlation in the time domain, a processed audio signal needs to be used. FIG. 5 shows a block diagram 500 of changes of a data stream in the system 200 in accordance with some implementations of the subject matter described herein. FIG. 5 will be described in conjunction with FIG. 2. As shown in FIG. 5, the time-frequency representation 201 includes the first time-frequency representation 501 of the processed audio signal and the second time-frequency representation 502 of the input audio signal 170. In the example of FIG. 5, the unprocessed input audio signal 170 involves 4 time periods in the time domain, which corresponds to the shaded second time-frequency representation 502.

The convolution layer 210 may generate a feature representation 511 based on the time-frequency representation 201, where a shaded part 503 is associated with the second time-frequency representation 502. That is, the part 503 is determined based on the second time-frequency representation 502. The part that is not shaded is not updated, so that data for processing a previous input audio signal may be used.

The feature representation 511 is fed to the first correlation unit 220-1. The correlation unit 220-1 applies one or more convolution layers (e.g., the convolution layers 311 to 314 shown in FIG. 3) to the feature representation 511 to generate a feature representation 512. In the feature representation 512, a shaded part 504 is associated with the second time-frequency representation 502. That is, the part 504 is determined based on the second time-frequency representation 502. The part that is not shaded is not updated, so that data for processing a previous input audio signal may be used.

Next, the correlation unit 220-1 applies DAB to the feature representation 512 to generate a feature representation 513. The feature representation 513 may only include the part associated with the second time-frequency representation 502. As seen from FIG. 5, the feature representation 513 is shortened in the time domain relative to the feature representation 512.

FIG. 6 shows a block diagram 600 of generating a feature representation in accordance with some implementations of the subject matter described herein. FIG. 6 will be described in conjunction with FIGS. 4 and 5. The feature representation 512 may be considered as one implementation of the feature representation 401. On the time path, the feature representation 512 is decomposed into multiple time-domain feature representations, and time correlation information 660 is determined by applying the attention mechanism along the time axis. The time correlation information 660 may be considered as one implementation of the time correlation information 460. Partial information (also referred to as the first partial information 661) in the time correlation information 660 is associated with the second time-frequency representation 502. That is, the first partial information 661 is determined in a way that is at least partly associated with the second time-frequency representation 502.

On the frequency path, the feature representation 512 is decomposed into multiple frequency-domain feature representations, and frequency correlation information 630 is determined by applying the attention mechanism along the frequency axis. The frequency correlation information 630 may be considered as one implementation of the frequency correlation information 430. Partial information (also referred to as the second partial information 631) in the frequency correlation information 630 is associated with the second time-frequency representation 502. For example, the second partial information 631 corresponds to the first partial information 661 in the time axis.

Next, the feature representation 513 shown in gray may be determined based on the first partial information 661, the second partial information 631 and the feature representation 512 (specifically, the part of the feature representation 512 which corresponds to the first partial information 661 and/or the second partial information 631 in the time axis). It should be appreciated that the part shown in a dashed box may not be determined or outputted. The feature representation 513 is determined in a way similar to the one described with reference to FIG. 4, which is not described here.

Still with reference to FIG. 5, as compared with the time-frequency representation 201, the feature representation 513 generated by the first correlation unit 220-1 is shortened in the time domain but remains unchanged in the frequency domain. Therefore, the attention mechanism is only applied in the frequency domain in the following correlation units 220-2 to 220-6, that is, only the frequency correlation is used.

Specifically, the correlation unit 220-2 applies one or more convolution layers (e.g., the convolution layers 311 to 314 shown in FIG. 3) to the feature representation 513 to generate a feature representation 514. In the example of FIG. 5, as compared with the feature representation 513, the feature representation 514 is shortened in the time domain because of a convolution operation. Next, the correlation unit 220-2 applies the attention mechanism in the frequency domain to generate a feature representation 515.

The correlation unit 220-3 applies one or more convolution layers (e.g., the convolution layers 311 to 314 shown in FIG. 3) to the feature representation 515 to generate a feature representation 516. In the example of FIG. 5, as compared with the feature representation 515, the feature representation 516 is shortened in the time domain because of a convolution operation. Next, the correlation unit 220-3 applies the attention mechanism in the frequency domain to generate a feature representation 517.

Similar operations are performed in correlation units that are not shown. The last correlation unit 220-6 applies the attention mechanism to a feature representation 519 in the frequency domain so as to determine a feature representation 520 as a target feature representation. As seen from FIG. 6, the feature representation 520 is associated with 4 time periods in the time domain, which corresponds to the duration of the input audio signal 170.

Example implementations of using the processed audio signal to process the input audio signal with a short duration have been described with reference to FIGS. 5 and 6. It should be appreciated that the number of correlation units and the number of associated time periods in the time domain are merely schematic and not intended to limit the scope of the subject matter described herein. In this way, the time correlation and the frequency correlation may be used in parallel so as to improve the performance of speech enhancement without affecting the timeliness requirement in online scenarios.

Training of Speech Enhancement System

Still with reference to FIG. 2, the system 200 may be trained using a reference audio signal including a reference speech signal and a reference noise signal. The reference speech signal may be a pure speech signal. The reference speech signal and the reference noise signal are synthesized into the reference audio signal to train the system 200.

In some implementations, the reference noise signal may include various types of noise. The system 200 trained as such is universal and applicable to speech enhancement in various scenarios.

In some implementations, the reference noise signal may include one or more specific types of noise (e.g., engine sound and the like). The system 200 trained as such is especially suitable for speech enhancement in specific scenarios, where there are one or more specific noise.

In some implementations, the system 200 may be trained under a T-F domain architecture. That is, a loss function may be calculated between the time-frequency representation Sⁱⁿand the masked time-frequency representation S^out.

In some implementations, a cross-domain training process may be implemented. That is, the loss function may be calculated between the input audio signal 170 and the output audio signal 180. In such implementations, the speech enhancement performance of the system 200 may further be improved. For example, both the signal distortion ratio (SDR) and the perceptual evaluation of speech quality (PESQ) may be improved.

Example Method and Example Implementation

FIG. 7 shows a flowchart of a method 700 for speech enhancement in accordance with some implementations of the subject matter described herein. The method 700 may be implemented by the computing device 100, for example, may be implemented at the audio processing module 122 in the memory 120 of the computing device 100.

As shown in FIG. 7, at block 710, the computing device 100 obtains a target time-frequency representation at least indicating intensities of an input audio signal at different frequencies over time. The input audio signal comprises a speech component and a noise component. At block 720, the computing device 100 determines frequency correlation information and time correlation information of the input audio signal based on the target time-frequency representation. At block 730, the computing device 100 generates a target feature representation for distinguishing the speech component and the noise component based on the frequency correlation information, the time correlation information, and the target time-frequency representation. At block 740, the computing device 100 generates an output audio signal based on the target feature representation and the target time-frequency representation. The speech component is enhanced relative to the noise component in the output audio signal.

In some implementations, generating the target feature representation comprises: generating an intermediate feature representation for distinguishing the speech component and the noise component based on the frequency correlation information, the time correlation information and the convolved target time-frequency representation; updating the frequency correlation information based on the intermediate feature representation; and determining the target feature representation based on the intermediate feature representation and the updated frequency correlation information.

In some implementations, generating the target feature representation comprises: generating an intermediate feature representation for distinguishing the speech component and the noise component based on the frequency correlation information, the time correlation information and the convolved target time-frequency representation; updating the frequency correlation information and the time correlation information based on the intermediate feature representation; and determining the target feature representation based on the intermediate feature representation, the updated frequency correlation information and the updated time correlation information.

In some implementations, obtaining the target time-frequency representation comprises: obtaining a first time-frequency representation of a processed audio signal occurring before the input audio signal, the first time-frequency representation indicating intensities of the processed audio signal at the different frequencies over time; determining a second time-frequency representation of the input audio signal, the second time-frequency representation indicating intensities of the input audio signal at the different frequencies over time; and combining the first time-frequency representation and the second time-frequency representation into the target time-frequency representation.

In some implementations, generating the target feature representation comprises: determining first partial information of the time correlation information associated with the second time-frequency representation; determining second partial information of the frequency correlation information associated with the second time-frequency representation; and determining the target feature representation based on the first partial information, the second partial information and the target time-frequency representation.

In some implementations, the target time-frequency representation is associated with a plurality of frequency bands in frequency-domain and a plurality of time periods in time-domain. Determining the frequency correlation information and the time correlation information comprises: obtaining a frequency-domain feature representation and a time-domain feature representation by processing the target time-frequency representation, the frequency-domain feature representation comprising a plurality of features at the plurality of frequency bands over one of the plurality of time periods, the time-domain feature representation comprising a plurality of features over the plurality of time periods at one of the plurality of frequency bands; determining the frequency correlation information based on the frequency-domain feature representation and frequency-domain weighting information, the frequency-domain weighting information indicating a correlation degree of a reference audio signal among the plurality of frequency bands; and determining the time correlation information based on the time-domain feature representation and time-domain weighting information, the time-domain weighting information indicating a correlation degree of a reference audio signal among the plurality of time periods.

In some implementations, the frequency-domain weighting information and the time-domain weighting information is determined based on the reference audio signal comprising a reference speech signal and a reference noise signal.

In some implementations, generating the output audio signal comprises: determining a masked time-frequency representation by applying a mask to at least one portion of the target time-frequency representation, the mask is generated based on the target feature representation; and converting the masked time-frequency representation into the output audio signal.

In some implementations, converting the masked time-frequency representation into the output audio signal comprises: converting the masked time-frequency representation into the output audio signal by applying the masked time-frequency representation to a trained decoder, wherein the trained decoder is configured to convert intensities of an audio signal at the different frequencies over time into a waveform over time.

As seen from the foregoing description, the speech enhancement solution in accordance with implementations of the subject matter described herein can make full use of the time-domain and frequency-domain correlation information of the audio signal. In this way, the performance of speech enhancement can be improved, which helps to obtain completely pure speech.

Some example implementations of the subject matter described herein are listed below.

In one aspect, the subject matter described herein provides a computer-implemented method. The method comprises: obtaining a target time-frequency representation at least indicating intensities of an input audio signal at different frequencies over time, the input audio signal comprising a speech component and a noise component; determining frequency correlation information and time correlation information of the input audio signal based on the target time-frequency representation; generating a target feature representation for distinguishing the speech component and the noise component based on the frequency correlation information, the time correlation information, and the target time-frequency representation; generating an output audio signal based on the target feature representation and the target time-frequency representation, the speech component is enhanced relative to the noise component in the output audio signal.