401. The terminal 401 and the terminal 402 are connected to the server 200 through a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

In some examples, a user may operate a speech enhancement control of a human-computer interaction interface (for example, a game client, a conference client, a livestreaming client, or an instant voice communication client) of the terminal 401, and the terminal 401 may transmit, in response to the user's operation on the speech enhancement processing control, a noisy speech signal generated by a client to the server 200 through the network 300. The server 200 may invoke the speech enhancement model to perform speech enhancement on the noisy speech signal transmitted by the terminal 401 through the network 300, to obtain an enhanced speech signal. The enhanced speech signal may be sent through the network 300 to the terminal 402 that makes the voice call with the terminal 401, so that the user using a client of the terminal 402 can hear the enhanced speech signal.

In some arrangements, the server 200 may be an independent physical server, a server cluster or a distributed system including a plurality of physical servers, or a cloud server that provides cloud services, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, big data, and basic cloud computing service such as an artificial intelligence platform. The terminal 401 or the terminal 402 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, or the like, and is not limited to such examples. The terminal and the server may be connected directly or indirectly in a wired communication manner or a wireless communication manner, but is not limited to such configurations.

The following description uses an example in which the electronic device in the training method for the speech enhancement model is a server, to provide details of a structure of the electronic device. FIG. 2A is a schematic diagram of an example structure of a server 200. The server 200 shown in FIG. 2A includes at least one processor 210, a memory 230, and at least one network interface 220. Various components in the server 200 may be coupled together through a bus system 240. It may be understood that the bus system 240 is configured to implement connection and communication between these components. In addition to a data bus, the bus system 240 may further include a power bus, a control bus, a status signal bus, and the like. Various buses are marked as the bus system 240 in FIG. 2A.

The processor 210 may be an integrated circuit chip having a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or another programmable logic device, a discrete gate or a transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

The memory 230 may be removable, non-removable, or a combination thereof. For example, hardware devices include a solid state memory, a hard disk drive, an optical disk drive, and the like. Optionally, the memory 230 may include one or more storage devices physically located away from the processor 210.

The memory 230 may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 230 may include any memory of proper type.

In some examples, the memory 230 can store data to support various operations, such as programs, modules, and data structures, or subsets or supersets thereof, as further described below.

An operating system 231 may include system programs used to handle various basic system services and execute hardware-related tasks, such as a framework layer, a core library layer, a driver layer, to implement various basic services and handle hardware-related tasks.

A network communication module 232 may be configured to reach another computing device via one or more (wired or wireless) network interfaces 220. For example, the network interface 220 includes Bluetooth, wireless compatibility certification (Wi-Fi), universal serial bus (USB), and the like.

In some arrangements, the training apparatus of the speech enhancement model may be implemented by using software. FIG. 2A shows an example training apparatus 233 of the speech enhancement model stored in the memory 230. The training apparatus 233 may be software in a form of programs, plugins, or the like, including the following software modules: an enhancement module 2331, an obtaining module 2332, a correction module 2333, a determining module 2334 and an update module 2335. These modules are logical, and may be combined or further disassembled in any way based on the functions implemented.

The following description of a structure of an electronic device uses an example in which an electronic device that performs the speech enhancement processing method is a terminal. FIG. 2B is a schematic diagram of an example structure of a terminal 401. The terminal 401 shown in FIG. 2B may include at least one processor 410, a memory 430, and at least one network interface 420.

An operating system 431 may include system programs used to handle various basic system services and execute hardware-related tasks, such as a framework layer, a core library layer, a driver layer, to implement various basic services and handle hardware-related tasks.

A network communication module 432 may be configured to reach another computing device via one or more (wired or wireless) network interfaces 420. For example, the network interface 420 includes Bluetooth, wireless compatibility certification (Wi-Fi), universal serial bus (USB), and the like.

In some examples, the speech enhancement processing apparatus may be implemented by using software. FIG. 2B shows an example speech enhancement processing apparatus 433 stored in the memory 430. The speech enhancement processing apparatus 433 may be software in a form of programs, plugins, or the like, including the following software modules: an enhancement module 4331, a mapping module 4332, and a reconstruction module 4333. These modules are logical, and may be combined or further disassembled in any way based on the functions implemented.

The following description of a training method for a speech enhancement model refers to an example application and implementation of the electronic device. The following method may be independently performed by the foregoing terminal 401 or the foregoing server 200, or may be collaboratively performed by the foregoing terminal 401 and the foregoing server 200.

FIG. 3A is a schematic flowchart of steps 101 to 105 of an example training method for a speech enhancement model. Descriptions are provided with reference to the steps 101 to 105 shown in FIG. 3A.

In step 101, the speech enhancement model may be invoked based on a noisy speech feature of a noisy speech signal, to perform speech enhancement processing, and obtain a plurality of first predicted mask values in an auditory domain.

The noisy speech signal may be obtained by adding noise to a clean speech signal (that is, a speech signal without noise). For example, sound collection may be performed on a sound source (for example, a human or an electronic device that plays human voice without noise) in a noise-free environment to obtain a clean speech signal, and then noise may be added to the clean speech signal to obtain a noisy speech signal. Alternatively, sound collection may be performed on a sound source in a noise-free environment to obtain a clean speech signal, and then sound collection may be performed on the same sound source in a noisy environment to obtain a noisy speech signal. Sounds emitted by the sound source in the noise-free environment and in the noisy environment may be the same, for example, when a human reads the same text by using the same three elements of sound (namely, loudness, pitch, and timbre), or an electronic device plays a noise-free speech of a human reading the same text.

The noisy speech feature may be obtained by performing feature extraction on the noisy speech signal after the noisy speech signal is obtained. The following examples are described.

First, the noisy speech signal may be transformed from time domain to frequency domain to obtain a spectrum. The spectrum may include an amplitude spectrum and a phase spectrum. An independent variable of the amplitude spectrum may be frequency, and a dependent variable of the amplitude spectrum may be an amplitude. This may represent that the amplitude changes with the frequency. An independent variable of the phase spectrum may be frequency, and a dependent variable of the phase spectrum may be phase. This may represent that the phase changes with the frequency. The transform process from the time domain to the frequency domain may include a Fourier transform (FT), including fast Fourier transform (FFT) and discrete Fourier transform (DFT).

Second, a first amplitude corresponding to a plurality of frequency points in the frequency domain may be read from the amplitude spectrum. A frequency of the frequency point may be related to a quantity of sampling times of the amplitude spectrum. Specifically, it may be calculated based on Formula 1 below. Frequency band combination may be performed on first amplitudes corresponding to the noisy speech signal at the plurality of frequency points in the frequency domain, to obtain a plurality of amplitudes corresponding to the noisy speech signal in the auditory domain. The frequency band combination may include: For each frequency point, a first amplitude at the frequency point is used as an amplitude of the corresponding frequency point in the auditory domain, to uniformly map amplitudes of frequency points with discrete amplitude spectrums to the auditory domain. For example, if two frequency points of 50 Hz and 100 Hz are obtained by sampling the amplitude spectrum, a first amplitude of the frequency point of 50 Hz may be assigned to the frequency point of 50 Hz in the auditory domain, and a first amplitude of the frequency point of 100 Hz may be assigned to the frequency point of 100 Hz in the auditory domain.

Subsequently, feature extraction may be performed based on the plurality of amplitudes corresponding to the noisy speech signal in the auditory domain, to obtain the noisy speech feature. For example, the plurality of amplitudes may be combined into a feature vector representing the noisy speech feature.

In some arrangements, before the feature extraction, the noisy speech signal can be further processed: A dynamic range including the plurality of amplitudes corresponding to the noisy speech signal in the auditory domain may be compressed into a smaller dynamic range, to reduce complexity of subsequent calculation. For example, the plurality of amplitudes may be substituted into a logarithmic function as a real number A in a logarithmic function (lgA), and an obtained index may be used as a new amplitude to replace the original amplitude.

In some examples, before the feature extraction, the noisy speech signal may be further processed: Spectral information of a plurality of noisy speech signals adjacent in a time sequence may be spliced, to form a new noisy speech signal, and replace the original noisy speech signal. The spectral information may be a spectrogram. The abscissa of the spectrogram may be time, and the ordinate may be an amplitude of an audio signal. Through splicing, a time domain view can be enlarged when training the speech enhancement model below, so that the speech enhancement model can be trained on a larger time domain view.

After obtaining the noisy speech feature, the speech enhancement model may be invoked based on the noisy speech feature, to perform speech enhancement processing, to obtain the plurality of first predicted mask values in the auditory domain. Different first predicted mask values may correspond to different frequency bands in the auditory domain. The first predicted mask value may represent a boundary value corresponding to the noisy speech feature, and the first predicted mask value may be used to solve a binary classification problem, e.g., distinguishing between noise and human voice.

A speech enhancement model may be various machine learning models based on deep learning, and a training target may be the IBM. For example, the speech enhancement model may be a deep neural network (DNN) model. Through a plurality of cascaded fully connected layers included in the DNN model, cascaded fully connected calculation may performed on the noisy speech feature a plurality of times, and a mask calculation may be performed on a calculation result output by the last fully connected layer (for example, to normalize the calculation result output by the fully connected layer by using a softmax function), to obtain a plurality of first predicted mask values in the auditory domain, where a value range is a closed interval between 0 and 1.

In some examples, the speech enhancement model may alternatively be a recurrent neural network (RNN) model. Through a plurality of cascaded cyclic units included in the RNN model, a cascaded recursive calculation may be performed on the noisy speech feature a plurality of times, and a mask calculation may be performed on a calculation result output by the last cyclic unit, to obtain a plurality of first predicted mask values in the auditory domain.

Further, the speech enhancement model may alternatively be a convolutional neural network (CNN) model. Through a plurality of cascaded convolution layers included in the CNN model, a cascaded convolution calculation may be performed on the noisy speech feature a plurality of times, and a mask calculation may be performed on a calculation result output by the last convolutional layer, to obtain a plurality of first predicted mask values in the auditory domain.

For example, the auditory domain may include a plurality of different frequency bands, and the plurality of first predicted mask values output by the speech enhancement model may respectively correspond to different frequency bands in the auditory domain. An example in which the auditory domain is Bark domain is used. In the Bark domain, based on features of a human car structure that resonates with 24 frequency points, a frequency of the speech signal is divided into 24 critical frequency bands. The frequency bands included in the Bark domain are close to linear at low frequency, and are close to an exponential distribution at high frequency. Table 1 below shows some frequency bands in the Bark domain.

TABLE 1

Some frequency bands in the Bark domain

Center
Cut-off frequency
Bandwidth of

frequency
of each frequency
each frequency

Sequence
point (Hz)
band (Hz)
band (Hz)

20

1
50
100
80

2
150
200
100

3
250
300
100

4
350
400
100

5
450
510
110

6
570
630
120

7
700
770
140

8
840
920
150

The foregoing Table 1 shows some frequency bands in the Bark domain. The center frequency point is a frequency value in the middle of the frequency values corresponding to a frequency band. As shown in Table 1, the 1^stfrequency band of the Bark domain is 20 to 100 Hz, and its center frequency point is 50 Hz and a bandwidth is 80 Hz. The 2^ndfrequency band is 100 to 200 Hz, it's a center frequency point thereof is 150 Hz and a bandwidth thereof is 100 Hz. A third frequency band is 200 to 300 Hz, a center frequency point thereof is 250 Hz, and a bandwidth thereof is 100 Hz.

In step 102, a first amplitude and first phase corresponding to each frequency point of the noisy speech signal may be obtained, and a second amplitude second phase corresponding to each frequency point of the clean speech signal may be obtained.

The noisy speech signal at a plurality of frequency points in the frequency domain separately corresponds to the first amplitude and the first phase (the term “first” is used herein to distinguish from the second amplitude and the second phase of the clean speech signal below, but does not indicate a specific amplitude and phase).

In some arrangements, frequency domain conversion processing may be performed respectively on the noisy speech signal and the clean speech signal, such as Fourier transform, to obtain the amplitude spectrum and the phase spectrum. The first amplitude and the first phase corresponding to the noisy speech signal at each frequency point in the frequency domain may be obtained from the amplitude spectrum through reading. The second amplitude and the second phase corresponding to the clean speech signal at each frequency point in the frequency band may be obtained from the phase spectrum through reading.

The noisy speech signal may be obtained by adding a noise signal to the clean speech signal. The noisy speech signal and the clean speech signal may be sampled by using the same sampling frequency and the same quantity of sampling times, so that a quantity of sampling points of the noisy speech signal is the same as a quantity of sampling points of the clean speech signal.

Frequency of each sampling point (namely, a frequency point) of the noisy speech signal and the clean speech signal may be calculated based on the following formula:

fs
⁡
(
n
)

=

(

n
-
1

)

*
Fs

N

Formula
⁢

1

fs(n) represents frequency of an N^thsampling point of the noisy speech signal and the clean speech signal, where 1≤n≤N, and N represents sampling times of the noisy speech signal or the clean speech signal, that is, the quantity of sampling points of the noisy speech signal and the clean speech signal. Fs represents the sampling frequency of the noisy speech signal and the clean speech signal, and a value of Fs meets a Nyquist law, that is, Fs is greater than twice the highest frequency of the noisy speech signal.

Because the sampling frequency and the quantity of sampling times may be the same when the noisy speech signal and the clean speech signal are sampled, it can be learned from Formula 1 that, the frequency of the N^thsampling point of the noisy speech signal is the same as the frequency of the N^thsampling point of the clean speech signal. Therefore, the noisy speech signal and the clean speech signal may have a plurality of identical frequency points in the frequency domain, so that each frequency point might correspond not only to the first amplitude and the first phase of the noisy speech signal, but also to the second amplitude and the second phase of the clean speech signal.

In step 103, a phase difference between the clean speech signal and the noisy speech signal at each frequency point may be determined based on the first phase and the second phase corresponding to each frequency point, and the second amplitude corresponding to each frequency point may be corrected based on the phase difference at each frequency point, to obtain a corrected second amplitude corresponding to each frequency point.

For example, after obtaining the first phase, the second phase, the first amplitude, and the second amplitude that are corresponding to each frequency point, a phase difference between the clean speech signal and the noisy speech signal at the frequency point may be determined based on the first phase and the second phase that are corresponding to each frequency point, and the second amplitude corresponding to the frequency point may be corrected based on the phase difference of the frequency point, to obtain a corrected second amplitude corresponding to the frequency point.

For example, a corrected second amplitude of an i^thfrequency point may be calculated based on the following formula:

S
⁡
(
i
)

=

❘
"[LeftBracketingBar]"

si

❘
"[RightBracketingBar]"

⁢

cos
⁡
(

θ

s
⁢
i

-

θ

x
⁢
i

)

Formula
⁢

2

S(i) represents the corrected second amplitude corresponding to the i^thfrequency point, |si| represents the second amplitude corresponding to the i^thfrequency point of the clean speech signal, θ_sirepresents the second phase corresponding to the i^thfrequency point of the clean speech signal, θ_xirepresents the first phase corresponding to the i^thfrequency point of the noisy speech signal, and (θ_si−θ_xi) represents a phase difference between the clean speech signal and the noisy speech signal at the i^thfrequency point.

In step 104, a loss value may be determined based on the plurality of first predicted mask values, and the first amplitude and the corrected second amplitude corresponding to each frequency point.

In one or more examples, after obtaining the corrected second amplitude corresponding to each frequency point, the loss value may be determined based on the plurality of first predicted mask values, and the first amplitude and the corrected second amplitude corresponding to each frequency point in the auditory domain.

Aspects described herein provide two exemplary solutions for determining the loss value. A first solution is to map the plurality of first predicted mask values in the auditory domain to the frequency domain, and determine a corresponding loss value in the frequency domain based on a plurality of second predicted mask values in the frequency domain, and the first amplitude and the corrected second amplitude corresponding to each frequency point in the frequency domain. A second solution is to map the first amplitude and the corrected second amplitude corresponding to each frequency point in the frequency domain to the auditory domain, determine a corresponding second target mask value in the auditory domain based on the first amplitude and the corrected second amplitude corresponding to the auditory domain, and determine a corresponding loss value in the auditory domain based on the second target mask value in the auditory domain and the first predicted mask value in the auditory domain.

The following separately describes the two example solutions for determining the loss value.

The following describes, with reference to the accompanying drawings, the first solution for determining the loss value. FIG. 3B is a schematic flowchart of steps 1041A and 1042A of an example training method for a speech enhancement model. Based on FIG. 3A, step 104 shown in FIG. 3B may be implemented through steps 1041A and 1042A. The following provides descriptions with reference to steps 1041A and 1042A shown in FIG. 3B.

In step 1041A, the plurality of first predicted mask values may be mapped to obtain second predicted mask values corresponding to each frequency point.

For example, in a process of determining the loss value corresponding to the frequency domain, first, the plurality of first predicted mask values may be mapped, to obtain the second predicted mask values corresponding to each frequency point in the frequency domain.

Because each first predicted mask value may correspond to a frequency band in the auditory domain, the first predicted mask values in the auditory domain can be mapped to the second predicted mask values corresponding to each frequency point in the frequency domain through mapping processing.

The mapping processing may mean estimating, based on a set of known discrete point data (for example: the first predicted mask value), approximate values (for example: the second predicted mask value) corresponding to another discrete point (for example: frequency point) within a data range corresponding to discrete point data. According to one or more aspects, the mapping processing may be implemented in a manner of copying or linear weighting. The following provides detailed descriptions.

In some arrangements, step 1041A may be implemented in the following copying manner: In a plurality of pre-divided frequency bands in the auditory domain (for example, pre-divided frequency bands in the BARK domain or the Mel domain), a first frequency band to which the frequency point belongs in the auditory domain may be determined, and a first predicted mask value corresponding to the first frequency band may be determined as the second predicted mask value corresponding to the frequency point.

For example, when determining the second predicted mask value corresponding to each frequency point, first, a frequency band to which each frequency point belongs in the auditory domain may be determined, that is, the first frequency band to which the frequency point belongs in the auditory domain may be determined. For example, when the frequency point is 60 Hz, it can be learned from Table 1 that the frequency point belongs to the 1^stfrequency band of 20 to 100 Hz in the Bark domain. Therefore, the frequency band of 20 to 100 Hz may be determined as the first frequency band to which this frequency point belongs. Then, after the first frequency band is determined, the first predicted mask value corresponding to the first frequency band may be determined as the second predicted mask value corresponding to the frequency point. For example, a first predicted mask value corresponding to the frequency band of 20 to 100 Hz may be determined as the second predicted mask value corresponding to the frequency point of 60 Hz.

In some arrangements, step 1041A may be alternatively implemented in the following linear weighting manner: In a plurality of pre-divided frequency bands in the auditory domain, a first frequency band to which the frequency point belongs in the auditory domain may be determined, and at least one reference frequency band adjacent to the first frequency band in the auditory domain (for example, at least one of a reference frequency band lower than the first frequency band and a reference frequency band higher than the first frequency band) may be determined. Weighted summation may be performed on the first predicted mask value corresponding to the first frequency band and the first predicted mask value corresponding to the at least one reference frequency band, to obtain a second predicted mask value corresponding to the frequency point.

For example, when determining the second predicted mask value corresponding to each frequency point, first, a frequency band to which each frequency point belongs in the auditory domain may be determined, that is, the first frequency band to which the frequency point belongs in the auditory domain may be determined; second, at least one adjacent frequency band of the first frequency band may be determined, and the at least one adjacent frequency band may be determined as a reference frequency band; and finally, the weighted summation may be performed on the first predicted mask value corresponding to the first frequency band and the first predicted mask value corresponding to the reference frequency band, to obtain the second predicted mask value corresponding to the frequency point. The weight corresponding to each of the first predicted mask values may be positively correlated with a distance between the following two elements: the frequency point and a center frequency point of the frequency band corresponding to the first predicted mask value.

For example, when the frequency point is 120 Hz, first, it can be learned from Table 1 that the frequency point belongs to the 2^ndfrequency band of 100 to 200 Hz in the Bark domain. Therefore, a frequency band of 100 to 200 Hz may be determined as the first frequency band to which the frequency point belongs. Second, it can be learned from Table 1 that adjacent frequency bands of the frequency band of 100 to 200 Hz may include two frequency bands of 20 to 100 Hz and 200 to 300 Hz. Therefore, the two frequency bands of 20 to 100 Hz and 200 to 300 Hz may be both determined as reference frequency bands. Finally, one of the two reference frequency bands may be selected, and weighted summation may be performed on a first predicted mask value corresponding to the selected reference frequency band and the first predicted mask value corresponding to the first frequency band, to obtain a second predicted mask value corresponding to the frequency point of 120 Hz. In this case, a center frequency point of the reference frequency band of 20 to 100 Hz is 50 Hz, and the frequency point of 120 Hz is 70 Hz away from the center frequency point; a center frequency point of the frequency band of 100 to 200 Hz is 150 Hz, and the frequency point of 120 Hz is 30 Hz away from the center frequency point; and a center frequency point of the reference frequency band of 200 to 300 Hz is 250 Hz, and the frequency point of 120 Hz is 130 Hz away from the center frequency point. Because a weight of the first predicted mask value may be positively correlated with a distance between the frequency point and the center frequency point of the frequency band, in the process of determining the second predicted mask value corresponding to the frequency point of 120 Hz, a weight of a first predicted mask value corresponding to the reference frequency band of 200 to 300 Hz may be greater than a weight of a first predicted mask value corresponding to the reference frequency band of 20 to 100 Hz; and similarly, the weight of the first predicted mask corresponding to the reference frequency band of 20 to 100 Hz may be greater than a weight of a first predicted mask value corresponding to the frequency band of 100 to 200 Hz.

Determining the second predicted mask value corresponding to each frequency point in the frequency domain in one of the foregoing methods can accurately represent an impact degree of the distance between the frequency point and the center frequency point on the second predicted mask value of the frequency point, so that accuracy of the determined second predicted mask value can be improved, to facilitate accurately determining the loss value based on the accurate second predicted mask value.

In step 1042A, the loss value may be determined based on the second predicted mask value, the first amplitude, and the corrected second amplitude that are corresponding to each frequency point.

For example, after determining the second predicted mask value corresponding to each frequency point, the loss value may be determined based on the second predicted mask value corresponding to each frequency point, and the first amplitude value and the corrected second amplitude corresponding to the frequency point.

In one or more arrangements, in the process of determining the loss value in the frequency domain in the foregoing first solution, there may be the following two implementations: The first implementation may include obtaining the loss value in the frequency domain through calculation based on the amplitude corresponding to each frequency point in the frequency domain; and the second implementation may include obtaining the loss value in the frequency domain through calculation based on the mask corresponding to each frequency point in the frequency domain.

The following separately describes the two example implementations corresponding to the first solution for determining the loss value.

The following describes an example first implementation corresponding to the first solution for determining the loss value. FIG. 3C is a schematic flowchart of steps 10421A and 10422A of an example training method for a speech enhancement model. Based on FIG. 3B, step 1042A shown in FIG. 3C may be implemented through steps 10421A and 10422A. The following provides descriptions with reference to steps 10421A and 10422A shown in FIG. 3C.

In step 10421A, the second predicted mask value and the first amplitude corresponding to each frequency point may be multiplied to obtain a third amplitude corresponding to each frequency point.

For example, FIG. 4A is a schematic flowchart of an example first implementation of determining a loss value corresponding to frequency domain. The Frequency domain dot product shown in FIG. 4A may be implemented in the following manner: In the process of determining the loss value in the frequency domain, the second predicted mask value G_icorresponding to each frequency may be multiplied by the first amplitude |xi| corresponding to the frequency point to obtain a third amplitude G_i|xi| corresponding to the frequency point, where 1≤i≤I, and I represents a total quantity of frequency points.

In step 10422A, the third amplitude and the corrected second amplitude corresponding to each frequency point may be substituted into the first target loss function for calculation to obtain the loss value.

For example, referring to FIG. 4A, after obtaining the third amplitude corresponding to each frequency point, the third amplitude corresponding to each frequency point and the corrected second amplitude corresponding to the frequency point may be substituted into the first target loss function for calculation to obtain the loss value.

The first target loss function may include but is not limited to: a mean square error (MSE) function, an MSE function based on exponential weighting, an MSE function based on logarithmic weighting, a mean absolute error (MAE) function, and a cross entropy loss function. The first target loss function may alternatively be a weighted combination of a plurality of loss functions, such as a weighted combination of the MSE function and the MAE function.

An example in which the first target loss function is the MSE function is used. The first target loss function may be calculated based on the following formula:

L
⁢
1

=

1
I

⁢

∑

i
=
1

I

⁢

(

❘
"[LeftBracketingBar]"

si

❘
"[RightBracketingBar]"

⁢

cos
⁡
(

θ
si

-

θ
xi

)

-

G
i

⁢

❘
"[LeftBracketingBar]"

xi

❘
"[RightBracketingBar]"

)

2

Formula
⁢

3

L1 represents the first target loss function, |si|cos(θ_si−θ_xi) represents the corrected second amplitude corresponding to the i^thfrequency point, G_i|xi| represents the third amplitude corresponding to the i^thfrequency point, and I represents the total quantity of frequency points.

The loss value can be obtained by substituting the corrected second amplitude and the third amplitude corresponding to each frequency point into the foregoing Formula 3 for calculation. In the foregoing manner, the amplitude and the phase of the speech signal can be both integrated into the loss value, to better guide the speech enhancement model in learning.

The following describes an example second implementation of determining the loss value. FIG. 3D is a schematic flowchart of steps 10423A and 10424A of an example training method for a speech enhancement model. Based on FIG. 3B, step 1042A shown in FIG. 3D may alternatively be implemented through steps 10423A and 10424A. The method shown in FIG. 3D and the method shown in FIG. 3C are two concurrent methods. The following provides descriptions with reference to steps 10423A and 10424A shown in FIG. 3D.

In step 10423A, a ratio of the corrected second amplitude corresponding to each frequency point to the first amplitude corresponding to each frequency point may be determined as the first target mask value corresponding to each frequency point.

For example, FIG. 4B is a schematic flowchart of an example second implementation of determining a loss value corresponding to frequency domain. As shown in FIG. 4B, in the process of determining the loss value of the frequency domain, a ratio of the corrected second amplitude corresponding to each frequency point to the first amplitude corresponding to the frequency point may be determined as the first target mask value corresponding to each frequency point.

For example, the first target mask value may be calculated based on the following formula:

M
⁢
1
⁢

(
i
)

=

❘
"[LeftBracketingBar]"

si

❘
"[RightBracketingBar]"

⁢

cos
⁡
(

θ
si

-

θ
xi

)

❘
"[LeftBracketingBar]"

xi

❘
"[RightBracketingBar]"

Formula
⁢

4

M1(i) represents the first target mask value corresponding to the i^thfrequency point, |si|cos(θ_si−θ_xi) represents the corrected second amplitude corresponding to the i^thfrequency point, and |xi| represents the first amplitude corresponding to the i^thfrequency point.

In step 10424A, the second predicted mask value corresponding to each frequency point and the first target mask value corresponding to the frequency point may be substituted into the second target loss function for calculation to obtain the loss value.

For example, referring to FIG. 4B, after determining the first target mask value corresponding to each frequency point, the first target mask value corresponding to each frequency point and the second predicted mask value corresponding to the frequency point may be substituted into the second target loss function for calculation to obtain the loss value. An example in which the second target loss function is the MSE function is used. For details, refer to descriptions of the foregoing Formula 3.

For example, the second target loss function may include but is not limited to: the MSE function, the MSE function based on exponential weighting, the MSE function based on logarithmic weighting, the MAE function, and the cross entropy loss function. The second target loss function may alternatively be a weighted combination of a plurality of loss functions, such as a weighted combination of the MSE function and the MAE function.

In the foregoing manner, the amplitude and the phase of the speech signal can be both integrated into the loss value, to better guide the speech enhancement model in learning.

The following provides descriptions of the example second solution for determining the loss value. FIG. 3E is a schematic flowchart of steps 1041B to 1044B of an example training method for a speech enhancement model. Based on FIG. 3A, step 104 shown in FIG. 3E may alternatively be implemented through steps 1041B to 1044B. The method shown in FIG. 3E and the method shown in FIG. 3B may be two concurrent implementations in step 104. The following provides descriptions with reference to steps 1041B to 1044B shown in FIG. 3E.

In step 1041B, the first amplitude and the corrected second amplitude corresponding to each frequency point may be mapped to the frequency band corresponding to the auditory domain.

For example, in the process of determining the loss function, the first amplitude and the corrected second amplitude corresponding to each frequency point may be first mapped to the frequency band corresponding to the auditory domain.

In some arrangements, step 1041B may be implemented in the following manner: the second frequency band to which each frequency point belongs in the auditory domain is determined; and the first amplitude and the corrected second amplitude that are corresponding to each frequency point are mapped to the second frequency band to which the frequency point belongs in the auditory domain.

For example, in the process of mapping the first value and the corrected second amplitude corresponding to each frequency point, first, a frequency band to which each frequency point belongs in the auditory domain (such as the Bark domain and the Mel domain) may be determined. That is, the second frequency band to which each frequency point belongs in the auditory domain may be determined. Second, the first amplitude and the corrected second amplitude corresponding to each frequency may be mapped to the second frequency band to which the frequency point belongs in the auditory domain.

For example, when the frequency point is 120 Hz, it can be learned from Table 1 that the frequency point of 120 Hz belongs to a frequency band of 100 to 200 Hz in the Bark domain, so that the frequency band of 100 to 200 Hz may be determined as the second frequency band, and a first amplitude and a corrected second amplitude corresponding to the frequency point of 120 Hz may be mapped to the second frequency band.

Because a quantity of frequency bands included in the auditory domain is less than a quantity of frequency points in the frequency domain, first amplitudes and corrected second amplitudes corresponding to a plurality of frequency points can be combined into first amplitudes and corrected second amplitudes corresponding to a small quantity of frequency bands by mapping the first amplitudes and the corrected second amplitudes corresponding to the frequency points to corresponding frequency bands in the auditory domain. In this way, the frequency band combination may be implemented through this mapping manner, and dimensions of to-be-processed information may be effectively reduced, so that computing resources can be effectively reduced in the process of calculating the loss value.

In step 1042B, first energy corresponding to each frequency band may be determined based on the first amplitude mapped to each frequency band.

For example, after mapping the first amplitude corresponding to each frequency point to the corresponding frequency band in the auditory domain, the first energy corresponding to each frequency band may be determined based on the first amplitude mapped to each frequency band. The first energy may be a weighted summation result of the following parameters: squares of first amplitudes mapped to each frequency bands.

Energy corresponding to a frequency point may be a square of an amplitude corresponding to the frequency point, and first energy corresponding to a frequency band may be a weighted summation result of squares of all first amplitudes included in the frequency band.

In one or more examples, a first energy corresponding to each frequency band may be calculated based on the following formula:

E
1

=

∑

p
=
1

P

⁢

w
p

⁢

E
p

Formula
⁢

5

E₁represents the first energy corresponding to each frequency band, w_prepresents a weight of energy corresponding to a first amplitude corresponding to a p^thfrequency point in the frequency band, Ep represents the energy corresponding to the first amplitude corresponding to the p^thfrequency point in the frequency band (that is, a square of the first amplitude corresponding to the p^thfrequency point), and P is a quantity of frequency points included in the frequency band.

A weight corresponding to a square of each first amplitude may be obtained through average distribution. To be specific, when the frequency band includes P frequency points, the weight of the square of the first amplitude corresponding to each frequency point may be 1/P. The weight corresponding to the square of each first amplitude can alternatively be determined based on a distance between a frequency point corresponding to the first amplitude and a center frequency point of the frequency band. In this case, a weight of the square of the first amplitude corresponding to the p^thfrequency point in the frequency band may be calculated based on the following formula:

w
p

=

α

❘
"[LeftBracketingBar]"

f
p

-

f
c

❘
"[RightBracketingBar]"

Formula
⁢

6

w_prepresents the weight of the square of the first amplitude corresponding to the p^thfrequency point in the frequency band, a is a constant, f_prepresents the p^thfrequency point in a frequency band, and f_crepresents a center frequency point of the frequency band.

In step 1043B, second energy corresponding to each frequency point may be determined based on the corrected second amplitude mapped to each frequency point.

For example, after mapping the corrected second amplitude corresponding to each frequency point to the corresponding frequency band in the auditory domain, second energy corresponding to each frequency band may be determined based on the corrected second amplitude mapped to each frequency band. The second energy may be a weighted summation result of the following parameters: squares of corrected second amplitudes mapped to each frequency band.

The second energy corresponding to the frequency band may be the weighted summation result of the squares of all corrected second amplitudes included in the frequency band. A manner of calculating the second energy corresponding to each frequency band may be similar to the manner for calculating the first energy corresponding to each frequency band. Accordingly, details of such a process are not described again.

In step 1044B, the loss value is determined based on the first predicted mask value, the first energy, and the second energy that are corresponding to each frequency band.

In one or more examples, after determining the first energy and the second energy corresponding to each frequency band, the loss value may be determined based on the first predicted mask value corresponding to each frequency band, the first energy and the second energy corresponding to the frequency band.

According to one or more aspects, the process of determining the loss value in the auditory domain in the second solution may be performed according to the following two example implementations: The first implementation may include obtaining the loss value in the auditory domain through calculation based on the mask corresponding to each frequency band in the auditory domain; and the second implementation may include obtaining the loss value in the auditory domain through calculation based on energy information corresponding to each frequency band in the auditory domain.

The following separately describes the two example implementations corresponding to the second solution for determining the loss value.

The following describes an example first implementation corresponding to the second solution for determining the loss value. FIG. 3F is a schematic flowchart of steps 10441B and 10442B of an example training method for a speech enhancement model. Based on FIG. 3E, step 1044B shown in FIG. 3F may be implemented through steps 10441B and 10422B. The following provides descriptions with reference to steps 10441B and 10422B shown in FIG. 3F.

In step 10441B, a second target mask value corresponding to each frequency band may be determined based on the first energy and the second energy that are corresponding to each frequency band.

For example, FIG. 4C is a schematic flowchart of an example first implementation of determining a loss value corresponding to an auditory domain. As shown in FIG. 4C, in the process of determining the loss value in the auditory domain, the second target mask value corresponding to each frequency band may be determined based on the first energy and the second energy corresponding to each frequency band.

In some arrangements, step 10441B may be implemented in the following manner: A ratio of the second energy corresponding to the frequency band to the first energy corresponding to the frequency band may be determined as the second target mask value corresponding to the frequency band.

For example, a ratio of the second energy corresponding to each frequency band to the first energy corresponding to each frequency band may be determined as the second target mask value corresponding to the frequency band. The second target mask value may be calculated based on the following formula:

M
⁢
2

=

E
⁢
2

E
⁢
1

Formula
⁢

7

M2 represents the second target mask value corresponding to each frequency point, E2 represents the second energy corresponding to the frequency band, and E1 represents the first energy corresponding to the frequency band.

In some arrangements, step 10441B may be implemented in the following manner: A difference between the first energy and the second energy corresponding to the frequency band may be determined as a third energy corresponding to the frequency band. A first summation result may be obtained by summing a square of the second energy corresponding to the frequency band and a square of the third energy corresponding to the frequency band, and a ratio of the square of the second energy to the first summation result may be determined as the second target mask value corresponding to the frequency band.

For example, first, the difference between the first energy and the second energy the corresponding to each frequency band may be determined as the third energy corresponding to the frequency band. Second, the first summation result may be obtained by summing the square of the second energy and the square of the third energy corresponding to the frequency band. Finally, the ratio of the square of the second energy corresponding to the frequency band to the first summation result may be determined as the second target mask value corresponding to the frequency band. The second target mask value corresponding to each frequency band may be calculated based on the following formula:

M
⁢
2

=

(

E
⁢
2

)

2

(

E
⁢
2

)

2

+

(

E
⁢
3

)

2

Formula
⁢

8

M2 represents the second target mask value corresponding to each frequency band, E2 represents the second energy corresponding to the frequency band, E3 represents the third energy corresponding to the frequency band, E3 is the difference between the first energy and the second energy that are corresponding to the frequency band, that is, E1−E2, and E1 represents the first energy corresponding to frequency band.

In step 10442B, the first predicted mask value corresponding to each frequency band and the second target mask value corresponding to the frequency band may be substituted into the third target loss function for calculation to obtain the loss value.

For example, referring to FIG. 4C, after determining the second target mask value corresponding to each frequency band, the first predicted mask value corresponding to each frequency band and the second target mask value corresponding to the frequency band may be substituted into the third target loss function for calculation to obtain the loss value.

For example, the third target loss function may include but is not limited to: the MSE function, the MSE function based on exponential weighting, the MSE function based on logarithmic weighting, the MAE function, and a cross entropy loss function. The third target loss function may alternatively be a weighted combination of a plurality of loss functions, such as a weighted combination of the MSE function and the MAE function. In the foregoing manner, the amplitude and the phase of the speech signal can be integrated into the loss value, so that the speech enhancement model can be better guided to learn.

The following describes an example second implementation corresponding to the second solution for determining the loss value. FIG. 3G is a schematic flowchart of steps 10443B and 10444B of an example training method for a speech enhancement model. Based on FIG. 3E, step 1044B shown in FIG. 3G may alternatively be implemented through steps 10443B and 10444B. The method shown in FIG. 3G and the method shown in FIG. 3F may be two concurrent methods. The following provides descriptions with reference to steps 10443B and 10444B shown in FIG. 3G.

In step 10443B, the first predicted mask value corresponding to each frequency band and the first energy corresponding to the frequency band may be multiplied to obtain fourth energy corresponding to each frequency band.

For example, FIG. 4D is a schematic flowchart of the example second implementation of determining a loss value corresponding to auditory domain. As shown in FIG. 4D, in the process of determining the loss value in the auditory domain, the first predicted mask value corresponding to each frequency band and the first energy corresponding to the frequency band may be multiplied to obtain the fourth energy corresponding to each frequency band.

In step 10444B, the second energy corresponding to each frequency band and the fourth energy corresponding to the frequency band may be substituted into the fourth target loss function for calculation to obtain the loss value.

For example, referring to FIG. 4D, after obtaining the fourth energy corresponding to each frequency band, the second energy corresponding to each frequency band and the fourth energy corresponding to the frequency band may be substituted into the fourth target loss function for calculation to obtain the loss value. The fourth target loss function may include but is not limited to: the MSE function, the MSE function based on exponential weighting, the MSE function based on logarithmic weighting, the MAE function, and the cross entropy loss function. The fourth target loss function may alternatively be a weighted combination of a plurality of loss functions, such as a weighted combination of the MSE function and the MAE function.

In the foregoing manner, the amplitude and the phase of the speech signal can be both integrated into the loss value, to better guide the speech enhancement model in learning.

In step 105, backpropagation may be performed in the speech enhancement model based on the loss value, to update parameters of the speech enhancement model.

For example, after obtaining the loss value, backpropagation may be performed in the speech enhancement model based on the loss value, to update the parameters of the speech enhancement model.

Following the above, an example in which the speech enhancement model is a convolutional neural network (CNN) model is used. The convolutional neural network model may include a plurality of cascaded convolutional layers. After obtaining the loss value through calculation, a backpropagation (BP) algorithm may be used to backpropagate the loss value in a direction from an output layer to an input layer, and parameters of the convolutional layer may be calculated by using a gradient descent algorithm during the backpropagation process.

The foregoing steps 101 to 105 may be repeatedly performed until a set quantity of training times is reached, or the loss value is less than a set threshold. Training of the speech enhancement model may be ended, to obtain a trained speech enhancement model. Then, the trained speech enhancement model can be used for speech enhancement.

According to one or more aspects, the amplitude of the clean speech signal at each frequency point may be corrected based on the phase difference between the noisy speech signal and the clean speech signal at each frequency point. The loss value may be determined based on the amplitude of the corrected clean speech signal, the amplitude of the noisy speech signal, and the plurality of first predicted mask values output by the speech enhancement model. Therefore, the phase and the amplitude of the speech signal may be fully considered in the process of calculating the loss value, so that the speech enhancement model is enabled to learn comprehensively based on the loss value integrating the phase and the amplitude. This breaks through the technical barrier that phase cannot be introduced to the auditory domain-based system in the related art. The enhanced speech signal determined based on the predicted mask value output by the trained speech enhancement model might not only be close to the clean speech signal in amplitude, but also be close to the clean speech signal in phase, to improve the noise reduction effect.

The following describes an example speech enhancement processing method and an example implementation of the electronic device. It may be understood that the following method may be independently performed by the foregoing terminal 401 or the foregoing server 200, or may be collaboratively performed by the foregoing terminal 401 and the foregoing server 200.

FIG. 3H is a schematic flowchart of an example speech enhancement processing method. The following provides descriptions with reference to steps 201 to 203 shown in FIG. 3H.

In step 201, a to-be-processed speech feature of a to-be-processed speech signal may be obtained, and a speech enhancement model may be invoked, to obtain a plurality of third predicted mask values in auditory domain.

For example, the to-be-processed speech feature may be obtained in the following manner: First, the to-be-processed speech signal may be transformed from time domain to frequency domain, to obtain a spectrum of the to-be-processed speech signal. The spectrum may include an amplitude spectrum and a phase spectrum. Amplitudes corresponding to a plurality of frequency points in the frequency domain may be read from the amplitude spectrum. Frequency of a frequency point may be related to a quantity of sampling times, and may be specifically calculated based on the foregoing formula 1. A corresponding phase of the plurality of frequency points may be read from the phase spectrum. Then, frequency band combination may be performed on third amplitudes corresponding to the plurality of frequency points of the to-be-processed speech signal in the frequency domain, to obtain a plurality of third amplitudes corresponding to the to-be-processed speech signal in the auditory domain, and the frequency band combination may be implemented in the following example manner: For each frequency point, a third amplitude of the frequency point may be used as an amplitude of the corresponding frequency point in the auditory domain, to uniformly map amplitudes of frequency points with discrete amplitude spectrums to the auditory domain. Finally, feature extraction may be performed based on the plurality of third amplitudes corresponding to the to-be-processed speech signal in the auditory domain, to obtain the to-be-processed speech feature. For example, the plurality of third amplitudes may be combined into a feature vector representing a noisy speech feature.

The speech enhancement model may be trained in the training method for a speech enhancement model, and a value range of a mask value may be a closed interval between 0 and 1.

In step 202, mapping processing may be performed based on a plurality of third predicted mask values in the auditory domain to obtain a mapping processing result.

For example, a plurality of mask values in the auditory domain may be in a one-to-one correspondence with a plurality of frequency points in the auditory domain. After obtaining the plurality of third predicted mask values in the auditory domain, first, the plurality of third predicted mask values in the auditory domain may be mapped to obtain the mapping processing result. The mapping processing result may include the mask values corresponding to the foregoing plurality of frequency points in the frequency domain. The mapping processing manner herein may be similar to the foregoing mapping processing manner of step 1041A. For details not described below, refer to the foregoing descriptions of step 1041A.

In some arrangements, step 202 may be implemented in the following copying manner: for each frequency point: In the plurality of pre-divided frequency bands in the auditory domain (for example, pre-divided frequency bands in BARK domain or Mel domain), the frequency band to which the frequency point belongs in the auditory domain (referred to as the frequency band for short below) may be determined, and a third predicted mask value corresponding to the frequency band may be determined as a fourth predicted mask value corresponding to the frequency point, namely, the mapping processing result.

In some arrangements, step 202 may be further implemented in the following linear weighting manner: In a plurality of pre-divided frequency bands in the auditory domain, a frequency band to which a frequency point belongs in the auditory domain may be determined, and at least one reference frequency band (for example, at least one of a reference frequency band lower than the frequency band and a reference frequency band higher than the frequency band) adjacent to the frequency band in the auditory domain may be determined. Weighted summation may be performed on the third predicted mask value corresponding to the frequency band and a third predicted mask value corresponding to the at least one reference frequency band, to obtain the fourth predicted mask value corresponding to the frequency point, namely, the mapping processing result.

In step 203, signal reconstruction may be performed based on the mapping processing result and the phase spectrum of the to-be-processed speech signal, to obtain the enhanced speech signal.

For example, after obtaining mask values corresponding to the plurality of frequency points in the frequency domain, the fourth predicted mask values corresponding to the plurality of frequency points in the frequency domain and an amplitude spectrum of the to-be-processed speech signal may be multiplied. To be specific, the fourth predicted mask values of the frequency points and amplitudes of corresponding frequency points in the amplitude spectrum may be multiplied, and amplitudes of other frequency points in the amplitude spectrum may remain unchanged, to obtain an enhanced amplitude spectrum. Signal reconstruction may be performed based on the enhanced amplitude spectrum and the phase spectrum of the to-be-processed speech signal, for example, inverse Fourier transform (IFFT) may be performed, to obtain the enhanced speech signal, that is, the speech signal after noise reduction.

According to one or more aspects, because the speech enhancement model used for speech enhancement processing may be obtained through training based on the loss value that integrates the phase and the amplitude, the enhanced speech signal determined based on the mask value obtained through speech enhancement processing based on the speech enhancement model might not only be close to the clean speech signal in amplitude, but may also be close to the clean speech signal in phase, so that the noise in the to-be-processed speech signal can be effectively reduced, and a signal-to-noise ratio of the speech signal can be improved, to improve the noise reduction effect.

The following describes an exemplary speech enhancement processing application scenario.

FIG. 5 is a schematic diagram of example speech enhancement processing in a game client. As shown in FIG. 5, the game client integrates a game voice SDK. A core engine algorithm of the game voice SDK may include one or more aspects of the speech enhancement processing methods based on deep learning described herein. The game voice SDK can provide a noise suppression enabling interface for the game client.

FIG. 5 illustrates a microphone control and a speaker control 501 in the game client. A user can enable a speech enhancement function, that is, enable a noise suppression function, by operating on the microphone control and/or the speaker control. For example, when the user uses the game client, if the user controls the microphone control and/or the speaker control of the game client through touching, in response to the user's touch operation on the microphone control and/or the speaker control, a terminal may obtain a speech signal that is generated by the user in the game client and that is collected by the microphone control, invoke the noise suppression enabling interface in the game voice SDK, and execute the speech enhancement processing, to obtain an enhanced speech signal. The terminal can then send the enhanced speech signal to other players, and other players can play the enhanced speech signal by using the speaker control in the game client. In this way, environmental noise in the speech signal collected by a microphone can be effectively suppressed, to implement high-quality voice call between users and players in the game client.

FIG. 6 is a schematic diagram of example speech enhancement processing in a voice call process. As shown in FIG. 6, during a voice call between a user 1 using a terminal 1 and a user 2 using a terminal 2, voice signals collected by microphones of the terminal 1 and the terminal 2 may include various types of environmental noise in addition to human voice signals emitted by the users. By applying the speech enhancement processing method based on deep learning described herein on the terminal 1 and the terminal 2, the environmental noise may be effectively suppressed in the speech signals collected by the microphone, and only the human voice signal may be transmitted to a communication peer, to improve quality of the voice call.

In some arrangements, aspects of speech enhancement processing based on deep learning described herein may additionally or alternatively be used in the following scenarios:

(1) Video call scenario: A user makes a video call with another user by using a terminal device in a public place, and various types of noise may be included in the environment. By applying speech enhancement processing based on deep learning, environmental noise in a speech signal can effectively suppressed, a human voice signal can be effectively retained, to improve the user experience of the video call, and improve the user's communication efficiency through the video call.

(2) Online meeting scenario: A plurality of users participate in an online meeting through an online meeting client installed on a terminal device. Because the users are in diversified environments, the speech signals collected by microphones may be mixed with environmental noise. By applying speech enhancement processing based on deep learning on the terminal device, the noise in the speech signals can be reduced, user experience of participating in the online meeting can be improved, and work efficiency of the online meeting can be improved.

The following describes example training and application processes of a speech enhancement model in speech enhancement processing. FIG. 7 is a schematic diagram of example training and application processes of a speech enhancement model. An example training method and an application method of the speech enhancement model shown in FIG. 7 may be executed by a terminal or a server separately, or by a terminal and a server collaboratively. An example in which the training method and the application method are executed by a server may be used.

As shown in FIG. 7, steps 701 to 704 may be training processes of the speech enhancement model, and steps 705 to 710 may be application processes of the speech enhancement model. The following provides descriptions with reference to all steps shown in FIG. 7.

In step 701, time-frequency conversion from time domain to frequency domain may be separately performed on a clean speech signal and a noisy speech signal, to obtain a first amplitude and first phase corresponding to a plurality of frequency points of the noisy speech signal in the frequency domain, and obtain a second amplitude and second phase corresponding to a plurality of frequency points of the clean speech signal in the frequency domain.

In step 702, frequency band combination may be performed on first amplitudes corresponding to the noisy speech signal at the plurality of frequency points in the frequency domain, to map the first amplitudes corresponding to the plurality of frequency points to a plurality of frequency bands in auditory domain; and frequency band combination may be performed on second amplitudes corresponding to the clean speech signal at the plurality of frequency points in the frequency domain, to map the second amplitudes corresponding to the plurality of frequency points to a plurality of frequency bands in the auditory domain. The auditory domain may include but is not limited to Bark domain and Mel domain.

In step 703, feature extraction may be performed on the first amplitude corresponding to the noisy speech signal in the auditory domain to obtain a noisy speech feature. For example, logarithmic or exponential weighting may be used for the first amplitude to reduce a dynamic range of a value of the first amplitude. Alternatively, the first amplitude may be spliced with spectral information of adjacent noisy speech signal frames in time sequence to enlarge a time domain view.

In step 704, the speech enhancement model may be invoked based on the noisy speech feature, to perform the speech enhancement processing, to obtain a plurality of first predicted mask values in the auditory domain.

After the first predicted mask value is obtained, a loss value may be determined based on the first predicted mask value. One or more aspects may provide two exemplary solutions for determining the loss value. An example first solution may include mapping the plurality of first predicted mask values in the auditory domain to the frequency domain, and determining the loss value corresponding to the frequency domain based on a plurality of second predicted mask values in the frequency domain, and a first amplitude and a corrected second amplitude corresponding to each frequency point in the frequency domain. An example second solution may include mapping a first amplitude and a corrected second amplitude corresponding to each frequency point in the frequency domain to the auditory domain, determining a corresponding second target mask value in the auditory domain based on a first amplitude and a corrected second amplitude corresponding to the auditory domain, and determining the loss value corresponding to the auditory domain based on the second target mask value in the auditory domain and the first predicted mask value in the auditory domain. For the process of determining the loss value, details may correspond to the foregoing steps 1042A and 1044B. According, those details are not repeated here.

After the loss value is determined in any one of the foregoing solutions, backpropagation may be performed in the speech enhancement model based on the loss value to update parameters of the speech enhancement model.

The foregoing steps 701 to 704 may be repeatedly performed until a set quantity of training times is reached, or the loss value is less than a set threshold. Training of the speech enhancement model may be ended, to obtain a trained speech enhancement model. Then, the trained speech enhancement model can be used for speech enhancement.

The following steps 705 to 710 are example application processes of the speech enhancement model.

In step 705, time-frequency conversion from time domain to frequency domain may be performed on the to-be-processed speech signal, to obtain amplitudes and phase corresponding to a to-be-processed speech signal at a plurality of frequency points in the frequency domain.

In step 706, frequency band combination may be performed on the amplitudes corresponding to the to-be-processed speech signal at the plurality of frequency points in the frequency domain, to map the amplitudes to a plurality of frequency bands in the auditory domain.

Due to a large scale and high algorithmic complexity of the speech enhancement model based on deep learning, in some cases, a voice enhancement requirement of a terminal may be high. To apply the speech enhancement model based on deep learning to the terminal, deep learning based on the auditory domain can effectively reduce dimensions of information to be processed through frequency band combination based on different sensitivities of the human car to different frequencies, thereby reducing the scale of the speech enhancement model, so that the speech enhancement model can be successfully deployed to the terminal.