perform operations enabling the system 200 of system 100 to implement mechanisms for end-to-end neural audio upsampling and bandwidth extension.

The one or more transceivers 220 transmit and receive communications signals that support mechanisms for end-to-end neural audio upsampling and bandwidth extension, according to some aspects, and may be coupled to the antenna 260. The antenna 260 may include one or more antennas that may be the same or different types. The one or more transceivers 220 allow the system 200 to communicate with other devices that may be wired and/or wireless. In some examples, the one or more transceivers 220 can include processors, controllers, radios, sockets, plugs, buffers, and like circuits/devices used for connecting to and communication on networks. According to some examples, the one or more transceivers 220 include one or more circuits to connect to and communicate on wired and/or wireless networks.

According to some aspects, the one or more transceivers 220 can include a cellular subsystem, a WLAN subsystem, and/or a Bluetooth™ subsystem, each including its own radio transceiver and protocol(s) as will be understood by those skilled arts based on the discussion provided herein. In some implementations, the one or more transceivers 220 can include more or fewer systems for communicating with other devices.

In some examples, the one or more transceivers 220 can include one or more circuits (including a WLAN transceiver) to enable connection(s) and communication over WLAN networks such as, but not limited to, networks based on standards described in IEEE 802.11. Additionally, or alternatively, the one or more transceivers 220 can include one or more circuits (including a Bluetooth™ transceiver) to enable connection(s) and communication based on, for example, Bluetooth™ protocol, the Bluetooth™ Low Energy protocol, or the Bluetooth™ Low Energy Long Range protocol.

Additionally, the one or more transceivers 220 can include one or more circuits (including a cellular transceiver) for connecting to and communicating on cellular networks. The cellular networks can include, but are not limited to, 3G/4G/5G networks such as Universal Mobile Telecommunications System (UMTS), Long-Term Evolution (LTE), and the like. For example, the one or more transceivers 220 can be configured to operate according to one or more of Rel-15, Rel-16, Rel-17, Rel-18, NR, or other of the 3GPP standards.

According to some aspects, the processor 210, alone or in combination with computer instructions stored within the memory 250, the one or more transceivers 220, and/or antennas 260 implements mechanisms for end-to-end neural audio upsampling and bandwidth extension, as discussed herein. For example, processor 210 can include end-to-end neural audio upsampler 108 for end-to-end neural audio upsampling and bandwidth extension.

FIGS. 3A-3D illustrate exemplary configurations of an end-to-end neural audio upsampler (also referred to herein as bandwidth extender), according to some aspects. Example end-to-end neural audio upsamplers (also referred to herein as bandwidth extenders) 300, 320, and 340 are provided for the purpose of illustration only and does not limit the disclosed aspects.

FIG. 3A illustrates one exemplary end-to-end neural audio upsampler 300. End-to-end neural audio upsampler 300 includes an encoder 301 and a decoder 303. As discussed above, end-to-end neural audio upsampler 300 can be part of a receiver device, such as receiver device 104 of FIG. 1. According to some aspects, encoder 301 receives an input signal 305. Input signal 305 can be an audio signal transmitted from a transmitter device, such as transmitter device 102 of FIG. 1. In some examples, input signal 305 can be an analog signal. In some examples, input signal 305 can be a digital signal. In some examples, input signal 305 is an NB signal. In some examples, input signal 305 is a VWB signal. In some examples, input signal 305 is an SWB signal.

According to some aspects, encoder 301 receives input signal 305 and samples input signal 305 at a first sampling rate (e.g., a first resolution). Encoder 301 samples input signal 305 to generate a sampled signal. In some examples, the first sampling rate is the same as the sampling rate of input signal 305. In some examples, the first sampling rate is different from the sampling rate of input signal 305. In some examples, the first sampling rate is based on the bandwidth of input signal 305. Encoder 301 is further configured to encode the sampled signal into a feature vector 309. According to some aspects, feature vector 309 is a feature representation of the sampled signal. According to some aspects, feature vector 309 can include a number of signal dimensional vectors/elements. In a non-limiting example, feature vector 309 can include 128 signal dimensional vectors/elements. But aspects of this disclosure are not limited to this example.

According to some aspects, encoder 301 can use neural network models, machine learning (ML) models, and/or artificial intelligent (AI) models to generate feature vector 309 from the sample signal. For example, encoder 301 can use a recurrent neural network, convolutional neural network, transformer neural network, or the like to generate feature vector 309 from the sample signal. In some aspects, encoder 301 can use a convolutional neural network to generate feature vector 309 from the sample signal to allow streaming. In other words, encoder 301 does not need to wait for the entirety of the input signal (that includes input signal 305) to be received in order to process input signal 305. For example, the convolutional neural network can use 20 ms of the input signal (that includes input signal 305) to process and deliver feature vector 309. The convolutional neural network architecture can make the bandwidth extension implementable in real time with little latency.

Decoder 303 is configured to receive feature vector 309 and upsample and bandwidth extend feature vector 309 to generate output signal 311. According to some aspects, decoder 303 upsamples and bandwidth extends feature vector 309 at a desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). For example, decoder 303 receives feature vector 309 and adds the frequencies that are missing from the desired resolution to generate output signal 311 at the desired sampling rate (e.g., a desired resolution—e.g., 24 kHz).

According to some aspects, decoder 303 can use neural network models, machine learning (ML) models, and/or artificial intelligent (AI) models to generate output signal 311 from feature vector 309 from the sample signal. For example, decoder 303 can use a recurrent neural network, convolutional neural network, transformer neural network, or the like to generate output signal 311 from feature vector 309. In some aspects, decoder 303 can use a convolutional neural network to generate output signal 311 from feature vector 309 to allow streaming. In other words, end-to-end neural audio upsampler 300 does not need to wait for the entirety of the input signal (that includes input signal 305) to be received in order to process input signal 305. The convolutional neural network architecture can make the bandwidth extension implementable in real time with little latency.

According to some aspects, input signal 305 is an NB signal and output signal 311 is a WB signal and end-to-end neural audio upsampler 300 is configured to bandwidth extend the NB signal to the WB signal. According to some aspects, input signal 305 is an NB signal and output signal 311 is an SWB signal and end-to-end neural audio upsampler 300 is configured to bandwidth extend the NB signal to the SWB signal. According to some aspects, input signal 305 is a WB signal and output signal 311 is an SWB signal and end-to-end neural audio upsampler 300 is configured to bandwidth extend the WB signal to the SWB signal.

According to some aspects, encoder 301 has 4 encoder blocks, with each block containing 6 convolutional neural network layers with increasing dilation followed by a strided convolutional neural network layers for downsampling. For example, encoder neural network of encoder 301 can include a cascade of several smaller neural networks referred to as encoder blocks. In an exemplary implementation, four encoder blocks with different downsampling factors can be used. For example, for a NB input sampled at 8 kHz, encoder 301 can include four encoder blocks with downsampling factors 2, 4, 4, and 5, respectively. According to some aspects, decoder 303 mirrors encoder 301's structure with strided convolutional neural network layers, replaced by transposed convolutions for upsampling. For example, decoder neural network of decoder 303 can include a cascade of smaller decoder neural networks called decoder blocks. Each decoder block is followed by an upsampling operation. For example, decoder 303 can be configured to first upsample by a factor 10, followed by 6, 4, and 2 to output a SWB signal sampled at 24 kHz.

Exemplary encoder blocks and decoder blocks are illustrated in FIG. 3E. FIG. 3E illustrates an exemplary end-to-end neural audio upsampler 380. Encoder 381 of end-to-end neural audio upsampler 380 can include 4 encoder blocks 382a-382d. According to some aspects, each encoder block 382 can include 6 convolutional neural network layers with increasing dilation followed by a strided convolutional neural network layers for downsampling. In a non-limiting example for a NB input signal 385 sampled at 8 kHz, encoder block 382a is configured to receive input signal 385 and downsample input signal 385 by a factor of 2 to generate a first downsampled signal 386a. Encoder block 382b is configured to receive the first downsampled signal 386a and downsample it by a factor of 4 to generate a second downsampled signal 386b. Encoder block 382c is configured to receive the second downsampled signal 386b and downsample it by a factor of 4 to generate a third downsampled signal 386c. Encoder block 382d is configured to receive the third downsampled signal 386c and downsample it by a factor of 5 to generate a fourth downsampled signal. According to some aspects, feature vector 389 can be the fourth downsampled signal and/or be generated based on the fourth downsampled signal.

According to some aspects, decoder 383 mirrors encoder 381's structure with strided convolutional neural network layers, replaced by transposed convolutions for upsampling. For example, decoder neural network of decoder 383 can include a cascade of smaller decoder neural networks called decoder blocks 384a-384d. In the non-limiting example of above, decoder block 384a can receive feature vector 389 and generate, based on feature vector 389, a first upsampled signal 388a that is upsampled by a factor of 10. Decoder block 384b can receive the first upsampled signal 388a and generate, based on the first upsampled signal 388a, a second upsampled signal 388b that is upsampled by a factor of 6. Decoder block 384c can receive the second upsampled signal 388b and generate, based on the second upsampled signal 388b, a third upsampled signal 388c that is upsampled by a factor of 4. Decoder block 384d can receive the third upsampled signal 388c and generate, based on the third upsampled signal 388c, a fourth upsampled signal that is upsampled by a factor of 2. Output signal 391 is the fourth upsampled signal and/or is generated based on the fourth upsampled signal. Output signal 391 can be a SWB signal sampled at 24 kHz.

Although four encoder blocks 382 and four decoder blocks 384 are illustrated in FIG. 3E, the aspects of this disclosure are not limited to these examples, and other number of encoder blocks and decoder blocks can be used.

According to some aspects, the neural network model(s), the ML model(s), and/or AI model(s) of encoder 301 and decoder 303 are trained together. For example, a database of SWB signals is used to train encoder 301 and decoder 303. NB signals and WB signals are generated from these SW signals. Encoder 301 can use the NB signals and/or WB signal to learn feature vector(s) 309. Decoder 303 can learn to map feature vector(s) 309 to the target SWB signals (e.g. output signals 311).

In a non-limiting example, a subset of the SWB signals in a training database of SWB signals are selected for training the neural network model(s), the ML model(s), and/or AI model(s) of encoder 301 and/or decoder 303. Each SWB signal of the subset of the SWB signals is converted to an NB signal. The NB signal is an input to encoder 301. Encoder 301 uses its neural network model(s), ML model(s), and/or AI model(s) to generate feature vector(s) (e.g., feature vector(s) 309) from the NB signal. The feature vector(s) are inputs to decoder 303. Decoder 303 upsamples and bandwidth extends the feature vector(s) to generate an output signal (e.g., output signal 311). According to some aspects, decoder 303 upsamples and bandwidth extends the feature vector(s) at a desired sampling rate. Decoder 303 uses its neural network model(s), ML model(s), and/or AI model(s) to generate the output signal. The generated output signal is compared with the corresponding SWB signal that was used to generate the NB signal. The results of the comparison can be used to update one or more parameters of the neural network model(s), the ML model(s), and/or the AI model(s) of encoder 301 and/or decoder 303. This training process can be repeated for each SWB signal of the training database of SWB signals.

Although the above example are discussed with respect to NB signals generated from the SWB signals of the training database, WB signals and/or SWB signals from the training database can be used to train encoder 301 and/or decoder 303.

According to some aspects, encoder 301 and decoder 303 for extending bandwidth of NB signal to WB signal can be trained using corresponding data (e.g., NB signals and WB signals). According to some aspects, encoder 301 and decoder 303 for extending bandwidth of NB signal to SWB signal can be trained using corresponding data (e.g., NB signals and SWB signals). According to some aspects, encoder 301 and decoder 303 for extending bandwidth of WB signal to SWB signal can be trained using corresponding data (e.g., WB signals and SWB signals). Additionally, or alternatively, the training of encoder 301 and decoder 303 for different signals can be combined.

The data used to train encoder 301 and decoder 303 can include audio signals from a wide number of speakers, languages, accents, hours of speech, music data, audio book data, and the like.

FIG. 3B illustrates another exemplary end-to-end neural audio upsampler 320. End-to-end neural audio upsampler 320 includes encoders 321a and 321b and a decoder 323. Encoders 321a and 321b can be similar to encoder 301 and decoder 323 can be similar to decoder 303. As discussed above, end-to-end neural audio upsampler 320 can be part of a receiver device, such as receiver device 104 of FIG. 1. According to some aspects, encoder 321a receives input signal 325a. Input signal 325a can be an audio signal transmitted from a transmitter device, such as transmitter device 102 of FIG. 1. In some examples, input signal 325a can be an analog signal. In some examples, input signal 325a can be a digital signal. In some examples, input signal 325a is an NB signal. In some examples, input signal 325a is a WB signal. In some examples, input signal 325a is an SWB signal.

Similar to FIG. 3A, encoder 321a can generate feature vector 329a based on input signal 325a. Decoder 323 can generate output signal 331 based on feature vector 329a. According to some aspects, decoder 323 upsamples and bandwidth extends feature vector 329a at a desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). For example, decoder 323 receives feature vector 329a and adds the frequencies that are missing from the desired resolution to generate output signal 331 at the desired sampling rate (e.g., a desired resolution—e.g., 24 kHz).

Additionally, or alternatively, encoder 321b receives input signal 325b. Input signal 325b can be an audio signal transmitted from a transmitter device, such as transmitter device 102 of FIG. 1. In some examples, input signal 325b can be an analog signal. In some examples, input signal 325b can be a digital signal. In some examples, input signal 325b is an NB signal. In some examples, input signal 325b is a WB signal. In some examples, input signal 325b is an SWB signal. Similar to FIG. 3A, encoder 321b can generate feature vector 329b based on input signal 325b. Decoder 323 can generate output signal 331 based on feature vector 329b. According to some aspects, decoder 323 upsamples and bandwidth extends feature vector 329b at a desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). For example, decoder 323 receives feature vector 329b and adds the frequencies that are missing from the desired resolution to generate output signal 331 at the desired sampling rate (e.g., a desired resolution—e.g., 24 kHz).

According to some aspects, one or more encoders (305 or 325) can be used for one or more sampling rates. But the encoders will share the same decoder. For example, a first encoder (e.g., encoder 321a) can be used for 8 kHz sampling rate, a second encoder (e.g., encoder 321b) can be used for 16 kHz sampling rate, a third encoder (not shown) can be used for 20 kHz sampling rate, and/or a fourth encoder (not shown) can be used for 24 kHz sampling rate. The encoders will share the same decoder (e.g., decoder 323) that uses the feature vectors to generate the SWB signal (e.g., output signal 331). Although two encoders are shown in FIG. 3B, any number of encoders can be used with a single decoder 323.

FIG. 3C illustrates another exemplary end-to-end neural audio upsampler 340. End-to-end neural audio upsampler 340 includes encoder 342, quantizer 341 and a decoder 343. Encoder 342 and quantizer 341 can operate similar to encoder 301, and decoder 343 can be similar to decoder 303. As discussed above, end-to-end neural audio upsampler 340 can be part of a receiver device, such as receiver device 104 of FIG. 1.

FIG. 3C also illustrates encoder 342 and quantizer 341. According to some aspects, encoder 342 and quantizer 341 can be part of a transmitter device 347 (e.g., transmitter device 102 of FIG. 1). In this exemplary architecture, encoder 342 receives an input signal 344 and generates encoded signal 345 that will be quantized with quantizer 341 and input to decoder 343 of the receiver device. In some examples, input signal 344 can be an analog signal. In some examples, input signal 344 can be a digital signal.

According to some aspects, encoded and quantized signal 349 (e.g., quantized feature vector 349) can be transmitted from the transmitter device 347 (e.g., transmitter device 102 of FIG. 1). In some examples, input signal 344 can be an analog signal. In some examples, input signal 344 can be a digital signal. In some examples, input signal 344 is an NB signal. In some examples, input signal 344 is a WB signal. In some examples, input signal 344 is an SWB signal.

Similar to FIG. 3A, decoder 343 can generate output signal 351 based on received quantized feature vector 349. According to some aspects, decoder 343 upsamples and bandwidth extends feature vector 349 at a desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). For example, decoder 343 receives feature vector 349 and adds the frequencies that are missing from the desired resolution to generate output signal 351 at the desired sampling rate (e.g., a desired resolution—e.g., 24 kHz).

FIG. 3D illustrates another exemplary end-to-end neural audio upsampler 360. End-to-end neural audio upsampler 360 includes encoder 361, quantizer 362, and a decoder 363. According to some aspects, encoder 361 and quantizer 362 can operate similar to encoder 301, and decoder 363 can be similar to decoder 303. End-to-end neural audio upsampler 360 can be distributed over two receiver devices. For example, encoder 361 and quantizer 362 can be part of a first receiver device 373 (e.g., receiver device 104 of FIG. 1). Decoder 363 can be part of a second receiver device 375. In a non-limiting example, first receiver device 373 can be a mobile phone, and second receiver device 375 can be a smart watch. However, aspects of this disclosure are not limited to this example, and first and second receiver devices 373 and 375 can be other types of devices.

According to some aspects, encoder 361 receives an input signal 365. Input signal 365 can be an audio signal transmitted from a transmitter device, such as transmitter device 102 of FIG. 1. In some examples, input signal 365 can be an analog signal. In some examples, input signal 365 can be a digital signal. In some examples, input signal 365 is an NB signal. In some examples, input signal 365 is a WB signal. In some examples, input signal 365 is an SWB signal.

According to some aspects, encoder 361 receives input signal 365 and samples input signal 365 at a first sampling rate (e.g., a first resolution). Encoder 361 samples input signal 365 to generate a sampled signal. In some examples, the first sampling rate is the same as the sampling rate of input signal 365. In some examples, the first sampling rate is different from the sampling rate of input signal 365. In some examples, the first sampling rate is based on the bandwidth of input signal 365. Encoder 361 is further configured to encode the sampled signal into a feature vector 369. According to some aspects, feature vector 369 is a feature representation of the sampled signal. According to some aspects, feature vector 369 can include a number of signal dimensional vectors/elements. In a non-limiting example, feature vector 369 can include 128 signal dimensional vectors/elements. But aspects of this disclosure are not limited to this example

Quantizer 362 can be configured to sample and quantize feature vector 369. Quantizer 362 can generate quantized feature vector 370 based on feature vector 369. For example, quantizer 362 can generate quantized feature vector 370 by quantizing feature vector 369 to generate a lower bit rate version of feature vector 369 for transmitting to second receiver device 375. In other words, instead of decoding to generate a bandwidth extended signal in first receiver device 373, quantized feature vector 370 is transmitted to second receiver device 375. Second receiver device is configured to generate the bandwidth extended signal.

Decoder 363 is configured to receive quantized feature vector 370 and can generate output signal 371 based on quantized feature vector 370. According to some aspects, decoder 363 upsamples and bandwidth extends quantized feature vector 370 at a desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). For example, decoder 363 receives quantized feature vector 370 and adds the frequencies that are missing from the desired resolution to generate output signal 371 at the desired sampling rate (e.g., a desired resolution—e.g., 24 kHz). According to some aspects, before upsampling and bandwidth extending quantized feature vector 370, decoder 363 is configured to inverse quantize quantized feature vector 370. For example, decoder 363 (and/or receiver device 375) can include an inverse quantizer configured to perform an inverse quantization on quantized feature vector 370 before decoder 363 upsamples and bandwidth extends the inverse quantized feature vector. According to some aspects, using decoder 363 in second receiver device 375 (instead of first receiver device 373) can reduce delay and complexity. Rather than first performing the bandwidth extension (to, e.g., SWB) and then re-encoding this SWB signal with a coder in first receiver device 373 and then transmitting the signal to second receiver device 375, the bandwidth extension is performed in a combined fashion, reducing delay and complexity.

According to some aspects, encoders 301, 321, 342, and/or 361 can have structure similar to encoder 381 of FIG. 3E. Additionally, or alternatively, decoders 303, 323, 343, and/or 363 can have structure similar to decoder 383 of FIG. 3E.

According to some aspects, encoders 301, 321, 342, 361, and/or 381 can include encoding circuitry. For example, one or more of encoders 301, 321, 342, 361, or 381 can include hardware circuitry (e.g., dedicated hardware circuitry) configured to perform the encoding operations for the bandwidth extension operations discussed herein. Additionally, or alternatively, one or more of encoders 301, 321, 342, 361, or 381 can include or can be a processing circuitry dedicated to neural network processing configured to perform the encoding operations for the bandwidth extension operations discussed herein.

Additionally, or alternatively, decoders 303, 323, 343, 363, and/or 383 can include decoding circuitry. For example, one or more of decoders 303, 323, 343, 363, or 383 can include hardware circuitry (e.g., dedicated hardware circuitry) configured to perform the decoding operations for the bandwidth extension operations discussed herein. Additionally, or alternatively, one or more of decoders 303, 323, 343, 363, or 383 can include or can be a processing circuitry dedicated to neural network processing configured to perform the decoding operations for the bandwidth extension operations discussed herein.

FIG. 4 illustrates a method 400 for end-to-end neural audio upsampling and bandwidth extension, according to some aspects. For illustrative purposes, the operations illustrated in method 400 will be described with reference to the example end-to-end neural audio upsampler (also referred to herein as bandwidth extender) 108, 300, 320, and/or 340 in FIGS. 1-3. Method 400 may also be performed by system 200 of FIG. 2 and/or computer system 600 of FIG. 6. Additional operations may be performed between various operations of method 400 and may be omitted merely for clarity and ease of description. Additional operations can be provided before, during, and/or after method 400; one or more of these additional operations are briefly described herein. Moreover, not all operations may be needed to perform the disclosure provided herein. Additionally, some of the operations may be performed simultaneously or in a different order than shown in FIG. 4. In some aspects, one or more other operations may be performed in addition to or in place of the presently-described operations.

At 410, an input signal is received. For example, an encoder (e.g., encoder 301, 321, or quantizer 341) of an end-to-end neural audio upsampler and bandwidth extender (e.g., 108, 300, 320, or 340) receives the input signal. The input signal can be an audio signal transmitted by a transmitter device (e.g., transmitter device 102 of FIG. 1). The input signal has a first bandwidth and a first sampling rate. For example, the input signal can be an NB signal with a bandwidth of 4 kHz or less and a sampling rate of 8 kHz. The input signal can be a WB signal with a bandwidth of 8 kHz or less and a sampling rate of 16 kHz. The input signal can be a SWB signal with a bandwidth of at least 12 kHz and a sampling rate of 24 kHz or higher.