206 may be referred to as “runtime memory.” Non-volatile memory 208 may include a non-volatile magnetic medium and/or a solid-state medium (typically called a “hard drive”). Non-volatile memory 208 also may include one or more disk drives capable of reading from and writing to various types of removable, non-volatile mediums such as a removable, non-volatile magnetic disk (e.g., a “floppy disk”) and/or a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media.

Memory 204 is capable of storing program instructions and/or data such that hardware processor 202 is capable of executing the program instructions to perform one or more operations as described within this disclosure. For example, the program instructions can include an operating system, one or more application programs, other program code such as an audio driver, and program data. Hardware processor 202, in executing the computer-readable program instructions, is capable of performing the various operations described herein that are attributable to a computer.

In one or more examples, architecture 200 includes an audio processor 214. Audio processor 214 may be implemented as a hardware processor as described herein in connection with hardware processor 202. Audio processor 214 is capable of, or dedicated to, processing audio. For example, audio processor 214 may be implemented as a digital signal processor (DSP) or an audio codec. In one or more examples, one or more or all of the operations described herein may be performed by hardware processor 202. In one or more examples, one or more or all of the operations described herein may be offloaded from hardware processor 202 and performed by audio processor 214. Though called an “audio processor,” audio processor 214 is capable of processing both audio and ultrasound signals.

Architecture 200 may include one or more Input/Output (I/O) interfaces 210. I/O interface(s) 210 allow architecture 200 to communicate with one or more external devices and/or communicate over one or more networks such as a local area network (LAN), a wide area network (WAN), and/or a public network (e.g., the Internet). Examples of I/O interfaces 210 may include, but are not limited to, network cards, modems, network adapters (whether wired and/or wireless), hardware controllers, etc. Examples of external devices also may include devices that allow a user to interact with architecture 200 (e.g., a display, a keyboard, and/or a pointing device) and/or other devices such as accelerator card.

Bus 212 represents one or more of any of a variety of communication bus structures. By way of example, and not limitation, bus 212 may be implemented as a Peripheral Component Interconnect Express (PCIe) bus. Bus 212 is capable of coupling to each of microphones 102, speakers 104, hardware processor 202, memory 204, I/O interface(s) 210, and audio processor 214 (if included). The respective devices coupled to bus 212 may be coupled through respective interface circuitry. Bus 212 may represent a plurality of buses that may be interconnected and/or hierarchically organized.

In one or more other examples, microphones 102 and speakers 104 may be coupled to bus 212 and/or other components illustrated in FIG. 2 directly by way of interface circuitry. For example, interface circuitry for microphones 102 may include analog-to-digital (A/D) conversion circuitry and a bus interface. Interface circuitry for speakers 104 may include a bus interface, digital-to-analog (D/A) conversion circuitry, and amplifier circuitry to drive speakers 104. Further, such interface circuitry may support multiple channels to drive each of speakers 104 on an individual speaker-by-speaker basis. Similarly, the interface circuitry for microphones 102 may support each microphone or microphone array as the case may be such that the results from each different microphone or microphone array may be provided to the relevant processor.

It should be appreciated that the interface circuitry may support A/D and D/A sampling rates sufficient to process audio and ultrasound in observance of the Nyquist rate. For example, if ultrasound signals of 20-22 kHz are used, the sampling rate used must be at least 44 kHz to support detection of ultrasound signals of 22 kHz. Accordingly, the particular frequencies of ultrasound signals used with the examples described herein must be supported by the sampling rate available in the hardware.

As discussed, the particular starting point for what is considered to be an ultrasound signal may depend on the particular implementation of electronic device 100. That is, digital filtering, e.g., a high pass filter, may be defined and implemented by hardware processor 202 or audio processor 214 to separate audio from ultrasound signals for purposes of detecting reflected ultrasound signals. The particular cut off frequency of the high pass filter may be selected as a frequency outside of the audible range.

Architecture 200 is only one example implementation and is not intended to suggest any limitation as to the scope of use or functionality of example implementations described herein. Architecture 200 is an example of computer hardware that is capable of performing the various operations described within this disclosure. In this regard, architecture 200 may include fewer components than shown or additional components not illustrated in FIG. 2 depending upon the particular type of device and/or system that is implemented. The particular operating system and/or application(s) included may vary according to device and/or system type as may the types of I/O devices included. Further, one or more of the illustrative components may be incorporated into, or otherwise form a portion of, another component. For example, a processor may include at least some memory.

FIG. 3 illustrates a method 300 of providing spatial audio using ultrasound signals in accordance with one or more implementations of the disclosed technology. Method 300 may be performed by electronic device 100 using an architecture the same as or similar to that described in connection with FIG. 2.

Method 300 may be performed in the context of electronic device 100 playing audio through speakers 104. Accordingly, method 300 may be implemented concurrently with the playing of audio so as to provide a mechanism for dynamically adjusting the audio played, e.g., performing spatial audio, based on user presence and/or position. Within the example of FIG. 3, for purposes of illustration, electronic device 100 is described as including two channels corresponding to a left channel and a right channel. It should be appreciated that the examples may be implemented for N different channels, where N is an integer value of two or more.

In block 302, electronic device 100 is capable of emitting ultrasound signals from speakers 104. More particularly, speakers 104 may play sound in the audible range concurrently with emitting ultrasound signals and do so under control of hardware processor 202 and/or audio processor 214. The particular frequency or frequencies of ultrasound signals emitted may be selected such that the hardware of electronic device 100 is capable of sampling those frequencies in terms of the Nyquist rate. The ultrasound signals may be streamed from all channels of speakers. For purposes of discussion and illustration, electronic device 100 is presumed to include one speaker for each of the N channels. As noted, in other examples, this relationship may differ. Accordingly, in the example of FIG. 3, ultrasound signals may be emitted from both the left channel and the right channel concurrently with audio.

In block 304, electronic device 100 is capable of detecting reflected ultrasound signals using microphones 102. As discussed, the output from microphones 102 may be sampled at or above the Nyquist rate for the selected frequency or frequencies of ultrasound signals. For the remainder of FIG. 3, the term “processor” is intended to refer to hardware processor 202, audio processor 214, or both hardware processor 202 and audio processor 214 working cooperatively. In block 304, for example, the processor is capable of applying a digital filter such as a high bandpass filter so as to extract or separate the ultrasound signals from the audio to facilitate detection of reflected ultrasound signals and the processing thereof.

In block 306, electronic device 100 is capable of detecting the presence of a user based on the reflected ultrasound signals. For example, in block 306, the processor is capable of comparing the original ultrasound signals with the reflected ultrasound signals detected by microphones 102. In response to detecting that a user is present, method 300 continues to block 308. In response to detecting that no human is present, method 300 continues to block 312.

In one or more examples, the comparing may be implemented using a cross-correlation technique. The cross-correlation may be calculated with respect to, or using, a predetermined window of time referred to as the “observation window.” The observation window is applied to detect signals within the observation regions as previously discussed. The observation window may be defined as the time necessary for an ultrasound signal, as emitted from speakers 104, to be reflected by a user human body located at no more than the predetermined distance from electronic device 100 (e.g., line 114 which may be approximately 0.5 meters in some example implementations) to be received by microphones 102. As discussed, the particular predetermined distance may vary with the type of electronic device 100, geometry of speakers and microphones, and the ability of electronic device 100 to emit ultrasound signals and detect reflected ultrasound signals.

For example, electronic device 100 is capable of detecting presence of a user within the observation region of electronic device 100. When a user is present within the observation region, the ultrasound signals are at least partially absorbed and/or attenuated by the user. The absorption means that ultrasound signals are reflected back toward microphones 102 with a reduced or lesser intensity compared to as transmitted. Further, the reflected ultrasound signals are reflected back toward microphones 102 at different angles. This means that in cases where the user is present, e.g., user presence is detected, reflected ultrasound signals have a lower cross-correlation value compared to the original ultrasound signals due to partial absorption by a human body.

In cases where the user is not present, e.g., presence is not detected, the reflected ultrasound signals will have a higher cross-correlation value with the original ultrasound signals. A higher cross-correlation value indicates that original ultrasound signals have not been deformed by partial absorption by a human body. The detected ultrasound signals will have a higher cross-correlation value due to the ultrasound signals reflecting off of non-absorbing surfaces such as walls and/or other objects in the sound environment. In this case, the reflected ultrasound signals typically have similar intensities as the original ultrasound signals.

Another indication that the reflected ultrasound signals were not reflected off of a user is that reflected ultrasound signals will take longer to be detected by microphones 102 because the ultrasound signals will typically travel farther and/or reflect off of multiple different surfaces before returning to microphones 102. That is, without the user being present within the predetermined distance from, or within the observation region of, electronic device 100, ultrasound signals will continue to propagate beyond that predetermined distance before reflecting back to microphones 102. This means that the reflected signals are received outside of the observation window previously described.

In cases where the user is not within the predetermined distance of electronic device 100, the ultrasound signals still may be at least partially absorbed by the user. In this scenario, those reflected ultrasound signals that are at least partially absorbed by the user are received outside of the observation window. Accordingly, in this scenario, electronic device 100 interprets this as the user not being present.

In one or more examples, the observation window may be defined by a buffer size used to collect sampled data from each of microphones 102. That is, the amount of time or length of time of the observation window is defined by the size of the buffer given a known sampling rate. The buffer size for each microphone 102 used to store sampled, reflected ultrasound signals may be a configurable parameter that may be increased or decreased based on the expected predetermined distance of the user and/or the particular geometry of microphones 102 and speakers 104. For example, longer predetermined distances may require longer observation windows and, as such, larger buffers. Different geometries of microphones 102 and speakers 104 also may require different buffer sizes.

The processor of electronic device 100 may perform cross-correlation as described using one or more different signal processing techniques. The signal processing technique used to perform cross-correlation is used by the processor to calculate a plurality of cross-correlation values.

In one or more examples, the processor implements a classic correlation technique that compares the original ultrasound signals with the reflected ultrasound signals. For purposes of illustration, consider the case in which

$x_{ref}^{k}$

is the k^thframe of microphone data received representing reflected ultrasound signals and

$x_{j}^{k}$

is the k^thframe of the original ultrasound signal on speakers 104. The cross-correlation of the k^thframe of

$x_{ref}^{k} and the x_{j}^{k}$

frame is denoted as

$c o r r_{j}^{k}$

and is defined by Expression 1 below.

$\begin{matrix} {corr}_{j}^{k} (n) = \sum_{p = 0}^{2 * sz - 1} x_{ref}^{k} (p + n) * x_{j}^{k} (n) {(n > 0), (\forall j \neq i)) & (1) \end{matrix}$

In Expression 1, sz is the size of the observation window, n is the sample, and p is the time lag or shift between the two signals being compared. The correlation function

$c o r r_{j}^{k}$

depicts the similarity of

$x_{r e f}^{k}$

compared to the time-shifted signal

$x_{j}^{k} .$

The higher the value of the cross-correlation, the more similar both signals are to each other. The operation illustrated in Expression 1 may be performed for each of the different microphones 102 resulting in a cross-correlation value for each microphone for each frame of ultrasound played via speakers 104. In one or more examples, a frame refers to a plurality of samples.

In one or more other examples, the cross-correlation values are generated using a Long Short-Term Memory (LSTM) based machine learning model. The LSTM model may be pre-trained using ultrasound signals to classify whether a particular microphone has detected presence of the user within the observation window. The LSTM model is suited to operate on sequential data that has longer term dependencies such as the original ultrasound signals and the reflected ultrasound signals over the observation window. The LSTM model is well-suited to memorize past ultrasound signal data and perform classification on the complete data corresponding to the observation window to make a classification decision (e.g., user present or not present for any particular microphone). The LSTM model may output a numeric value that, for purposes of discussion herein, is also referred to as a cross-correlation value. As was the case for the classic correlation technique, the LSTM model may be used to generate a cross-correlation value for each microphone for each frame of ultrasound played via speakers 104.

Accordingly, in one or more examples, electronic device 100 is capable of detecting presence of a user by calculating cross-correlation of the original ultrasound signals and the reflected ultrasound signals. The processor of electronic device 100 may use a predetermined threshold cross-correlation value (the “predetermined threshold”) to compare the cross-correlation result. A lesser cross-correlation value indicates that a user is present while a higher cross-correlation value indicates that the user is not present. Accordingly, in one or more examples, electronic device 100 compares the cross-correlation value with the predetermined threshold. A cross-correlation value greater than or equal to the predetermined threshold indicates that no user is present. A cross-correlation value less than the predetermined threshold indicates that the user is present.

By using a plurality of microphones 102 such as one or more microphone arrays, not only may presence of a user be detected as described, but a position of the user may also be detected. In general, the user is considered to be present or near the particular microphones 102 that detect reflected ultrasound signals with the highest correlation. The user is considered not to be present or near microphones that detect reflected ultrasound signals having lower cross-correlation. Detecting a position of the user is described and illustrated herein in greater detail in connection with FIGS. 4A, 4B, and 4C.

Continuing with block 312 in the case where no user presence is detected, the audio played by electronic device 100 is left unchanged. That is, because no user was detected within a predetermined distance of, or within the observation region for, electronic device 100, audio being played by electronic device 100 may be played or continue to play in its original form unaltered as there is no user present based on which electronic device 100 may dynamically adapt the audio being played.

Continuing with block 308 in the case where user presence is detected, the processor is capable of calculating the particular manner in which the audio in the audible playing through speakers 104 will be adjusted. In block 308, for example, the processor is capable of generating correlation distribution coefficients and channel function(s). Block 308 is performed based on the comparison of the original ultrasound signals with the reflected ultrasound signals as detected by microphones 102.

In one or more examples, the processor is capable of calculating the distribution coefficients using the cross-correlation values. The distribution coefficients may be generated for each speaker channel and may be generated for each frame of audio data that is to be played. For example, electronic device 100 is capable of generating the plurality of coefficients based on Expression 2 below.

$\begin{matrix} {dist}^{k} (n) = f (c o r r_{j}^{k} (n)) {(n > 0 & n \leq N), (\forall j)) & (2) \end{matrix}$

In one or more examples, the processor is capable of generating the distribution coefficients dist^k(n) by normalizing the cross-correlation values (e.g., whether obtained via classic correlation or via the LSTM model). Thus,

$f (c o r r_{j}^{k} (n))$

may be a normalization function applied to the cross-correlation values. Using Expression 2, the processor is capable of generating an array of cross-correlation value (e.g., normalized cross-correlation values) for each microphone. In one or more examples, the distribution coefficients may be calculated for each of a plurality of different speaker channel to microphone mappings.

In one or more examples, the cross-correlation techniques described herein may be used to calculate the correlation between each microphone and each speaker of electronic device 100. For example, in a simplified system including left and right microphones and left and right speakers, a correlation between each pairing of left microphone and left speaker; left microphone and right speaker; right microphone and left speaker; and right microphone and right speaker may be calculated. If the user is present in the direction of the left microphone and the left speaker, the cross-correlation is low due to absorption. In such cases, different cross-correlation values as normalized may be combined (e.g., summed or weighted and summed) depending on the particular speaker configuration (e.g., number of speakers, microphone to speaker mapping, and speaker orientations) where each different cross-correlation result may be mapped to a particular speaker/channel and weighted accordingly to obtain a final cross-correlation result as normalized to be used for the speaker/channel in calculating the gain mask for that speaker/channel.

As part of block 308, the processor is capable of generating gain masks from the distribution coefficients. In one or more examples, the processor is capable of generating the gain masks by inverting the respective distribution coefficients. For example, referring to the normalized distribution coefficients, the processor may generate the gain masks according to the expression (1−distribution_coefficient).

The processor is capable of applying the gain masks generated to the audio being played by the device to ensure that the audio being rendered through speakers 104 is steered toward the particular speaker deemed closest to user. By steering audio toward the speaker closest to the user (e.g., steering audio to speakers based on distance of the speaker to the user's position), and continuing to do so dynamically over time as the user moves and audio is played, the user is provided with a more consistent audio experience as the user moves about electronic device 100 or at least in the observation region. Appreciably, the steering may steer audio to one or more speakers based on distance of the speaker to the user. The user is better able to hear audio content from the different channels as each channel is adjusted by steering the audio toward the user.

In block 310, in response to the detecting the presence of the user and/or position of the user, the processor of electronic device 100 is capable of adjusting audio played through one or more of the plurality of speakers as audible sound. For example, the processor is capable adjusting audio of one or more channels of a plurality of channels played through respective ones of speakers 104. The adjusting, as performed by the processor, may include steering audio of one or more audio channels to at least one speaker of the plurality of speakers based on a distance of the at least one speaker and the position of the user.

For purposes of illustration, consider the case where the number of channels (of audio) that electronic device 100 may play are correlated with the speakers on a one-to-one basis. Consider an example in which the user is positioned in front of speaker 104-3. In that case, the gain masks may cause the left speaker to play the audio of the left channel at a reduced volume while diverting at least a portion of the audio of the left channel to the right channel to be played through speaker 104-3. The diverted portion of the left channel audio may be summed with the right channel audio and played through speaker 104-3. The gain masks effectively create a map of the reflected ultrasound signals detected by microphones 102 to speakers 104.

In block 314, electronic device 100 is capable of playing audio through speakers 104. In arriving at block 314 from block 312, the audio is played without modification or adjustment. In arriving at block 314 from block 310, the audio is played as adjusted in block 310.

In the case where audio being played is adjusted, the gain masks may be provided to an audio driver, plug-in, or other software component that may be executed by hardware processor 202 or audio processor 214 that controls rendering of audio to the different channels and, as such, to the different speakers 104. As noted, audio may be steered toward the user based on presence and position of the user. As an example, if the audio stream rendered by an application is represented as x(n) and y(n) represents the audio stream after adjustment based on user presence and position, then y(n) may be generated according to Expression 3.

$\begin{matrix} y^{k} (n) = f ({dist}^{k} (n), x^{k} (n)) {(n > 0), (\forall j)) & (3) \end{matrix}$

The steering of the audio stream toward the user may be performed linearly to avoid any sudden changes in audio the audio experience. To provide the linear behavior, the distribution function ƒ in Expression 3 above is implemented as a piece-wise linear function for the k^thframe. In one or more examples, the distribution function dist^k(n) generates a gain mask

$α_{j}^{k}$

for each frame per channel for the rendered audio stream as illustrated in Expression 4 below where k represents the frame, j represents the playback channel, and

$α_{j}^{k} (n)$

represents the gain mask derived for every k^thframe for the j^thchannel.

$\begin{matrix} y_{j}^{k} (n) = \sum_{n = 1}^{N} α_{j}^{k} (n) * x_{j}^{k} (n) {(n > 0 & n \leq N), (\forall j)) & (4) \end{matrix}$

For example, in case of N channel audio playback where N=2, Expressions 5 and 6 are illustrative of the audio output from each of the two channels.

$\begin{matrix} y_{1}^{k} (1) = α_{1}^{k} (1) * x_{1}^{k} (1) + α_{1}^{k} (2) * x_{2}^{k} (2) & (5) \end{matrix}$

$\begin{matrix} y_{2}^{k} (2) = α_{2}^{k} (1) * x_{1}^{k} (1) + α_{2}^{k} (2) * x_{2}^{k} (2) & (6) \end{matrix}$

In Expressions 5 and 6,

$x_{1}^{k} (1) and x_{2}^{k} (2)$

represent the original audio data for channels 1 and 2 respectively for each frame.

$y_{1}^{k} (1) and y_{2}^{k} (2)$

represent the modified spatial audio data for each frame generated from original audio data using gain mask sets

${α_{1}^{k} (1), α_{1}^{k} (2)} and {α_{2}^{k} (1), α_{2}^{k} (2)}$

for channels 1 and 2, respectively. It should be appreciated that the gain masks may be generated based on the ultrasound signal processing described and, as such, may be applied to the audio being rendered in real-time and/or substantially real-time. The examples described herein seek to maintain consistent audio quality across audio channels despite changes in the user position occurring over time.

Method 300 may continue to loop back to block 302 to continually and dynamically adjust audio played by electronic device 100 to provide a spatial audio experience as the user continues to move and/or be present (or not) relative to electronic device 100. The implementations described herein are capable of tracking user position based on presence detection data and, based on that data, adjust audio played from the different speakers by, at least in part, changing the distribution of audio across the different channels based on the user position.

FIGS. 4A, 4B, and 4C illustrate audio adjustments for spatial audio implemented by electronic device 100 based on user presence and position in accordance with one or more examples of the disclosed technology. Within each of FIGS. 4A, 4B, and 4C, speakers 104 are not illustrated as the examples may be applied to any of a variety of speaker numbers and/or geometries.

Appreciably, depending on the speaker numbers and/or geometries, certain parameters such as the observation window and/or the mapping of microphones to particular speakers (audio channels) may vary. The mappings may be one-to-one (one microphone/microphone array to one speaker), one-to-many (one microphone/microphone array to two or more speakers), or a combination thereof. In general, however, the greater the predetermined distance for which the user may be detected, the larger the buffer size (observation window) required. Further, in general, down-firing speakers (e.g., FIG. 1C) will utilize a larger observation window compared to up-firing speakers (e.g., FIG. 1B).

Within each of FIGS. 4A, 4B, and 4C, graph 404 illustrates the correlation values generated using cross-correlation for the microphones 102. Graph 406 illustrates the steering implemented to provide spatial audio based on graph 404.

In the example of FIG. 4A, the user is positioned in front of speaker 104-3 (on the right). As illustrated in graph 404-1, the correlation values calculated based on reflected ultrasound waves 402 from user 110 are lower for the microphones 102 toward the right than those on the left due to absorption. This causes the driver to steer audio toward speaker 104-3 and the user as illustrated by graph 406-1.

In the example of FIG. 4B, the user is positioned in front of speaker 104-1 (on the left). As illustrated in graph 404-2, the correlation values calculated based on reflected ultrasound waves 402 from user 110 are lower for the microphones 102 toward the left than those on the right due to absorption. This causes the driver to steer audio toward speaker 104-1 and the user as illustrated by graph 406-2.

In the example of FIG. 4C, the user is positioned in front of speaker 104-2 (in the center). As illustrated in graph 404-3, the correlation values calculated based on reflected ultrasound waves 402 from user 110 are lower for the microphones 102 on both the left and right edges than in the center due to absorption. This causes the driver to steer audio toward speaker 104-2 and the user as illustrated by graph 406-3.

FIG. 5 illustrates a simplified example of audio adjustments performed by electronic device 100 to implement spatial audio based on user presence and position in accordance with one or more examples of the disclosed technology. The example of FIG. 5 illustrates one example of how audio may be adjusted and delivered to different channel speakers 104-1 and 104-3. For purposes of illustration, only two speakers are illustrated in the example of FIG. 5. In the example of FIG. 5, the processor, by way of the gain masks, adjusts an amount of audio corresponding to each channel played through one or more of the plurality of speakers.

An audio driver 502 includes a channel 504 (e.g., a processing pipeline) for each speaker. As illustrated, audio driver 502 includes a channel 504-1 for speaker 104-1 and a channel 504-2 for speaker 104-3. In the example, each channel 504 receives both left channel audio and right channel audio. Channel 504-1 applies a gain mask α1 to the left channel audio and a gain mask α2 to the right channel audio. The results are summed and output as final left channel audio to speaker 104-1. Channel 504-2 applies a gain mask α3 to the left channel audio and a gain mask α4 to the right channel audio. The results are summed and output as final right channel audio to speaker 104-3.

Appreciably, the gain masks α1, α2, α3, and α4 are adjusted dynamically in real time, over time, based on detected presence of the user and, when presence is detected, the detected position of the user in front electronic device 100. FIG. 5 illustrates that the different channels of audio may be dynamically redirected and steered toward the user based on the user's position. In the example of FIG. 5, not only the gain or volume of each channel of audio is being adjusted, but also the particular channel speaker to which the audio is directed may change by adjusting the gain masks as illustrated. That is, the amount of each audio channel carried or played by a particular channel speaker may be adjusted dynamically in real time. This ensures that despite the position of the user in front of electronic device