11.

Further, for example, a footstep emitted from a spike or the like serving as the sound source has a directional characteristic (non-directivity) in which a sound propagates with substantially the same strength in all directions as indicated by an arrow Q12.

Furthermore, for example, a voice emitted from a mouth of a player serving as the sound source has a directional characteristic in which a sound strongly propagates toward the front and sides, that is, has a relatively strong front directivity as indicated by an arrow Q13.

Directional characteristic data indicating the directional characteristics of such sound sources can be obtained by acquiring a propagation characteristic (transfer characteristic) of a sound to the surroundings for each sound source type by using a microphone array in, for example, an anechoic chamber or the like. In addition, the directional characteristic data can also be obtained by, for example, performing a simulation on 3D data in which a shape of the sound source is simulated.

Specifically, the directional characteristic data is, for example, a gain function dir(i, ψ, θ) defined as a function of an azimuth angle ψ and elevation angle θ indicating a direction viewed from the sound source, the function being determined for a value i of an ID indicating the sound source type.

Further, a gain function dir(i, d, ψ, θ) having not only the azimuth angle ψ and the elevation angle θ but also a distance d from a discretized sound source as arguments may be used as the directional characteristic data.

In this case, when each argument is substituted into the gain function dir(i, d, ψ, θ), a gain value indicating a sound transfer characteristic (propagation characteristic) is obtained as an output of the gain function dir(i, d, ψ, θ).

The gain value indicates a characteristic (transfer characteristic) of a sound that is emitted from the sound source of the sound source type whose ID value is i, propagates in a direction of the azimuth angle ψ and elevation angle θ viewed from the sound source, and reaches a position (hereinafter, referred to as a position P) at the distance d from the sound source.

Therefore, in a case where audio data of the sound source type whose ID value is i is subjected to gain correction on the basis of the gain value, it is possible to reproduce the sound emitted from the sound source of the sound source type whose ID value is i and supposed to be actually heard at the position P.

In particular, in this example, in a case where the gain value serving as the output of the gain function dir(i, d, ψ, θ) is used, it is possible to achieve gain correction for adding the transfer characteristic indicated by the directional characteristic in which the distance from the sound source, that is, distance attenuation is considered.

Note that the directional characteristic data may be, for example, a gain function indicating the transfer characteristic in which a reverberation characteristic or the like is also considered. In addition, the directional characteristic data may be, for example, Ambisonics format data, that is, data including a spherical harmonic coefficient (spherical harmonic spectrum) in each direction.

The transmission device transmits the directional characteristic data prepared for each sound source type as described above to the signal processing device on the reproduction side.

Herein, a specific example of transmitting the metadata and the directional characteristic data will be described.

For example, the metadata is prepared for each frame having a predetermined time length of the audio data of the object, and the metadata is transmitted for each frame to the reproduction side by using a bitstream syntax illustrated in FIG. 3. Note that, in FIG. 3, uimsbf represents unsigned integer MSB first, and tcimsbf represents two's complement integer MSB first.

In an example of FIG. 3, the metadata includes sound source type information “Object_type_index”, sound source position information “Object_position[3]”, and sound source direction information “Object_direction[3]” for each object included in the content.

In particular, in this example, the sound source position information Object_position[3] is set as coordinates (x_o, y_o, z_o) of an xyz coordinate system (three-dimensional orthogonal coordinate system) taking, as an origin, a predetermined reference position in a target space in which the object is located. The coordinates (x_o, y_o, z_o) indicate an absolute position of the object in the xyz coordinate system, that is, in the target space.

Further, the sound source direction information Object_direction[3] includes the azimuth angle ψ_o, the elevation angle θ_o, and the tilt angle φ_oindicating an absolute direction of the object in the target space.

For example, in a free-viewpoint content, a viewpoint (listening position) changes with time during reproduction of the content. Therefore, it is advantageous to generate a reproduction signal when the position of the object is expressed by coordinates indicating the absolute position, instead of relative coordinates based on the listening position.

Meanwhile, for example, in a case of a fixed-viewpoint content, coordinates of a polar coordinate system including an azimuth angle and elevation angle indicating a direction of the object viewed from the listening position and a radius indicating a distance from the listening position to the object are preferably set as the sound source position information indicating the position of the object.

Note that the configuration of the metadata is not limited to the example of FIG. 3 and may be any other configuration. Further, it is only necessary to transmit the metadata at predetermined time intervals, and it is not always necessary to transmit the metadata for each frame.

Furthermore, the directional characteristic data of each sound source type may be stored in the metadata and then be transmitted, or may be transmitted in advance separately from the metadata and the audio data by using, for example, a bitstream syntax illustrated in FIG. 4.

In an example of FIG. 4, a gain function “Object_directivity[distance] [azimuth] [elevation]” having a distance “distance” from the sound source and an azimuth angle “azimuth” and elevation angle “elevation” indicating a direction viewed from the sound source as arguments are transmitted as directional characteristic data corresponding to a value of predetermined sound source type information.

Note that the directional characteristic data may be data in a format in which sampling intervals of the azimuth angle and elevation angle serving as the arguments are not equiangular intervals, or may be data in a higher order Ambisonmics (HOA) format, that is, in an Ambisonics format (spherical harmonic coefficient).

For example, directional characteristic data of a general sound source type is preferably transmitted to the reproduction side in advance.

Meanwhile, directional characteristic data of a sound source having a non-general directional characteristic, such as an object that is not defined in advance, may be included in the metadata of FIG. 3 and be transmitted as the metadata.

As described above, the metadata, the audio data, and the directional characteristic data are transmitted from the transmission device to the signal processing device on the reproduction side.

Next, the signal processing device, which is a device on the reproduction side, will be described.

For example, the signal processing device on the reproduction side is configured as illustrated in FIG. 5.

A signal processing device 11 of FIG. 5 generates a reproduction signal for reproducing a sound of content (object) at a listening position on the basis of the directional characteristic data acquired from the transmission device or the like in advance or shared in advance, and outputs the reproduction signal to a reproduction unit 12.

For example, the signal processing device 11 generates a reproduction signal by performing processing for vector based amplitude panning (VBAP) or wave field synthesis, head related transfer function (HRTF) convolution processing, or the like by using the directional characteristic data.

The reproduction unit 12 includes, for example, headphones, earphones, a speaker array including two or more speakers, and the like, and reproduces a sound of the content on the basis of the reproduction signal supplied from the signal processing device 11.

Further, the signal processing device 11 includes an acquisition unit 21, a listening position designation unit 22, a directional characteristic database unit 23, and a signal generation unit 24.

The acquisition unit 21 acquires the directional characteristic data, the metadata, and the audio data by, for example, receiving data transmitted from the transmission device or reading data from the transmission device connected by wire or the like.

Note that a timing of acquiring the directional characteristic data and a timing of acquiring the metadata and the audio data may be the same or different.

The acquisition unit 21 supplies the acquired directional characteristic data and metadata to the directional characteristic database unit 23 and also supplies the acquired metadata and audio data to the signal generation unit 24.

The listening position designation unit 22 designates a listening position in a target space and a direction of the listener (user) who is at the listening position, and supplies, as the designation result, listening position information indicating the listening position and listener direction information indicating the direction of the listener to the signal generation unit 24.

The directional characteristic database unit 23 records the directional characteristic data for each of a plurality of sound source types supplied from the acquisition unit 21.

Further, in a case where the sound source type information included in the metadata is supplied from the acquisition unit 21, the directional characteristic database unit 23 supplies, among the plurality of pieces of recorded directional characteristic data, directional characteristic data of a sound source type indicated by the supplied sound source type information to the signal generation unit 24.

The signal generation unit 24 generates a reproduction signal on the basis of the metadata and audio data supplied from the acquisition unit 21, the listening position information and listener direction information supplied from the listening position designation unit 22, and the directional characteristic data supplied from the directional characteristic database unit 23, and supplies the reproduction signal to the reproduction unit 12.

The signal generation unit 24 includes a relative distance calculation unit 31, a relative direction calculation unit 32, and a directivity rendering unit 33.

The relative distance calculation unit 31 calculates a relative distance between the listening position (listener) and the object on the basis of the sound source position information included in the metadata supplied from the acquisition unit 21 and the listening position information supplied from the listening position designation unit 22, and supplies relative distance information indicating the calculation result to the directivity rendering unit 33.

The relative direction calculation unit 32 calculates a relative direction between the listener and the object on the basis of the sound source position information and sound source direction information included in the metadata supplied from the acquisition unit 21 and the listening position information and listener direction information supplied from the listening position designation unit 22, and supplies relative direction information indicating the calculation result to the directivity rendering unit 33.

The directivity rendering unit 33 performs rendering processing on the basis of the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information and listener direction information supplied from the listening position designation unit 22.

The directivity rendering unit 33 supplies a reproduction signal obtained by the rendering processing to the reproduction unit 12 and causes the reproduction unit 12 to reproduce the sound of the content. For example, the directivity rendering unit 33 performs the processing for VBAP or wave field synthesis, the HRTF convolution processing, or the like as the rendering processing.

(Listening Position Designation Unit)

Next, each unit of the signal processing device 11 will be described in more detail.

The listening position designation unit 22 designates the listening position and the direction of the listener in response to a user operation or the like.

For example, in a case of the free-viewpoint content, the user who is viewing the content, that is, the listener operates a graphical user interface (GUI) or the like in a service, an application, or the like that is currently executed, thereby designating an arbitrary listening position or direction of the listener.

In this case, the listening position designation unit 22 sets the listening position and the direction of the listener designated by the user as the listening position (viewpoint position) serving as a viewpoint of the content and the direction in which the listener faces, that is, the direction of the listener as they are.

Further, for example, when the user designates a desired player from a plurality of predetermined players or the like, a position and direction of the player may be set as the listening position and the direction of the listener.

Furthermore, the listening position designation unit 22 may execute some automatic routing program or the like or acquire information indicating the position and direction of the user from a head mounted display including the reproduction unit 12, thereby designating an arbitrary listening position and direction of the listener without receiving a user operation.

As described above, in the free-viewpoint content, the listening position and the direction of the listener are set as an arbitrary position and arbitrary direction that can change with time.

Meanwhile, in the fixed-viewpoint content, the listening position designation unit 22 designates a predetermined fixed position and fixed direction as the listening position and the direction of the listener.

A specific example of the listening position information indicating the listening position is, for example, coordinates (x_v, y_v, z_v) indicating the listening position in an xyz coordinate system indicating an absolute position on the earth's surface or an xyz coordinate system indicating an absolute position in the target space.

Further, for example, the listener direction information can be information including an azimuth angle v and elevation angle θ_vindicating the absolute direction of the listener in the xyz coordinate system and a tilt angle φ_vthat is an angle of absolute rotation (tilt) of the listener in the xyz coordinate system, that is, can be an Euler angle.

In particular, in this case, in the fixed-viewpoint content, it is only necessary to set, for example, the listening position information (x_v, y_v, z_v)=(0, 0, 0) and the listener direction information (ψ_v, θ_v, φ_v)=(0, 0, 0).

Note that, hereinafter, description will be continued on the assumption that the listening position information is the coordinates (x_v, y_v, z_v) in the xyz coordinate system and the listener direction information is the Euler angle (ψ_v, θ_v, φ_v).

Similarly, hereinafter, description will be continued on the assumption that the sound source position information is the coordinates (x_o, y_o, z_o) in the xyz coordinate system and the sound source direction information is the Euler angle (ψ_o, θ_o, φ_o).

(Relative Distance Calculation Unit)

The relative distance calculation unit 31 calculates a distance from the listening position to the object as a relative distance d_ofor each object included in the content.

Specifically, the relative distance calculation unit 31 obtains the relative distance d_oby calculating the following expression (1) on the basis of the listening position information (x_v, y_v, z_v) and the sound source position information (x_o, y_o, z_o), and outputs relative distance information indicating the obtained relative distance d_o.

[

Math
.

1

]



d
o

=

sqrt
⁡
(

(

x
o

-

x
v

)

2

+

(

y
o

-

y
v

)

2

+

(

z
o

-

z
v

)

2

)

(
1
)

(Relative Direction Calculation Unit)

Further, the relative direction calculation unit 32 obtains relative direction information indicating a relative direction between the listener and the object.

For example, the relative direction information includes an object azimuth angle ψ_{i_obj}, an object elevation angle θ_{i_obj}, an object rotation azimuth angle ψ_rot_{i_obj}, and an object rotation elevation angle θ_rot_{i_obj}.

Herein, the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}are an azimuth angle and an elevation angle, each of which indicates a relative direction of the object viewed from the listener.

A three-dimensional orthogonal coordinate system, which takes a position indicated by the listening position information (x_v, y_v, z_v) as an origin and is obtained by rotating the xyz coordinate system by an angle indicated by the listener direction information (ψ_v, θ_v, φ_v), will be referred to as a listener coordinate system. In the listener coordinate system, the direction of the listener, that is, a front direction of the listener is set as a +y direction.

At this time, the azimuth angle and elevation angle indicating the direction of the object in the listener coordinate system are the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}.

Similarly, the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}are an azimuth angle and an elevation angle, each of which indicates a relative direction of the listener (listening position) viewed from the object. In other words, it can be said that the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}are information indicating how much a front direction of the object is rotated with respect to the listener.

A three-dimensional orthogonal coordinate system, which takes a position indicated by the sound source position information (x_o, y_o, z_o) as an origin and is obtained by rotating the xyz coordinate system by an angle indicated by the sound source direction information (ψ_o, θ_o, φ_o), will be referred to as an object coordinate system. In the object coordinate system, the direction of the object, that is, the front direction of the object is set as a +y direction.

At this time, the azimuth angle and elevation angle indicating the direction of the listener (listening position) in the object coordinate system are the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}.

Those object rotation azimuth angle ψ_rot_{i_obj}and object rotation elevation angle θ_rot_{i_obj}are an azimuth angle and elevation angle used to refer to the directional characteristic data during the rendering processing.

Note that, in the following description, a clockwise direction from the front direction (+y direction) of the azimuth angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system is set as a positive direction.

For example, in the xyz coordinate system, an angle that, after a target point such as the object is projected onto an xy plane, indicates a position (direction) of the projected target point based on the +y direction in the xy plane, that is, an angle between a direction of the projected target point and the +y direction is set as the azimuth angle. At this time, the clockwise direction from the +y direction is a positive direction.

Further, in the listener coordinate system or object coordinate system, the direction of the listener or object, that is, the front direction of the listener or object is the +y direction.

An upward direction of the elevation angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system is set as a positive direction.

For example, in the xyz coordinate system, an angle between the xy plane and a straight line passing through the origin of the xyz coordinate system and the target point such as the object is the elevation angle.

Further, in a case where the target point such as the object is projected onto the xy plane and a plane including the origin of the xyz coordinate system, the target point, and the projected target point is set as a plane A, a +z direction from the xy plane is set as the positive direction of the elevation angle on the plane A.

Note that, for example, in the case of the listener coordinate system or object coordinate system, the object or listening position serves as the target point.

Further, in a case where, after the elevation angle rotates, the tilt angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system rotates in an upper right direction while the +y direction serves as the front direction, such rotation is set as rotation in the positive direction.

Note that, herein, the azimuth angle, the elevation angle, and the tilt angle indicating the listening position, the direction of the object, and the like in the three-dimensional orthogonal coordinate system are defined as described above. However, the present technology is not limited thereto and does not lose generality even in a case where those angles are defined in another way by using quaternion, a rotation matrix, or the like.

Herein, specific examples of the relative distance d_o, the object azimuth angle ψ_{i_obj}, the object elevation angle θ_{i_obj}, the object rotation azimuth angle ψ_rot_{i_obj}, and the object rotation elevation angle θ_rot_{i_obj}will be described.

First, there will be described a case where only the azimuth angle is considered and the elevation angle and the tilt angle are not considered in the sound source direction information and the listener direction information, that is, a two-dimensional case.

For example, as illustrated in FIG. 6, a position of a point P21 in an xy coordinate system having an origin O as a reference is set as the listening position, and the object is located at a position of a point P22.

Further, a direction of a line segment W11 passing through the point P21, more specifically, a direction from the point P21 toward an end point of the line segment W11 opposite to the point P21 is set as the direction of the listener.

Similarly, a direction of a line segment W12 passing through the point P22 is set as the direction of the object. Further, a straight line passing through the point P21 and the point P22 is defined as a straight line L11.

In this case, a distance between the point P21 and the point P22 is set as the relative distance d_o.

Further, an angle between the line segment W11 and the straight line L11, that is, an angle indicated by an arrow K11 is the object azimuth angle ψ_{i_obj}. Similarly, an angle between the line segment W12 and the straight line L11, that is, an angle indicated by an arrow K12 is the object rotation azimuth angle ψ_rot_{i_obj}.

Further, in a case of a three-dimensional target space, the relative distance d_o, the object azimuth angle ψ_{i_obj}, the object elevation angle θ_{i_obj}, the object rotation azimuth angle ψ_rot_{i_obj}, and the object rotation elevation angle θ_rot_{i_obj}are as illustrated in FIGS. 7 to 9. Note that corresponding parts in FIGS. 7 to 9 are denoted by the same reference signs, and description thereof will be omitted as appropriate.

For example, as illustrated in FIG. 7, positions of points P31 and P32 in an xyz coordinate system having an origin O as a reference are set as the listening position and the position of the object, respectively, and a straight line passing through the point P31 and the point P32 is set as a straight line L31.

Further, a plane, which is obtained by rotating an xy plane of the xyz coordinate system by an angle indicated by the listener direction information (ψ_v, θ_v, φ_v) and then translating the origin O to a position indicated by the listening position information (x_v, y_v, z_v), is set as a plane PF11. The plane PF11 is an xy plane of the listener coordinate system.

Similarly, a plane, which is obtained by rotating the xy plane of the xyz coordinate system by an angle indicated by the sound source direction information (ψ_o, θ_o, φ_o) and then translating the origin O to a position indicated by the sound source position information (x₀, y_o, z_o), is set as a plane PF12. The plane PF12 is an xy plane of the object coordinate system.

Further, a direction of a line segment W21 passing through the point P31, more specifically, a direction from the point P31 toward an end point of the line segment W21 opposite to the point P31 is set as the direction of the listener indicated by the listener direction information (ψ_v, θ_v, φ_v).

Similarly, a direction of a line segment W22 passing through the point P32 is set as the direction of the object indicated by the sound source direction information (ψ_o, θ_o, φ_o).

In such a case, a distance between the point P31 and the point P32 is set as the relative distance d_o.

Further, as illustrated in FIG. 8, in a case were a straight line obtained by projecting the straight line L31 onto the plane PF11 is set as a straight line L41, an angle between the straight line L41 and the line segment W21 on the plane PF11, that is, an angle indicated by an arrow K21 is the object azimuth angle ψ_{i_obj}.

Furthermore, an angle between the straight line L41 and the straight line L31, that is, an angle indicated by an arrow K22 is the object elevation angle θ_{i_obj}. In other words, the object elevation angle θ_{i_obj}is an angle between the plane PF11 and the straight line L31.

Meanwhile, as illustrated in FIG. 9, in a case where a straight line obtained by projecting the straight line L31 onto the plane PF12 is set as a straight line L51, an angle between the straight line L51 and the line segment W22 on the plane PF12, that is, an angle indicated by an arrow K31 is the object rotation azimuth angle ψ_rot_{i_obj}.

Further, an angle between the straight line L51 and the straight line L31, that is, an angle indicated by an arrow K32 is the object rotation elevation angle θ_rot_{i_obj}. In other words, the object rotation elevation angle θ_rot_{i_obj}is an angle between the plane PF12 and the straight line L31.

Specifically, the object azimuth angle ψ_{i_obj}, the object elevation angle θ_{i_obj}, the object rotation azimuth angle ψ_rot_{i_obj}, and the object rotation elevation angle θ_rot_{i_obj}described above, that is, the relative direction information can be calculated as follows, for example.

For example, a rotation matrix describing rotation in the three-dimensional space is shown by the following expression (2).

[

Math
.

2

]



(

x
′

y
′

z
′

)

=

(

cos
⁢

ψ

0

sin
⁢

ψ

0

1

0

-
sin

⁢

ψ

0

cos
⁢

ψ

)

⁢

(

1

0

0

0

cos
⁢

θ

-
sin

⁢

θ

0

sin
⁢

θ

cos
⁢

θ

)

⁢

(

cos
⁢

ϕ

-
sin

⁢

ϕ

0

sin
⁢

ϕ

cos
⁢

ϕ

0

0

0

1

)

⁢

(

x

y

z

)

(
2
)

Note that, in the expression (2), coordinates (x, y, z) in an X₁Y₁Z₁space that is a space of a three-dimensional orthogonal coordinate system having predetermined X₁, Y₁, and Z₁axes are rotated by the rotation matrix, and rotated coordinates (x′, y′, z′) are obtained.

That is, in the calculation shown by the expression (2), the second matrix from the right on the right side is a rotation matrix for rotating the X₁Y₁Z₁space about the Z₁axis by the angle φ in an X₁Y₁plane to obtain a rotated X₂Y₂Z₁space. In other words, the coordinates (x, y, z) are rotated by an angle −φ on the X₁Y₁plane by the second rotation matrix from the right on the right side.

Further, the third matrix from the right on the right side of the expression (2) is a rotation matrix for rotating the X₂Y₂Z₁space about an X₂axis by the angle θ in a Y₂Z₁plane to obtain a rotated X₂Y₃Z₂space.

Furthermore, the fourth matrix from the right on the right side of the expression (2) is a rotation matrix for rotating the X₂Y₃Z₂space about a Y₃axis by the angle ψ in an X₂Z₂plane to obtain a rotated X₃Y₃Z₃space.

The relative direction calculation unit 32 generates the relative direction information by using the rotation matrixes shown by the expression (2).

Specifically, the relative direction calculation unit 32 calculates the following expression (3) on the basis of the sound source position information (x_o, y_o, z_o) and the listener direction information (ψ_v, θ_v, φ_v), thereby obtaining rotated coordinates (x_o′, y_o′, z_o′) of the coordinates (x_o, y_o, z_o) indicated by the sound source position information.

[

Math
.

3

]



(

x
o
′

y
o
′

z
o
′

)

=

(

cos
⁢

ψ

0

sin
⁢

ψ

0

1

0

-
sin

⁢

ψ

0

cos
⁢

ψ

)

⁢

(

1

0

0

0

cos
⁢

θ

-
sin

⁢

θ

0

sin
⁢

θ

cos
⁢

θ

)

⁢

(

cos
⁢

ϕ

-
sin

⁢

ϕ

0

sin
⁢

ϕ

cos
⁢

ϕ

0

0

0

1

)

⁢

(

x
o

y
o

z
o

)

(
3
)

In the calculation of the expression (3), φ=−φ_v, θ=−θ_v, and ψ=−ψ_vare set, and the rotation matrixes are calculated.

The coordinates (x_o′, y_o′, z_o′) thus obtained indicate the position of the object in the listener coordinate system. However, the origin of the listener coordinate system herein is not the listening position but is the origin O of the xyz coordinate system in the target space.

Next, the relative direction calculation unit 32 calculates the following expression (4) on the basis of the listening position information (x_v, y_v, z_v) and the listener direction information (ψ_v, θ_v, φ_v), thereby obtaining rotated coordinates (x_v′, y_v′, z_v′) of the coordinates (x_v, y_v, z_v) indicated by the listening position information.

[

Math
.

4

]



(

x
v
′

y
v
′

z
v
′

)

=

(

cos
⁢

ψ

0

sin
⁢

ψ

0

1

0

-
sin

⁢

ψ

0

cos
⁢

ψ

)

⁢

(

1

0

0

0

cos
⁢

θ

-
sin

⁢

θ

0

sin
⁢

θ

cos
⁢

θ

)

⁢

(

cos
⁢

ϕ

-
sin

⁢

ϕ

0

sin
⁢

ϕ

cos
⁢

ϕ

0

0

0

1

)

⁢

(

x
v

y
v

z
v

)

(
4
)

In the calculation of the expression (4), φ=−φ_v, θ=−θ_v, and ψ=−ψ_vare set, and the rotation matrixes are calculated.

The coordinates (x_v′, y_v′, z_v′) thus obtained indicate the listening position in the listener coordinate system. However, the origin of the listener coordinate system herein is not the listening position but is the origin O of the xyz coordinate system in the target space.

Further, the relative direction calculation unit 32 calculates the following expression (5) on the basis of the coordinates (x_o′, y_o′, z_o′) calculated from the expression (3) and the coordinates (x_v′, y_v′, z_v′) calculated from the expression (4).

[

Math
.

5

]



(

x
o
″

y
o
″

z
o
″

)

=

(

x
o
′

y
o
′

z
o
′

)

-

(

x
v
′

y
v
′

z
v
′

)

(
5
)

The expression (5) is calculated to obtain coordinates (x_o″, y_o″, z_o″) indicating the position of the object in the listener coordinate system taking the listening position as the origin. The coordinates (x_o″, y_o″, z_o″) indicate a relative position of the object viewed from the listener.

The relative direction calculation unit 32 calculates the following expressions (6) and (7) on the basis of the coordinates (x_o″, y_o″, z_o″) obtained as described above, thereby obtaining the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}.

[

Math
.

6

]



ψ

i
⁢
_
⁢
obj

=

arc
⁢
tan
⁢

(

y
o
″

/

x
o
″

)

(
6
)

[

Math
.

7

]



θ

i
⁢
_
⁢
obj

=

arc
⁢
tan
⁢

(

z
o
″

/

sqrt
⁡
(

x
o
″2

+

y
o
″2

)

)

(
7
)

In the expression (6), the object azimuth angle ψ_{i_obj}is obtained on the basis of x_o″ and y_o″ that are the x coordinate and the y coordinate.

Note that, more specifically, in the calculation of the expression (6), the object azimuth angle ψ_{i_obj}is calculated by performing proof-by-cases processing on the basis of a sign of y_o″ and a result of zero determination on x_o″ and performing exception processing on the basis of a result of the proof by cases. However, detailed description thereof will be omitted herein.

Further, in the expression (7), the object elevation angle θ_{i_obj}is obtained on the basis of the coordinates (x_o″, y_o″, z_o″). Note that, more specifically, in the calculation of the expression (7), the object elevation angle θ_{i_obj}is calculated by performing proof-by-cases processing on the basis of a sign of z_o″ and a result of zero determination on (x_o″²+y_o″²) and performing exception processing on the basis of a result of the proof by cases. However, detailed description thereof will be omitted herein.

In a case where the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}are obtained by the above calculation, the relative direction calculation unit 32 performs similar calculation to obtain the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}.

That is, the relative direction calculation unit 32 calculates the following expression (8) on the basis of the listening position information (x_v, y_v, z_v) and the sound source direction information (ψ_o, θ_o, φ_o), thereby obtaining the rotated coordinates (x_v′, y_v′, z_v′) of the coordinates (x_v, y_v, z_v) indicated by the listening position information.

[

Math
.

8

]



(

x
v

′

y
v

′

z
v

′

)

=

(

cos
⁢

ψ

0

sin
⁢

ψ

0

1

0

-
sin

⁢

ψ

0

cos
⁢

ψ

)

⁢

(

1

0

0

0

cos
⁢

θ

-
sin

⁢

θ

0

sin
⁢

θ

cos
⁢

θ

)

⁢

(

cos
⁢

ϕ

-
sin

⁢

ϕ

0

sin
⁢

ϕ

cos
⁢

ϕ

0

0

0

1

)

⁢
 

(

x
v

y
v

z
v

)

(
8
)

In the calculation of the expression (8), φ=−φ_o, θ=−θ_o, and ψ=−ψ_oare set, and the rotation matrixes are calculated.

The coordinates (x_v′, y_v′, z_v′) thus obtained indicate the listening position (position of the listener) in the object coordinate system. However, the origin of the object coordinate system herein is not the position of the object but is the origin O of the xyz coordinate system in the target space.

Next, the relative direction calculation unit 32 calculates the following expression (9) on the basis of the sound source position information (x_o, y_o, z_o) and the sound source direction information (ψ_o, θ_o, φ_o), thereby obtaining the rotated coordinates (x_o′, y_o′, z_o′) of the coordinates (x_o, y_o, z_o) indicated by the sound source position information.

[

Math
.

9

]



(

x
o

′

y
o

′

z
o

′

)

=

(

cos
⁢

ψ

0

sin
⁢

ψ

0

1

0

-
sin

⁢

ψ

0

cos
⁢

ψ

)

⁢

(

1

0

0

0

cos
⁢

θ

-
sin

⁢

θ

0

sin
⁢

θ

cos
⁢

θ

)

⁢

(

cos
⁢

ϕ

-
sin

⁢

ϕ

0

sin
⁢

ϕ

cos
⁢

ϕ

0

0

0

1

)

⁢
 

(

x
o

y
o

z
o

)

(
9
)

In the calculation of the expression (9), φ=−φ_o, θ=−θ_o, and ψ=−ψ_oare set, and the rotation matrixes are calculated.

The coordinates (x_o′, y_o′, z_o′) thus obtained indicate the position of the object in the object coordinate system. However, the origin of the object coordinate system herein is not the position of the object but is the origin O of the xyz coordinate system in the target space.

Further, the relative direction calculation unit 32 calculates the following expression (10) on the basis of the coordinates (x_v′, y_v′, z_v′) calculated from the expression (8) and the coordinates (x_o′, y_o′, z_o′) calculated from the expression (9).

[

Math
.

10

]



(

x
v

″

y
v

″

z
v

″

)

=

(

x
v

′

y
v

′

z
v

′

)

-

(

x
o

′

y
o

′

z
o

′

)

(
10
)

The expression (10) is calculated to obtain coordinates (x_v″, y_v″, z_v″) indicating the listening position in the object coordinate system taking the position of the object as the origin. The coordinates (x_v″, y_v″, z_v″) indicate a relative position of the listening position viewed from the object.

The relative direction calculation unit 32 calculates the following expressions (11) and (12) on the basis of the coordinates (x_v″, y_v″, z_v″) obtained as described above, thereby obtaining the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}.

[

Math
.

11

]



ψ_rot

i
⁢
_
⁢
obj

=

arctan
⁡
(

y
v

″

/

x
v

″

)

(
11
)

[

Math
.

12

]



θ_rot

i
⁢
_
⁢
obj

=

arctan
⁡
(

z
v

″

/

sqrt
⁡
(

x
v

″2

+

y
v

″2

)

)

(
12
)

The expression (11) is calculated in a similar manner to the expression (6) to obtain the object rotation azimuth angle ψ_rot_{i_obj}. Further, the expression (12) is calculated in a similar manner to the expression (7) to obtain the object rotation elevation angle θ_rot_{i_obj}.

The relative direction calculation unit 32 performs the processing described above on each frame of the audio data for the plurality of objects.

Therefore, it is possible to obtain the relative direction information including the object azimuth angle ψ_{i_obj}, the object elevation angle θ_{i_obj}, the object rotation azimuth angle ψ_rot_{i_obj}, and the object rotation elevation angle θ_rot_{i_obj}of each object for each frame.

Using the relative direction information obtained as described above makes it possible to localize a sound image of each object in accordance with the listening position, the direction of the listener, and movement and rotation of the object, thereby providing a higher realistic feeling.

(Directional Characteristic Database Unit)

The directional characteristic database unit 23 records directional characteristic data for each type of object, that is, for each sound source type.

The directional characteristic data is, for example, a function that uses the azimuth angle and elevation angle viewed from the object as arguments and obtains a gain in a propagation direction and a spherical harmonic coefficient indicated by the azimuth angle and elevation angle.

Note that, instead of the function, the directional characteristic data may be data in a table format, that is, for example, a table in which the azimuth angle and elevation angle viewed from the object are associated with the gain in the propagation direction and the spherical harmonic coefficient indicated by the azimuth angle and elevation angle.

(Directivity Rendering Unit)

The directivity rendering unit 33 performs rendering processing on the basis of the audio data of each object, the directional characteristic data, the relative distance information, and the relative direction information obtained for each object, the listening position information, and the listener direction information, and generates a reproduction signal for the corresponding reproduction unit 12 serving as a target device.

Next, an operation of the signal processing device 11 will be described.

That is, the content reproduction processing performed by the signal processing device 11 will be described below with reference to a flowchart of FIG. 10.

Note that, herein, description will be provided on the assumption that content to be reproduced is free-viewpoint content and directional characteristic data of each sound source type is acquired and recorded in advance in the directional characteristic database unit 23.

In step S11, the acquisition unit 21 acquires metadata and audio data for one frame of each object included in the content from the transmission device. In other words, the metadata and audio data are acquired at predetermined time intervals.

The acquisition unit 21 supplies sound source type information included in the acquired metadata of each object to the directional characteristic database unit 23, and supplies the acquired audio data of each object to the directivity rendering unit 33.

Further, the acquisition unit 21 supplies sound source position information (x_o, y_o, z_o) included in the acquired metadata of each object to the relative distance calculation unit 31 and the relative direction calculation unit 32, and supplies sound source direction information (ψ_o, θ_o, φ_o) included in the acquired metadata of each object to the relative direction calculation unit 32.

In step S12, the listening position designation unit 22 designates a listening position and a direction of the listener.

That is, the listening position designation unit 22 determines the listening position and the direction of the listener in response to an operation or the like of the listener, and generates listening position information (x_v, y_v, z_v) and listener direction information (ψ_v, θ_v, φ_v) indicating the determination result.

The listening position designation unit 22 supplies the resultant listening position information (x_v, y_v, z_v) to the relative distance calculation unit 31, the relative direction calculation unit 32, and the directivity rendering unit 33, and supplies the resultant listener direction information (ψ_v, θ_v, φ_v) to the relative direction calculation unit 32 and the directivity rendering unit 33.

Note that, in a case of fixed-viewpoint content, for example, the listening position information is set to (0, 0, 0), and the listener direction information is also set to (0, 0, 0).

In step S13, the relative distance calculation unit 31 calculates a relative distance d_oon the basis of the sound source position information (x_o, y_o, z_o) supplied from the acquisition unit 21 and the listening position information (x_v, y_v, z_v) supplied from the listening position designation unit 22, and supplies relative distance information indicating the calculation result to the directivity rendering unit 33. For example, in step S13, the expression (1) described above is calculated for each object, and the relative distance d_ois calculated for each object.

In step S14, the relative direction calculation unit 32 calculates a relative direction between the listener and the object on the basis of the sound source position information (x_o, y_o, z_o) and sound source direction information (ψ_o, θ_o, φ_o) supplied from the acquisition unit 21 and the listening position information (x_v, y_v, z_v) and listener direction information (ψ_v, θ_v, φ_v) supplied from the listening position designation unit 22, and supplies relative direction information indicating the calculation result to the directivity rendering unit 33.

For example, the relative direction calculation unit 32 calculates the expressions (3) to (7) described above for each object, thereby obtaining the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}for each object.

Further, for example, the relative direction calculation unit 32 calculates the expressions (8) to (12) described above for each object, thereby obtaining the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}for each object.

The relative direction calculation unit 32 supplies information including the object azimuth angle ψ_{i_obj}, the object elevation angle θ_{i_obj}, the object rotation azimuth angle ψ_rot_{i_obj}, and the object rotation elevation angle θ_rot_{i_obj}obtained for each object as the relative direction information to the directivity rendering unit 33.

In step S15, the directivity rendering unit 33 acquires the directional characteristic data from the directional characteristic database unit 23.

For example, in a case where the metadata is acquired for each object in step S11 and the sound source type information included in the metadata is supplied to the directional characteristic database unit 23, the directional characteristic database unit 23 outputs the directional characteristic data for each object.

That is, the directional characteristic database unit 23 reads, for each piece of the sound source type information supplied from the acquisition unit 21, the directional characteristic data of the sound source type indicated by the sound source type information from the plurality of pieces of recorded directional characteristic data, and outputs the directional characteristic data to the directivity rendering unit 33.

The directivity rendering unit 33 acquires the directional characteristic data output for each object from the directional characteristic database unit 23 as described above, thereby obtaining the directional characteristic data of each object.

In step S16, the directivity rendering unit 33 performs rendering processing on the basis of the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information (x_v, y_v, z_v) and listener direction information (ψ_v, θ_v, φ_v) supplied from the listening position designation unit 22.

Note that the listening position information (x_v, y_v, z_v) and the listener direction information (ψ_v, θ_v, φ_v) only need to be used for the rendering processing as necessary, and may not necessarily be used for the rendering processing.

For example, the directivity rendering unit 33 performs the processing for VBAP or wave field synthesis, the HRTF convolution processing, or the like as the rendering processing, thereby generating a reproduction signal for reproducing a sound of the object (content) at the listening position.

Herein, an example of performing VBAP as the rendering processing will be described. Therefore, in this case, the reproduction unit 12 includes a plurality of speakers.

Further, an example where a single object is included in the content will be described herein for simplicity of description.

First, the directivity rendering unit 33 calculates the following expression (13) on the basis of the relative distance d_oindicated by the relative distance information, thereby obtaining a gain value gain_{i_obj}for reproducing distance attenuation.

[

Math
.

13

]



gain

i
⁢
_
⁢
obj

=

1.
/

power
(

d
0

,
2.

)

(
13
)

Note that power (d_o, 2.0) in the expression (13) represents a function for calculating a square value of the relative distance d_o. Herein, an example of using an inverse-square law will be described. However, calculation of the gain value for reproducing the distance attenuation is not limited thereto, and any other method may be used.

Next, the directivity rendering unit 33 calculates, for example, the following expression (14) on the basis of the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}included in the relative direction information, thereby obtaining a gain value dir_gain_{i_obj}according to the directional characteristic of the object.

[

Math
.

14

]



dir_gain

i
⁢
_
⁢
obj

=

dir
⁡
(

i
,

ψ_rot

i
⁢
_
⁢
obj

,

θ_rot

i
⁢
_
⁢
obj

)

(
14
)

In the expression (14), dir(i, ψ_rot_{i_obj}, θ_rot_{i_obj}) represents a gain function corresponding to a value i of the sound source type information supplied as the directional characteristic data.

Therefore, the directivity rendering unit 33 calculates the expression (14) by substituting the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}into the gain function, thereby obtaining the gain value dir_gain_{i_obj}as the calculation result.

That is, in the expression (14), the gain value dir_gain_{i_obj}is obtained from the object rotation azimuth angle φ_rot_{i_obj}, the object rotation elevation angle θ_rot_{i_obj}, and the directional characteristic data.

The gain value dir_gain_{i_obj}obtained as described above achieves gain correction for adding a transfer characteristic of a sound propagating from the object toward the listener, in other words, gain correction for reproducing sound propagation according to the directional characteristic of the object.

Note that a distance from the object may be included as an argument (variable) of the gain function serving as the directional characteristic data as described above, thereby achieving gain correction that reproduces not only the directional characteristic but also the distance attenuation by using the gain value dir_gain_{i_obj}that is an output of the gain function. In this case, the relative distance d_oindicated by the relative distance information is used as the distance that is the argument of the gain function.

Further, the directivity rendering unit 33 obtains a reproduction gain value VBAP_gain_{i_spk}of a channel corresponding to each of the plurality of speakers included in the reproduction unit 12 by performing VBAP on the basis of the object azimuth angle ψ_{i_obj}and object elevation angle θ_{i_obj}included in the relative direction information.

Then, the directivity rendering unit 33 calculates the following expression (15) on the basis of audio data obj_audio_{i_obj}of the object, the gain value gain_{i_obj}of the distance attenuation, the gain value dir_gain_{i_obj}of the directional characteristic, and the reproduction gain value VBAP_gain_{i_spk}of the channel corresponding to the speaker, thereby obtaining a reproduction signal speaker_signal_{i_spk}to be supplied to the speaker.

[

Math
.

15

]



speaker_signal
i_spk

=

obj_audio
i_obj

×

VBAP_gain
i_spk

×

gain
i_obj

×
 

dir_gain
i_obj

(
15
)

Herein, the expression (15) is calculated for each combination of the speaker included in the reproduction unit 12 and the object included in the content, and the reproduction signal speaker_signal_{i_spk}is obtained for each of the plurality of speakers included in the reproduction unit 12.

Therefore, the gain correction for reproducing the distance attenuation, the gain correction for reproducing sound propagation according to the directional characteristic, and the processing of VBAP for localizing a sound image at a desired position are achieved.

Meanwhile, in a case where the gain value dir_gain_{i_obj}obtained from the directional characteristic data is a gain value in which both the directional characteristic and the distance attenuation are considered, that is, in a case where the relative distance d_oindicated by the relative distance information is included as an argument of the gain function, the following expression (16) is calculated.

That is, the directivity rendering unit 33 calculates the following expression (16) on the basis of the audio data obj_audio_{i_obj}of the object, the gain value dir_gain_{i_obj}of the directional characteristic, and the reproduction gain value VBAP_gain_{i_spk}, thereby obtaining the reproduction signal speaker_signal_{i_spk}.

[

Math
.

16

]



speaker_signal
i_spk

=

obj_audio
i_obj

×

VBAP_gain
i_spk

×

dir_gain
i_obj

(
16
)

In a case where the reproduction signal is obtained as described above, the directivity rendering unit 33 finally performs overlap addition of the reproduction signal speaker_signal_{i_spk}obtained for the current frame with the reproduction signal speaker_signal_{i_spk}of a previous frame of the current frame, thereby obtaining a final reproduction signal.

Note that the example of performing VBAP as the rendering processing has been described herein, but, also in a case where the HRTF convolution processing is performed as the rendering processing, reproduction signals can be obtained by performing similar processing.

Herein, there will be described a case where reproduction signals of headphones are generated in consideration of the directional characteristic of the object by using an HRTF database including an HRTF for each user according to the distance, azimuth angle, and elevation angle indicating a relative positional relationship between the object and the user (listener).

In particular, herein, the directivity rendering unit 33 holds the HRTF database including an HRTF from a virtual speaker corresponding to a real speaker used when measuring the HRTF, and the reproduction unit 12 is headphones.

Note that a case where the HRTF database is prepared for each user in consideration of a difference in a personal characteristic of each user will be described herein. However, an HRTF database common to all users may be used.

In this example, a personal ID information for identifying an individual user is set as j, and azimuth angles and elevation angles indicating directions of arrival of a sound from a sound source (virtual speaker), that is, from the object to ears of the user will be denoted by ψ_Land ψ_Rand θ_Land θ_R, respectively. Herein, the azimuth angle ψ_Land the elevation angle θ_Lare an azimuth angle and elevation angle indicating a direction of arrival to a left ear of the user, and the azimuth angle ψ_Rand the elevation angle θ_Rare an azimuth angle and elevation angle indicating a direction of arrival to a right ear of the user.

Further, an HRTF serving as a transfer characteristic from the sound source to the left ear of the user will be particularly denoted by HRTF(j, ψ_L, θ_L), and an HRTF serving as a transfer characteristic from the sound source to the right ear of the user will be particularly denoted by HRTF(j, ψ_R, θ_R).

Note that the HRTF to each of the left and right ears of the user may be prepared for each direction of arrival and distance from the sound source, and the distance attenuation may also be reproduced by HRTF convolution.

Further, the directional characteristic data may be a function indicating a transfer characteristic from the sound source to each direction or may be a gain function as in the example of VBAP described above, and, in either case, the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}are used as arguments of the function.

In addition, the object rotation azimuth angle and the object rotation elevation angle may be obtained for each of the left and right ears in consideration of a convergence angle between the left and right ears of the user with respect to the object, that is, a difference in an angle of arrival of a sound between the object and each ear of the user caused by a facial width of the user.

The convergence angle herein is an angle between a straight line connecting the left ear of the user (listener) and the object and a straight line connecting the right ear of the user and the object.

Hereinafter, among the object rotation azimuth angles and the object rotation elevation angles included in the relative direction information, the object rotation azimuth angle and object rotation elevation angle obtained for the left ear of the user will be particularly denoted by ψ_rot_{i_obj_1}and θ_rot_{i_obj_1}, respectively.

Similarly, hereinafter, among the object rotation azimuth angles and the object rotation elevation angles included in the relative direction information, the object rotation azimuth angle and object rotation elevation angle obtained for the right ear of the user will be particularly denoted by ψ_rot_{i_obj_r}and θ_rot_{i_obj_r}, respectively.

First, the directivity rendering unit 33 calculates the expression (13) described above, thereby obtaining the gain value gain_{i_obj}for reproducing the distance attenuation.

Note that, in a case where the HRTF is prepared for each direction of arrival of a sound and distance from the sound source as the HRTF database and the distance attenuation can be reproduced by the HRTF convolution, the gain value gain_{i_obj}is not calculated. In addition, the distance attenuation may be reproduced by convolution of the transfer characteristic obtained from the directional characteristic data, instead of the HRTF convolution.

Next, the directivity rendering unit 33 acquires the transfer characteristic according to the directional characteristic of the object on the basis of, for example, the directional characteristic data and the relative direction information.

For example, in a case where a function for obtaining the transfer characteristic is supplied as the directional characteristic data and the function uses a distance, azimuth angle, and elevation angle as arguments, the directivity rendering unit 33 calculates the following expressions (17) on the basis of the relative distance information, the relative direction information, and the directional characteristic data.

[

Math
.

17

]



dir_func

i_obj
⁢
_l

=

dir
⁡
(

i
,

d
i_obj

,

ψ_rot

i_obj
⁢
_l

,

θ
⁢
_rot

i_obj
⁢
_l

)

dir_func

i_obj
⁢
_r

=

dir
⁢

(

i
,

d
i_obj

,

ψ_rot

i_obj
⁢
_r

,

θ
⁢
_rot

i_obj
⁢
_r

)

(
17
)

That is, in the expressions (17), the directivity rendering unit 33 sets the relative distance d_oindicated by the relative distance information as d_{i_obj}.

Then, the directivity rendering unit 33 substitutes the relative distance d_o, the object rotation azimuth angle ψ_rot_{i_obj_l}, and the object rotation elevation angle θ_rot_{i_obj_l}into a function dir(i, d_{i_obj}, ψ_rot_{i_obj_l}, θ_rot_{i_obj_l}) for the left ear supplied as the directional characteristic data, thereby obtaining a transfer characteristic dir_func_{i_obj_l}of the left ear.

Similarly, the directivity rendering unit 33 substitutes the relative distance d_o, the object rotation azimuth angle ψ_rot_{i_obj_r}, and the object rotation elevation angle θ_rot_{i_obj_r}into a function dir(i, d_{i_obj}, ψ_rot_{i_obj_r}, θ_rot_{i_obj_r}) for the right ear supplied as the directional characteristic data, thereby obtaining a transfer characteristic dir_func_{i_obj_r}of the right ear.

In this case, the distance attenuation is also reproduced by convolution of the transfer characteristics dir_func_{i_obj_l}and dir_func_{i_obj_r}.

Further, the directivity rendering unit 33 obtains the HRTF(j, ψ_L, θ_L) for the left ear and the HRTF(j, ψ_R, θ_R) for the right ear from the held HRTF database on the basis of the object azimuth angle ψ_{i_obj}and the object elevation angle θ_{i_obj}. Herein, for example, the HRTF(j, ψ_L, θ_L) in which ψ_L=ψ_{i_obj}and θ_L=θ_{i_obj}are set is read from the HRTF database. Note that the object azimuth angle and the object elevation angle may also be obtained for each of the left and right ears.

In a case where the transfer characteristics and HRTFs of the left and right ears are obtained by the above processing, reproduction signals for the left and right ears to be supplied to the headphones serving as the reproduction unit 12 are obtained on the basis of the transfer characteristics, the HRTFs, and the audio data obj_audio_{i_obj}of the object.

Specifically, for example, in a case where the transfer characteristics dir_func_{i_obj_l}and dir_func_{i_obj_r}are obtained from the directional characteristic data in consideration of both the directional characteristic and the distance attenuation, that is, in a case where the transfer characteristics are obtained from the expressions (17), the directivity rendering unit 33 calculates the following expressions (18) to obtain a reproduction signal HPout_Lfor the left ear and a reproduction signal HPout_Rfor the right ear.

[

Math
.

18

]



HP

out
L

=

obj_audio
i_obj

*

dir_func

i_obj
⁢
_l

*

HRTF
⁡
(

j
,

ψ
L

,

θ
L

)

HP

out
R

=

obj_audio
i_obj

*

dir_func

i_obj
⁢
_r

*
HRTF
⁢

(

j
,

ψ
R

,

θ
R

)

(
18
)

Note that, in the expressions (18), * represents convolution processing.

Therefore, herein, the transfer characteristic dir_func_{i_obj_l}and the HRTF(j, ψ_L, θ_L) are convolved to the audio data obj_audio_{i_obj}to obtain the reproduction signal HPout_Lfor the left ear. Similarly, the transfer characteristic dir_func_{i_obj_r}and the HRTF(j, ψ_R, θ_R) are convolved to the audio data obj_audio_{i_obj}to obtain the reproduction signal HPout_Rfor the right ear. Further, also in a case where the distance attenuation is reproduced by the HRTFs, the reproduction signals are obtained by calculation similar to that of the expressions (18).

Meanwhile, for example, in a case where the transfer characteristics obtained from the directional characteristic data and the HRTFs are obtained without considering the distance attenuation, the directivity rendering unit 33 calculates the following expressions (19) to obtain reproduction signals.

[

Math
.

19

]



HP

out
L

=

obj_audio
i_obj

*

dir_func

i_obj
⁢
_l

*

HRTF
⁡
(

j
,

ψ
L

,

θ
L

)

*

gain
i_obj

HP

out
R

=

obj_audio
i_obj

*

dir_func

i_obj
⁢
_r

*
HRTF
⁢

(

j
,

ψ
R

,

θ
R

)

*

gain
i_obj

(
19
)

In the expressions (19), the audio data obj_audio_{i_obj}is subjected not only to the convolution processing performed in the expressions (18) but also to processing for convolving the gain value gain_{i_obj}for reproducing the distance attenuation. Therefore, the reproduction signal HPout_Lfor the left ear and the reproduction signal HPout_Rfor the right ear are obtained. The gain value gain_{i_obj}is obtained from the expression (13) described above.

In a case where the reproduction signals HPout_Land HPout_Rare obtained by the above processing, the directivity rendering unit 33 performs overlap addition of the reproduction signals with reproduction signals of the previous frame, thereby obtaining final reproduction signals HPout_Land HPout_R.

Further, in a case where the processing for wave field synthesis is performed as the rendering processing, that is, in a case where a sound field including a sound of the object is formed by wave field synthesis by using a plurality of speakers serving as the reproduction unit 12, reproduction signals are generated as follows.

Herein, there will be described an example where speaker drive signals to be supplied to the speakers included in the reproduction unit 12 are generated as reproduction signals by using spherical harmonics.

An external sound field at a position outside a certain radius r from a predetermined sound source, that is, at a position where a radius (distance) from the sound source is r′ (where r′>r) and an azimuth angle and elevation angle indicating a direction viewed from the sound source are ψ and θ, that is, a sound pressure p(r′, ψ, θ) can be shown by the following expression (20).

[

Math
.

20

]



p
⁡
(

r
′

,
ψ
,
θ

)

=

∑

n
=
0

∞

∑

m
=

-
n

n

P
nm

(
r
)

⁢

h
n

(
1
)

(

kr
′

)

h
n

(
1
)

(
kr
)

⁢

Y
n
m

(

ψ
,
θ

)

⁢

X
⁡
(
k
)

(
20
)

Note that, in the expression (20), Y_n^m(ψ, θ) represents a spherical harmonic function, and n and m represent a degree and order of the spherical harmonic function. Further, h_n⁽¹⁾(kr) is a Hankel function of the first kind, and k represents a wave number.

Furthermore, in the expression (20), X(k) represents a reproduction signal represented in a frequency domain, and P_nm(r) represents a spherical harmonic spectrum of a sphere having a radius (distance) r. Herein, the signal X(k) in the frequency domain corresponds to the audio data of the object.

For example, in a case where a measurement microphone array for measuring a directional characteristic has a spherical shape having the radius r, a sound pressure at a position of the radius r of a sound propagating in all directions from the sound source existing at the center of the sphere (measurement microphone array) can be measured by using the measurement microphone array. In particular, because the directional characteristic varies depending on the sound source, an observation sound including directional characteristic information is obtained by measuring the sound from the sound source at each position.

The spherical harmonic spectrum P_nm(r) can be shown by the following expression (21) by using such a measured observation sound pressure p(r, ψ, θ) measured by the measurement microphone array.

[

Math
.

21

]



P
nm

(
r
)

=

∫

∂

Ω

p
⁡
(

r
,
ψ
,
θ

)

⁢

Y
n
m

(

ψ
,
θ

)

*
dr

(
21
)

Note that, in the expression (21), ∂Ω represents an integral range and particularly represents an integral on the radius r.

Such a spherical harmonic spectrum P_nm(r) is data indicating the directional characteristic of the sound source. Therefore, in a case where, for example, the spherical harmonic spectrum P_nm(r) of each combination of the degree n and the order m in a predetermined domain is measured in advance for each sound source type, it is possible to use a function shown by the following expression (22) as directional characteristic data dir(i_obj, d_{i_obj}).

[

Math
.

22

]



dir
⁡
(

i_obj
,

d
i_obj

)

=

P
nm

(
r
)

⁢

h
n

(
1
)

(

kd
i_obj

)

h
n

(
1
)

(
kr
)

(
22
)

Note that, in the expression (22), i_obj represents a sound source type, d_{i_obj}represents a distance from the sound source, and the distance d_{i_obj}corresponds to the relative distance d_o. Such a set of pieces of the directional characteristic data dir(i_obj, d_{i_obj}) of the respective degrees n and orders m is data indicating the transfer characteristic in each direction determined on the basis of the azimuth angle ψ and the elevation angle θ in consideration of an amplitude and a phase, that is, in all directions.

In a case where the relative positional relationship between the object and the listening position does not change, a reproduction signal in which the directional characteristic is also considered can be obtained from the expression (20) described above.

However, even in a case where the relative positional relationship between the object and the listening position changes, a sound pressure p(d_{i_obj}, ψ, θ) at a point (d_{i_obj}, ψ, θ) determined on the basis of the azimuth angle ψ, the elevation angle θ, and the distance d_{i_obj}can be obtained by subjecting the directional characteristic data dir(i_obj, d_{i_obj}) to a rotation operation based on the object rotation azimuth angle ψ_rot_{i_obj}and the object rotation elevation angle θ_rot_{i_obj}, as shown by the following expression (23).

[

Math
.

23

]



p
⁡
(

d
i_obj

,
ψ
,
θ

)

=

∑

n
=
0

∞

∑

m
=

-
n

n

P
nm

(
r
)

⁢

h
n

(
1
)

(

kd
i_obj

)

h
n

(
1
)

(
kr
)

⁢

Y
n
m

(

ψ
+

ψ_rot
i_obj

,

θ
+
 

θ
⁢
_rot

i_obj

)

⁢

X
⁡
(
k
)

(
23
)

Note that, in the calculation of the expression (23), the relative distance d_ois substituted into the distance d_{i_obj}and the audio data of the object is substituted into X(k), and thus the sound pressure p(d_{i_obj}, ψ, θ) is obtained for each wave number (frequency) k. Then, the sum of the sound pressures p(d_{i_obj}, ψ, θ) of each object, which are obtained for the respective wave numbers k, is calculated to obtain a signal of the sound observed at the point (d_{i_obj}, ψ, θ), that is, a reproduction signal.

Therefore, in order to generate reproduction signals for wave field synthesis, the expression (23) is calculated for each wave number k for each object as the processing in step S16, and reproduction signals are generated on the basis of the calculation result.

In a case where the reproduction signals to be supplied to the reproduction unit 12 are obtained by the rendering processing described above, the processing proceeds from step S16 to step S17.

In step S17, the directivity rendering unit 33 supplies the reproduction signals obtained by the rendering processing to the reproduction unit 12 and causes the reproduction unit 12 to output a sound. Therefore, the sound of the content, that is, the sound of the object is reproduced.

In step S18, the signal generation unit 24 determines whether or not to terminate the processing of reproducing the sound of the content. For example, in a case where the processing is performed on all the frames and reproduction of the content ends, it is determined that the processing is to be terminated.

In a case where it is determined in step S