1. FIELD OF THE INVENTION

The field of the invention is that of digital holography, in particular the adaptation of a pre-existing hologram in order to make it displayable by a particular holographic reproduction system.

The invention may in particular, but not exclusively, be applied to the uses of virtual or augmented reality to reproduce the input hologram on a head-mounted display and possibly to superimpose it to a virtual scene that the user perceives through a transparent screen of his head-mounted display HMD.

2. PRESENTATION OF THE PRIOR ART

Nowadays, with the appearance of hologram libraries, digital holograms of varied resolutions and sizes can be obtained.

The person skilled in the art knows how to reproduce a pre-existing hologram on a holographic screen device provided that the hologram in question has the same resolution and the same size as the screen.

On the other hand, if the input hologram has resolution and size that are different from those of the holographic screen of the reproduction system or if the reproduction system that integrates the holographic screen comprises optical elements arranged between the plane of the holographic screen and a plane in which the hologram is viewed, then the hologram could appear very deformed or unusable.

Moreover, it is known from J. J. Healy et al., “Linear Canonical Transforms—Theory and Applications”, edited by New York: Springer-Verlag, first edition, in 2016, pp. 347-366, a method of optical generation of a hologram on a charge-coupled sensor or CCD (for “Charge-Coupled Device”), in which a reference beam and an object beam, resulting from the reflection of the reference beam on the objects of the scene, are conventionally made to interfere with each other, on a surface consisted of a set of pixels.

The propagation of the object beam up to the CCD sensor is made through an arrangement of several optical elements (lenses, beam splitters, etc.).

This method seeks to optimize the parameters of the optical system by simulating the propagation of the light field through the optical elements that constitute it. That way, the parameters of the optical system are adjusted to obtain a light field having the desired characteristics.

It is based on a so-called “Linear Canonical Transform” (LCT) theory, described for example in the above-mentioned Healy et al. document, pages 39-80. This theory makes the link between two types of mathematical transformations, which are:

- The ray transfer matrices, linear transformations expressing the ray direction change induced by the passing through an optical element. They integrate the parameters of the optical element. If each ray is represented by spatial and angular parameters, the transfer matrix of an optical element is the linear transform of these parameters that, to each input ray, associates the output ray. The transfer matrices of the conventional optical elements are known from the person skilled in the art. It may be mentioned, for example:
  - A free propagation, in vacuum or in the air, over a distance d:

Mv
=

(

I

λ
⁢

⁢
dI

0

I

)

,

- - A thin lens of focal length f:

Ml
=

(

I

0

-
I

/
λ

⁢

⁢
f

I

)

,

- - where I represents the identity matrix

(

1

0

0

1

)

.

- The integral operators:
- To each transfer matrix

M
=

(

A

B

C

D

)

- is associated an integral operator T that, to a function ƒ representing a field, associates a function ƒ′ such that:

ƒ′=T(ƒ) (1)

ƒ′ being defined by

ƒ′(X′)=det(iB)^−1/2∫ƒ(X)e^iπP(X,X′)dX, (2)

with

P(X,X′)=B⁻¹AX+DB⁻¹X′−2X^tB⁻¹X′ (3)

X and X′ being two-component real vectors and A, B, C and D being 2×2 matrices.

According to this theory, the obtained operator performs the transformation operated on the light field by the optical element in question.

Hence, a cascade of optical elements can be represented by an overall ray transfer matrix, product of the ray transfer matrices of the optical elements successively passed through. The resulting overall matrix that is obtained can be interpreted as a transform of the space/frequency distribution of the light field.

According to this theory, the overall transfer matrix can be associated with a single integral operator, which describes the propagation of a light field through the totality of the optical system.

3. DRAWBACKS OF THE PRIOR ART

The prior proposes no solution to adapt an existing hologram to a holographic reproduction system.

There hence exists a need for a solution for adapting a pre-existing digital hologram to a particular holographic reproduction system, which is simple, which does not deform the hologram viewed by the user with respect to the existing hologram, and which is usable in wearable reproduction systems of the HMD type.

4. OBJECTS OF THE INVENTION

The invention helps in improving the situation.

The invention has in particular for object to compensate for these drawbacks of the prior art.

More precisely, an object of the invention is to propose a solution for adapting a pre-existing hologram to a particular display device, which is simple and which does not deform the hologram.

Another object of the invention is to propose a solution that is able to be integrated in a reproduction system wearable by the user, such as a head-mounted display, and that adapts itself in real time to a displacement of the user.

5. DISCLOSURE OF THE INVENTION

These objects, as well as others that will appear hereinafter, are achieved by means of a method for processing at least one input hologram associated with at least one input plane in a World reference frame, in order to obtain an output hologram displayable on a holographic screen placed in a so-called output plane of a reproduction system and viewable by an observer from a viewing plane of said system, said system comprising at least one optical element arranged between the viewing plane (PV) and the output plane.

Such a method is particular in that it comprises the following steps:

- obtaining the input hologram and a pose of the associated input plane in the World reference frame;
- obtaining at least one first ray transfer matrix, called extrinsic matrix, representative of a propagation in free space between the input plane and the viewing plane of the reproduction system;
- obtaining at least one second ray transfer matrix, called intrinsic matrix, representative of a propagation between the viewing plane and the output plane of the reproduction system through at least one said optical element;
- calculating at least one overall ray transfer matrix of a light field emitted by the input hologram between the input plane and the output plane of the reproduction system, by a product of at least two of said matrices; and
- transforming a light field of the input hologram into a light field of the output hologram by application of at least one linear integral operator determined from the at least one calculated overall matrix, as a function of input sizes and input resolutions of the input hologram and of output sizes and output resolutions of the holographic screen.

The invention hence proposes a perfectly new and inventive approach that makes it possible to adapt in a simple way a pre-existing hologram to a particular reproduction system, in such a manner that the output hologram displayed on the holographic screen of this system reproduces to an observer the same view of the reconstructed 3D scene as that of the input hologram in the input plane.

The invention uses the properties of the “Linear Canonical Transform” theory to explain rather simply, in the form of 4×4 ray transfer matrices, the distortion undergone by a light field emitted by any input hologram during its reverse propagation from the input plane to the viewing plane, on the one hand, and from the viewing plane to the output plane, on the other hand, through the succession of optical elements constituting the holographic reproduction system. The compensation for this distortion is expressed as one or several linear transformation integral operator(s) appliable to any input hologram.

The linear integral operator(s) take into account the resolutions and sizes of the input hologram to transform it into an output hologram whose resolutions and sizes correspond to those of the holographic screen. That way, the invention makes it possible to make any input hologram displayable by the holographic reproduction system.

Due to its simplicity, the invention is well adapted to real-time applications.

According to one aspect of the invention, when the holographic reproduction system comprises a module for the non-linear processing of the light field of the input hologram, the step of obtaining at least one intrinsic matrix comprises obtaining a first matrix representative of a propagation of the light field through at least one linear optical element located upstream from the non-linear processing module and obtaining a second intrinsic matrix representative of a propagation of the processed light field through at least one linear optical element located downstream from the filtering module, the step of calculating at least one overall matrix comprises calculating a first overall matrix by a product of the at least one extrinsic matrix and the first intrinsic matrix and a second overall matrix from the second intrinsic matrix, the step of transforming the light field of the input hologram comprises the successive application of a first linear integral operator from the first overall matrix, an operator of the non-linear processing and a second linear integral operator from the second overall matrix (M_A).

An advantage lies in the taking into account of a light field non-linear processing integrated to the reproduction system, as for example a filtering, while proposing a simple compensation, by a minimum number of linear integral operators, for the linear distortions undergone by the input hologram.

According to another aspect of the invention, the method comprises a step of obtaining a pose of the observer in the World reference frame, a step of evaluating a visibility of the at least one input hologram by the observer, and the steps of obtaining ray transfer matrices and transforming the input hologram are implemented when the at least one input hologram is evaluated as visible by the observer.

An advantage is that the output hologram is recalculated only when necessary.

According to still another aspect of the invention, after detection of a change of the observer's pose in the World reference frame, the method updates at least one extrinsic matrix and the at least one overall transformation matrix as a function of said change, transforms the input hologram by application of at least one linear integral operator determined from the at least one updated overall ray transfer matrix.

An advantage is that only the extrinsic matrix is impacted by the displacement of the reproduction system and that the updating of the overall transformation operator is uncomplicated and operable in real time. The invention is hence well adapted to a mobile reproduction system, for example worn by a moving user.

According to still another aspect of the invention, when the output plane of the reproduction system is not parallel to the input plane of the input hologram, the method comprises a step of obtaining a matrix of rotation of the light field between the input plane and a corrected plane, parallel to the viewing plane, a step of determining a non-linear operator of transformation of the light field by composition of a Fourier transform, a rotation frequency operator from said rotation matrix and an inverse Fourier transform, and the transformation step further comprises the application of said non-linear operator previously to said at least one linear integral operator.

An advantage is to allow the user to position/turn as he chooses with respect to the reference plane of the input hologram.

According to still another aspect of the invention, the step of calculating a first overall matrix implements the product of the inverse Fourier transform, the extrinsic matrix and the first intrinsic matrix.

An advantage is to group all the linear operations consecutively applied to the light field into a single pre-calculated operator.

According to still another aspect of the invention, the light field of the input hologram comprising a plurality of samples function of the input sizes and input resolutions, the step of transforming the light field of the input hologram by at least one linear operator comprises the application to said samples of a discrete Fourier transform and the transformed light field is sampled as a function of the output resolutions and output sizes of the output hologram.

A first advantage is the adaptation of the input light field to the sizes and resolution of the output plane, which guarantees the observer to see the totality of the scene reconstructed by the input hologram. A second advantage is that, by grouping the transformations to be applied to the input hologram into a minimum number of linear integral operators, the number of Fourier transforms to be calculated and hence the overall complexity of the processing is hence reduced.

According to still another aspect of the invention, when a plurality of input holograms is associated with a plurality of input planes forming the faces of a geometric solid, the method comprises a step of selecting input planes corresponding to faces visible from the output plane, the steps of obtaining at least one extrinsic matrix, the step of calculating at least one overall ray transfer matrix, and the transformation step transforms the input holograms of the selected input planes by application of at least one linear integral operator determined from the at least one calculated overall matrix, and sums the transformed fields to obtain the output hologram.

An advantage is that, when several input holograms of a same 3D scene are placed on faces of a cube or another geometric solid, the invention makes it possible to transform only the input holograms corresponding to faces visible from the observer's position and direction.

According to still another aspect of the invention, the at least one obtained extrinsic matrix and the at least one obtained intrinsic matrix are stored into a memory.

An advantage is that they can be at least partly reused for a new input hologram or for a new pose of the observer. If the input hologram has different sizes and resolutions, the operator(s) will have to be updated.

Correspondingly, the invention also relates to a device for implementing the method for processing at least one input hologram according to the invention as defined hereinabove. This device can of course include the different features relating to the method according to the invention. Hence, the features and advantages of this device are the same as those of the processing method, and will not be further detailed.

According to a particular embodiment of the invention, such a device is comprised in a holographic reproduction system comprising a module for obtaining an input hologram associated with an input plane in a World reference frame, an output plane in which is placed a holographic screen intended to reproduce a hologram, called output hologram, a viewing plane in which the output hologram is intended to be viewed by a user, and at least one optical element, placed between the viewing plane and the output plane.

It is, for example, a projector, that is fixed and that does not take into account the observer's position.

According to an aspect of the invention, the holographic reproduction system is a head-mounted device, intended to be worn on the head of a user, the viewing plane is placed in front of the user's eyes, the output plane is placed out of a vision cone of the user, and the holographic reproduction system comprises at least the following optical elements:

- a module for receiving an input hologram having input sizes and input resolutions;
- a module for obtaining a pose of the observer in a World reference frame;
- a holographic screen in the output plane, having output sizes and output resolutions;
- an optical filtering system, of the 4F type, comprising a non-linear filtering module and two thin lenses;
- a field lens intended to enlarge a viewing field of the light field perceived by the observer on the viewing plane; and
- a beam splitter intended to redirect the light field of the hologram in the direction of the viewing plane.

The invention also relates to a computer program including instructions for implementing the steps of a method for processing at least one input hologram as described hereinabove, when this program is executed by a processor.

This program may use any programming language. It may be loaded from a communication network and/or recorded on a computer-readable medium.

The invention finally relates to a recording medium, readable by a processor, integrated or not to the processing device according to the invention, potentially removable, memorizing a computer program implementing the processing method according to the invention, as described hereinabove.

6. LIST OF FIGURES

Other advantages and features of the invention will appear more clearly upon reading of the following description of a particular embodiment of the invention, given by way of simple illustrative and non-limitative example, and of the appended drawings, among which:

FIG. 1 schematically illustrates the processing of an input hologram made by the invention to adapt it to a holographic reproduction system;

FIG. 2A schematically describes the arrangement of a first example of holographic reproduction system implementing the invention;

FIG. 2B describes in further detail the arrangement of the optical elements constituting this first holographic reproduction system;

FIG. 3 schematically shows the steps of processing an input hologram according to a first embodiment of the invention;

FIG. 4A schematically describes the arrangement of a second example of holographic reproduction system implementing the invention;

FIG. 4B describes in further detail the arrangement of the optical elements constituting this second holographic reproduction system;

FIG. 5A schematically illustrates the visibility cone of an input hologram;

FIG. 5B schematically illustrates the vision cone of an observer through a holographic reproduction system;

FIG. 6 illustrates an example of reproduction of a 3D scene based on a plurality of holograms displayed on the faces of a cube, according to a third exemplary embodiment of the invention;

FIG. 7 schematically describes the method for processing a plurality of holograms associated with the faces of a geometric solid according to this third embodiment of the invention;

FIG. 8 schematically illustrates a first example of hardware structure of an input hologram processing device according to the invention; and

FIG. 9 schematically illustrates a second example of hardware structure of an input hologram processing device according to the invention.

7. DESCRIPTION OF A PARTICULAR EMBODIMENT OF THE INVENTION

In relation with FIG. 1, a 3D scene is considered, from which an input hologram H_Ehas been generated. It is associated with an input plane P_E, or plane of the input hologram, in which it is intended to be displayed, with given sizes and resolution. In other words, the input data forming this hologram H_Ecorrespond to a plurality of discrete values or samples of a complex light field in a quadrilateral Q_Eof this plane, this plurality being defined by the sizes and resolution of the input hologram.

This input plane P_Eis associated with a pose, i.e. known position and direction in a reference frame, called the World reference frame RM. This pose is generally received with the input hologram, when the latter comes for example from a hologram library, and it allows reproducing the hologram in good conditions, i.e. viewing a reconstructed scene at the good scale. Of course, another pose of the input plane P_Ecan be chosen by a particular application or use, for example virtual or augmented reality or also videoconference, which uses this input hologram.

For example, the pose of the input plane can be modified as a function of that of a holographic reproduction system (SRH) in such a manner that the face of the corresponding quadrilateral Q_Eat the output of the light field is visible by the latter.

Three examples of holographic reproduction systems SRH have been represented, which are a projector PRO, and body-worn display systems or HMD (for “Head-Mounted Display”), such as Augmented Reality Glasses LRA or a Virtual Reality Helmet VC.

The general principle of the invention is based on the rapid adaptation of an input hologram H_Efor the reproduction thereof by such a holographic system comprising at least one optical element arranged between the hologram display device, also called holographic screen, of the Spatial Light Modulator or SLM type, and a virtual viewing plane P_Vof the latter. This adaptation is obtained by transforming the light field of the input hologram in the input plane into an output hologram H_Sintended to be displayed by the SLM, by means of an overall linear operator, that compensates for the passing through of the holographic reproduction system optical elements arrangement by the light field.

Hereinafter, the image, formed by the eye of an observer Ob, of the light field that passes through a quadrilateral Q_Vof the viewing plane P_Vis called reconstructed scene SC_R. This light field is initially formed on the output plane P_Sof the SLM by the product of a plane monochromatic wave of wavelength λ with the complex function representative of the hologram H_S, called output hologram, then transformed by the arrangement of optical elements through which it passes up to the viewing plane P_V. This complex function is recalculated by the processing method according to the invention so as to compensate for the transformation induced by the passing through the just-described optical element arrangement constituting the HMD. That way, the light field that passes through the viewing plane P_Vcoincides with that of the input hologram that passes through the input plane.

In order to perform the rapid calculating of the output hologram H_S, the invention performs two propagations:

- a direct propagation of the original hologram from the input plane H_Eto the viewing plane P_V; then
- a so-called “reverse” propagation of the light field from the viewing plane P_Vto the output plane P_S.

Then, to model these two propagations, the invention relies on the above-mentioned “Linear Canonical Transform” (LCT) theory. According to this theory, it is searched to represent a continuous succession of linear optical transformations due to the passing through optical elements, by a ray transfer matrix that is then associated with a single light propagation operator.

In relation with FIGS. 2A and 2B, a first exemplary embodiment of the invention is considered, when the holographic reproduction system SRH is of the projector PRO type PRO. The input hologram H_Eis associated with a plane P_E, called input plane, and with a quadrilateral Q_Ein the plane P_E. This plane is positioned in the World reference frame RM, like the holographic reproduction system SHR.

The projector PRO is fixed and comprises the following elements, arranged along an axis Ax and parallel to the input plane:

- Means (not shown) for receiving the input hologram H_E. For example, the system SRH comprises wired or wireless means for the connection to a communication network. The input hologram H_Eis associated with an input plane P_Ewhose pose is supposed to be fixed in the World reference frame RM. It is provided as a discretized form on a quadrilateral Q_Eof sizes Sx_Eand Sy_Eincluding Nx_E×Ny_Epixels, inscribed in the input plane P_E. By way of example, the holograms available in publicly accessible data libraries are of the order of 8K×4K pixels for sizes Sx_Eand Sy_Eof the order of about one cm. The greatest holograms available may be close to 72K×36K pixels for sizes Sx_Eand Sy_Eof the order of about ten cm;
- A holographic screen of the SLM type placed in a so-called output plane P_S;
- A field lens L_Cof focal length f intended to enlarge the view angle of the light field emitted by the SLM, located at a distance d₄from the plane P_S. Typically, f and d₄are of the order of 5 cm;
- A reflective film FR placed at a distance d₃from the field lens whose purpose is to reflect the image coming from the holographic projector, as would do a cinema screen. Typically, the distance d₃is chosen equal to 3 metres;
- A virtual viewing plane P_Vplaced at a distance d₂in front of the reflective film. The distance d₂corresponds to a minimum distance of the observer with respect to the reflective film. Typically, the distance d₂is chosen equal to 50 centimetres. The distance between the viewing plane P_Vand the input plane P_Eis given by d₁, of the order of 50 centimetres.

In relation with FIG. 3, we will now describe the steps of a method for processing an input hologram H_Eaccording to this first embodiment of the invention. During a step E₁, an input hologram H_Eis received by the holographic reproduction system SRH.

At this stage, a distinction is made between the following two successive propagations:

- a first propagation, called “extrinsic direct propagation”, in free space, of the light field from the input hologram H_Eto the viewing plane of the reproduction system SRH;
- a second propagation, called “intrinsic reverse propagation”, of the light field through the arrangement of optical elements constituting the SRH up to the plane P_Sof the holographic screen SLM.

At E4, an extrinsic ray transfer matrix M_EXrelating to the first propagation is obtained. It is not specific to the input hologram H_E. It can hence have been calculated in a preliminary phase then stored into a local memory MEM. In the example of the projector PRO, it consists of a propagation in free space over a distance d₁between the input plane P_Eand the viewing plane P_Vof the projector.

It is hence simply expressed as:

M
EX

=

(

I

λ
⁢

⁢

d
1

⁢
I

0

I

)

with I the identity matrix.

At E5, at least one intrinsic ray transfer matrix M_INrelating to the intrinsic reverse propagation of the light field of the hologram H_E, inside the SRH, is obtained, as a product of the ray transfer matrices of the optical elements that constitute it:

M_IN=M₆·M₅·M₄·M₃·M₂,

with:

-

⁢

⁢

M
2

=

(

I

λ
⁢

⁢

d
2

⁢
I

0

I

)

- for the propagation in free space over a distance d₂between the viewing plane P_Vand the reflective film FR;

-

⁢

⁢

M
3

=

(

I

0

0

-
I

)

- for the reflection or the rays by the reflective film FR;

-

⁢

⁢

M
4

=

(

I

λ
⁢

⁢

d
3

⁢
I

0

I

)

- for the propagation in free space over a distance d₃between the reflective film FR and the field lens L_C;

-

⁢

⁢

M
5

=

(

I

0

-
I

/
λ

⁢

⁢
f

I

)

- for the propagation of the light field of the hologram H_Ethrough the field lens L_C; and

-

⁢

⁢

M
6

=

(

I

λ
⁢

⁢

d
4

⁢
I

0

I

)

- for the propagation in free space over a distance d₄between the field lens L_Cand the output plane P_S. It will be noted that such a ray transfer matrix makes it possible to model a so-called linear propagation, in the sense that the expression of the rays at the output of the optical element is linear as a function of the expression of the input rays, these rays being expressed by a point in space, respectively input and output ones, and angles, respectively input and output ones.

In the case where the system SRH would comprise a non-linear processing module interposed between two linear optical elements, not only it would be required to model the effect of this non-linear processing on the light field by means of a general light field transformation operator, but it would also be necessary to determine a first ray transfer matrix to model the effect of the optical elements located upstream from the non-linear transfer module on the light field of the hologram, and a second ray transfer matrix to model that of the optical elements located downstream from the non-linear processing module. An example will be described in more details in relation with FIGS. 4A and 4B.

It is herein supposed that the holographic reproduction system PRO comprises no non-linear processing module. Hence, a single intrinsic ray transfer matrix M_INis obtained.

At E6, an overall ray transfer matrix M_Gof a light field emitted by the input hologram H_Efrom the input plane P_Eto the output plane P_Sof the reproduction system SRH is calculated, by a product of the extrinsic ray transfer matrix M_EXand the intrinsic ray matrix M_IN:

M_G=M_EX·M_IN

At E7, the input hologram H_Eis transformed by application of an integral transformation operator T obtained from the calculated overall matrix M_G, in accordance with the above-described equations (1) to (3). This operator T performs the transformation of the light field of the input hologram H_Ethrough the arrangement of optical elements of the SRH up to the output plane P_Sof the holographic screen.

Hence, if the overall matrix M_Gis equal to

(

A

B

C

D

)

,

with A, B, C and D 2×2 matrices, the operator T is defined in theory as the operator that transforms the field ƒ of the input hologram H_Einto the field ƒ′ of the output hologram

ƒ′(X′)=T(ƒ)(X′)=det(iB)^−1/2∫ƒ(X)e^iπP(X,X′)dX, (2)

with

P(X,X′)=B⁻¹AX+DB⁻¹X′−2X^tB⁻¹X′, (3)

and X, X′ the vectors of spatial components of the light fields of the input plane P_Eand the output plane P_S, respectively.

In practice, the operator T is discretized by transforming the integrals of the equation (2) into discrete sums to take into account the resolution Nx_E, Ny_Eand the sizes Sx_E, Sy_Eof the input hologram ƒ, and the resulting field ƒ′ is sampled according to the resolution Nx_S, Ny_Sand the sizes Sx_S, Sy_Sof the output hologram H_S, as follows:

f

k
,
l

′

=

det
⁡

(
iB
)

-

1
2

⁢

∑

i
=
1

NxE

⁢

∑

j
=
1

NyE

⁢

f

i
,
j

⁢

e

i
⁢

⁢
π
⁢

⁢

P
(

X

i
,
j

,

X

k
,
l

′

)

⁢

Δ
X

(
4
)

where k, l are the indices of the sample ƒ′_k,l, which corresponds to the discretized version of the light field ƒ′ transformed by the linear operator T;

with

ƒ_i,j=ƒ(X_i,j) (5)

f
′

k
,
l

=

f
′

⁡

(

X
′

k
,
l

)

(
6
)

X

i
,
j

=

(

iΔ
x

jΔ
y

)

(
7
)

X
′

k
,
l

=

(

kΔ
′

x

jΔ
′

y

)

(
8
)

Δ
′

x

=

(

kΔ
′

x

jΔ
′

y

)

(
9
)

and

Δ
X

=

S
x

⁢

S
y

NxE
·
NyE

(
10
)

Advantageously, the function ƒ′_k,lis calculated according to a known method that consists in extracting therefrom a discrete Fourier transform:

f
′

k
,
l

=

det
⁡

(
iB
)

-

1
2

⁢

e

i
⁢
π
⁢
D
⁢

B

-
1

⁢

X
′

k
,
l

⁢

Δ
X

⁢
T
⁢
F
⁢

D
B

⁡

(
G
)

⁢

(

X
′

k
,
l

)

(
11
)

TFD_Bdenotes the discrete Fourier transform staggered by a factor B⁻¹, G being the following discrete function:

e

i
⁢
π
⁢

B

-
1

⁢
A
⁢

X

i
,
j

⁢

f

i
,
j

(
12
)

A number of samples ƒ′_k,lcorresponding to the resolutions Nx_S, Ny_Sand sizes Sx_S, Sy_Sof the output holographic screen SLM is hence calculated.

The light field of the output hologram that is to be reproduced by the SLM for an observer Ob to visualize correctly the scene reconstructed by the input hologram H_Ein the viewing plane P_Vis hence obtained.

It is understood that the fact to group into a single linear operator all the transformations to be applied to the input light field makes it possible to limit the total number of Fourier transforms to be calculated and hence to reduce the complexity of the system.

In relation with FIGS. 4A and 4B, we will now describe a second exemplary embodiment of the processing method according to the invention, when the holographic reproduction system SRH is a device intended to be worn on the head of a user, such as a head-mounted display or augmented-reality glasses. It will be called hereinafter HMD.

In this example, the HMD comprises:

- a module (not shown) for receiving an input hologram H_E, similar to the above-described one. The means for the connection to the communication network are herein preferentially of the wireless type;
- a module (not shown) for obtaining a pose T_Ob, R_Obof the observer Ob in the reference frame RM. As already mentioned, this pose is generally expressed as 6 parameters representative of a translation T_Oband a rotation R_Obin the reference frame RM. It can be obtained with a position tracking device, of the IMU (“Inertial Measurement Unit”) type, known from the person skilled in the art, placed on the HMD;
- a display device of the holographic screen SLM type, placed in an output plane P_S. The SLM has the sizes Sx_S, Sy_Sand it is discretized into Nx_S×Ny_Spixels. The usual values of N×s and Ny_Svary between 720 and 4096, for sizes Sx_Sand Sy_Sof the order of about one cm. It is generally placed on the side or on the top of the observer's head so as to be out of his viewing field;
- a filtering system of the 4F type, know from the person skilled in the art, intended to suppress frequencies associated with directions of emission of undesirable diffraction orders. It comprises a non-linear processing module MTNL and two thin lenses L₁and L₂;
- a field lens L_Cintended to enlarge the light field emitted by the SLM; and
- a beam splitter SF intended to redirect the light field of the hologram in the direction of a viewing plane P_Vbehind which the eye of an observer Ob is positioned. Indeed, the SLM is placed on the side or on the top of the observer's head, whereas the viewing plane of the hologram is located in front of the observer's eyes, as a glass of a pair of glasses. The viewing plane P_Vand the output plane P_Sare hence not parallel to each other. That way, the possibility for the observer Ob to see his real environment by transparency in superimposition on a 3D scene SC_Rreconstructed by the hologram is preserved.

The HMD/observer unit being mobile in the World reference frame RM, the input plane P_Eand viewing plane P_Vare this time not necessarily parallel to each other.

In relation with FIG. 4B, the propagation path of the light field of the input hologram from the input plane P_Eto the output plane of the SLM is decomposed as follows:

- a rotation EO₁between the input plane P_Eand a corrected plane P_Rparallel to the viewing plane P_V;
- a propagation EO₂in free space from the corrected plane P_Rup to the viewing plane P_V, over a length d₁;
- a propagation EO₃in free space, over a length d₂from the viewing plane P_Vup to the beam splitter SF. The passing through the beam splitter SF is omitted because it acts as a mirror;
- a propagation EO₄in free space, from the beam splitter SF up to the field lens L_C, over a length d₃;
- a passing EO₆through the field lens L_C;
- a propagation EO₆in free space between the field lens L_Cand the first lens L₁of the filtering system 4F, over a distance d₄;
- a passing EO₇through the first lens L₁of the filtering system 4F;
- a propagation EO₈in free space between the first lens L₁of the filtering system 4F and the non-linear processing module MTNL, over a distance d₅;
- a processing EO₉of non-linear filtering MTNL;
- a propagation EO₁₀in free space between the non-linear processing module MTNL and the second lens L₂of the system 4F, over a distance d₆;
- a passing EO₁₁through the second lens L₂of the system 4F; and
- a propagation EO₁₂in free space between the second lens L₂of the filtering system 4F and the output plane P_Sof the SLM, over a distance d₇.

Each of the optical elements or processing operations mentioned hereinabove is denoted EO₁, EO₂, EO₃, EO₄, EO₅, EO₆, EO₇, EO₈, EO₉, EO₁₀, EO₁₁and EO₁₂respectively, and

the propagation operator representing the passing through the optical element EO_i, i.e. the operator that, with a light field arriving at the input of the optical element EO_i, associates the light field that exits therefrom, is denoted T_i. The propagation from the input hologram H_Eto the SLM on the output plane H_Sis hence represented by the operator:

T=T₁₂∘T₁₁∘T₁₀∘T₉∘ . . . ∘T₁,

where ∘ denotes the composition of the operators.

This chain of operators is composed of a so-called extrinsic part, relating to the direct propagation of the light field, in free space between the input plane P_Eand the viewing plane P_V, and of a so-called intrinsic part, relating to the reverse propagation of the light field, inside the augmented reality system SRH. The extrinsic part depends on the position and direction of the observer with respect to the input plane P_Eof the hologram H_Eand hence changes at each displacement of the observer Ob, whereas the intrinsic part is independent of the movements thereof and can be modelled once for all in a preliminary phase. The extrinsic part corresponds to the processing by the operators T₁and T₂, the intrinsic part corresponds to the processing by the product of the operators T₃to T₁₂.

In this example, two processing operations undergone by the light field on the path thereof between the input plane P_Eand the output plane P_Scannot be represented by a ray transfer matrix. They are:

- 1) the rotation EO₁of the light field between the input plane P_Eand a corrected plane P_R, parallel to the viewing plane P_Vcorresponding to the relative direction between the HMD/observer system and the plane P_Eof the input hologram;
- 2) the filtering of the light field by the non-linear processing module MTNL, which corresponds to the optical element EO₉.

It is understood that, in this second example, it is not possible to model the transformation of the light field by a single operator of the LCT type, due to the fact that these two non-linear processing operations segment the chain of propagation operators into several propagation operators, 3 of which are of the LCT type.

The steps of the method for processing a input hologram according to the invention will now be described in this particular exemplary embodiment.

At E1, a hologram H_Eis received.

At E2, a pose of the observer Ob in the World reference frame RM is obtained. It is used to evaluate, at E3, if the input hologram is visible by the observer Ob. It is herein considered that the observer is integral with the HMD system that he wears on his head. That way, the pose of the observer corresponds to the pose of the HMD system.

More precisely, in relation with FIGS. 5A and 5B, it is considered that the hologram is visible by the observer Ob if, on the one hand, the quadrilateral Q_Eof the input hologram H_Eis in his vision cone CV_o, and if, on the other hand, the observer Ob himself is in the emission or visibility cone CV_Eof the hologram H_E, i.e. inside an angular sector in which the hologram is capable of emitting a light signal.

In other words, the matter is to make a test of reciprocal visibility, which may, for example, be calculated as follows:

Four planes PH₁, PH₂, PH₃and PH₄inclined by an angle φ with respect to the quadrilateral Q_Eof the plane P_Eand respectively intersecting the four edges of Q_Eare defined. The angle φ may be chosen as the maximum diffraction angle given by φ=arcsin(X/(λ/(2Sx_E/Nx_E)). It is, for example, of 4.5 degrees for a hologram of resolution 72K with a size Sx_Eof the order of 30 cm. Likewise, four planes PO₁, PO₂, PO₃and PO₄inclined by an angle Υ with respect to the quadrilateral Q_V, Υ being the maximum viewing angle of the holographic reproduction system HMD, are defined. For the existing HMDs, Υ is of the order of 30 degrees. The visibility cone CV_Eis delimited by the quadrilateral Q_Eand the planes PH₁, PH₂, PH₃and PH₄, and the vision cone CV_Oof the observer is delimited by the quadrilateral Q_Vand the planes PO₁, PO₂, PO₃and PO₄. The plane of the hologram P_Eand the viewing plane P_Vare then decreed mutually visible if at least one of the vertices of the quadrilateral Q_Eis inside CV_O, and if one of the vertices of the quadrilateral Q_Vdefining P_Vis inside CV_E. We call inner side of a cone plane the side that is located inside the cone. For that purpose, it is verified that at least one vertex in question is located on the inner side of all the planes defining the concerned cone.

If the visibility test is positive, then the method continues with step E4. Otherwise, it is stopped until a new pose of the observer Ob is received.

At E4, the ray transfer matrices M₁₂, M₁₁, M₁₀, M₈, M₇, M₆, M₅, M₄, M₃, M₂, M_f-1and M_f, respectively associated with the above-listed optical elements EO₁₂to EO₁₀and EO₈to EO₂are obtained.

At E5, a first overall ray transfer matrix M_Acorresponding to the path of the light field between the output of the filtering module EP₉and the output plane P_Sis calculated:

M_A=M₁₂·M₁₁·M₁₀

- with

M

1
⁢
2

=

(

I

λ
⁢

d
7

⁢
I

0

I

)

- for the propagation or the light field in free space over a distance d₇. It is for example possible to choose d₇=2 cm;

M

1
⁢
1

=

(

I

0

-
I

/
λ

⁢

f
2

I

)

- for the propagation of the light field through the thin lens L₂of focal length f₂. It is for example possible to place a lens of focal length f₂=5 cm;

M
10

=

(

I

λ
⁢

d
6

⁢
I

0

I

)

- for the propagation of the light field in free space over a distance d₆. It is for example possible to choose d₆=5 cm.

And at E6, a second overall ray transfer matrix M_Bcorresponding to the path of the light field between the corrected input plane P_Rand the input of the non-linear filtering module EO₉:

M_B=M₈M₇M₆M₅M₄M₃M₂M_f-1

- with

M

8
=

⁡

(

I

λ
⁢

d
5

⁢
I

0

I

)

- for the propagation of the light field in free space over a distance d₅. It is for example possible to choose d₅=5 cm;

M
7

=

(

I

0

-
I

/
λ

⁢

f
1

I

)

- for the propagation of the light field through the thin lens L₁of focal length f₁. It is for example possible to place a lens of focal lens f₁=5 cm;

M
6

=

(

I

λ
⁢

d
4

⁢
I

0

I

)

- for the propagation or the light field in free space over a distance d₄. It is for example possible to choose d₄=1 cm;

M
5

=

(

I

0

-
I

/
λ

⁢

f
C

I

)

- for the propagation of the light field through the thin lens L_Cof focal length f_C. It is for example possible to place a lens of focal lens f_C=2 cm;

M
4

=

(

I

λ
⁢

d
3

⁢
I

0

I

)

- for the propagation of the light field in free space over a distance d₃. It is for example possible to choose d₃=1 cm;

M
3

=

(

I

λ
⁢

d
2

⁢
I

0

I

)

- for the propagation of the light field in free space over a distance d₂. It is for example possible to choose d₂=1 cm;

M
2

=

(

I

λ
⁢

d
1

⁢
I

0

I

)

- for the propagation of the light field in free space over a distance d₁. d₁is of the order of about ten centimetres;
- M_f⁻¹denotes an inverse Fourier transform.

M_Ais intrinsic and hence independent of the observer's displacements.

M_Bis partly extrinsic and intrinsic and depends on the observer's displacements, due to the matrix M₂.

It must be recalculated at each displacement of the observer Ob.

A non-linear operator T₁for the transformation of the light field by the optical element EO₁is considered. It consists of a rotation of the direction of the input plane P_Etowards the direction of the viewing plane P_V. The operator T₁can hence be decomposed as follows:

T₁=U∘T_f

where:

- T_fdenotes the 2D Fourier transform. This is a linear operator;
- U is the operator that, to a function ƒ, associates ƒ∘H, where H is the function that, to two spatial frequencies u and v, associates the two first components of the following product:

R
·

(

u

v

λ

-
2

-

u
2

-

ν
2

)

,

where R is a rotation matrix such that its application to the plane P_Ecorrects the latter for it to be parallel to the viewing plane P_Vaccording to a method known by the person skilled in the art.

An operator LCT T_Acorresponding to the matrix M_Aand an operator LCT T_Bcorresponding to the matrix M_Bare also considered, in accordance with the matching established by the equations (1) to (3).

We have T_A=T₁₂∘T₁₁∘T₁₀and T_B=T₈∘ . . . T₂∘T_f-1.

It is noted that the inverse Fourier transform T_f-1can be herein integrated to the operator T_B, because it is a linear operation.

An overall operator T for transforming the light field is deduced by means of the following expression:

T=T_A∘T₉∘T_B∘U∘T_f.

For example, the filtering T₉can be defined as the operator that, to a function ƒ, associates ƒ·H (product of functions), where H is the function that, to a couple (u, v) of real numbers, associates 1 if v>0 and 0 otherwise.

For example, it is a directional filtering blocking the rays directed upward, u and v representing the spatial frequency variables of the light signal. As a variant, a filter of bass-band type could also be used.

At E7, the output hologram H_Sto be displayed on the screen SLM of the output plane so that the observer Ob can see, in the viewing plane P_Vof his HMD, the 3D scene SC_Ras reconstructed by the input hologram in the input plane, is now calculated.

For that purpose, the samples of the light field of the input hologram are transformed by successive applications to them of the operators T_A, T₉, T_B, U and T_f. As regards the linear operators, the successive transformations are made, as in the first exemplary embodiment, from the overall matrices M_Aand M_B, by application to the plurality of samples of the input light field of a discrete Fourier transform according to the above-described equation (11). The transformed light field is resampled to obtain the number of samples f′_k,lcorresponding to the output resolutions Nx_S, Ny_Sand output sizes Nx_S, Ny_Sof the output hologram H_S.

In a variant embodiment of the invention, we consider now the case in which the observer Ob wearing the system HMD moves in the World reference frame RM. It is supposed that the system HMD is equipped with a module for obtaining the position/direction of the observer wearing the HMD in the World reference frame, which allows it to obtain a regular updating of this position/direction and to detect a displacement of the observer Ob with respect to the plane of the input hologram P_E. The existing HMDs allow for example an updating of this position/direction with a temporal frequency of the order of a few milliseconds. It is supposed that the just-described method according to the invention has been implemented at the instant t₀, then that a displacement of the observer Ob has been detected at an instant t₁with respect to an instant t₀.

This detection triggers the updating of the operator U of rotation of the input plane P_Ewith respect to the viewing plane P_V, of the matrix M₂corresponding to the propagation in free space over a distance d₁(t₁), and hence of the overall matrix M_B.

A new operator T_Bis calculated from the updated version of the overall matrix M_B(t₁) at the instant t₁.

The aggregation of the operators by multiplication of the ray transfer matrices reduces the number of Fourier transforms to be calculated, which contributes to reduce the overall calculation time of the transformed light field. Indeed, as explained hereinabove, the application of each operator T_ito the complex light field of the input hologram H_Eis performed by calculating a discrete Fourier transform. Thanks to the invention, the fact to translate the transformations induced by the passing through a plurality of optical elements by means of a limited number of overall operators, reduces proportionally the number of Fourier transforms to be applied, and hence the complexity of the processing method.

In relation with FIGS. 6 and 7, a third exemplary embodiment of the invention will now be described. In this third exemplary embodiment, several input holograms H_Ei, with i an integer comprised between 1 and N, are received, which are associated with the face planes of a geometric solid, such as for example a rectangular parallelepiped. In other words, there are as many reconstructed scenes as solid faces, which makes it possible to offer the observer a view of the scene from a wider range of points of view.

It is understood that this third embodiment is particular suited to a reproduction system of the HMD type, according to which the observer can move about the plane P_Eof the input hologram, and hence view certain faces of the geometric solid as a function his position/direction in the World Coordinate System RM.

In the example of FIG. 6, the geometric solid considered is a cube and the holographic reproduction system receives 6 holograms H_Ei, with i an integer comprised between 1 and I=6, the hologram H_Eibeing associated with one of the six faces Fi of the cube. In relation with FIG. 7, the method for processing this plurality of input holograms {H_Ei} according to this second embodiment of the invention will now be described.

During a step E1′, the plurality of holograms {H_Ei} is received.

At E2′, a pose of the observer Ob is obtained. At E3′, at least one face F_j, with j comprised between 1 and N, of the cube of which the observer is liable to view the associated hologram H_Ejis selected. In other words, it searched if there exist at least one face Fi_jfor which the field of the associated hologram H_Ei_jis viewable by the observer Ob. Advantageously, the above-described visibility test is performed for each of the faces Fi.

Two cases are contemplated:

- A single visible face Fi_jis selected and a single corresponding hologram H_Ei_jwill be recalculated by the method according to the invention. This face will be for example the closest to the observer or that which passes the best the above-described visibility test;
- Several visible faces Fi₁, Fi₂. . . Fi_jare selected, with i an integer comprised between 1 and I, and the corresponding holograms are recalculated in order to compose an output hologram from the recalculated holograms. The contributions of each field are then summed into a total output hologram H_S.

In the first case, it is supposed that the face Fi₀has been selected and then the method for processing the selected hologram H_Ei₀is then executed as described in relation with FIG. 2, which corresponds to steps E4′ to E8′ of FIG. 7.

In the second case, steps E4′ to E8′ are repeated for each of the selected input holograms H_Ei_j, as long as all the selected faces have not been processed (E9′, E10′). Once all the output holograms H_Si_jcalculated, an output hologram H_Sis composed from the J holograms H_Si_j, at E11′.

With the invention, it becomes possible for an observer wearing a system HMD to turn about the scene reconstructed by the plurality of holograms associated with the faces of the geometric solid.

It will be noted that the just-described invention can be implemented by means of software and/or hardware components. From this perspective, the terms “module” and “entity”, used in this document, can correspond either to a software component, or to a hardware component, or also to a set of hardware and/or software components, adapted to implement the function(s) described for the concerned module or entity.

In relation with FIGS. 8 and 9, we will now present the simplified structure of a processing device 100 adapted to implement the method for processing an input hologram according to any one of the particular embodiments of the invention that have just been described in relation with FIGS. 2 to 7.

The processing device is adapted to process at least one input hologram and to produce an output hologram.

The processing device 100 is in particular configured to:

- receive at least one input hologram and one associated pose in the World Coordinate System;
- obtain at least one first matrix, called extrinsic geometric matrix (Me), representative of a propagation in free space between the reference plane and the output plane of the reproduction system, said matrix being function of the reference position and the output position;
- obtain at least one second matrix, called intrinsic geometric matrix (Mi), representative of a propagation between the output plane and the input plane of the reproduction system through at least one said optical element;
- calculate at least one overall matrix M_Gfor geometric transformation of a light field emitted by the input hologram between the reference plane and the input plane of the reproduction system, by a product of at least two of said matrices;
- transform the input hologram into the output hologram by application of at least one linear integral operator determined from the at least one calculated matrix, as a function of an input resolution Nx_E, Ny_Eand of input sizes of the input hologram (P_E) and of output resolutions Nx_S, Ny_Sand of output sizes Sx_S, Sy_Sof the holographic screen.

According to a particular embodiment of the invention illustrated by FIG. 8, the processing device 100 has the conventional architecture of a computer and comprises in particular a processing unit 110, equipped with a processor μ₁, and piloted by a computer program Pg₁120, stored in a memory 130 and implementing the method according to the invention.

At the initialization, the code instructions of the computer program Pg₁120 are for example loaded in a memory RAM before being executed by the processor of the processing unit 110. The processor of the processing unit 110 implements the steps of the above-described method, according to the instructions of the computer program 120.

According to another particular embodiment of the invention illustrated by FIG. 9, the processing method is implemented by functional modules. For that purpose, the processing device 100 comprises at least the following functional modules:

- a module REC. for receiving one or several input holograms H_E;
- a module OBT. M_EXfor obtaining at least one first, so-called extrinsic, ray transfer matrix, representative of a propagation in free space between the reference plane and the output plane of the reproduction system;
- a module OBT. M_INfor obtaining at least one second, so-called intrinsic, ray transfer matrix, representative of a propagation between the output plane and the input plane of the reproduction system through at least one said optical element;
- a module CALC. M_Gfor calculating at least one overall ray transfer matrix of a light field emitted by the input hologram between the reference plane and the input plane of the reproduction system, by a product of said matrices; and
- a module TRANS for transforming the light field of the input hologram H_Eby application of at least one linear integral operator determined from the at least one calculated matrix, as a function of an input resolution Nx_E, Ny_Eand input sizes of the input hologram (P_E) and of an output resolution Nx_S, Ny_Sand output sizes Sx_S, Sy_Sof the holographic screen.

Optionally, the device 100 comprises a module OBT. T_Ob, R_Obfor obtaining a pose of the observer in the World reference frame and a unit SEL H_jfor selecting at least one input hologram, visible by the observer.

According to an embodiment of the invention, the device 100 comprises a module COMPO for composing several output holograms obtained by transformation of input holograms selected as visible by the observer into a single output hologram.

The device 100 further comprises a module M1 for storing the ray transfer matrices calculated by the invention, as well as operators determined from these matrices.

The units are piloted by the processor μ1 of the processing unit 110.

The processing unit 110 cooperates with the different above-described functional modules and the memory MEM1 in order to implement the steps of the processing method. The different above-described functional modules may be in hardware and/or software form. In software form, such a functional module may include a processor, a memory and program code instructions to implement the function corresponding to the module when the code instructions are executed by the processor. In hardware form, such a functional module may be implemented by any type of adapted coding circuits, such as for example and non-limitatively, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), a logical unit wiring.

Advantageously, such a device 100 may be integrated to a holographic reproduction system SRH. The device 100 is then arranged to cooperate at least with the following module of the system SRH:

- a data transmission/reception module E/R, through which the input hologram(s) H_Eare received via a telecommunication network, for example a wired network or a wireless network; and
- as the case may be, a module POS for tracking the position of the observer in the World reference frame RM.

It is obvious that the above-mentioned embodiments have been given for purely informative and non-limiting purposes, and that many modifications can be easily made by the person skilled in the art without thereby departing from the framework of the invention.