The term “data library” describes in context of the present disclosure a collection of numerical methods, machine learning models or artificial intelligence-based algorithms. In the present disclosure the term “first data library” describes a collection of trained machine learning models for identifying an object within a scene context, the trained machine learning model comprises at least one identification probability factor. The identification probability factor for an object can be stored in a table or any other form of an information storage within the first data library. An example for the first data library is TensorFlow, Image Net, or PyTorch.

The machine learning model has been trained to recognize and identify at least one object and the context in which the object is detected. An object can be for example a computer keyboard and the context can be a computer keyboard on an office desk. The computer keyboard and the office desk together with the surroundings do represent a scene. As used herein, the term “trained machine learning model” refers to a model of a machine learning model, specifically comprising at least one neural network, that is trained, particularly trained to determine at least one identification probability factor.

The term “identification probability factor” refers to a certain probability that the trained machine learning model generates from analyzing the recognized object within the second data set. The previously mentioned example comprises a computer keyboard on an office desk. In this example the trained machine learning model generates an identification probability factor, that the recognized object is a computer keyboard on an office desk. The identification probability factor can contain a value in the range from 0 to 1, with 0 and 1 included, which equals to 0% to 100%. An identification probability factor of 0 is equal to 0% and means that the first data library does not know which object is recognized in the context. Hence, there is no such data available for training the machine learning model to identify for example a computer keyboard on a desk. In this case the machine learning model may be trained in a further step. An identification probability factor of 1 is equal to 100% and means that the first data library does know with the highest possible probability which object is recognized in the context. Hence, there is enough data available for training the machine learning model to identify for example a computer keyboard on a desk. In this case the machine learning model does not need to be trained in a further step. The identification probability factor comprises, typically, the identification probability of at least one object within a context or, alternatively, the identification probability factor of at least one object without a context.

The term “generating” or any grammatical variation thereof refers to a process of generating a certain information within the scope of this disclosure. This can refer for example to generate an identification probability factor or to generate a data set. An information can be generated by calculation, recording, extracting, assigning, deriving, processing, loading, predicting or any other possible step of information generation.

In a further step of the identifying process, the identification probability factor is assigned to each object of the plurality of objects based on the first data library. At the end of the further step of the identification process, an identification probability factor is assigned to recognized objects in the scene.

The term “assigning” or any grammatical variation thereof refers to a process of merging one piece of information with a second piece of information. For example, the second data set can provide the existence of an object. The first data library generates the identification probability factor of the object. By assigning a particular identification probability factor of a certain percentage to the recognized object, the computer-implemented method knows that e.g. a computer keyboard on a desk is identified within the scene with a probability of the certain percentage. The certain percentage is in a range from 0% to 100%.

In a further step of the identifying process, the identification probability factor of each identified object of the plurality of objects is compared with a predefined identification probability threshold. At the end of the further step of the identification process, all objects with an identification probability factor below the predefined identification probability threshold are ignored for further steps. If no identified object comprises a larger identification probability factor compared to the predefined identification probability threshold than, i) the predefined identification probability threshold may be decreased to the highest assigned identification probability factor of the recognized objects in order to ensure the further processing of the computer-implemented method or ii) the computer-implemented method stops at this step, does not tune the tunable lens and proceeds with the next scene in order to tune the tunable lens of the optometry device. Alternatively, if no object can be identified, the tunable lens can be tuned to a default distance, for example to infinite, or to another distance based on raw data of the first data set. If more than one identified object comprises a larger identification probability factor compared to the predefined identification probability threshold, then all such objects will pass to the next step of the identifying process.

The term “predefined identification probability threshold” refers to a filter step for further processing of identified objects. The predefined identification probability threshold can contain a value in the range from 0 to 1, with 0 and 1 included, which equals to 0% to 100%. In an exemplary embodiment the value of the predefined identification probability threshold is between 0.5 and 0.9, which equals to 50% to 90%. In an exemplary embodiment, the value of the predefined identification probability threshold is between 0.6 and 0.8, which equals to 60% to 80%. In a most preferred embodiment, the value of the predefined identification probability threshold is 0.7, which equals to 70%.

In a further step of the identifying process, a second data library for generating a selection probability factor for each identified object of the plurality of objects, or at least for all objects having an identification probability factor equal or larger that the identification probability threshold, is used. At the end of the further step of the identification process, identified objects are selected, the selected objects comprise a certain selection probability of being observed by the user of the optometry device.

In the present disclosure the term “second data library” describes a collection of trained machine learning models for selecting an object within a scene context, the trained machine learning model comprises at least one selection probability factor. The selection probability factor for an object can be stored in a table or any other form of an information storage within the second data library. The selection probability factor of the second data library is based on the selection probability of the user of the optometry device.

Moreover, the second data library comprises individual characteristics of a user. For example, a user above the age of retirement shows a smaller selection probability factor for changing the visual field to a computer compared to a younger user who works with computers on daily basis. The younger user will show a larger selection probability factor for computers as compared to the user above the age of retirement. Alternatively, a past behavior of the user, taken from the second data library, shows that the user never looks at computers. Hence, the selection probability factor of that user for a computer is small. Other parameters like time of the day, day of the week etc. which impact the visual behavior, and thus selected objects can also be comprised in the second data library. Examples for the second data library are DeepGaze or SCEGRAM.

The machine learning model has been trained to select at least one object out of the context in which the object is detected. An object can be for example a computer keyboard and a computer mouse and the context can be a computer keyboard and a computer mouse on an office desk. As used herein, the term “trained machine learning model” refers also to a model of a machine learning model, specifically comprising at least one neural network, that is trained, particularly trained to determine at least one selection probability factor.

The term “selection probability factor” refers to a certain probability that the trained machine learning model generates from analyzing the recognized and identified object(s) within the second data set. The previously mentioned example comprises a computer keyboard and a computer mouse on an office desk. In this example the trained machine learning model generates a selection probability factor, for the recognized and identified objects. In other words, the machine learning method generates a value which comprises a selection probability for a plurality of objects within a scene. The selection probability factor can contain a value in the range from 0 to 1, with 0 and 1 included, which equals to 0% to 100%. A selection probability factor of 0 is equal to 0% and means that based on the second data library the selection probability that the user will focus the recognized and identified object in this context does not exist. A selection probability factor of 1 is equal to 100% and means that based on the second data library the selection probability that the user will focus the recognized and identified object in this context is 100%. The selection probability factor comprises, preferably the selection probability of at least one object within a context or alternatively, the selection probability of at least one object without a context. The selection probability factor for a particular object, accordingly, is a measure for the probability or likelihood that a user will focus on this particular object.

In a further step of the identifying process, the selection probability factor of at least each identified object of the plurality of objects having an identification probability factor larger than the predefined identification probability threshold is compared with a predefined selection probability threshold. At the end of the further step of the identifying process, all objects with a selection probability factor below the predefined selection probability threshold are ignored for consecutive further steps.

If no identified object comprises a larger selection probability factor compared to the predefined selection probability threshold than, i) the predefined selection probability threshold may be decreased to the highest assigned selection probability factor of the recognized and identified objects in order to ensure the further processing of the computer-implemented method or ii) the computer-implemented method stops at this step, does not change the focus of the tunable lens and proceeds with the next scene in order to tune the tunable lens of the optometry device. If more than one identified object comprises a larger selection probability factor compared to the predefined selection probability threshold, then the object with the larger selection probability factor will pass to the next step of the identifying process.

The term “predefined selection probability threshold” refers to a filter step for further processing of identified object. The predefined selection probability threshold can contain a value in the range from 0 to 1, with 0 and 1 included, which equals to 0% to 100%. In an exemplary embodiment the value of the predefined selection probability threshold is between 0.5 and 0.9, which equals to 50% to 90%. In an exemplary embodiment, the value of the predefined selection probability threshold is between 0.6 and 0.8, which equals to 60% to 80%. In a most preferred embodiment, the value of the predefined selection probability threshold is 0.7, which equals 70%.

In a further step of the identifying process, the distance of the user of the optometry device to the identified object with the largest selection probability factor is determined based on the step of comparing the selection probability factor of at least each identified object of the plurality of objects having an identification probability factor larger than the predefined identification probability threshold with a predefined selection probability threshold and selecting the object with the highest selection probability factor. At the end of the further step of the identifying process, the computer-implemented method is able to tune a tunable lens of the optometry device based on the distance of a user of the optometry device to the object with the highest selection probability factor based set, on the second data

The further step of identifying the object, enables the computer-implemented method not only to recognize but also to identify an object within the recorded scene. Hence, the further step of identifying the object leads to a further step of tuning the tunable lens of the optometry device based on a known and identified object. This results in a more natural and more accurate adjustment of focus of the tunable lens of the optometry device and leads to a better comfort of wear for the user of the optometry device. Moreover, tuning a tunable lens may be achieved before the user is changing the view to another object.

In an exemplary embodiment, the computer-implemented method is characterized in that machine learning algorithms are used to determine the identification probability factor stored in the first data library and the selection probability factor stored in the second data library.

The further step of identifying the object, enables the computer-implemented method to load an identification probability factor or a selection probability factor from the first data library or the second data library without the need of calculating a new identification probability factor or a new selection probability factor. This results in a faster adjustment of focus of the tunable lens of the optometry device and leads to a better comfort of wear for the user of the optometry device. Moreover, tuning a tunable lens can be achieved before the user is changing the view to another object.

In an exemplary embodiment, the computer-implemented method is characterized by generating a third data set, the third data set comprising gaze information of the user of the optometry device, predicting a next gaze of the user based on the third data set, and tuning the tunable lens of the optometry device based on the predicted next gaze of the user.

In a particular embodiment of the present aspect, the term “third data set” refers to an output of an eye tracking sensor. This output may be represented by an RGB image or a numerical information about a saccade and/or a fixation point of a user in the context of a scene. The term “saccade” refers to a movement of at least one eye between two or more phases of fixation in the same direction. The term “fixation” or “fixation point” refers to a moving behavior of the eye. During a “fixation” a spatial location of the gaze is situated constantly on a single location. In other words, if a fixation is reached, the eye does not move in a recognizable fashion.

The term “gaze information” refers to the visual field of the user of the optometry device and may refer to the change in the line of sight of the person. Based on DIN ISO 13666:2019, Section 3.2.24, the term “line of sight” refers to a path from a point of interest, i.e. a point of fixation (or fixation point), in object space to a center of an entrance pupil of the eye of the person and, further, comprises a continuation in image space from a center of an exit pupil to a retinal point of fixation, generally the foveola, in the eye of the person. The term “predicted next gaze” refers to the estimated next visual field of the user of the optometry device.

The term “next fixation point” describes in context of the present disclosure the end of a saccade and the beginning of a new fixation. The term “predicted next fixation point” refers to a next fixation point which is generated based on the third data set.

The third data set of the computer-implemented method recorded by an eye tracker sensor enables an evaluation of the eye movement of the user based on the recorded scene represented by the first and second data set. Hence, the further step of tuning a tunable lens of the optometry device is based and optimized by the predicted next fixation point of the user. This results in a faster adjustment of focus of the tunable lens of the optometry device and leads to a better comfort of wear for the user of the optometry device. Moreover, tuning a tunable lens can be achieved before the user is changing the view to another object.

In an exemplary embodiment, the computer-implemented method is characterized by predicting the next gaze of the user comprising the following steps:

- determining a current fixation point of the user of the optometry device based on the third data set and,
- assigning the object based on the second data set and the distance of the object to the user of the optometry device based on the first data set to the current fixation point of the user of the optometry device and,
- tracking the current fixation point of the user of the optometry device based on the third data set until a next saccade of the user of the optometry device is recognized by detecting a change in gaze position and,
- predicting a next fixation point of the user of the optometry device based on the current fixation point of the user of the optometry device and the recognized next saccade of the user of the optometry device and,
- tuning the optometry device according to the predicted next fixation point.

In a first step of the predicting process, a current fixation point of the user of the optometry device based on the third data set is determined. At the end of the first step of the predicting process, the fixation point is determined for the beginning of a new saccade, represented by the time t=0.

The term “current fixation point” refers to i) the fixation point which is already established when the predicting process starts or to ii) a fixation point which is detected after the predicting process starts.

In a further step of the predicting process, the object identified and selected based on the second data set and the distance of the object to the user of the optometry device based on the first data set are assigned to the current fixation point of the user of the optometry device. Alternatively, with the further step of the predicting process, the current fixation point is assigned to an object identified and selected based on the second data set and to a distance of the user of the optometry device to the object based on the first data set.

In a further step of the predicting process, the current fixation point of the user of the optometry device is tracked based on the third data set until a next saccade of the user of the optometry device is recognized by detecting a change in gaze position. At the end of the further step of the predicting process, the beginning of a new saccade is detected.

The term “tracking” or any grammatical variation thereof refers to a process of identifying a change in information within the scope of this disclosure. This can refer for example to track a scene or a gaze position. A change of information can be tracked by observing, detecting, analyzing, searching or any other possible step of identifying a change in information.

In a further step of the predicting process, a next fixation point of the user of the optometry device is estimated based on the current fixation point of the user of the optometry device and the recognized next saccade of the user of the optometry device. At the end of the further step of the predicting process, the next fixation point of the user of the optometry device is predicted.

The term “predicting” or any grammatical variation thereof refers to a process of estimating an information within the scope of this disclosure. This can refer for example to predict a scene or a fixation point. Estimating an information can be carried out by calculating, interpolating, comparing, guessing or any other step of estimating an information.

In a further step of the predicting process, the optometry device is tuned according to the predicted next fixation point.

The further step of identifying the object, enables the computer-implemented method to estimate the next fixation point of the user of the optometry device. This results in a faster adjustment of focus of the tunable lens of the optometry device and leads to a better comfort of wear for the user of the optometry device. Moreover, tuning a tunable lens can be achieved before the user is changing the view to another object.

In an exemplary embodiment, the computer-implemented method is characterized in that predicting the next fixation point comprising at least one of the following steps:

- determining a gaze direction of the recognized next saccade and predicting the next fixation point based on previous gaze information of the user
- determining a gaze direction of the recognized next saccade and predicting the next fixation point based on the next selectable object within the determined gaze direction
- determining a gaze direction of the recognized next saccade and predicting the next fixation point based on an object within the determined gaze direction, the object comprising the highest selection probability factor.
- using a predefined next scene of the second data library and assigning the distance of the object to the user of the optometry device based on the first data set and adjusting the estimation of the next fixation point based on the predefined next scene;
- estimating a next scene by generating a fourth data set, the fourth data set comprising electrical brain activity information of the user of the optometry device and adjusting the estimation of the next fixation based on estimated next scene;
- estimating a next scene by generating a fifth data set, the fifth data set comprising inertial measurement information of the optometry device (110, 710) and adjusting the estimation of the next fixation point (365, 465, 565, 665) based on estimated next scene.

A step of predicting the next fixation point may be realized by determining a gaze direction of the recognized next saccade and predicting the next fixation point based on previous saccade information of the user.

The term “gaze direction” refers to spatial localization detection of the current saccade. Starting from the current fixation point, the next saccade can move in every spatial direction, e.g. in the northeast direction. The gaze direction can be specified in the form of celestial directions, global or local coordinate systems or by spatial directions.

The term “previous gaze information” refers to a sample of previous gaze information of the user of the optometry device or of a standard user of the optometry device. The previous gaze information can comprise data of a saccade distance length, a saccade duration, a fixation point and a fixation duration. The previous gaze information is generated and stored in the third data set.

Starting from a current fixation point and the beginning of a next saccade, the next fixation point is predicted by analyzing the last saccade distance length and/or the last saccade duration.

An alternative step of predicting the next fixation point may be realized by determining a gaze direction of the recognized next saccade and predicting the next fixation point based on the next selectable object within the determined gaze direction.

The term “next selectable object” refers to an object within the saccade direction. Furthermore, the next selectable object is identifiable and selectable based on the second data set. Hence, the identifiable and selectable object comprises an identification probability factor larger than the predefined identification probability threshold and a selection probability factor larger than the predefined selection probability threshold.

Starting from a current fixation point and the beginning of a next saccade, the next fixation point is predicted by analyzing the saccade direction. For example, if the beginning of the next saccade indicates a gaze direction in northeast direction and an object in the northeast direction is identifiable and selectable based on the second data set, then the next fixation point will be set at the object. If more than one objects within the saccade direction are identifiable and selectable based on the second data set, then the next fixation point is set on the object closer to the current fixation point. The object closer to the current fixation point represents in the context of the disclosure the “next selectable object.”

If no object within the saccade direction is identifiable and selectable based on the second data set, then i) the predefined identification probability threshold and/or the predefined selection probability threshold may be decreased to the highest assigned identification probability factor and/or the next fixation point is set to the object with the highest assigned selection probability factor of the recognized and identified objects in order to ensure the further processing of the computer-implemented method or ii) the computer-implemented method stops at this step, does not change the focus of the tunable lens and proceeds with the next scene in order to tune the tunable lens of the optometry device.

An alternative step of predicting the next fixation point may be realized by determining a gaze direction of the recognized next saccade and predicting the next fixation point based on an object within the determined gaze direction, the object comprising the highest selection probability factor.

Starting from a current fixation point and the beginning of a next saccade, the next fixation point is predicted by analyzing the saccade direction. For example, if the beginning of the next saccade indicates a gaze direction in northeast direction and a plurality of objects in the northeast direction are identifiable and selectable based on the second data set, then the next fixation point will be set at an object comprising the highest selection probability factor of the plurality objects within the gaze direction.

If no object within the saccade direction is identifiable and selectable based on the second data set, then i) the predefined identification probability threshold and/or the predefined selection probability threshold may be decreased to the highest assigned identification probability factor and/or to the highest assigned selection probability factor of the recognized and identified objects in order to ensure the further processing of the computer-implemented method or ii) the computer-implemented method stops at this step, does not change the focus of the tunable lens and proceeds with the next scene in order to tune the tunable lens of the optometry device.

An alternative step of predicting the next fixation point may be realized by using a predefined next scene of the second data library and assigning the distance of the object to the user of the optometry device based on the first data set and adjusting the estimation (or prediction) of the next fixation point based on the predefined next scene.

The term “predefined next scene” describes an item of the second data library. This item is configured to train the machine learning model or to provide testing scenes for the user of the optometry device. Such testing scenes are displayed on a screen, e.g. a monitor or a mobile device display. The “predefined next scene” is evaluated by the computer-implemented method before the scene is shown to the user of the optometry device. The identification probability factor and the selection probability factor of objects within the predefined next scene are determined prior to the recognition of the scene by the user of the optometry device. Hence, the tunable lens of the optometry device is tuned based on the predefined next scene prior to the recognition of the scene by the user of the optometry device.

An alternative step of predicting the next fixation point may be realized by estimating a next scene by generating a fourth data set, the fourth data set comprising electrical brain activity information of the user of the optometry device and adjusting the estimation (or prediction) of the next fixation point based on the estimated next scene.

In a particular view of the present aspect, the term “fourth data set” refers to an output of a sensor comprising electrical brain activity information such as data of an electroencephalography method or an electrooculography method. This output may be represented by voltage signals from different regions of the brain activity, comprising information about the amplitude and the frequency of the voltage signals and providing information about the next gaze and next fixation point of the user.

The starting point for estimating a next scene comprises the detection of the current gaze of the user of the optometry device, e.g. the right side of the visual view of the user. If the user decides to change his view to the left side of the visual view, an electrical brain activity signal is sent to the eye of the user. The brain activity signal is detected and analyzed by the sensor used for generating the fourth data set.

Moreover, the tunable lens of the optometry device is tuned based on the detected and analyzed brain activity signal of the fourth data set. Hence, the fixation of the user is trailing the tuning step of the tunable lens.

An alternative step of predicting the next fixation point may be realized by estimating a next scene by generating a fifth data set, the fifth data set comprising inertia measurement information of the optometry device and adjusting the estimation (or prediction) of the next fixation point based on the estimated next scene.

In a particular view of the present aspect, the term “fifth data set” refers to an output of an inertia measurement unit comprising inertia measurement information such as an angular rate and the orientation of the optometry device by using a combination of accelerometers, gyroscopes, and may be magnetometers, and providing information about the next gaze and next fixation point of the user.

The starting point for estimating a next scene comprises the detection of the current gaze of the user of the optometry device and the associated inertia measurement information to the current gaze of the user. If the user decides to change his view from one side to another, a change in the fifth data set is generated. The change in the fifth data set is detected and analyzed by the inertia measurement unit to generate the fifth data set.

Moreover, the tunable lens of the optometry device is tuned based on the detected and analyzed inertia measurement information of the fifth data set. Hence, the fixation point of the user is trailing the tuning step of the tunable lens.

The first aspect of the disclosure is fully solved by the computer-implemented method of the above described type.

In a second aspect, the present disclosure relates to a system, comprising:

- an optometry device,
- a first device configured to generate a first data set, the first data set comprising a distance of a user of the optometry device to an object,
- a control unit configured to tune the optometry device according to the distance of the user of the optometry device to the object based on the first data set.

The system is characterized in that the first device is a LiDAR sensor.

As generally used, the term “system” refers to at least one apparatus or a plurality of apparatus, particularly connected via a computer network. The plurality of apparatus may be connected via a network by using at least one connection interfaces at any one of the apparatus of the plurality of apparatus.

The term “control unit” refers to a device configured to tune the optometry device according to the data set generated by the other devices. An example for a “control unit” is an Optotune device driver controlled by a PC or a Raspberry Pi.

In an exemplary embodiment, the system comprises a second device, wherein the second device is a camera.

In an exemplary embodiment, the system comprises a second device, wherein the second device is an RGB camera.

In an exemplary embodiment of the system the control unit is configured to tune the optometry device based on the object identified in a second data set recorded by the second device.

In an exemplary embodiment, the system comprising a third device, the third device is an eye-tracker device configured to predict a next gaze of the user based on the third data set.

In an exemplary embodiment of the system the control unit is configured to select the object based on an identification probability factor and a selection probability factor determined as described above with reference to the computer-implemented method.

The second aspect of the disclosure is fully solved by system of the above described type.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, properties, and advantages of the present disclosure will become clear from the following description of exemplary embodiments of the disclosure in conjunction with the accompanying drawings.

FIG. 1 shows a first exemplary embodiment of a top view onto a user of an optometry device with a tunable lens;

FIG. 2 shows a block diagram of a first exemplary embodiment of the method for operating a tunable lens by using a LiDAR sensor and an RGB camera;

FIG. 3 shows a block diagram of a second exemplary embodiment of the method for operating a tunable lens by using a LiDAR sensor, an RGB camera and an eye tracking device;

FIGS. 4A, 4B, and 4C show various scenes to illustrate a first exemplary embodiment of a predicting process;

FIGS. 5A, 5B, and 5C show various scenes to illustrate a second exemplary embodiment of a predicting process;

FIGS. 6A, 6B, and 6C show various scenes to illustrate a third exemplary embodiment of a predicting process; and

FIG. 7 shows a sectional view of a second exemplary embodiment of an optometry device with a tunable lens.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

A first exemplary embodiment of the system 100 for operating a tunable lens 101 is shown in FIG. 1. The system 100 is worn by user 110 and comprises a tunable lens 101, a LiDAR sensor 103, an RGB camera 105, an eye tracking device 117 and a control unit (not shown in FIG. 1) which is located within the tunable lens 101.

As shown in FIG. 1 the LiDAR sensor 103 and the RGB camera 105 are recording simultaneously a scene 107, which is observed by the user 110. The scene 107 can comprise several objects 109, 111 and 115. The data set recorded by the LiDAR sensor 103 comprises data representing respective distances of the user 110 to each of the objects 109, 111 and 115. The data set recorded by the RGB camera 105 comprises data representing images of each object 109, 111 and 115. The eye tracking device 117 records the movement of the eye of the user 110 based on the observed scene 107.

Based on the second data set recorded by the RGB camera 105 the objects 107 and 109 are recognized, identified and selected by the control unit of the system 100. Furthermore, based on the first data set recorded by the LiDAR sensor 103 the respective distances of the user 110 to each of the objects 107 and 109 are determined by the control unit of the system 100. The control unit of the system 100 assigns in a next step each distance to the respective object 109, 111 and 115. Based on the selection by the control unit and the determined distance of the selected object, the tunable lens 101 is tuned according to the data of the LiDAR sensor and the data of the RGB camera.

Optionally the data of the eye tracking device 117 may be processed by the control unit of the system 100 and used for tuning the tunable lens 101 according to the data of the LiDAR sensor, the data of the RGB camera and the data of the eye tracking device.

FIG. 2 shows the process steps of a first exemplary embodiment of the method 200 for operating a tunable lens 101, 701 by using a LiDAR sensor 103, 703 and an RGB camera 105, 705 of the systems 100 and 700.

In a first data generating step 201 of the first exemplary embodiment of the method 200 the first data set 211 and the second data set 212 are generated. The first data set 211 is generated by a LiDAR sensor and the second data set 212 is generated by an RGB camera. The first data set 211 and the second data set 212 both comprise the same scene 107 (see FIG. 1), wherein the scene comprises the objects 109, 111 and 115.

Based on the second data set 212, the identification step 203 is executed. In a first sub-step of the identification step 203 an identification probability factor 283 is determined for each of the objects 109, 111 and 115 of the scene 107, based on the second data set 212 and the first data library 281.

In an exemplary embodiment the object 109 can comprise an identification probability factor 283 of 0.8, the object 111 can comprise an identification probability factor 283 of 0.7 and the object 115 can comprise an identification probability factor 283 of 0.6.

In a second sub-step of the identification step 203, the identification probability factor 283 of each identified object 109, 111 and 115 is compared with a predefined identification probability threshold 285. At the end of the second sub step of the identification step 203, the objects with an identification probability factor 283 below the predefined identification probability threshold 285 are ignored for further steps. In the above exemplary embodiment, the predefined identification probability threshold 285 can comprise a value of 0.7. Hence, the object 109 and the object 111 which comprise an identification probability factor 283 equal to or higher than 0.7 will be considered for further steps and the object 115 which comprises an identification probability factor 283 of 0.6 will not be considered for further steps.

Based on the second data set 212 and the identification step 203, the selection step 205 is executed.

In a first sub-step of the selection step 205 a selection probability factor 293 is assigned for the remaining objects 109 and 111 of the scene 107 having an identification probability factor equal to or larger than the identification probability threshold, based on the second data set 212 and the second data library 291. In the above exemplary embodiment, the object 109 may comprise a selection probability factor 293 of 0.8 and the object 111 may comprise a selection probability factor 293 of 0.7.

In a second sub-step of the selection step 205, the selection probability factor 293 of each identified object 109 and 111 is compared with a predefined selection probability threshold 295. At the end of this second sub step of the selection step 205, objects with a selection probability factor 293 below the predefined selection probability threshold 295 are ignored for further steps. If more than one object exceeds the predefined selection probability threshold 295, than the object 297 comprising the higher or highest selection probability factor 293 is selected. In the above exemplary embodiment, the predefined selection probability threshold 295 may comprise a value of 0.7. Hence, the object 109 and the object 111 which comprise a selection probability factor 293 equal to or higher than 0.7 may be considered for further steps. The selection probability factor 293 of object 109 is higher than the selection probability factor 293 of object 111. Hence, the object 109 will be considered for further steps.

Based on the first data set 211 and the selected object 297, the assigning step 207 is executed. In the assigning step 207 the distance of the selected object 297 is assigned to the selected object 297 based on the first data set 211

Based on and following the assigning step 207 the tuning step 209 is executed. In the tuning step 209 the tunable lens 101 is tuned based on the selected object 297 and the distance of the selected object 297 based on the first data set 211. This results in a tuned tunable lens 101.

FIG. 3 shows a second exemplary embodiment of the method 300 for operating a tunable lens by using a LiDAR sensor 103, an RGB camera 105 and an eye tracking device 117.

In a data generating step 301 of the second exemplary embodiment of the method 300 the first data set 311, the second data set 312 and the third data set 313 are generated. The first data set 311 is generated by the LiDAR sensor 103, the second data set 312 is generated by the RGB camera 105 and the third data 313 set is generated by an eye tracking device 117. The first data set 311 and the second data set 312 both comprise the same scene 107, wherein the scene comprises the objects 109, 111 and 115 (see FIG. 1). The third data set 313 comprises saccade information and fixation point information of the user 110 in the context of the scene 107. In other words, the first data set 311 and the second data set 312 represent the scene 107, whereas the third data set 313 represents the movement of the eyes of the user that are triggered by the scene 107.

Based on the third data set 313, a determining step 315 is executed. In the determining step 315 a a current fixation point 361 of the user 110 of the tunable lens 101 is obtained, based on the third data set 313. In an exemplary embodiment the current fixation point 361 is set to the object 109 at the left side of the recognized scene 207.

Based on the current fixation point 361 and the third data set 313, a change in gaze position 362 is detected by a tracking step 317. The change in gaze position 363 may lead to a new fixation point. In the above exemplary embodiment, the gaze position changes from the left side of the recognized scene 107 to the right side of the recognized scene 107.

Based on the change in gaze position 363 detected by the tracking step 317, an estimated fixation point 365 is estimated by a predicting step 319. In this exemplary embodiment the estimated fixation point 365 is set to the object 107 at the right side of the recognized scene 107.

Based on the second data set 312 an identification step 303, which is similar to the identification step 203 of the previous first exemplary embodiment for operating a tunable lens and a selection step 305, which is similar to the selection step 205 of the previous first exemplary embodiment for operating a tunable lens are executed. At the end of the selection step 305 one or more selectable objects 397, which comprise a larger selection probability factor compared to the predefined selection probability threshold, can be suitable for setting the next fixation point.

Based on the first data set 311, the selectable objects 397 based on the second data set 312 and the estimated fixation point 365, an assigning step 307 is executed. In the assigning step 307 the distance of an estimated fixated object 107 to the user 110 based on the first data set 311 and the position of the estimated fixated object 107 is assigned to the new fixation point 365. In other words, the computer-implemented method generates in the assigning step 307 the information that the estimated fixation point 365 is set to the object 107 if the object 107 has a larger selection probability factor as the predefined selection probability threshold.

Based on the assigning step 307 the tuning step 309 is executed. In the tuning step 309 the tunable lens 101 is tuned based on the estimated fixated object 107 and the distance of the estimated fixated object 107 based on the first data set 311. This results in a tuned tunable lens 101.

With the aid of FIGS. 4A to 4C, a first exemplary embodiment of a predicting process 400 based on the method is explained in more detail. Each of FIGS. 4A to 4C represents a same scene observed by the user at different times t₁to t₃. This first exemplary embodiment is based on determining a gaze direction of a recognized next saccade and predicting a next fixation point based on the next selectable object within the determined gaze direction.

FIG. 4A shows a scene 407 comprising the objects 409, 411 and 415 and represents the beginning of the predicting process at a time t₁=0 s. A current fixation point 461, at the time t₁=0 s is assumed to be set at the object 409. The current fixation point 461 is represented as a circle, the circle comprises all possible micro saccades within a fixation point. The current fixation point 461 will remain set at the object 409, as long as the current fixation point 461 at least partially covers the object 409.

FIG. 4B shows the scene 407 comprising the objects 409, 411 and 415 and represents a current saccade 463 within the predicting process at a time t₂=0.5 s. The current saccade 463, at the time t₂=0.5 s is moving along a direction d₁(see arrow in FIG. 4B) The current saccade 463 will remain moving until the user of the tunable lens fixates a new object.

Based on the detected saccade the computer-implemented method predicts a new fixation point 465 of the user by determining the direction d₁of the current saccade 463 and detecting the next object within the direction d₁of the current saccade 463 which comprises i) an identification probability factor not lower than a predefined identification probability threshold and ii) a selection probability factor not lower than a predefined selection probability threshold.

In a numerical example of the exemplary first embodiment of a predicting process 400, the predefined identification probability threshold is assumed to be set to 0.7 and the predefined selection probability threshold is assumed to be set to 0.7. If one object 415 in the direction d₁of the detected saccade comprises an identification probability factor of 0.8 and a selection probability factor of 0.8, and another object 411 in the direction d₁of the detected saccade comprises an identification probability factor of 0.8 and a selection probability factor of 0.9, both objects 415 and 411 are suitable for setting the predicted fixation point 465.

FIG. 4C shows the scene 407 comprising the objects 409, 411, and 415 and represents the end of the predicting process at a time t_3=1.0s. A predicted fixation point 465, at the time t_3=1.0s is set at the object 415. The predicted fixation point 465 is set at the object 415 due to the fact, that the object 415 is the next selectable object within the direction d₁of the saccade at the time t₂. If the object 415 would not be in the direction d₁of the saccade at the time t₂than the predicted fixation point 465 would be set at the object 411.

With the aid of FIGS. 5A to 5C, a second exemplary embodiment of a predicting process 500 based on the method is explained in more detail. Each of FIGS. 5A to 5C represents a same scene observed by the user at different times t₁to t₃. This second exemplary embodiment is based on determining a gaze direction of the recognized next saccade and predicting the next fixation point based on an object within the determined gaze direction, the object comprising the highest selection probability factor.

FIG. 5A shows the scene 507 comprising the objects 509, 511, and 515 and represents the beginning of the predicting process at a time t=0 s. A current fixation point 561, at the time t₁=0 s is assumed to be set at the object 509. The current fixation point 561 is represented as a circle, the circle comprises all possible micro saccades within a fixation point. The current fixation point 561 will remain set at the object 509, as long as the current fixation point 561 at least partially covers the object 509.

FIG. 5B shows the scene 507 comprising the objects 509, 511, and 515 and represents a current saccade 563 within the predicting process at a time t₂=0.5 s. The current saccade 563, at the time t₂=0.5 s is moving along a direction d₁(see arrow in FIG. 5B). The current saccade 563 will remain moving until the user of the tunable lens fixates a new object.

Based on the detected saccade the computer-implemented method predicts a new fixation point 565 of the user by determining the direction d₁of the current saccade 563 and detecting the objects within the direction d₁of the current saccade 563 which comprise i) an identification probability factor not lower than an predefined identification probability threshold and ii) a selection probability factor not lower than an predefined selection probability threshold and in the case of a plurality of objects, the highest selection probability of the objects.

Like in the example as described with reference to FIGS. 4A to 4C, the predefined identification probability threshold is assumed to be set to 0.7 and the predefined selection probability threshold is assumed to be set to 0.7. If, the object 515 comprises an identification probability factor of 0.8 and a selection probability factor of 0.8 and the object 511 comprises an identification probability factor of 0.8 and a selection probability factor of 0.9, both objects 515 and 511 are suitable for setting the predicted fixation point 565.

FIG. 5C shows the scene 507 comprising the objects 509, 511 and 515 and represents the end of the predicting process at a time t