101. The color event data may be generated within an event detection time interval that is the same as the exposure or integration time interval of the first color imaging pixel. In some embodiments, the color event data is received within a specific time interval after a time at which the color signal is received. In some embodiments, the color event data is generated within a specific time interval before a time at which the color signal is generated. The specific time interval maybe configured based on at least the exposure (or integration) time interval of the first color imaging pixel 101 and the readout time of the color signal.

In further some embodiments, a first set of color event data may be received within a first time interval before a time at which the color signal is generated, and a second set of color event data may be received within a second time interval after a time at which the color signal is generated. In such embodiments, the image signal of the first color imaging pixel (e.g., first color imaging pixel 101) of the plurality of color imaging pixels is determined based on the color signal of the first color image pixel 101, the first color event data of one or more color event pixels of the plurality of color event pixels, and the second color event data of one or more color event pixels of the plurality of color event pixels. The first time interval and the second time interval may be configured in view of an exposure or integration interval of the first color imagining pixel 101.

In some embodiments of the present disclosure, the one or more color event pixels 111, 113, 114 are ones of the plurality of color event pixels, 111, 112, 113, 114 being correlated to the first color image pixel 101 (e.g., spatially or temporally correlated to the first color image pixel 101).

In some embodiments of the present disclosure, the one or more color event pixels 111, 113, 114 are ones of the plurality of color event pixels, 111, 112, 113, 114 being located in close proximity to the first color imaging pixel 101. In some embodiments of the present disclosure, a color sensed by the first color image pixel 101 is different from a color associated with one of the one or more color event pixels 111, 113, 114. However, in some embodiments of the present disclosure, a color sensed by the first color image pixel 101 may be identical to a color of one of the one or more color event pixels 111, 113, 114 that are used to determine the image signal of the first color imaging pixel 101.

In some embodiments of the present disclosure, image signals of each color imaging pixel included in the plurality of color imaging pixels in the hybrid image sensor 10 are determined based on a color signal of the color image pixel and the color event data of one or more color event pixels included in the plurality of color event pixels 111, 112, 113, 114. In some embodiments of the present disclosure, the one or more color event pixels are correlated to each color imaging pixel of the plurality of color imaging pixels. In some embodiments of the present disclosure, the first color imaging pixel 101 is positioned within a region of interest of the plurality of color image pixels 101, 102, 103, 104, 105.

FIG. 2 illustrates a flow chart 20 of a method of color imaging, in accordance with some embodiments of the present disclosure. The flow chart 20 may be executed by a processor included in a hybrid image sensor.

The method may start from block 200. In some embodiments of the present disclosure, color CIS data is captured from the plurality of color imaging pixels 101, 102, 103, 104, 105 in block 201. In block 203, the color CIS data is stored into a CIS data buffer. The CIS data buffer may be included in a memory bank on the hybrid image sensor. Similarly, in some embodiments of the present disclosure, color EVS data is captured from the plurality of color event pixels 111, 112, 113, 114 in block 202. In block 204, the color EVS data is stored into an EVS buffer. The EVS data buffer may be included in a memory bank on the hybrid image sensor. In embodiments, block 202 may be implemented after block 201.

To combine the color CIS data and the color EVS data, the color EVS data stored in the EVS buffer is converted to the CIS color space in step 206. In block 210, the fusion of the color CIS data and the color EVS data is performed. The method ends at block 211. In some embodiments of the present disclosure, the method of FIG. 2 is performed in real time.

FIG. 3 illustrates time diagrams of the log pixel illuminance and the event flow, in accordance with some embodiments of the present disclosure.

FIG. 3 shows an EVS-to-CIS color space conversion illustrating how color events from color event pixels 111, 112, 113, 114 are used to determine a given CIS color channel. In some embodiments of the present disclosure, the events are accumulated through event time integral shown as follows:

$\begin{matrix} E_{p_{i}} (t) = f (E_{C} (t)) & (1) \end{matrix}$

wherein p_i∈P, c_i∈C·P is a set of CIS color channels, and C is a set of EVS color channels. In some embodiments of the present disclosure, when there are four CIS color channels and four EVS color channels.

$\begin{matrix} P = {p_{1}, p_{2}, p_{3}, p_{4}} & (2) \end{matrix}$

$\begin{matrix} C = {c_{1}, c_{2}, c_{3}, c_{4}} & (3) \end{matrix}$

Accordingly, an arbitrary combination of the four EVS color channels to obtain a given CIS color channel can be represented as follows:

$\begin{matrix} E_{p_{i}} (t) = f (E_{C} (t)) = f (E_{c_{1}} (t), E_{c_{2}} (t), E_{c_{3}} (t), E_{c_{4}} (t)) & (4) \end{matrix}$

That is, the event integral for any CIS color channel p_iis a function of the event integrals of the event color channels c_i.

The function of the event time integral can be shown as follows:

$\begin{matrix} E (t) = \int_{f}^{t} e (s) d s & (5) \end{matrix}$

wherein f is a reference time, t represents the endpoint time for event accumulation.

L(f) is the logarithm of the pixel illuminance of the given CIS color channel as reference. The logarithm of the temporally aligned image L(t) can be calculated as follows:

L(t)=L(f)exp(cE(t)),

wherein c is an event detection threshold value.

Accordingly, the CIS data at the reference time f and the event integral between time f and time t are combined to obtain a temporal estimation for a given color event pixel at the time t.

FIG. 3 includes two time diagrams 31 and 32. As shown in FIG. 3, the vertical axis 311 of time diagram 31 is the logarithm of the pixel illuminance, and the vertical axis 321 of the time diagram 32 is the event flow e(t). Each of the impulses in time diagram 32 means a detected event of one of the color event pixels. The detected events include positive events and negative events corresponding to the change in intensity. For example, impulse 323 is a positive event (e.g., increase in light intensity) and impulse 325 is a negative event (e.g., decrease in light intensity).

Curve 313 in time diagram 31 is the actual pixel illuminance received by the given CIS color channel. Curve 315 in time diagram 31 is the reconstructed pixel illuminance of the given CIS color channel. As shown in time diagram 31, curve 315 corresponds to the integral of the impulses in time diagram 32. At the time of each positive impulse, curve 315 steps up a positive detection threshold value c, and, at the time of each negative impulse, curve 315 steps down a negative detection threshold value c. As shown in time diagram 31, curve 315 can fit curve 313 so that the reconstructed pixel illuminance can be deemed as corresponding to the actual pixel illuminance received by the given CIS color channel.

In some embodiments of the present disclosure, such process can be applied to correct one or multiple color channels in an image sensor. In some embodiments of the present disclosure, such process can be applied to correct all color channels in an image sensor. In some embodiments of the present disclosure, such process can be applied to correct all pixel locations in an image sensor. In some embodiments of the present disclosure, such process can be applied to a specific (e.g., a region of interest) in an image sensor, which may be enabled by activity monitoring, object recognition or feature recognition, etc.

In some embodiments of the present disclosure, input data formats other than event integral can be used for the color space conversion. For example, events can be represented as voxels or pseudo-frames. The number of color channels for CIS and EVS can also be arbitrary and not limited to three or four. That is, the CFA patterns for CIS and EVS can be arbitrary. In some embodiments of the present disclosure, in case that the CFA pattern for EVS is an RGGB CFA pattern, the CFA pattern of CIS can be (but not limited to) an RGGB CFA pattern or an RGBC CFA pattern. In some embodiments of the present disclosure, in case that the CFA pattern for CIS is an RGGB CFA pattern, the CFA pattern of EVS can be (but not limited to) a BCGR CFA pattern, a BYYR CFA pattern or a CyYYMg CFA pattern. Other variations of the input data formats might also include the resolution (different or the same between CIS and EVS) and the spatial arrangement. In some embodiments of the present disclosure, the spatial location of the pixels to be used may depend on the color space conversion scheme.

FIG. 4 illustrates time diagrams of the log pixel illuminance and the event flows, in accordance with some embodiments of the present disclosure.

FIG. 3 shows reconstructing the actual pixel illuminance with detected events of one of the color event pixels. FIG. 4 further shows reconstructing the actual pixel illuminance with detected events of more than one of the color event pixels. FIG. 4 includes four time diagrams 41, 42, 43 and 44. Similar to FIG. 3, the vertical axis of time diagram 41 is the logarithm of the pixel illuminance, and the vertical axes of the time diagrams 42, 43 and 44 are the event polarity as a function of time e(t). The impulses in time diagrams 42, 43 and 44 represent detected events of each of the color event pixels 111, 113 and 114 in FIG. 1. In some embodiments of the present disclosure, the color event pixel 111 is used to detect red light, the color event pixel 113 is used to detect green light, and the color event pixel 114 is used to detect blue light.

In one example, for a given color imaging pixel 101, a combination of nearby color EVS pixels such as red EVS pixel data of color event pixel 111, green EVS pixel data of color event pixel 113, and blue EVS pixel data of color event pixel 114 can be used to obtain equivalent white (W) imaging data associated with the given color imaging pixel 101. The detected events accumulated between reference time f and time t also include positive events and negative events. In one exemplary embodiments of the present disclosure, the arbitrary combination of the three color event pixels can be shown as follow:

$E_{W} (t) = 0.2 5 E_{R} (t) + 0.5 E_{G} (t) + 0.2 5 E_{B} (t)$

wherein the weights 0.25, 0.5 and 0.25 are color-weighting factors that can be obtained based on an off-line color calibration process which will be explained in more detail later in FIG. 6.

As shown in time diagram 42, during the time interval from reference time f to time t, there are four positive events and one negative event. Therefore, assuming that the positive and negative contrast detection threshold are the same, the total number of impulses counted for the time interval should be three. The weight for red color event pixel is 0.25. Therefore, the weighted impulses for the time interval in time diagram 42 is 0.75 (i.e., E_R(t)=0.25*3=0.75).

As shown in time diagram 43, during the time interval from reference time f to time t, there are five positive events and three negative events. Therefore, assuming that the positive and negative contrast detection threshold are the same, the total number of impulses counted for the time interval should be two. The weight for green color event pixel is 0.5. Therefore, the weighted impulses for the time interval in time diagram 43 is 1 (i.e., E_G(t)=0.5*2=1).

As shown in time diagram 44, during the time interval from time f to time t, there are two positive events and one negative event. Therefore, assuming that the positive and negative contrast detection threshold are the same, the total number of impulses counted for the time interval should be one. The weight for blue color event pixel is 0.25. Therefore, the weighted impulses for the time interval in time diagram 44 is 0.25 (i.e., E_B(t)=0.25*1=0.25).

Accordingly, the arbitrary combination of the three color event pixels E_W(t) is 2 (i.e., E_W(t)=0.75+1+0.25). That is, as shown in time diagram 41, curve 415, in reference to the initial starting value at reference time f, steps up two positive detection threshold value c increments at time t.

FIG. 5 is a flow chart 50 illustrating a method for converting color event data to color space data, in accordance with some embodiments of the present disclosure. FIG. 5 shows the details of step 206 in FIG. 2.

The method starts from step 500. In some embodiments of the present disclosure, the destination of a CIS color channel (color imaging pixel) is first defined in step 501. The target color to be computed is also defined. In step 502, the EVS pixels (the color event pixels) in close proximity to the CIS color channel (associated with the given color imaging pixel) are selected and event data of the selected EVS pixels in proximity are received from the EVS buffer as shown in FIG. 2. In step 503, a set of color conversion model parameters is retrieved from a register bank so that they can be applied to the event data of the EVS pixels to generate color space data. The register bank stores data from an offline calibration which will be explained in FIG. 6. The color EVS data is then converted to the destination CIS color space. In step 504, the color space data is returned so as to fuse the color space data with the image signal of the CIS color channel. The method ends at step 505. In some embodiments of the present disclosure, the method of FIG. 5 can be performed in real time on an image sensor like the method of FIG. 2.

FIG. 6 illustrates a flow chart 60 of an error minimization process, in accordance with some embodiments of the present disclosure.

The error minimization process starts from step 600. In some embodiments of the present disclosure, color CIS data is captured from the color imaging pixels in step 601. In step 603, the color CIS data is stored into a CIS buffer. Similarly, in some embodiments of the present disclosure, color EVS data is captured from the color event pixels in step 602. In step 604, the color EVS data is stored into an EVS buffer. In step 605, differences between the color CIS data of consecutive CIS frames are computed. In step 607, the patches where the color changes in the color imaging pixels of different CIS frames are found. In step 606, the integrals of the color EVS data within time intervals between the corresponding CIS frames are computed to obtain accumulated color EVS information. Then, the set of color conversion model parameters are optimized so as to minimize the errors between the pixel value differences and the event integrals in step 610, and the set of color conversion model parameters is stored to the register bank in step 620. The error minimization process ends at step 630. In some embodiments of the present disclosure, the error minimization process of FIG. 6 is performed in an offline manner. That is, the color conversion model parameters is predetermined through the error minimization process so that they can be stored in the register bank for later use of the method in FIG. 5.

In comparison to the methods in FIGS. 2 and 5, the error minimization process is an offline calibration and the method in FIGS. 2 and 5 are real time processes. In the offline calibration, the parameters needed for color space conversion are computed by considering CIS pixel locations where color changes and their respective event data in the same time interval. A loss function (e.g., the error between the target pixel from CIS data and its estimation) together with a predefined conversion scheme (e.g., linear, quadratic, neural network, or look-up table, etc.) guide such process. Additionally, the method in in