₁, W₂, . . . , W_N. However, the present disclosure is not limited to that example. The wavelength bands included in the target wavelength range W may be set as desired. For instance, the wavelength bands may have nonuniform band widths. A gap or an overlap may exist between adjacent two of the wavelength bands. In the example illustrated in FIG. 3B, the wavelength bands have different band widths, and a gap exists between adjacent two of the wavelength bands. Thus, the wavelength bands just need to be different from each other, and a manner of determining the wavelength bands may be selected as desired.

FIG. 4A is an explanatory view illustrating characteristics of the spectral transmittance in a certain region of the filter array 110. In the example illustrated in FIG. 4A, the spectral transmittance has local maximum values P1 to P5 and local minimum values with respect to wavelengths in the target wavelength range W. In the example illustrated in FIG. 4A, the spectral transmittance is normalized such that a maximum value of the spectral transmittance in the target wavelength range W is 1 and a minimum value is 0. In the example illustrated in FIG. 4A, the spectral transmittance has local maximum values in some wavelength ranges, for example, the wavelength band W₂and the wavelength band W_N-1. Thus, the spectral transmittance in each region has local maximum values in at least two among the wavelength bands W₁to W_N. In the example illustrated in FIG. 4A, the local maximum values P1, P3, P4, and P5 are greater than or equal to 0.5.

As described above, the optical transmittance in each region is different depending on the wavelength. Accordingly, the filter array 110 allows a component of the incident light in a certain wavelength range to pass therethrough in a large amount, but it does not allow another component of the incident light in another wavelength range in a so large amount. For instance, the transmittance may be greater than 0.5 for the light in k bands among N wavelength bands and may be smaller than 0.5 for the light in the remaining N-k wavelength bands. Here, k is an integer satisfying 2≤k<N. If the incident light is white light including even visible wavelength components, the filter array 110 modulates the incident light to light with discrete intensity peaks in terms of wavelength for each region and outputs obtained multiwavelength light in a superimposed fashion.

FIG. 4B illustrates, by way of example, a result of averaging the spectral transmittance illustrated in FIG. 4A for each of the wavelength bands W₁, W₂, . . . , W_N. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing an integrated value by the band width of the relevant wavelength band. In this Specification, a value of the transmittance averaged for each wavelength band as mentioned above is defined as the transmittance in the relevant wavelength band. In the illustrated example, the transmittance is remarkably high in three wavelength ranges taking the local maximum values P1, P3, and P5. Particularly, the transmittance exceeds 0.8 in two wavelength ranges taking the local maximum values P3 and P5.

The examples illustrated in FIGS. 2A to 2D are premised on the transmittance distribution with a gray scale in which the transmittance in each region can take an arbitrary value of greater than or equal to 0 and smaller than or equal to 1. However, it is not always required to use the transmittance distribution with a gray scale. For instance, a transmittance distribution with a binary scale in which the transmittance in each region can take either one of a value of substantially 0 or a value of substantially 1 may also be used. In the case of the transmittance distribution with a binary scale, each region allows most part of light in at least two of the wavelength ranges included in the target wavelength range to pass therethrough and does not allow most part of light in the remaining wavelength ranges to pass therethrough. Here, the wording “most part” indicates part of greater than or equal to 80%.

Part of all cells, for example, a half of the cells, may be replaced with transparent regions. The transparent regions allow the lights in all the wavelength bands W₁to W_Nincluded in the target wavelength range W to pass therethrough with a comparable high transmittance, for example, a transmittance of greater than or equal to 80%. In such a configuration, the transparent regions may be arranged in, for example, a checkerboard pattern. In other words, the region with the optical transmittance being different depending on the wavelength and the transparent region may be alternately arrayed in two array directions of the regions in the filter array 110.

Data representing the above-described spatial distribution of the spectral transmittance of the filter array 110 is obtained in advance as design data or with actual calibration and is stored in the storage medium included in the processing device 200. That data is utilized in a calculation process described later.

The filter array 110 may be formed using, for example, a multilayer film, an organic material, a grating structure, or a microstructure containing a metal. When the multilayer film is used, a dielectric multilayer film or a multilayer film containing a metal layer may be used, for example. In this case, the multilayer film is formed such that at least one of a thickness, a material, or a layer laminating order of the multilayer film is different for each cell. In such a manner, spectral characteristics different depending on the cell can be realized. Using the multilayer film enables sharp rising and falling of the spectral transmittance to be realized. The configuration using the organic material may be realized by selecting different pigments or dyes to be contained in the cells, or by laminating different types of materials for each cell. The configuration using the grating structure may be realized by forming individual grating structures to be different in grating pitch or depth for each cell. When the microstructure containing the metal is used, the microstructure can be fabricated by utilizing dispersion of light due to the plasmon effect.

An example of signal processing executed by the processing device 200 will be described below. The processing device 200 reconstructs the multiwavelength spectral image 220 based on both the image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance of the filter array 110 for each wavelength. Here, the word “multiwavelength” indicates wavelength ranges in larger number than, for example, three-color wavelength ranges of RGB that are obtained by a usual color camera. The number of those wavelength ranges may be, for example, greater than or equal to 4 and smaller than or equal to about 100. The number of the wavelength ranges is referred to as the “band number”. The band number may exceed 100 depending on the application.

Data to be acquired is data of the spectral image 220, and that data is denoted by f Assuming the band number to be N, f is data resulting from integrating the image data f₁, f₂, . . . , f_Nin the individual bands. Here, as illustrated in FIG. 1A, a horizontal direction of the image is assumed to be an x-direction, and a vertical direction of the image is assumed to be a y-direction. Assuming that the number of pixels in the x-direction of the image data to be acquired is denoted by n and the number of pixels in the y-direction is denoted by m, each of the image data f₁, f₂, . . . , f_Nis two-dimensional data of n×m pixels. Accordingly, the data f is three-dimensional data with the number of elements being n×m×N. This three-dimensional data is referred to as a “hyperspectral data cube” or a “hyperspectral cube”. On the other hand, the number of elements of data g of the image 120 obtained after coding and multiplexing by the filter array 110 is n×m. The data g can be expressed by the following equation (1).

$\begin{matrix} ℊ = Hf = H [\begin{matrix} f_{1} \\ f_{2} \\ ⋮ \\ f_{N} \end{matrix}] & (1) \end{matrix}$

Here, each of f₁, f₂, . . . , f_Nis the data including (n×m) elements. Speaking exactly, therefore, a vector on the right side is a one-dimensional vector of (n×m×N) rows and one column. The vector g is expressed and calculated through conversion to a one-dimensional vector of (n×m) rows and one column. A matrix H represents conversion of coding and intensity-modulating each component f₁, f₂, . . . , f_Nof a vector fin accordance with coding information (hereinafter also referred to as “mask information”) per wavelength band and adding the modulated results. Thus, H is a matrix of (n×m) rows and (n×m×N) columns. In this Specification, the matrix H is referred to as a “system matrix”) in some cases.

It seems that, if the vector g and the matrix H are given, f can be calculated by solving an inverse problem of the equation (1). However, because the number (n×m×N) of elements of the data f to be acquired is greater than the number (n×m) of elements of the data g, that problem is an ill-posed problem and cannot be solved as it is. In consideration of the above point, the processing device 200 determines a solution with a compressed sensing method by utilizing redundancy of the image included in the data f. More specifically, the data f to be acquired is estimated by solving the following equation (2).

$\begin{matrix} f^{'} = \underset{f}{\arg \min} {{ ℊ - Hf }_{l_{2}} + τΦ (f)} & (2) \end{matrix}$

Here, f denotes estimated data off. The first term in the parenthesis of the above equation denotes a deviation between an estimated result Hf and the obtained data g, namely the so-called residual term. While the square sum is used here as the residual term, an absolute value or a square root of sum of squares, for example, may be used as the residual term. The second term in the parenthesis is a regularization term or a stabilization term. The equation (2) indicates that f minimizing the sum of the first term and the second term is to be determined. The processing device 200 can calculate the final solution f by converging the solution with recursive and iterative calculation.

The first term in the parenthesis of the equation (2) indicates calculation of determining the square sum of the difference between the obtained data g and Hf resulting from transforming fin an estimation process with the matrix H. Φ(f) in the second term represents a restriction condition in regularization of f and is a function reflecting sparse information of the estimated data. This function is effective in smoothing or stabilizing the estimated data. The regularization term may be expressed with, for example, discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) off. For instance, when the total variation is used, stable estimated data can be obtained while an influence of noise of the observed data g is suppressed. Sparseness of the object 70 in a space of each regularization term is different depending on the texture of the object 70. The regularization term making the texture of the object 70 more sparse in the space of the regularization term may be selected. Alternatively, two or more regularization terms may be included in the calculation. τ is a weight factor. As the weight factor τ increases, a reduction of redundant data increases and a compression rate increases. As the weight factor τ decreases, convergence to the solution weakens. The weight factor τ is set to an appropriate value at which f is converged to some extent without causing excessive compression.

With the configurations illustrated in FIGS. 1B and 1C, an image after being coded by the filter array 110 is obtained in a blurred state on the imaging surface of the image sensor 160. Accordingly, the spectral image 220 can be reconstructed by holding blur information in advance and by reflecting the blur information on the above-described system matrix H. Here, the blur information is expressed by a PSF (Point Spread Function). The PSF is a function specifying an extent of spread to peripheral pixels around a point image. For instance, when a point image corresponding to one pixel on the image spreads to an area of k×k pixels around the one pixel due to a blur, the PSF may be specified as a group, i.e., a matrix, of coefficients indicating influences on light intensities of the individual pixels in the area. The spectral image 220 can be reconstructed by reflecting the influences of the blur of a coding pattern, caused by the PSF, on the system matrix H. While a layout position of the filter array 110 can be selected as desired, a position at which the coding pattern of the filter array 110 does not disappear due to excessive dispersion may be selected.

With the above-described configuration, the hyperspectral data cubes representing two-dimensional images for all the wavelength bands W₁to W_Nincluded in the target wavelength range W are generated as illustrated in FIGS. 3A and 3B. Depending on the application, however, the images for all those wavelength bands are not required. In such a case, generating the images with high wavelength resolution for all the wavelength bands is inefficient.

FIG. 5 is a schematic view illustrating scenes where a hyperspectral camera is used. In one scene, a user desires to obtain color information in a red wavelength band, e.g., a wavelength band of longer than or equal to 600 nm and shorter than or equal to 700 nm, for the purpose of estimating a sugar content of an apple. In another scene, a user desires to obtain color information in a green wavelength band, e.g., a wavelength band of longer than or equal to 500 nm and shorter than or equal to 600 nm, for the purpose of finding an exact spectrum of a leaf. In still another scene, a user desires to obtain color information in a blue wavelength band, e.g., a wavelength band of longer than or equal to 400 nm and shorter than or equal to 500 nm, for the purpose of finding a color fading condition of a blue commodity.

To obtain color information in different wavelength ranges in the above-described scenes, one of the following methods (1) or (2) is used in the related-art hyperspectral camera. (1) Utilize an image sensor capable of independently obtaining information in each of the wavelength ranges.

(2) Obtain color information widely with a camera capable of obtaining information over a wide wavelength range and display only the information in a necessary wavelength range.

According to the method (1), an additional image sensor needs to be prepared in a different application in which color information different from that required in some application is required.

According to the method (2), color information in a wider wavelength range than a required wavelength range is obtained, and an image is generated for each of many wavelength bands included in the wide wavelength range. Therefore, a lot of calculation is executed for unnecessary wavelength ranges as well, and the calculation cost increases beyond an allowable level.

In consideration of the above point, the embodiment of the present disclosure is designed such that the user can designate, based on image data obtained by a hyperspectral imaging device, one or more sub-wavelength ranges each being part of the target wavelength range W. A signal processing device generates hyperspectral data cubes representing two-dimensional images for wavelength bands included in the one or more designated sub-wavelength ranges. As a result, detailed spectral information in the sub-wavelength range(s) demanded by the user can be obtained while the calculation cost is suppressed.

FIG. 6A illustrates an example of the target wavelength range W and a designated sub-wavelength range Wa. In this example, only one sub-wavelength range Wa is designated. The sub-wavelength range Wa is part of the target wavelength range W and includes wavelength bands W_a1, W_a2, . . . , W_ai. Here, “i” denotes the number of wavelength bands included in the sub-wavelength range Wa. The signal processing device generates the hyperspectral data cube representing the two-dimensional image for each of the wavelength bands W_a1, W_a2, . . . , W_ai.

FIG. 6B illustrates an example of the case of designating, in addition to a first sub-wavelength range W_a, a second sub-wavelength range W_b. The first sub-wavelength range W_aand the second sub-wavelength range W_bare away from each other and are each included in the target wavelength range W. The second sub-wavelength range W_bincludes wavelength bands W_b1, W_b2, . . . , W_bj. Here, “j” denotes the number of wavelength bands included in the second sub-wavelength range Wb. Thus, two or more sub-wavelength ranges may be designated.

As described above, the signal processing device generates, based on the image data generated by the imaging device, the images of the wavelength bands included in the designated sub-wavelength range. The generated images of the wavelength bands are displayed on a display. That operation can reduce the calculation cost that is taken to generate unnecessary spectral images in the application scene of interest.

Embodiments of the Present Disclosure are Summarized as Follows

A signal processing method according to one embodiment of the present disclosure is executed by a computer. The method includes obtaining compressed image data including two-dimensional image information that is obtained by compressing hyperspectral information corresponding to wavelength bands included in a target wavelength range, obtaining setting data including information designating one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the compressed image data, two-dimensional images corresponding to wavelength bands included in the one or more sub-wavelength ranges.

The wording “hyperspectral information in a target wavelength range” indicates information of images corresponding to wavelength bands included in a predetermined target wavelength range. The wording “compressing hyperspectral information” includes not only compressing pieces of image information in wavelength bands into one monochrome two-dimensional image by using a coding element such as the above-mentioned the filter array 110, but also compressing pieces of image information in wavelength bands into one monochrome two-dimensional image through a software process.

According to the above-described method, data of two-dimensional images corresponding to the wavelength bands, namely hyperspectral data cubes, included in the one or more sub-wavelength ranges designated by a user, for example, can be generated. Therefore, the above-described method can generate only necessary hyperspectral data cubes depending on the application or the purpose.

The wavelength bands included in the target wavelength rang may be four or more wavelength bands included in the target wavelength range, and the two-dimensional image information may be data items of pixels included in the compressed image data. Information items corresponding to four or more wavelength bands may be superimposed in each of the data items of the pixels. In other words, each of the data items of the pixels of the compressed image data may include one value in which the pieces of information in the four or more wavelength bands included in the target wavelength range are superimposed. Pieces of information in wavelength bands of more than or equal to 10 or less than or equal to 100 may be superimposed in the data for each of the pixels of the compressed image data depending on the application.

The setting data may include information designating one or more wavelength resolutions corresponding to the one or more sub-wavelength ranges. The two-dimensional images may be generated with the one or more designated wavelength resolutions. With the feature described above, the user can designate, in addition to the sub-wavelength range, the wavelength resolution for each sub-wavelength range. Therefore, the user can make flexible adjustment such as increasing the wavelength resolution in the sub-wavelength range where detailed spectral information is required.

The one or more sub-wavelength ranges may include a first sub-wavelength range and a second sub-wavelength range. The two-dimensional images may include two-dimensional images corresponding to the first sub-wavelength range and two-dimensional images corresponding to the second sub-wavelength range.

The one or more sub-wavelength ranges may include a first sub-wavelength range and a second sub-wavelength range. The one or more wavelength resolutions may include a wavelength resolution corresponding to the first sub-wavelength range and a wavelength resolution corresponding to the second sub-wavelength range, the wavelength resolution corresponding to the first sub-wavelength range and the wavelength resolution corresponding to the second sub-wavelength range being determined independently. The two-dimensional images may include two-dimensional images corresponding to the first sub-wavelength range and generated with the wavelength resolution corresponding to the first sub-wavelength range, and two-dimensional images corresponding to the second sub-wavelength range and generated with the wavelength resolution corresponding to the second sub-wavelength range.

The first sub-wavelength range and the second sub-wavelength range may be away from each other. Alternatively, the first sub-wavelength range and the second sub-wavelength range may be adjacent to each other or may partially overlap each other.

The method may further include displaying, on a display connected to the computer, a graphical user interface (GUI) on which the user inputs the setting data. On the GUI displayed, the user can easily perform operations of designating the sub-wavelength range and designating the wavelength resolution for each sub-wavelength range.

The method may further include displaying the two-dimensional images on a display connected to the computer. This feature enables the user to easily check the spectral image generated for each of the wavelength bands.

The compressed image data may be generated by taking an image with an image sensor and a filter array including optical filters different in spectral transmittance and an image sensor. The method may further include obtaining mask data reflecting a spatial distribution of the spectral transmittance of the filter array. The two-dimensional images may be generated based on the compressed image data and the mask data.

The method may further include obtaining mask data including information of mask images that are obtained by taking images of backgrounds with the image sensor through the filter array, the backgrounds corresponding to unit bands included in the target wavelength range. The two-dimensional images may be generated based on the compressed image data and the mask data.

The mask data may be data specifying, for example, the matrix H in the above-described equation (2). The form of the mask data may be different depending on the configuration of the imaging system. The mask data may represent the spatial distribution of the spectral transmittance of the filter array or information to calculate the spatial distribution of the spectral transmittance of the filter array. For instance, the mask data may include information of the background image for each unit band in addition to the above-described information of the mask images. Information of the transmittance distribution can be obtained for each unit band by dividing the mask image by the background image per pixel. The mask data may include only the information of the mask images. The mask images represent a distribution of values resulting from multiplying the transmittances of the filter array by sensitivity of the image sensor. In a configuration in which the filter array and the image sensor are arranged to closely face each other, that type of mask data may be used.

The mask data may include mask information. The mask information may represent a spatial distribution of transmittance of the filter array corresponding to each of unit bands included in the target wavelength range. The method may further include generating combined mask information by combining parts of the mask information, the parts corresponding to unit bands included in a not-designated wavelength range in the target wavelength range other than the one or more sub-wavelength ranges, and generating a combined image corresponding to the not-designated wavelength range based on both the compressed image data and the combined mask information.

The method may further include generating a combined mask image by combining the mask images for the unit bands included in the not-designated wavelength range in the target wavelength range other than the one or more designated sub-wavelength ranges, and generating combined image data for the not-designated wavelength range based on both the compressed image data and the combined mask image.

According to the above-described method, a detailed spectral image is not generated for the not-designated wavelength range, and the detailed spectral image is generated only for the designated sub-wavelength range. Therefore, the calculation time necessary for generating the spectral images can be shortened.

The mask data may further include information of background images that are obtained by taking images of backgrounds with the image sensor without interposition of the filter array. The method may further include generating a combined background image resulting from combining the background images. The combined image may be generated based on the compressed image data, the combined mask image, and the combined background image.

The mask data may include background images and mask images. The background images may be obtained, for example, by capturing backgrounds with the image sensor without interposition of the filter array, the background images corresponding one-to-one to the backgrounds. The mask images may be obtained by capturing the backgrounds with the image sensor through the filter array, the mask images corresponding one-to-one to the backgrounds. The combined mask information may be generated based on the mask images and the background images.

The method may further include displaying the combined image on a display connected to the computer. This feature enables the user to easily check a rough image for the not-designated wavelength range.

The mask data may include mask information. The mask information may represent a spatial distribution of transmittance of the filter array corresponding to each of unit bands included in the target wavelength range. The setting data may include information designating large sub-wavelength ranges each being part of the target wavelength range and small sub-wavelength ranges included in one of the large sub-wavelength ranges. The method may further include generating, for each of the large sub-wavelength ranges, first combined mask information by combining parts of the mask information, the parts corresponding to unit bands included in each of the large sub-wavelength ranges, and generating a first combined image for each of the large sub-wavelength ranges based on both the compressed image data and the first combined mask information. The two-dimensional images may correspond to the small sub-wavelength ranges and are based on the first combined image. The method may further include generating, for each of the small sub-wavelength ranges, second combined mask information resulting from combining pieces of the mask information for the unit bands included in the small sub-wavelength range, and generating, for each of the small sub-wavelength ranges, a second combined image based on both the first combined image for the at least one of the designated large sub-wavelength ranges and the second combined mask information.

According to the above-described method, it is possible to generate the combined image for each of the large sub-wavelength ranges included in the target wavelength range and for each of the small sub-wavelength ranges included in at least one of the large sub-wavelength ranges, those large and small sub-wavelength ranges being designated by the user, for example. Therefore, the user can make flexible adjustment such as setting the small sub-wavelength ranges only for the wavelength range where detailed color information is necessary and setting only the large sub-wavelength ranges for the wavelength range where detailed color information is not necessary.

The target wavelength range may include a visible wavelength range. The method may further include generating, based on the compressed image data and the combined mask information, an image corresponding to a red wavelength range, an image corresponding to a green wavelength range, and an image corresponding to a blue wavelength range, and displaying, on a display connected to the computer, an RGB image based on the image corresponding to the red wavelength range, the image corresponding to the green wavelength range, and the image corresponding to the blue wavelength range. This feature enables the user to check the RGB image of an object in addition to the detailed spectral image for the designated sub-wavelength range.

A method according to another embodiment of the present disclosure is a method of generating mask data. The mask data is used to generate spectral image data for each wavelength band from compressed image data obtained by an imaging device that includes a filter array including optical filters different in spectral transmittance. In other words, the method is to generate the mask data, which is used to generate the spectral image data for each wavelength band, from the compressed image data obtained by the imaging device that includes the filter array including plural types of optical filters different in spectral transmittance. The method includes obtaining first mask data to generate first spectral image data corresponding to a first wavelength band group in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the first mask data and the setting data, second mask data to generate second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges.

The first wavelength band group may be a set of all or parts of the wavelength bands included in the target wavelength range. The second wavelength band group may be a set of all or parts of the wavelength bands included in the sub-wavelength range. The first wavelength band group and the second wavelength band group may be each an assembly of combined bands in each of which two or more unit wavelength bands are combined. When such band combination is performed, a conversion process of the mask data is executed depending on the form of the band combination. The setting data may include information regarding the form of the band combination, the information being used to execute the conversion process of the mask data.

The first mask data may include first mask information reflecting a spatial distribution of spectral transmittance of the filter array corresponding to the first wavelength band group. The second mask data may include second mask information reflecting a spatial distribution of spectral transmittance of the filter array corresponding to the second wavelength band group.

The second mask data may further include third mask information obtained by combining pieces of information. The pieces of the information each reflect the spatial distribution of the spectral transmittance in a corresponding wavelength band that is included in a not-designated wavelength range in the target wavelength range other than the one or more sub-wavelength ranges.

A signal processing device according to still another embodiment of the present disclosure includes a processor and a memory storing a computer program executed by the processor. The computer program causes the processor to execute obtaining compressed image data including two-dimensional image information that is obtained by compressing hyperspectral information in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the compressed image data, two-dimensional images corresponding to wavelength bands included in the one or more sub-wavelength ranges.

A signal processing device according to still another embodiment of the present disclosure includes a processor and a memory storing a computer program executed by the processor. The computer program causes the processor to execute obtaining first mask data to generate first spectral image data corresponding to a first wavelength band group in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the first mask data and the setting data, second mask data to generate second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges.

An imaging device according to still another embodiment of the present disclosure includes the above-described signal processing device and an imaging device generating the compressed image data.

A computer program according to still another embodiment of the present disclosure includes causing a computer to execute obtaining compressed image data including two-dimensional image information that is obtained by compressing hyperspectral information corresponding to wavelength bands included in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the compressed image data, two-dimensional images corresponding to wavelength bands included in the one or more sub-wavelength ranges.

A computer program according to still another embodiment of the present disclosure includes causing a computer to execute obtaining first mask data to generate first spectral image data corresponding to a first wavelength band group in a target wavelength range from compressed image data obtained by an imaging device that includes a filter array including plural types of optical filters different in spectral transmittance, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the first mask data and the setting data, second mask data to generate second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges.

A non-transitory computer-readable recording medium according to still another embodiment of the present disclosure stores a program causing a computer to execute processes including obtaining compressed image data including two-dimensional image information that is obtained by compressing hyperspectral information corresponding to wavelength included in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the compressed image data, two-dimensional images corresponding to wavelength bands included in the one or more sub-wavelength ranges.

A non-transitory computer-readable recording medium according to still another embodiment of the present disclosure stores a program causing a computer to execute processes including obtaining first mask data to generate first spectral image data corresponding to a first wavelength band group in a target wavelength range, obtaining setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range, and generating, based on the first mask data and the setting data, second mask data to generate second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges.

More specific embodiments of the present disclosure will be described below. However, detailed description more than a necessary level is omitted in some cases. For instance, detailed description of the already well-known matters and duplicate description of substantially the same components are omitted in some cases. This is intended to avoid the following description from becoming excessively redundant and to make those skilled in the art more easily understand the present disclosure. The inventors provide the attached drawings and the following description for allowing those skilled in the art to sufficiently understand the present disclosure, and the attached drawings and the following description are not purported to restrict the main subjects stated in Claims. In the following description, the same or similar constituent elements are denoted by the same reference signs. The following description is made using xyz-coordinates denoted in the drawings.

EMBODIMENT

FIG. 7 illustrates a configuration of an imaging system according to an exemplary embodiment of the present disclosure. The system includes an imaging device 100, a processing device 200, a display device 300, and an input user interface (UI) 400. The processing device 200 corresponds to the signal processing device according to the present disclosure.

The imaging device 100 includes an image sensor 160 and a control circuit 150 controlling the image sensor 160. Although not illustrated in FIG. 7, the imaging device 100 further includes a filter array 110 and at least one optical system 140 as illustrated in FIGS. 1A to 1D. The filter array 110 and the optical system 140 may be arranged as per illustrated in any of FIGS. 1A to 1D. The image sensor 160 obtains a monochrome image based on light of which intensity is modulated by the filter array 110 for each region. Pieces of information in the wavelength bands included in the target wavelength range W are superimposed in data of each pixel of the monochrome image. Accordingly, the monochrome image can be said as being an image resulting from compression of hyperspectral information in the target wavelength range W into a two-dimensional image. That monochrome image is an example of a “compressed image” in this Specification. Furthermore, in this Specification, the data representing the compressed image is referred to as “compressed image data”.

The processing device 200 includes a signal processing circuit 250 and a memory 210 such as a RAM or a ROM. The signal processing circuit 250 may be an integrated circuit including a processor such as a CPU or a GPU. The signal processing circuit 250 executes a reconstruction process based on the compressed image data that is output from the image sensor 160. This reconstruction process is basically the same as that executed by the processing device 200 illustrated in FIGS. 1A to 1D, but in this embodiment, the reconstruction process is executed in accordance with reconstruction conditions input from the input UI 400. The signal processing circuit 250 generates the image data with high wavelength resolution only for the designated sub-wavelength range in the target wavelength range. Accordingly, the calculation time can be shortened. The memory 210 stores computer programs executed by the processor included in the signal processing circuit 250, various data referred to by the signal processing circuit 250, and various data generated by the signal processing circuit 250.

The display device 300 includes a memory 310, an image processing circuit 320, and a display 330. The memory 310 temporarily stores setting data representing the reconstruction conditions that are sent from the input UI 400. The image processing circuit 320 causes an image reconstructed by the signal processing circuit 250 to be displayed on the display 330 after executing necessary processes on the reconstructed image. The display 330 may be any suitable display such as a liquid-crystal or organic LED display, for example.

The input UI 400 includes hardware or software for setting various conditions such as imaging conditions and reconstruction conditions. The imaging conditions may include, for example, an image resolution, a gain, and an exposure time. The reconstruction conditions may include, for example, a lower limit wavelength and an upper limit wavelength of each sub-wavelength range, the number of wavelength bands included in each sub-wavelength range, and the number of calculations. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 controls the image sensor 160 to take an image in accordance with the imaging conditions. Thus, the image sensor 160 generates the compressed image in which pieces of information in the wavelength bands included in the target wavelength range W are superimposed. The input reconstruction conditions are further sent to the signal processing circuit 250 and to the memory 310 to be recorded there. The signal processing circuit 250 executes the reconstruction process in accordance with the set reconstruction conditions and generates the hyperspectral data cubes for the designated sub-wavelength range. The image processing circuit 320 causes the display 330 to display the image for each of the wavelength bands included in the designated sub-wavelength range in accordance with the set reconstruction conditions.

In the reconstruction process, the signal processing circuit 250 uses mask data previously recorded on the memory 210 after converting the mask data as required in accordance with the input reconstruction conditions that are entered from the input UI 400. The mask data is data representing the spatial distribution of the spectral transmittance of the filter array 110 and includes information corresponding to the matrix H in the above-described equation (2). The generated spectral image is processed as required by the image processing circuit 320. The image processing circuit 320 causes the spectral image to be displayed on the display 330 after executing processes of, for example, determining a layout of the image in a screen, setting linkage with band information, and coloring the image corresponding to wavelengths.

In this embodiment, the signal processing circuit 250 generates the image for each of the wavelength bands only for at least one designated sub-wavelength range in the target wavelength range W. For the wavelength ranges in the target wavelength range W other than the designated sub-wavelength range, calculation is executed by regarding the continuous wavelength ranges as one collective wavelength range. Thus, the calculation cost can be suppressed. The signal processing circuit 250 may generate the image for each of the wavelength bands over the entirety of the target wavelength range W. In such a case, the image processing circuit 320 may extract data for the designated sub-wavelength range from the image data input from the signal processing circuit 250 and may display the extracted data.

FIG. 8 is a flowchart illustrating operation of the system according to this embodiment. In this embodiment, first, the user inputs, in step S101, the imaging conditions and the reconstruction conditions through the input UI 400 (step S101). Data representing the input imaging conditions is sent to the control circuit 150. Data representing the input reconstruction conditions is sent to the signal processing circuit 250 and the memory 310. The memory 310 temporarily stores the reconstruction conditions. When the image is displayed, the stored reconstruction conditions are referred to confirm the linkage between the image and the set condition for the wavelength band. Then, the imaging device 100 takes an object image in accordance with the imaging conditions and obtains a compressed image (step S102).

After the compressed image is obtained, the signal processing circuit 250 determines, based on the input reconstruction conditions, whether conversion of the mask data is required (S103). If the conversion is necessary, the signal processing circuit 250 converts the mask data that is previously stored in the memory 210 (step S104). Here, the word “conversion” indicates that pieces of the mask information for the wavelength ranges are combined and are handled as the mask information for one wavelength range. Details of combination of the mask information will be described later with reference to FIG. 14. If the conversion is not necessary, step S104 is omitted. The signal processing circuit 250 executes the reconstruction process in accordance with the input reconstruction conditions by using the compressed image and the mask data after being converted as required (step S105). Thus, a spectral image is generated from the compressed image. Then, the image processing circuit 320 of the display device 300 links the generated spectral image with the reconstruction conditions stored in the memory 310 for labeling (step S106). For instance, the image processing circuit 320 generates image data in which a label indicating a corresponding wavelength range is added to each of the generated spectral images. Then, the image processing circuit 320 outputs the generated image data to the display 330 to be displayed thereon (step S107).

FIG. 9 illustrates an example of the mask data before the conversion, the mask data being stored in the memory 210. The mask data in the illustrated example includes mask information representing the spatial distribution of the transmittance for each of unit bands included in the target wavelength range. The mask data in the illustrated example includes the mask information for each of many unit bands divided per 1 nm and information regarding conditions in obtaining the mask information. Each of the unit bands is specified by a lower limit wavelength and an upper limit wavelength. The mask information includes a mask image and a background image. The mask images illustrated in FIG. 9 are obtained by taking images of backgrounds corresponding respectively to the unit bands with the image sensor 160 through the filter array 110. The background images are obtained by taking images of the backgrounds with the image sensor 160 without interposition of the filter array 110. Thus-obtained data of the mask images and the background images are recorded in advance for each unit band. The information regarding the conditions in obtaining the mask information includes information about an exposure time and a gain. In the example illustrated in FIG. 9, the data of the mask images and the background images are recorded for each of the unit bands with a width of 1 nm. The width of each unit band is not limited to 1 nm and may be determined as any desired value. When the background image has very high uniformity, the mask data does not need to include the information of the background image. For instance, in a configuration in which the image sensor 160 and the filter array 110 are integrated to closely face each other, since the mask information substantially matches the mask image, the mask data does not need to include the background image.

Examples of a graphical user interface (GUI) displayed in accordance with a program executing the above-described information processing will be described below with reference to FIGS. 10 to 13. Images providing those examples of the GUI are generated by the signal processing circuit 250 and the image processing circuit 320 and are displayed on the display 330.

FIG. 10 illustrates an example of a GUI screen on which the imaging conditions are to be input. In the illustrated example, prior to taking a hyperspectral image, the user sets the image resolution, the gain, the exposure time, and the frame rate. The image resolution indicates the numbers of pixels in vertical and horizontal directions of a displayed image. The image resolution may be designated, for example, by the user selecting one of names, such as VGA, HD, and 4K, from a pulldown menu or directly inputting the numbers of pixels in the vertical and horizontal directions. The gain is designated as a rational number of greater than or equal to 0 and may be input as a value resulting from addition, subtraction, multiplication, or division of rational numbers. For instance, when 8/3 is input, the gain is set as 2.6666 . . . dB. As to the exposure time and the frame rate, it is not necessary to input both of them. The user may input at least one of the exposure time or the frame rate, and when competition occurs (e.g., when the exposure time of 100 msec and the frame rate of 30 fps are input as conditions), priority may be given to one of them. The function of automatically adjusting the gain, the exposure time, and the frame rate may be prepared in addition to input setting for the above-described four conditions. For instance, average light intensity may be automatically adjusted to ½ of maximum light intensity. As seen from the example illustrated in FIG. 10, the GUI for inputting the imaging conditions may have the functions of saving and loading the set imaging conditions. The GUI may further have the function of displaying in real time the compressed image that is obtained under the set imaging conditions. Here, it is not always required to display the compressed image itself. Any desired image obtained under the set imaging conditions at that time may be displayed. In an example, pixels outputting only values of red (R), green (G), and blue (B) may be arranged, and an obtained RGB image may be displayed using only the values of those pixels. In another example, reconstruction in three bands including a first band of longer than or equal to 400 nm and shorter than or equal to 500 nm, a second band of longer than or equal to 500 nm and shorter than or equal to 600 nm, and a third band of longer than or equal to 600 nm and shorter than or equal to 700 nm may be executed with a unit-band combination process described later, and a reconstruction result may be displayed as an RGB image.

FIGS. 11 and 12 each illustrate an example of a GUI on which the reconstruction conditions are to be input. In the example illustrated in FIG. 11, the user inputs the sub-wavelength range, the wavelength resolution or the number of band divisions, and the number of calculations. Here, the number of calculations indicates the number of iterations of the reconstruction process expressed by the equation (2). As illustrated in FIG. 11, the sub-wavelength range may be designated by setting the lower limit wavelength and the upper limit wavelength with drag and drop. In the example illustrated in FIG. 11, the sub-wavelength range of longer than or equal to 420 nm and shorter than or equal to 480 nm and the sub-wavelength range of longer than or equal to 600 nm and shorter than or equal to 690 nm are designated. Instead of using drag and drop for the designation, as illustrated in FIG. 12, a span of one or more sub-wavelength ranges and a span of each wavelength band in each sub-wavelength range may be set by inputting numerical values. An area where the span of the one or more sub-wavelength ranges and the span of each wavelength band in each sub-wavelength range are input may be displayed as an independent window or may be arranged in a screen on which other setting items are input.

In the example illustrated in FIG. 11, the user inputs either one of the wavelength resolution and the number of band divisions. The number of calculations is designated as an arbitrary integer of greater than or equal to 1. Typically, a value greater than or equal to about 10 and smaller than or equal to 10000 may be designated as the number of calculations. In the example illustrated in FIG. 11, an estimated calculation time is also displayed. The estimated calculation time is not input by the user, but is automatically calculated from the resolution, the number of band divisions, and the number of calculations, which have been set, and is then displayed. The functions of inputting the number of calculations and displaying the estimated calculation time may be omitted. As an alternative, several modes, such as a high-precision mode (low speed), a balance mode (medium speed), and a high-speed mode (high speed), may be prepared, and one of those modes may be selected from a pull-down menu, for example. As illustrated in FIG. 11, the GUI may have the functions of saving and loading the set reconstruction conditions.

FIG. 13 illustrates an example of a screen displaying spectral images that are generated as a result of executing reconstruction calculation. The generated spectral images are linked with the set reconstruction conditions and are displayed in the form allowing each of the spectral images to be discriminated per set band. As illustrated in FIG. 13, by way of example, in addition to the reconstructed image in each band, the lower limit wavelength and the upper limit wavelength of the band may be displayed as numerical values. Alternatively, each band may be indicated as a number counted from a short wavelength side or a long wavelength side. The image in each band may be displayed in color belonging to the band. Physical values expressed in terms of wavelength (nm) in the above-described examples may be all expressed in terms of wave number (e.g., cm⁻¹) or frequency (e.g., Hz).

FIG. 14 is an explanatory view illustrating an example of a method of combining pieces of mask information for bands and converting those pieces to new mask information. In the illustrated example, mask information for unit bands #1 to 20 is previously stored as mask information before the conversion in the memory 210 as illustrated in FIG. 9. In the example illustrated in FIG. 14, the combination process is not executed for the unit bands #1 to 5 and is executed for the unit bands #6 to 20. For the unit bands #1 to 5, the transmittance distribution of the filter array 110 is calculated by dividing a value in each area of the mask image by a value in a corresponding area of the background image. Here, data of each mask image stored in the memory 210 is referred to as “unit mask image data”, and data of each background image stored therein is referred to as “unit background image data”. For the bands #6 to 20, a combined transmittance distribution is obtained by dividing data resulting from adding the unit mask image data for the bands #6 to 20 per pixel by data resulting from adding the unit background image data for the bands #6 to 20 per pixel. By executing the above-described operation, pieces of the mask information can be combined for any desired bands. When the background image has very high uniformity, the mask information substantially matches the mask image. In such a case, the data resulting from adding the mask image data for the bands #6 to 20, or data resulting from averaging the added data may be used as combined mask data for the bands #6 to 20.

In the example illustrated in FIG. 9, the mask information is recorded for each of many unit bands with the width of 1 nm. On other hand, in the example illustrated in FIG. 12, each of the wavelength bands designated by the user has a relatively wide width of 30 nm. In the latter case, the signal processing circuit 250 executes the reconstruction by combining pieces of the mask information for the unit bands for each of the designated wavelength bands.

A process of converting the mask data through combination may be executed in an environment where the end user uses the signal processing device, or may be executed in a production site, for example, a system production factory. When the process of converting the mask data is executed in the production site, the mask data after the conversion is previously stored in the memory 210 instead of or in addition to the mask data before the conversion.

As described above, the mask data is used to generate the spectral image data for each wavelength band from the compressed image data obtained by the imaging device that includes the filter array including the plural types of optical filters with the spectral transmittances different from one another. A method of converting the mask data in this embodiment includes steps described below.

- Obtain first mask data to generate first spectral image data corresponding to a first wavelength band group in the target wavelength range.
- Obtain setting data to designate one or more sub-wavelength ranges that are parts of the target wavelength range.
- Generate, based on the first mask data and the setting data, second mask data to generate second spectral image data corresponding to a second wavelength band group in the sub-wavelength range.

The first mask data may be, for example, data to reconstruct, from the compressed image, the spectral image for each of all unit bands included in the target wavelength range. The second mask data may be, for example, data to reconstruct, from the compressed image, the spectral image for each of all unit bands included in each designated sub-wavelength range. The second mask data may be data to reconstruct, from the compressed image, the spectral image for each combined band resulting from combining the unit bands. When the above-described combination is executed, the setting data may include data with respect to a scheme of the band combination. In the following description, the combined band resulting from combining the unit bands is also referred to as a “post-edit band”).

The first mask data and the second mask data are each data reflecting the spatial distribution of the spectral transmittance of the filter array. The first mask data includes first mask information reflecting the spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second mask data includes second mask information reflecting the spatial distribution of the spectral transmittance corresponding to the second wavelength band group.