For example, in a case where the property score for “presence or absence of a spicula” which is one of the property items of the abnormal shadow is equal to or greater than a predetermined threshold value (for example, 0.5), the property for “presence or absence of a spicula” of the abnormal shadow specifies “presence of a spicula (positive)”, and in a case where the property score for “presence or absence of a spicula” is less than the threshold value, the property for the presence or absence of a spicula of the abnormal shadow specifies “absence of a spicula (negative)”. The threshold value of 0.5 used for the property determination is merely an example, and is set to an appropriate value for each property item. In a case where the property score is near the threshold value (for example, 0.4 or more and 0.6 or less), a false positive may be specified.

FIG. 4 is a diagram showing an example of an analysis result derived by the analysis unit 22. As shown in FIG. 4, the property information specified by the analysis unit 22 includes the property items of a location of the abnormal shadow, a type of an absorption value, a spicula, calcification, a cavity, and a pleural invagination, and the properties for the property items are “upper left section”, “solid type”, “presence of a spicula”, “presence of calcification”, “presence of a cavity”, and “absence of a pleural invagination”. In FIG. 4, “+” is given in a case of “presence”, that is, positive, and “−” is given in a case of “absence”, that is, negative.

As the learning model 22A, in addition to the convolutional neural network, for example, any learning model such as a support vector machine (SVM) can be used.

Further, a learning model for detecting an abnormal shadow from the medical image G0 and a learning model for discriminating a property of the abnormal shadow may be separately constructed.

The sentence generation unit 23 generates a medical sentence related to the patient based on the analysis result derived by the analysis unit 22 and the diagnostic information acquired by the information acquisition unit 21. In the present embodiment, the sentence generation unit 23 consists of a learning model 23A constructed by machine learning so as to generate a finding sentence to be described in an interpretation report as a medical sentence from input information. As the learning model 23A, for example, a neural network such as a recurrent neural network described in U.S. Pat. Nos. 10,181,098B or 10,268,671B can be used. In the present embodiment, the learning model 23A is constructed by training the recurrent neural network by supervised learning. The training data used in this case is data in which combinations of various analysis results and various types of diagnostic information are associated with various training sentences to be generated from the analysis results and the diagnostic information.

FIG. 5 is a diagram schematically showing a recurrent neural network. As shown in FIG. 5, the recurrent neural network 40 consists of an encoder 41 and a decoder 42. Labels of the analysis result derived by the analysis unit 22 and the diagnostic information are input to the encoder 41. The 1-hot expressions of the labels of the analysis result and the diagnostic information are input to the encoder 41. In the 1-hot expression, each label is represented by a vector having one component of 1 and the remaining components of 0. For example, in a case where the 1-hot expression is a vector consisting of three elements, (1, 0, 0), (0, 1, 0), and (0, 0, 1) respectively represent three different labels.

The encoder 41 transforms the 1-hot expression of each label by an embedding matrix to derive a vector representation of each label. Each element of the embedding matrix is a learning parameter. The learning parameter is determined by machine learning of the recurrent neural network 40.

The decoder 42 is configured by connecting a plurality of networks consisting of an input layer, an intermediate layer, and an output layer. In each network, a vector representation xt output from the encoder 41 and an output ht of a network in a previous stage are input. In the intermediate layer, calculation shown in Expression (1) below is performed. In Expression (1), Wh, Wx, and b are learning parameters and are determined by learning. tanh is an activation function. The activation function is not limited to tanh, and a sigmoid function or the like may be used.

ht=tanh(ht−1·Wh+xt·Wx+b) (1)

Here, it is assumed that the analysis result of the training data and the diagnostic information are “under left lung pleura”, “4.2 cm”, “spicula+”, and “mass”. Since “under left lung pleura” is a term representing a location in the lung, a label representing the location is given. Since “4.2 cm” is a size of a diameter, a label indicating the size is given. “spicula+” is given a label indicating that the spicula is positive, and “mass” is given a label indicating a medically small mass. These labels are input to the encoder 41, and a vector representation of each label is output.

In the decoder 42, the output of the previous stage and the vector representation are input to the input layer of each neural network, and a finding sentence “[small mass] with [size] diameter having [spicula] is recognized at [location]” is output. The sentence generation unit 23 embeds information on the analysis result and the diagnostic information into the label included in the finding sentence output from the learning model 23A, to generate a sentence “a mass having a diameter of 4.2 cm with a spicula is recognized under the left lung pleura”.

Here, the diagnostic information used in the present embodiment will be described. The diagnostic information is information relating to diagnosis of the patient other than the analysis result derived by the analysis unit 22. Specifically, examples of the diagnostic information include information (referred to as first information D1) confirmed regarding a lesion included in the medical image, confirmed information (referred to as second information D2) other than the information related to the lesion included in the medical image, information (referred to as third information D3) representing a determination result for the medical image by a radiologist, and information (referred to as fourth information D4) representing an examination result performed on the patient.

The first information D1 confirmed for the lesion included in the medical image G0 includes, for example, measurement information such as the size (vertical and horizontal lengths or an area) of the lesion included in the medical image G0, a definitive diagnosis result for the lesion, a history of the patient from whom the medical image G0 has been acquired, and contents of treatment performed on the patient. Regarding the size of the lesion, information representing a change over time from the size of the lesion included in the medical image acquired in the past for the same patient can be used as the first information D1. The information representing the change is information representing whether the size of the lesion is increasing, decreasing, or not changing. The definitive diagnosis result regarding the lesion is a diagnosis result confirmed by a doctor, such as whether the lesion is cancer or a benign tumor. The history of the patient is a history of diseases suffered in the past by the patient from whom the medical image G0 has been acquired. Examples of the contents of treatment performed on the patient include contents of surgery performed on the patient, a type of medicine used, an amount of medicine, a medication period, and the like.

Regarding the generation of the finding sentence using the first information D1, the diagnostic information may be directly included in the finding sentence (for example, there is a history of primary lung cancer or colorectal cancer with a major axis of 10 mm). In this case, the learning model 23A may be constructed by training the neural network using training data that includes labels of a size, definitive diagnosis of malignity, and a history as the first information D1 in the diagnostic information and includes a training sentence in which the labels are described.

In addition, for example, since it is difficult to evaluate an internal property other than calcification in a nodule having a major axis less than 10 mm, a finding sentence may be generated so as not to include internal negative findings (an air bronchogram, a cavity, fat, and the like) as the analysis result. In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using training data that includes a label of, for example, the size as the first information D1 in the diagnostic information, includes labels of “nodule, air bronchogram−, and cavity−” in the analysis result, and includes a training sentence that does not include descriptions of the labels of the air bronchogram and the cavity among the labels.

Further, in a case where primary lung cancer is confirmed as the diagnosis result, a change (increase or decrease) in size over time is more important than pictorial properties (presence of a spicula or the like). Therefore, in the finding sentence, only a change in size over time may be described without describing an analysis result representing properties, or an analysis result inconsistent with the definitive diagnosis (for example, a finding suggesting benignity in a case where there is a definitive diagnosis of primary lung cancer) may not be described. In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using training data that includes a label of definitive diagnosis of malignity such as primary lung cancer and a label of “increase” as the first information D1 in the diagnostic information, includes a label of a positive property item as the analysis result, and includes a training sentence that includes a description of the label of increase of the definitive diagnosis of malignity but does not include a description of the label of a positive property item among the labels.

The confirmed second information D2 other than the information related to the lesion included in the medical image G0 is information not related to the lesion regarding the medical image G0, and specific examples thereof include an examination purpose for acquiring the medical image G0 and an image condition in a case of acquiring the medical image G0. The acquisition of the medical image G0 is performed as a part of an examination for determining a state of a patient, and has various examination purposes such as a thorough examination or a follow-up observation. In a case where the medical image G0 is acquired, various conditions (a window level, a window width, and a slice interval) are prepared in order to make it easy to see a target organ in various modalities. The image condition is a condition in a case of generating the medical image G0 such as the window level, the window width, and the slice interval.

Regarding the generation of the finding sentence using the second information D2, the content to be described in the finding sentence varies depending on whether the examination purpose is the thorough examination or the follow-up observation. For example, the finding sentence may be generated such that detailed contents are described in the finding sentence in a case of the thorough examination, and a change (increase or decrease) in size over time is described in a case of the follow-up observation. In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using, for example, training data that includes a label indicating that the examination purpose is a thorough examination as the second information D2 in the diagnostic information and includes a training sentence in which labels of all property items included in the analysis result are described, or training data that includes a label indicating that the examination purpose is a follow-up observation as the second information D2 in the diagnostic information and includes a training sentence in which only a label of a change in size is described among property labels included in the analysis result.

In addition, in a case where a slice thickness of a CT image is 5 mm, pixel values are averaged due to a partial volume effect to decrease a contrast, which makes it difficult to evaluate internal properties and edge properties. In such a case, in a case where a finding sentence with an assertive ending is generated, a creator feels uncomfortable with the finding sentence. For this reason, a finding sentence may be generated such that an analysis result which is difficult to evaluate may or may not be described according to the image condition, or described in such a way as to decrease or increase a certainty factor (change the ending from “recognized” to “suspected”), or described in a blurred or asserted expression (a frosted glass nodule is set as a light nodule, and a nodule is set as a density increased region). In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using training data that includes a label indicating that a slice thickness of a CT image is 5 mm as the second information D2 in the diagnostic information, includes a label of a positive property item among labels of property items included in the analysis result, and includes a training sentence in which “~ is suspected” rather than “~ is recognized” is described at the ending of the label of the positive property item.

The third information D3 representing the determination result for the medical image by the radiologist is information representing an interpretation result for the lesion included in the medical image G0 by the radiologist who interprets the medical image G0. Specifically, examples thereof include an unconfirmed diagnosis result for the medical image G0, relevance between a lesion and a tissue other than the lesion, and a selection result of the radiologist for a plurality of analysis results.

Regarding the generation of the finding sentence using the third information D3, in a case where the third information D3 is the unconfirmed diagnosis result or the relevance between a lesion and a tissue other than the lesion, a finding sentence may be generated such that the third information D3 is directly included in the finding sentence. For example, a finding sentence with such a description that primary lung cancer is suspected, mediastinal invasion is suspected, or intrapulmonary metastasis is suspected may be generated. In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using training data that includes, for example, a label indicating that “mediastinal invasion is suspected” as the third information D3 in the diagnostic information and includes a training sentence in which the label indicating that “mediastinal invasion is suspected” is described.

In addition, regarding the selection result of the radiologist for a plurality of analysis results, a finding sentence including only the property item selected by the radiologist may be generated. The property item not selected by the radiologist may not be described in the finding sentence, but the finding sentence may be generated in such a way as to lower a certainty factor, lower importance, or blur the description.

Examples of the fourth information 4 representing the examination result performed on the patient include an interpretation result for a medical image acquired by an imaging apparatus different from the imaging apparatus that acquires the medical image G0 and an examination result other than the examination using the image such as a blood examination.

Regarding the generation of a finding sentence using the fourth information D4, discrimination of tuberculoma, which is one of pulmonary diseases, is performed not only by image diagnosis but also in combination with blood examination. Therefore, diagnostic information such as a blood examination result may be directly described in a finding sentence. For example, in a case where the blood examination result is “negative quantiferon” and information on a symptom suspected based on the blood examination result is “nontuberculous mycobacteriosis”, a finding sentence such as “nontuberculous mycobacteriosis is suspected due to negative quantiferon” may be described. In this case, the learning model 23A may be constructed by training the recurrent neural network 40 using training data that includes, for example, a label of “blood examination” and a label of a suspected symptom as the fourth information D4 in the diagnostic information and includes a training sentence in which these labels are described. In this case, the learning model 23A may be constructed so as to generate a finding sentence including a label of a property item included in the analysis result, or the learning model 23A may be constructed so as to generate a finding sentence not including the label of the property item.

In addition, even in a case where an examination result and an image analysis result obtained by using another diagnostic apparatus such as a contrast CT apparatus or a PET-CT apparatus other than the CT apparatus are used as diagnostic information, the diagnostic information may be similarly described in the finding sentence. For example, in a case where the examination result is “FDG accumulation in mPET is low” and the information on the symptom suspected based on the examination result is “round atelectasis or organizing pneumonia”, a finding sentence such as “FDG accumulation in mPET is low and round atelectasis or organizing pneumonia is suspected” may be described. Further, the analysis result related to the diagnostic information may be described in the finding sentence, and the analysis result not related to the patient information may not be described in the finding sentence. Further, depending on whether or not the analysis result is related to the fourth information D4, the finding sentence may be generated such that the analysis result related to the fourth information D4 may or may not be described in the finding sentence, or described in such a way as to decrease or increase a certainty factor, decrease or increase importance, or blur or assert.

Hereinafter, examples of a combination of the diagnostic information and the analysis result and a finding sentence generated by the sentence generation unit 23 will be described. FIG. 6 is a diagram showing an example of the diagnostic information and the analysis result. The diagnostic information shown in FIG. 6 is the first information D1. As shown in FIG. 6, in the first information D1, a type of lesion is “mass”, a diameter is “maximum diameter 46 mm”, definitive diagnosis of malignity is “primary lung cancer”, a treatment content is “after gefitinib treatment”, and a change in size is “increase”. These labels are Nodule, Diameter, Malignant, Treated, and Progress, respectively.

In addition, the analysis result includes a segment of the lesion in the lung which is “upper left section”, and a type of an absorption value which is a solid type, presence of a spicula, presence of calcification, absence of a cavity, and presence of a pleural contact. These labels are Segment, Solid, Spiculated+, Calcification+, Cavity−, and PleuralContact+, respectively.

The diagnostic information shown in FIG. 6 includes a definitive diagnosis of malignity of primary lung cancer. In this case, the change in size of the lesion over time is more important than the analysis result of internal properties such as the presence of a spicula. For this reason, the learning model 23A is constructed so as to generate a finding sentence to include only a change in size over time and not to include an analysis result indicating internal properties, or generate a finding sentence not to include an analysis result inconsistent with a definitive diagnosis (for example, a finding suggesting benignity in a case where there is a definitive diagnosis of primary lung cancer).

For example, the learning model 23A is constructed by training the recurrent neural network using training data including the labels of the diagnostic information and the analysis result shown in FIG. 6 and a training sentence “[Malignant] of [Segment] is [Treated]. [Nodule] is further increased in [Diameter]”. In the constructed learning model 23A, in a case where the diagnostic information and the analysis result are input, the analysis result is selected based on the diagnostic information and the finding sentence is generated. Specifically, the learning model 23A is constructed by training performed such that in a case where the diagnostic information and the analysis result shown in FIG. 6 are input, a finding sentence “[Malignant] of [Segment] is [Treated]. [Nodule] is further increased in [Diameter]” is output. The sentence generation unit 23 embeds the diagnostic information and the analysis result into the label of the finding sentence output from the learning model 23A to generate a finding sentence “The primary lung cancer in the upper left section is in a condition after gefitinib treatment. A mass is further increased to a maximum diameter of 46 mm”.

In order to reduce the certainty factor for the analysis result in the finding sentence described in the second information D2, the third information D3, and the fourth information D4, a finding sentence described in a non-assertive expression may be generated. Conversely, in order to increase the certainty factor for the analysis result, a finding sentence described in an assertive expression may be generated. The non-assertive expression is, for example, “~ is suspected”, and the assertive expression is, for example, “~ is recognized”. As described above, in the generated finding sentence, in order to decrease or increase the certainty factor for the analysis result, the learning model 23A may be constructed by training the recurrent neural network 40 using a training sentence in which the certainty factor for the analysis result is increased or a training sentence in which the certainty factor for the analysis result is decreased, according to the diagnostic information.

For example, as shown in FIG. 7, it is assumed that the labels of the diagnostic information are Nodule [MASS] and Diameter [MAJOR AXIS 48 mm], and the labels of the analysis result are Segment [LOWER RIGHT LOBE S6], Solid [SOLID TYPE], IrregularForm [IRREGULAR FORM], Spiculated+ [PRESENCE OF SPICULA], Lobulated+ [PRESENCE OF LOBULATED SHAPE], Airbronchogram+ [PRESENCE OF AIR BRONCHOGRAM], Cavity+ [PRESENCE OF CAVITY], Calcification− [ABSENCE OF CALCIFICATION], and PleuralContact+ [PRESENCE OF PLEURAL CONTACT]. The content in parentheses represents a specific content of each label. In this case, the finding sentence generated by the sentence generation unit 23 is “An irregular solid mass having a major axis of 48 mm in contact with the pleura is recognized in the lower right lobe S6. The mass is lobulated and accompanied by a spicula. An air bronchogram and a cavity are recognized inside. The calcification is not recognized”. In the generated finding sentence, for a property item whose presence or absence is clear, the sentence ends with “~ is recognized”, “~ is not recognized”, or “~ is accompanied by ~”.

On the other hand, as shown in FIG. 8, it is assumed that the labels of the diagnostic information are Nodule [MASS] and Diameter [MAJOR AXIS 48 mm], and the labels of the analysis result are Segment [LOWER RIGHT LOBE S6], Solid [SOLID TYPE], IrregularForm [IRREGULAR FORM], Spiculated+ [PRESENCE OF SPICULA], Lobulated+[PRESENCE OF LOBULATED SHAPE], Airbronchogram? [AIR BRONCHOGRAM UNCLEAR], Cavity? [CAVITY UNCLEAR], Calcification− [ABSENCE OF CALCIFICATION], and PleuralContact+[PRESENCE OF PLEURAL CONTACT]. The property item to which “?” is attached represents a false-positive analysis result.

In this case, the finding sentence generated by the sentence generation unit 23 is “An irregular solid mass having a major axis of 48 mm in contact with the pleura is recognized in the lower right lobe S6. The mass is lobulated and accompanied by a spicula. A low absorption area is recognized and an air bronchogram and a cavity are suspected inside. The calcification is not recognized”. In the generated finding sentence, for a property item whose presence or absence is clear, the sentence ends with “~ is recognized”, “~ is not recognized”, or “~ is accompanied by ~”. However, for a property whose presence or absence is unclear, that is, a false-positive property item, the sentence ends with “~ is suspected”.

In order to change the certainty factor of the sentence generated in this way, the 1-hot expressions of “Airbronchogram? [AIR BRONCHOGRAM UNCLEAR]” and “Cavity? [CAVITY UNCLEAR]” are defined in advance, the learning model 23A is constructed by training the neural network using training data that includes the labels of “Airbronchogram? [AIR BRONCHOGRAM UNCLEAR]” and “Cavity? [CAVITY UNCLEAR]” and includes a training sentence that ends with “~ is suspected”, whereby it is possible to generate a finding sentence with a certainty factor different according to the label.

In addition, by adding an expression according to the certainty factor, that is, an expression vector of “suspected” to the vector representation xt of the air bronchogram and the cavity input to the decoder 42 shown in FIG. 5, it is possible to generate a finding sentence that ends with “suspected”. Alternatively, it is also possible to generate a finding sentence that ends with “suspected” by deriving a vector representation in which a component corresponding to 1 in the 1-hot expressions of “air bronchogram” and “cavity” is changed according to the certainty factor. In this case, for example, in a case where the 1-hot expression of “air bronchogram” is (1,0,0), the 1-hot expression may be changed to (0.5,0,0) in accordance with the certainty factor.

As a result, the learning model 23A is constructed which generates a sentence that ends with a high certainty factor for a property item whose presence or absence is clear, but generates a sentence that ends with a low certainty factor for a property item whose presence or absence is unclear.

On the other hand, regarding the importance, an order according to the importance is given to each item included in the diagnostic information and the analysis result. In the present embodiment, it is assumed that a table defining the importance for each item of the diagnostic information and the analysis result is stored in the storage 13. The sentence generation unit 23 gives an order according to the importance for each item of the diagnostic information and the analysis result with reference to the table stored in the storage 13. Then, the learning model 23A is constructed by training the recurrent neural network 40 such that the importance is changed according to whether the property is negative or positive and further according to the diagnostic information and a finding sentence including a predetermined number of analysis results is generated in order of the importance.

For example, the calcification is generally a benign property. Further, a negative property has less importance than a positive property item. For this reason, the learning model 23A is constructed such that the importance of calcification and negative property items included in the analysis result is decreased, the importance of positive property items is increased, and a finding sentence is generated using a predetermined number of property items having high importance. Depending on the confirmed diagnosis result included in the diagnostic information, it may be better to describe a negative property item for a specific property item in the finding sentence. In this case, the learning model 23A is constructed such that a finding sentence is generated with high importance even in a case where a specific property is negative according to the diagnostic information.

For example, as shown in FIG. 9, it is assumed that the labels of the diagnostic information are Nodule [MASS] and Diameter [MAJOR AXIS 24 mm], and the labels of the analysis result are Segment [LOWER RIGHT LOBE S6], Solid [SOLID TYPE], Lobulated+ [PRESENCE OF LOBULATED SHAPE], Airbronchogram− [ABSENCE OF AIR BRONCHOGRAM], Cavity− [ABSENCE OF CAVITY], Calcification− [ABSENCE OF CALCIFICATION], and PleuralContact+ [PRESENCE OF PLEURAL CONTACT]. Further, as shown in FIG. 9, it is assumed that numbers are given to respective items of the diagnostic information and the analysis result in order of the importance. In the present embodiment, it is assumed that the learning model 23A is constructed so as to include high-order n (for example, five) labels having high importance for the diagnostic information and the analysis result.

In this case, as shown in FIG. 9, in a case where the diagnostic information and the analysis result to which the importance is given are input to the learning model 23A, a finding sentence “[Nodule] of [Lobulated+] with [PleuralContact+] [Diameter] is recognized in [Segment]” is output. The sentence generation unit