411 and a second reflective surface 412. In general, the reflective surface is part of a background object.

The image frame 402 comprises pixels representing an image of the object 410 which is detected as a person, i.e., an object to be masked. However, the image frame 402 also comprises pixels representing an image of the first and second reflective surfaces 411, 412.

FIG. 7b represents the scene of FIG. 7a captured with the second image frame 403, i.e., slightly after the first image frame 402. In FIG. 7b the object 410 has moved slightly with respect to the background. Also the image of the object 410 in the reflective surface 411 has moved.

FIG. 8 illustrates content of the first image frame 402 and content of the second image frame 403. The image-processing device 120, 260 may define the identified object 410 in the image frame 402 by a two-dimensional bounding box 413. In FIG. 8 the object 410 is illustrated as a rectangle for simplicity. The two-dimensional bounding box 413 may be defined by its coordinates in the image frame 402. Further, a surface of the two-dimensional bounding box 413 may be divided into surface elements 414. Each surface element 414 of the two-dimensional bounding box 413 may be defined by its two-dimensional coordinate in the image frame 402 and by a colour value, such as a hue value. The colour value may be obtained by calculating an average colour value of pixels within the surface element 414.

Embodiments herein are based on finding reflective surfaces that potentially may reflect the object into the camera 120 by tracing rays in a three-dimensional model of the scene. To do so embodiments herein define a three-dimensional bounding box 415 of the defined object 410 in the three-dimensional model of the background scene, e.g., based on the defined object 410 in the image frame 402 an obtained position of the object 410 in the three-dimensional model and by extrapolating the defined object 410 in a plane extending along the normal of the image plane, i.e. along the depth plane of the image.

FIG. 9a illustrates a simplified three-dimensional model of the scene including the three-dimensional bounding box 415, the camera 120 and the reflective surface 411 of the background. The reflective surface 411 of the background is drawn as a two-dimensional sheet in FIG. 9a for simplicity.

The two-dimensional bounding box 413 and the surface elements 414 of the two-dimensional bounding box 413 may be used to extrapolate the defined object 410 in the depth plane. For example, the three-dimensional bounding box 415 may be generated by extrapolating the two-dimensional bounding box 413. Surfaces of the three-dimensional bounding box 415 may be generated based on the surface elements 414 of the two-dimensional bounding box 413.

FIG. 9b schematically illustrates a part of the background model which comprises the first reflective surface 411. FIG. 9b further schematically illustrates how a light ray from a back surface 416 of the three-dimensional bounding box 415 is reflected from the first reflective surface 411 into the camera 120. FIG. 9b further schematically illustrates parts of the background image frame 401b, overlaid on the background model, which correspond to at least a part of the first reflective surface 411. The back surface 416 of the three-dimensional bounding box 415 may comprise surface elements 417 of the three-dimensional bounding box 415 which are copies of the surface elements 414 of the two-dimensional bounding box 413. Side surfaces of the three-dimensional bounding box 415 may be generated as copies of bounding surface elements 414 of the two-dimensional bounding box 413. A colour of the copied surface elements may be preserved.

FIG. 10 illustrates a flowchart describing a method performed in the image-processing device for detecting the reflection of the identified object 410 in the image frame 402 captured by the camera 120.

The method may be performed by an image-processing device, such as the video camera 120 or the video server 260.

The below actions may be taken in any suitable order, e.g., in another order than the order presented below.

Action 1001

The background image frame 401b may be generated according to known methods. Background objects may be identified and defined in the background image frame 401b.

Preferably, the background image frame 401b does not comprise the foreground object 410.

Action 1002

The image frame 402 may be obtained. The object 410 in the image frame 402 may be identified from image information in the image frame 402. For example, an object identification algorithm may be used to find objects in the scene.

Action 1003

A three-dimensional model of the background scene of the image frame 401 is generated based on obtained three-dimensional information about the background scene. The background scene may for example be a room.

The three-dimensional model of the background scene comprises spatial information about the background objects, such as position, size and orientation of surfaces of the background objects. For example, the three-dimensional model of the background scene may comprise information about the direction of a normal of a surface of a background object. The normal may be used for ray tracing.

The three-dimensional information about the background scene may comprise three-dimensional coordinates of surfaces in the background scene, a respective normal vector of the surfaces, three-dimensional coordinates of the camera 120 and an orientation of the camera 120. For example, the three-dimensional model of the background scene may comprise information about a position of the camera 120 in relation to a position of the background object.

The three-dimensional information about the background scene may be generated from Light Detection And Ranging (LIDAR).

In some other embodiments the three-dimensional information about the background scene may be obtained by running a Neural Network with the background image frame 401b as input data.

The three-dimensional model of the background scene is linked to the background image frame 401b. For example, there is a mapping between three-dimensional coordinates of the surfaces of the background model and the corresponding pixels of the background image frame 401b. Mapping image data on top of for example LIDAR data is known in the art.

The three-dimensional model of the background scene may be updated when the background image frame 401b is updated.

In some embodiments herein the three-dimensional model of the background scene is modified by removing surfaces of the background objects which normal is directed towards the camera 120 since reflected light from these surfaces is not likely to reach the camera.

Action 1004a

The image-processing device 120, 260 defines the identified object 410 in the image frame 402 based on image information in the image frame 402.

For example, the image-processing device 120, 260 may define the identified object 410 in the image frame 402 by the two-dimensional bounding box 413. A position of the identified object 410 in the image frame 402 may be defined.

Action 1004b

The image-processing device 120, 260 defines the three-dimensional bounding box 415 of the defined object 410 in the three-dimensional model of the background scene. The three-dimensional bounding box 415 may be a cuboid. A bounding box may contain any shape and may simplify the computation. An arbitrary shape may fit inside the bounding box 415. In some embodiments herein the three-dimensional bounding box 415 comprises one or more voxels. The three-dimensional bounding box 415 may be based on the defined object 410 in the image frame 402, an obtained position of the object 410 in the three-dimensional model and by extrapolating the defined object 410 in a plane extending along the normal of the image plane. The plane extending along the normal of the image plane may also be referred to as the depth plane of the image.

The image-processing device 120, 260 may extrapolate the object 410 by extrapolating bounding pixels of the object 410 from the captured image frame 402.

In some example embodiments pixels of the object 410 are extrapolated a certain distance in the plane of the normal of the image plane. For instance, a car has a fairly uniform length, which may be used to carry out the extension.

The back surface 416 of the three-dimensional bounding box 415 may comprise pixels being copies of the pixels of the object 410 of the image frame 402, that is pixels within the two-dimensional bounding box 413. Side surfaces of the three-dimensional bounding box 415 may comprise pixels being copies of the pixels of the bounding pixels of the object 410 of the image frame 402. A colour of the copied pixels is preserved.

The obtained position of the object 410 in the three-dimensional model may be obtained by three-dimensional detection of the object 410, such as based on LIDAR, RADAR, or depth estimation Artificial Intelligence (AI) based on the image information in the image frame 402.

In some other embodiments the three-dimensional bounding box 415 is based directly on a three-dimensional mapping of the object 410, such as based on LIDAR, or depth estimation Artificial Intelligence (AI).

Action 1005

The image-processing device 120, 260 defines surface elements 417 of the three-dimensional bounding box 415. Smaller surface elements 417 means better accuracy, but worse performance.

A respective surface element 417 being defined by a centre coordinate 418 in the three-dimensional model of the background scene and a colour value.

The colour value may be a hue value or a combination of Cb and Cr values in the YUV format. Other colour values are also possible.

Hue is one of the main properties called colour appearance parameters of a colour, defined technically in the CIECAM02 model as “the degree to which a stimulus can be described as similar to or different from stimuli that are described as red, orange, yellow, green, blue, violet,” within certain theories of colour vision. Hue may typically be represented quantitatively by a single number, often corresponding to an angular position around a central or neutral point or axis on a colour space coordinate diagram, such as a chromaticity diagram, or colour wheel.

A respective surface element 417 of the three-dimensional bounding box 415 corresponds to multiple pixels from the captured image frame 402. The colour value of the respective surface element 417 is calculated as an average of colour values of the corresponding multiple pixels.

The colour value of the surface element 417 may be defined by averaging colour values of pixels within another surface element from which the surface element 417 has been derived. For example, the surface element 417 may be on the backside and may be derived from another surface element on the front side which is derived from a group of pixels of the image frame 402.

Action 1006

The image-processing device 120, 260 determines a three-dimensional coordinate of the surface 411 in the three-dimensional model of the background scene which reflects light from the surface element 417 of the three-dimensional bounding box 415 of the object 410 into the camera 120. That is, the determined three-dimensional coordinate of the surface 411 in the three-dimensional model of the background scene is positioned such that light that comes from the surface element 417 and is reflected by the surface 411 at this three-dimensional coordinate will be captured by the camera 120. However, an actual lighting condition of the scene may determine whether or not it is possible to detect a reflection of the object 410 in pixels of the image frame 402 corresponding to the three-dimensional coordinate of the surface 411. Thus, action 1006 is about finding candidate reflective coordinates of the surface 411.

The determined three-dimensional coordinate of the surface 411 in the three-dimensional model of the background scene may be stored.

The determining is performed by tracing rays from the centre coordinate 418 of the surface element 417 of the three-dimensional bounding box 415 of the object 410 and based on a normal of the surface in the three-dimensional model of the background scene at said three-dimensional coordinate.

The ray tracing may be repeated for multiple surface elements 417. For example, ray tracing may be performed from all surface elements of the three-dimensional bounding box 415. Then all corresponding three-dimensional coordinates of the surface 411 in the three-dimensional model of the background scene which reflect light from the surface elements 417 may be found.

In some other embodiments ray tracing is only performed for surface elements 417 that define the three-dimensional bounding box 415. In this way a contour of the projection of the three-dimensional bounding box 415 onto the surface 411 may be found and used later for determining which pixels of the image frame 402 to mask. This latter option requires less computations.

Ray tracing may be performed by known methods. Ray tracing is a method for calculating the path of waves or particles through a system with regions of varying propagation velocity, absorption characteristics, and reflecting surfaces. Under these circumstances, wavefronts may bend, change direction, or reflect off surfaces. Ray tracing solves the problem by repeatedly advancing idealized narrow beams called rays through the medium by discrete amounts.

When applied to problems of electromagnetic radiation, such as light, ray tracing often relies on approximate solutions to Maxwell's equations that are valid as long as the light waves propagate through and around objects whose dimensions are much greater than the light's wavelength.

Ray tracing works by assuming that the particle or wave can be modeled as a large number of very narrow beams (rays), and that there exists some distance, possibly very small, over which such a ray is locally straight. The ray tracer may advance the ray over this distance, and then use a local derivative of the medium to calculate the ray's new direction. From this location, a new ray is sent out and the process is repeated until a complete path is generated. If the simulation includes solid objects, the ray may be tested for intersection with them at each step, making adjustments to the ray's direction if a collision is found.

Action 1007

The image-processing device 120, 260 identifies a first pixel 431 in the image frame 402 which corresponds to the determined three-dimensional coordinate of the surface 411. The identified first pixel may be a candidate reflection pixel.

The image-processing device 120, 260 may identify multiple first pixels 431 in the image frame 402 which correspond to multiple determined three-dimensional coordinates of the surface 411.

Action 1008

The image-processing device 120, 260 detects the reflection of the object 410 in the image frame 402 when a mixing of a first colour value of the identified first pixel 431 and the colour value of the surface element 417 of the object 410 and a ground truth colour value of the identified first pixel 431 fulfils a mixing criteria. That is, the identified first pixel 431 is detected as a pixel reflecting the object if mixing of a first colour value of the identified first pixel 431 and the colour value of the surface element 417 of the object 410 and a ground truth colour value of the identified first pixel 431 fulfils a mixing criteria. The mixing criteria may for example be that the colour value of the identified first pixel 431 of the background image frame 401b is roughly an addition of the colour value of the surface element 417 of the object 410 and the ground truth colour value. This may be checked by subtracting the ground truth colour value from the colour value of the identified first pixel 431 and comparing the obtained value with the colour value of the surface element 417 of the object 410.

The ground truth colour value is a colour value of the identified first pixel 431 when there is no reflection from foreground objects.

The ground truth colour value may be obtained from one or more background image frames 401b or from one or more neighbouring pixels to the identified first pixel 431 in the image frame 402.

The ground truth colour value from the one or more background image frames 401b may be a value taken over time to produce an average value. The ground truth value may be stored in memory.

The ground truth may be a heatmap that is built up over time. If some of the one or more background image frames 401b comprises the object 410 then the one or more background image frames 401b may be averaged to not contain any significant reflection of the object 410.

Action 1008 may be repeated for multiple identified first pixels 431.

FIG. 11 illustrates a further flowchart describing a further method performed in the image-processing device for detecting the reflection of the identified object 410 in the image frame 402 captured by the camera 120. The method of FIG. 11 is also for masking a pixel area that has been determined to represent a reflection of the object 410.

The method of FIG. 11 may be performed by the image-processing device, such as the video camera 120 or the video server 260.

The below actions may be taken in any suitable order, e.g., in another order than the order presented below.

Action 1109

In response to detecting the reflection of the object 410 the image-processing device applies a mask to a pixel area 441 of the image frame 402. The pixel area 441 comprises the identified first pixel 431. The pixel area 441 is illustrated in FIG. 9b. As mentioned above in relation to action 1008 image-processing device 120, 260 detects the reflection of the object 410 in the image frame 402 when a mixing of the first colour value of the identified first pixel 431 and the colour value of the surface element 417 of the object 410 and the ground truth colour value of the identified first pixel 431 fulfils the mixing criteria. For example, if the colour value of the identified first pixel 431 of the background image frame 401b is roughly an addition of the colour value of the surface element 417 of the object 410 and the ground truth colour value, then the identified first pixel 431 and surrounding pixels of the pixel area 441 may be masked.

FIG. 12 schematically illustrates the part of the background model which comprises the first reflective surface 411. FIG. 12 further schematically illustrates parts of the background image frame 401b, overlaid on the background model, which correspond to at least the part of the first reflective surface 411. In some embodiments herein the masked pixel area 441 comprises all pixels corresponding to a projection 421 of the surface element 417 of the three-dimensional bounding box 415 onto the surface 411 or the second surface 412 in the three-dimensional model of the background scene. In some embodiments herein the masked pixel area 441 is limited to the pixels corresponding to the projection 421 of the surface element 417 of the three-dimensional bounding box 415 onto the surface 411 or the second surface 412. The projection is reflected into the camera 120. The projection 421 and an enlarged view of the projection 421 is also illustrated in FIG. 12. The enlarged view of the projection 421 has been illustrated as filled with pixels of the image frame 402 which correspond to the part of the surface 411 which is covered by the projection 421. Thus, the masked pixel area 441 may comprise the pixels of the image frame 402 which correspond to the part of the surface 411 which is covered by the projection 421.

A method for corroborating the detection of the reflection of the identified object 410 in the image frame 402 will now be described with reference to FIG. 11 and FIG. 13 and may be performed by analysing whether or not the detected reflection moves between the first image frame 402 and at least a further image frame, such as the second image frame 403.

The below method may be triggered when motion of the object 410 has been detected based on the second image frame 403 and the image frame 402.

The image-processing device 120, 260 may increase a probability value of having found a reflection of the object 410 if both motion of the object 410 and motion of the reflection is detected.

Action 1110

In some embodiments herein the image-processing device 120, 260 obtains the second image frame 403 of the video sequence 400 comprising the image frame 402. The second image frame 403 comprises the identified object 410. The second image frame 403 may be an image frame following directly after the first image frame 402 in the video sequence 400.

Action 1111

The image-processing device 120, 260 defines the identified object 410 in the second image frame 403 based on image information in the second image frame 403.

Action 1112

The image-processing device 120, 260 determines a respective second centre coordinate 420 of the surface element 417 of the three-dimensional bounding box 415 of the object 410 in the three-dimensional model of the background scene based on an obtained second position of the object 410 in the three-dimensional model. The second position of the object 410 in the three-dimensional model may be obtained in the same way as described above in relation to action 1004b. The second position of the object 410 in the three-dimensional model may correspond to a position of the object 410 in the second image frame 403.

Action 1113

The image-processing device 120, 260 determines a second three-dimensional coordinate of the surface 411 or of the second surface 412 in the three-dimensional model of the background scene, which reflect light from the surface element 417 of the three-dimensional bounding box 415 of the object 410 into the camera 120 and differs from the determined three-dimensional coordinate of the surface 411.

The determining is performed by tracing rays from the second centre coordinate 420 of the surface element 417 and based on a second normal of the surface 411 or of the second surface 412 in the three-dimensional model of the background scene at said second three-dimensional coordinate.

Action 1114

The image-processing device 120, 260 identifies a second pixel 432 in the second image frame 403 which corresponds to the determined second three-dimensional coordinate of the surface or of the second surface 412.

Action 1115

The image-processing device 120, 260 obtains a second colour value of the second pixel 432. The second colour value of the second pixel 432 may be stored.

Action 1116a

The image-processing device 120, 260 may confirm detection of the reflection of the object 410 when a mixing of the first colour value and the second colour value fulfils a second mixing criteria.

For example, for the next frame 403 look at the saved three-dimensional position from the previous frame 402, and compare the saved colour value to its current colour value. If there is a change roughly corresponding to the colour of the object 410, it is a strong indication that the reflection is moving. If this is concluded, it indicates that the area is reflective, which may increase a probability to add a mask.

Action 1116a

In some other embodiments herein the image-processing device 120, 260 rejects detection of the reflection of the object 410 when the mixing of the first colour value and the second colour value does not fulfil the second mixing criteria. A colour value of the first pixel 431 in the second image frame 403 equals the first colour value.

Thus, if the second colour value is different from the first colour value, but the colour value of the pixel area that was analysed for the previous frame 402 is the same, this indicates a false positive.

This means that the object is moving, but there is not a moving reflection. If the object is moving, but the reflection is not, this indicates a false positive and removal of the reflective mask at this area may be considered.

Action 1117

In response to confirming detection of the reflection of the object 410, the image-processing device 120, 260 may apply a mask to a second pixel area 442 of the second image frame 403. The second pixel area 442 comprises the identified second pixel 432. In some embodiments herein the masked second pixel area 442 is limited to the pixels corresponding to a second projection of the surface element 417 of the three-dimensional bounding box 415 onto the surface 411 or the second surface 412.

Action 1118

In response to rejecting detection of the reflection of the object 410, the image-processing device 120, 260 may determine to not apply a mask to the second pixel area 442 comprising the identified second pixel 432.

With reference to FIG. 14, a schematic block diagram of embodiments of an image-processing device 600 is shown. The image-processing device 600 corresponds to any of the image-processing devices of FIG. 1 or FIG. 2. Thus, the image-processing device 600 may comprise or be any of a camera, such as a surveillance camera, a monitoring camera, a camcorder, a network video recorder, and the wireless communication device 130. In particular, the image-processing device 600 may be the camera 120, such as a surveillance video camera, or the video-server 260.

As mentioned above, the image-processing device 600 is configured to perform the method according to FIG. 10 or FIG. 11.

The image-processing device 600 may further comprise a processing module 601, such as a means for performing the methods described herein. The means may be embodied in the form of one or more hardware modules and/or one or more software modules.

The image-processing device