122 and 124 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid-state magnetic device, and so forth. The storage devices 122 and 124 can be the same type of storage device or different types of storage devices.

A speaker 132 is operatively coupled to system bus 102 by the sound adapter 130. A transceiver 142 is operatively coupled to system bus 102 by network adapter 140. A display device 162 is operatively coupled to system bus 102 by display adapter 160. One or more video cameras 156 can be further coupled to system bus 102 by any appropriate connection system or method (e.g., Wi-Fi, wired, network adapter, etc.), in accordance with aspects of the present invention.

A first user input device 152 and a second user input device 154 are operatively coupled to system bus 102 by user interface adapter 150. The user input devices 152, 154 can be one or more of any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. One or more video cameras 156 can be included, and the video cameras can include one or more storage devices, communication/networking devices (e.g., WiFi, 4G, 5G, Wired connectivity), hardware processors, etc., in accordance with aspects of the present invention. In various embodiments, other types of input devices can also be used, while maintaining the spirit of the present principles. The user input devices 152, 154 can be the same type of user input device or different types of user input devices. The user input devices 152, 154, and camera 156 are used to input and output information to and from system 100, in accordance with aspects of the present invention. A neural network/transfer learning trainer 164 can be operatively connected to the system 100 for retraining of a deep neural network using transfer learning to improve accuracy of video analytics tasks, in accordance with aspects of the present invention.

Of course, the processing system 100 may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements. For example, various other input devices and/or output devices can be included in processing system 100, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art. These and other variations of the processing system 100 are readily contemplated by one of ordinary skill in the art given the teachings of the present principles provided herein.

Moreover, it is to be appreciated that systems 200, 300, 400, 700, and 800, described below with respect to FIGS. 2, 3, 4, 7, and 8, respectively, are systems for implementing respective embodiments of the present invention. Part or all of processing system 100 may be implemented in one or more of the elements of systems 200, 300, 400, 700, and 800, in accordance with aspects of the present invention.

Further, it is to be appreciated that processing system 100 may perform at least part of the methods described herein including, for example, at least part of methods 200, 300, 500, 600, and 800, described below with respect to FIGS. 2, 3, 5, 6, and 8, respectively. Similarly, part or all of systems 200, 300, 400, 700, and 800 may be used to perform at least part of methods 200, 300, 500, 600, 700, and 800 of FIGS. 2, 3, 5, 6, and 8, respectively, in accordance with aspects of the present invention.

As employed herein, the term “hardware processor subsystem”, “processor”, or “hardware processor” can refer to a processor, memory, software, or combinations thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and/or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem can include one or more on-board memories (e.g., caches, dedicated memory arrays, read only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that can be on or off board or that can be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input/output system (BIOS), etc.).

In some embodiments, the hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and/or one or more applications and/or specific code to achieve a specified result.

In other embodiments, the hardware processor subsystem can include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or programmable logic arrays (PLAs).

These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

Referring now to FIG. 2, a high-level system and method 200 for real-time surveillance and identification of objects and people with increased accuracy of video analytics applications (e.g., object/people/animal detection, tracking, identification, etc.) by reducing object and people detection count fluctuations across video frames using transfer learning and/or adaptive tuning of a confidence threshold, is illustratively depicted in accordance with embodiments of the present invention.

In an illustrative embodiment, a camera system 202 can be utilized to monitor an area of interest and/or capture video and/or images for processing for video analytics applications (e.g., object/person/animal detection, tracking, identification, etc.), and can include a wired and/or wireless connectivity device for providing connectivity with one or more computing networks (e.g., Internet, LAN, WAN, etc.), and local or remote computing devices (e.g., server 206, computer for vehicle 210, etc.). While the single camera system 202 is shown in FIG. 2 for the sakes of illustration and brevity, it is to be appreciated that multiple camera systems can be also used, in accordance with aspects of the present invention.

In various embodiments, the camera system 202 can be mounted on a mounting entity 204. For the sake of illustration, the mounting entity 204 is depicted as a pole. While a pole 204 is shown (as a mounting entity) for the sake of illustration, any other mounting entity can be used, as readily appreciated by one of ordinary skill in the art given the teachings of the present invention provided herein. For example, the camera system 202 can be mounted in or on any of the following: a building, a tree, a drone, etc., in accordance with aspects of the present invention, noting that the preceding examples of various mounting entities are merely illustrative. The camera system 202 can be a wireless camera system having its own antenna(s) or can use one or more antennas included on the pole 204 (or other mounting entity (e.g., building, drone, etc.) to which the camera system 202 is mounted or proximate).

In various embodiments, the system and method 200 can include a local or remote computing device, depicted herein for ease of illustration as a server 206, for processing video captured by the camera system 202 for any or a plurality of video analytics tasks (e.g., object/person/animal detection, tracking, identification, etc.). It is to be appreciated that any of a plurality of types or amount of computing devices can be used in block 206, as readily appreciated by one of ordinary skill in the art given the teachings of the present invention provided herein. For example, the computing device 206 can be one or more of any of the following: a server, a main-frame, a network of servers, a cloud, a personal computer, etc., in accordance with aspects of the present invention, noting that the preceding examples of various computing devices 206 are merely illustrative.

In various embodiments, the server 206 can perform video analytics applications, including detecting, tracking, and identifying objects (e.g., vehicle 210) and people 208 across multiple frames of a video captured by the camera system 202, with reduced fluctuations in accuracy of such video analytics applications across video frames by retraining a pretrained Deep Neural Network (DNN) using transfer learning and/or adaptive tuning of a confidence threshold, in accordance with aspects of the present invention.

In various embodiments, the system and method 200 can accurately perform video analytics applications, including, for example, detection, tracking, and identification of objects, people, and animals in an area of interest in real time, and can transmit results of such video analytics applications to any of a plurality of computing devices for use in any of a plurality of application, including, for example, providing access to secure areas (e.g., using facial recognition), tracking a number of people in a particular area, detecting a particular person in a crowd, etc., in accordance with aspects of the present invention.

Such detection, tracking, and/or facial recognition can be with respect to a secured object such as, for example, a facility. The facility can be a secured facility or a non-secured facility. In one embodiment, the facility can be a secured facility implemented as an airport. Of course, other facilities (e.g., mass transit hub, sports complex, etc.) can also be monitored for detection and tracking of people 208 or objects 210, for facial recognition tasks, etc., in accordance with aspects of the present invention.

The server 206 can be located remote from, or proximate to, the camera system 202, and can include a processor 212 operatively coupled to a memory 214, a networking device 216, a detector/quantifier 218 configured to detect and quantify fluctuations in video analytics accuracy across video frames, and a neural network transfer learning retrainer/confidence threshold tuner 220 configured for increasing accuracy of video analytics tasks in real-time by reducing fluctuations in accuracy of video analytics applications across video frames by retraining a pretrained Deep Neural Network (DNN) using transfer learning and/or adaptive tuning of a confidence threshold, in accordance with aspects of the present invention.

In an illustrative embodiment, the server 206 can be configured to perform object 210 and person 208 counts for a particular area of interest (e.g., train station to determine daily traffic) based on video data received from the camera system 202 by (e.g., the networking device 216) of the server 206 to identify and provide a list 222 of object 210 and person 208 counts in the area of interest. In facial recognition embodiments, the server 206 can identify and store a list 222 of a plurality of different faces captured by the video camera and/or can store or access a database of known faces (e.g., person to grant/deny access for a facility) for comparison with the faces captured by the video camera and stored in the list 222, in accordance with aspects of the present invention. Such video analytics/facial recognition tasks can involve one or more actions performed responsive to the list 222, as readily appreciated by one of ordinary skill in the art. For example, an alert (local and/or remote) can be provided, one or more doors and/or windows can be closed and locked to secure the person 208 or vehicle 210 within a specific area or to keep the person 208 from accessing a specific area, a person containment procedure (e.g., all doors to a room closing and locking with person inside) can be automatically performed, etc., in accordance with aspects of the present invention.

Accordingly, some exemplary suitable environments to which the present invention can be applied can include any environments where detection, tracking, and/or identification of objects, people, animals, etc. can prove useful (e.g., mass transit hubs, border crossings, subways, transportation hubs, airports, ship ports, sports facilities, etc.), in accordance with aspects of the present invention. It is to be appreciated that the preceding environments are merely illustrative and, thus, other environments can also be used, while maintaining the spirit of the present invention.

In an illustrative embodiment, the system and method 200 can be used for object/people detection tasks, noting that such object detection is one of the most common tasks in video analytics pipelines. For example, detecting cars or people in real-time can be critical for many real-world applications, including, for example, video surveillance, retail, health care monitoring and intelligent transportation systems. It is to be appreciated that although object/people detection tasks are discussed hereinbelow for ease of illustration as an example of a particular video analytics task performed by the present invention, object/person detection is merely one of one of a plurality of video analytics tasks capable of being performed and improved by the present invention, as would be appreciated by one of ordinary skill in the art.

Although conventional systems and methods for video analytics applications can detect objects 210 and people 208 in a video, experiments have shown that conventional video analytics systems, which utilize widely-used object detectors (e.g., YOLOv5 (small and large variant), EfficientDet-v0, and EfficientDet-v8) for analysis of videos for detection of objects 210 and people 208, exhibit low accuracy in practice when compared to the ground truth (e.g., the actual number of cars and people at different times (i.e., frames) in the video). The detector prediction by the deep learning model (e.g., number of cars and people detected by the deep learning model at different times (e.g., frames) in the video), can be significantly lower than the ground truth for all considered object detectors using conventional systems and methods. It is notable that light-weight models (e.g., Yolov5—small or Yolov5—large) exhibit a much higher comparative range of detection fluctuations than heavier-weight models (e.g., EfficientDet), but the heavier-weight deep learning models require significantly more computing resources than the light-weight models in real-world utilization for video analytics tasks.

It is notable that the behavior of a Deep Neural Network (DNN) model can be deterministic in the sense that if a frame is processed multiple times by the DNN model, then the DNN inferences are identical, and similar consecutive frames should produce similar results. However, conventional systems and methods exhibit low video analytics accuracy across consecutive frames during processing using a DNN model in practice. In accordance with various embodiments, internal and external factors to a video camera that can cause accuracy fluctuations in video analytics applications for consecutive video frames can be identified and/or mitigated to improve accuracy of any of a plurality of video analytics applications by the server 206, in accordance with aspects of the present invention.

In various embodiments, camera external factors that can affect the quality of the frames of a video processed by the DNNs during video analytics tasks can include, for example, motion in the field of view of the camera 202, which can affect the quality because of blurring caused by the movement of objects, lossy video compression methods (e.g., H.264, AV1, VP9, etc.), in which the quality of decoded frames can noticeably differ from the pre-compression frames, and environmental conditions (e.g., lighting, fog, camera angle, etc.).

In accordance with various embodiments, an illustrative example of an environmental lighting condition which can cause reduction in video analytics accuracy is the flicker exhibited by fluorescent lighting, which can thus affect the quality of frames captured by the camera 202, in addition to contributing to detrimental fluctuations in the analytics accuracy of video analytics tasks, even though most people cannot notice the flicker in fluorescent lights, at least in part due to the flicker rate of approximately 120 cycles per second (120 Hz) being generally imperceptible by the human eye. In some embodiments, motion of objects or people in a video can cause detrimental fluctuations in video analytics accuracy across video frames (e.g., because of blurring caused by the movement of objects), and eliminating motion by, for example, utilizing still frames from a video stream captured by a video camera for executing video analytics tasks. In some embodiments, detection fluctuation of objects and people for video analytics applications can be reduced by eliminating video compression by fetching uncompressed frames directly from the camera 202 instead of fetching a compressed video stream and decoding the stream to obtain frames that can be processed by a DNN model, in accordance with aspects of the present invention.

In various embodiments, eliminating compression in a video can reduce the adverse effect on accuracy fluctuations, and can improve accuracy and results of deep learning models, in accordance with aspects of the present invention. This improvement can be shown by an analyzing the detection counts with and without compression by using the t-test for repeated measures. For example, let A be the sequence of true-positive object detection counts (per frame) for the experiment where video compression is used. Let B be the sequence of true-positive object detection counts for the case when no compression is used. A third sequence D can be computed as a sequence of pair-wise differences between the true-positive object count without compression and with compression (e.g., B−A).

Essentially, the use of difference scores converts a two-sample problem with A and B into a one-sample problem with D. Our null hypothesis states that compression has no effect on object detection counts (and we hypothesize a population mean of 0 for the difference scores). However, in an experiment with a sample size of 200 frames showed that we can reject the null hypothesis at the 0.01 level of significance (99% confidence), suggesting that the elimination of compression does reduce the accuracy fluctuations. Similar results were observed for sample sizes of 100 and 1000 frames. While the above-discussed t-test measures the statistical difference between two distributions, it does not reflect on the fluctuations observed in repeated measures or multiple frames (e.g., group of frames, consecutive frames, temporally separated frames, etc.) in a video.

In various embodiments, the server 206 can quantify the observed fluctuations across a multiple video frames using the following novel metrics: (1) Equation F2 can quantify the observed fluctuations by



tp
⁡
(
i
)

-

tp
⁡
(

i
+
1

)



mean
(

gt
⁡
(
i
)

,

gt
⁡
(

i
+
1

)

)

for frame i, where tp(i), gt(i) are true positive object detection count and ground-truth object count respectively on frame i (on a moving window of 2 frames), and (2) Equation F10 which is defined as



max
⁢

(

tp
⁡
(
i
)

,
…

,

tp
⁡
(

i
+
9

)

)

-

min
⁡
(

tp
⁡
(
i
)

,
…

,

tp
⁡
(

i
+
9

)

)



mean
(

gt
⁡
(
i
)

,
…

,

gt
⁡
(

i
+
9

)

)

(on a moving window of 10 frames), in accordance with aspects of the present invention. By eliminating video compression (e.g., by fetching uncompressed frames directly from the camera 202, the maximum variation in object count on a static scene can be reduced from 17.4% to 13.0% (F2) and from 19.0% to 17.4% (F10), showing a quantified improvement in video analytics accuracy and thus the results of deep learning models, in accordance with aspects of the present invention.

Eliminating light flickering in addition to motion and compression can reduce the maximum (F2) and (F10) variations from 13.0% to 8.7% and 17.4% to 13.0% respectively, quantifiably showing that after eliminating motion and video compression, fluorescent light with flicker continues to have an adverse effect on video analytics accuracy fluctuations.

In various embodiments, as described above, the system and method 200 can improve object and person detection results by eliminating motion, video compression and flicker. However, even after controlling for motion, video compression and flickering light, noticeable fluctuations in object detection counts still remain, and are notably present when using multiple different camera models from different vendors and using different deep learning models. Thus, in some embodiments, internal camera settings causing fluctuations in video analytics accuracy can be identified and/or mitigated in accordance with aspects of the present invention.

Various auto-parameter setting in modern cameras can have an adverse effect on video from a video camera 202 for use for video analytics applications. Along with exposing endpoints to retrieve streaming videos (e.g., RTSP stream URL), IP cameras can also expose APIs to set various camera parameters (e.g., VAPIX API for Axis camera models). These camera settings can change the quality of images and video captured and produced by the camera. Camera vendors expose these APIs because they do not know in advance which environment their camera would be utilized, or what camera settings would be ideal for that environment. Therefore, camera settings are generally set by the manufacturer to default values, and end users can then determine what settings would work best for their environment.

In practice, there are essentially two types of camera settings that are exposed by camera vendors. Firstly, are Type 1 parameters, which can include parameters which affect the manner in which raw images are captured (e.g., exposure, gain, shutter speed, etc.). These parameters generally can be adjusted automatically by the camera with little or no interaction or control by end users, and only allow end users to set maximum value. However, within a range of this value, the camera can internally change the settings dynamically to produce a visually pleasing video output for the human eye. Hereinafter such parameters will be referred to as automated parameters (AUTO) for ease of illustration. Secondly, are Type 2 parameters, which can include parameters which affect processing of raw data in order to produce the final frame (e.g., image specific parameters such as brightness, contrast, sharpness, color saturation, and video specific parameters such as compression, GOP length, target bitrate, FPS). For these parameters, camera vendors often expose fine control to end users, who can set a specific value for such parameters. Hereinafter such parameters will be referred to as non-automated parameters (NAUTO) for ease of illustration. The distinction between AUTO and NAUTO parameters can be useful for identifying and adjusting camera internal factors causing adverse video analytics accuracy fluctuations across frames, in accordance with aspects of the present invention.

A main purpose of a video camera is to capture videos, rather than still images, for viewing by human eyes. Hence, irrespective of how the scene in front of the camera looks (e.g., whether the scene is static or dynamic), a video camera always tries to capture a video, which assumes changes in successive frames. To capture a visually pleasing and smooth (e.g., to human eyes) video, the camera tries to automatically determine the optimal exposure time or shutter speed. High shutter speed (e.g., low exposure time) can freeze motion in each frame, resulting in very crisp individual images. However, when such frames are played back at usual video frame rates, it can appear as hyper-realistic and provide a very jittery, unsettled feeling to the viewer in practice. However, low shutter speed can cause moving objects to appear blurred, and can builds up noise in the capture of the video. To maintain an appropriate amount of motion blur and noise in the capture, video cameras can have another setting called gain, which indicates the amount of amplification applied to the capture. For example, a high gain can provide better images in a low-light scenario but can also increase the noise present in the capture. For these reasons, the optimal values of AUTO parameters like exposure and gain are internally adjusted by the camera to output a visually pleasing smooth video. Thus, it has been identified that video capture is fundamentally different from still image capture and the exact values of exposure and gain used by the camera for each frame are not known to end users or analytics applications running on the video output from the camera. This loose control over maximum shutter time and maximum gain parameters can be a major cause for fluctuations in video analytics accuracy, and can cause an unintentional adversarial effect for the camera system 202, and thus, a detrimental effect on accuracy of video analytics tasks.

It is noted that SOTA object detectors (e.g., Yolov5 or EfficientDet) are trained on still image datasets (e.g., COCO and VOC datasets), and it has been observed by experimentation that the accuracy of insights from such deep learning models fluctuate significantly when used for video analytics tasks. This suggests that deep learning models trained on still image datasets are not capable of being directly used for accurate results for video analytics tasks.

In various embodiments, the system and method 200 can mitigate the accuracy fluctuations in video analytics tasks due to the use of image-trained DNN models by retraining models trained using images for extracting insights from videos using video frames that are captured under a plurality of different scenarios. As an illustrative example, transfer learning can be executed by a neural network transfer learning retrainer/confidence threshold tuner 220 to train a Yolov5 model using a plurality of videos captured under different scenarios, in accordance with aspects of the present invention. These videos can contain objects from person and vehicle super-category (e.g., including car, truck, bus, train, etc. categories), captured by the cameras at different deployment sites (e.g., traffic intersection, airport, mall, etc.) during different times-of-the-day (e.g., day, afternoon, night) and also under different weather conditions (e.g., rainy, foggy, sunny). As an illustrative example, assume a total of 34K consecutive frames have been extracted from these video snippets, and these frames form a training dataset.

In various embodiments, the retrainer 220 can execute transfer learning from images to videos for training, in accordance with aspects of the present invention. In some embodiments, rather than using independent images or synthetically transformed images for training (as in conventional systems and methods), real video frames, which can take into account the distribution shift observed in practice for video analytics, can be utilized for training by the retrainer 220, in accordance with aspects of the present invention.

In this illustrative embodiment, it can be assumed that the first 23 modules (corresponding to 23 layers) of a deep learning model can be initialized using weights from COCO-trained Yolov5 model, and these weights can be frozen. In some embodiments, during training, only the weights in the last detect module are updated (noting that in other embodiments the weights can be updated in any appropriate manner, as appreciated by those of ordinary skill in the art. For the transfer learning, a learning rate of 0.01 with a weight decay value of 0.0005 was utilized in this exemplary embodiment. A Yolov5 model can be utilized, and can be trained on, for example, a NVIDIA GeForce RTX 2070 GPU server for a predetermined epoch amount and batch size (e.g., 50 epochs with a batch size of 32). In real-world practice, this lightweight training can significantly reduce processing power requirements and increase detection speed as compared to conventional systems and methods. In this exemplary embodiment, the transfer learning training can be completed using only 1.6 GB GPU memory, and can finish 50 epochs in less than 1.5 hours, which is a significant increase of speed while using less processing power as compared to conventional systems and methods.

In various embodiments, the transfer learning retrained Yolov5 model can be utilized to detect vehicles 210 and people 208 in an area of interest, and shows a large improvement in detection counts (as compared to conventional systems and methods) due at least in part to the utilization of the transfer learning trained model (e.g., Yolov5) for video analytics tasks, in accordance with aspects of the present invention. A comparison of the sequence of object detection counts for the original Yolov5 model (e.g., trained on images with no transfer learning) and the transfer-learning trained Yolov5 model, as in the present invention, can be performed by using a t-test for repeated measures. Then, the size of the beneficial effect due to transfer-learning can be determined (e.g., using equation F2 and/or F10). The results show that at a 0.01 level of significance, the improvement is 2.32 additional object detections (e.g., 14.3% improvement over the mean detections of the original Yolov5 model). For this experiment, the camera can have parameters set for automatic adjustment (e.g., AUTO) of camera parameters to produce a visually pleasing video, but the transfer-learning trained Yolov5 detector was able to detect more objects than when using the image trained model (as utilized by conventional systems and methods) despite the unintentional adversarial effect of the camera caused by the AUTO settings, in accordance with aspects of the present invention.

In various embodiments, in practical deployments of video analytics systems that operate 24×7 in real-world environments, it is difficult to control motion or environmental conditions, and the default video compression settings also vary from camera to camera, and can be unknown to users. Thus, although compensating for motion, environmental conditions, and video compression do improve the accuracy of video analytics applications, the largest improvement, and the most practical for use in real-world, dynamic environments, in particular with unknown variables (e.g., video compression rate for a camera), can be achieved using the transfer learning trained DNN model, in accordance with aspects of the present invention.

In various embodiments, experimental results show that the true-positive object detections by the transfer-learning trained Yolov5 model show significantly less range of fluctuations than the original Yolov5 model (e.g., trained on images). A comparison of the sequence of object detection counts for the original Yolov5 model (e.g., trained on images with no transfer learning) and the transfer-learning trained Yolov5 model, as in the present invention, can be performed by using a t-test for repeated measures. Then, the size of the beneficial effect due to transfer-learning can be determined (e.g., using equation F2 and/or F10). The results show that at a 0.01 level of significance, the improvement is 1 additional object detection (e.g., 9.6% improvement over the mean detections due to the original Yolov5 model). The newly transfer learning trained Yolov5 model can reduce the maximum variation of correctly detected object counts from 47.4% to 33.2% (F10), and 42.1% to 32.5% (F2), which is a significant improvement over conventional systems and methods, in accordance with aspects of the present invention.

Generally, object trackers (e.g., MOT SORT tracker) assign the same track-id to an object appearing in contiguous frames, and if an object is not detected in a frame, then the object's track is terminated. If the object is detected again in subsequent frames, a new track-id can be assigned to the object. In some embodiments, for illustrative purposes, it can be assumed that the number of track-ids assigned by a tracker can be an indicator of the quality of object detections. Experimental results show that a tracker reported 157 track-ids when the original Yolov5 model was used for detecting objects in a video with movement. In contrast, the same tracker reported 94 track-ids when the transfer-learning trained Yolov5 model was used (i.e., 40.1% fewer mistakes in tracking), which shows significant, quantifiable improvement. The ground truth for this example can be assumed to be 29 tracks, and experimental results show that the transfer-learning based Yolov5 model leads to significantly better performance on object tracking tasks in practice, in accordance with aspects of the present invention.

In some embodiments, a confidence threshold can be adaptively tuned based on a difficulty level to detect in particular frames, and the confidence threshold will be described in further detail herein below with reference to FIG. 5.

Referring now to FIG. 3, with continued reference to FIG. 2, a diagram showing a high-level view of an exemplary video surveillance and security system 300, including one or more video cameras 308, is illustratively depicted in accordance with embodiments of the present invention.

In an illustrative embodiment, a facility (e.g., an office, hospital, etc.) can perform object and person tracking using the video surveillance and security system 300, in accordance with aspects of the present invention. The video surveillance and security system 300 can include a video processing system similar to 206, as described with reference to FIG. 2 herein above. The video surveillance and security system 300 can deploy one or more video cameras 308 (hereafter “camera”) in, for example, a hallway, to monitor the area. The camera 308 can monitor the hallway for security purposes. The video acquired using the camera 308 can be analyzed using a transfer learning retrained DNN to increase accuracy of video analytics applications, including, for example, facial detection/identification, in accordance with aspects of the present invention. This can allow the video surveillance and security system 300 for a plurality of real-world tasks. For example, the video surveillance and security system 300 can unlock one or more doors 306 (hereafter “door”) when a known user 302 approaches the door 306. Though the door 306 may not have an access panel for entry, it can be unlocked by the video security system 300 when the known user 302 approaches the door 306. The video processing system 206 can recognize a known user 302 even if the known user 302 is next to another user 304 by utilizing, the transfer learning retrained DNN for increased accuracy of such video analytics tasks, in accordance with aspects of the present invention. It is to be appreciated that the preceding description utilizing the present invention in an office environment is merely illustrative and, thus, the video surveillance and security system 300 can be deployed in any sort of other environments, while maintaining the spirit of the present invention.

In some embodiments, the video surveillance and security system 300 can be utilized for facial detection/recognition tasks, which can be critical to many real-world applications (e.g., identifying a person of interest in airports, hospitals or arenas, authenticating individuals based on face-recognition for face-based payments, etc.), in accordance with aspects of the present invention. Experimental results show that the number of faces detected by conventional DNN models is significantly lower than the ground truth, and even though the ground truth barely changes, there is a significant fluctuation in the number of detections in consecutive frames using conventional systems and methods. Conversely, experimental results of a number of object and/or faces using a transfer learning retrained DNN, significantly and quantifiably reduces such detection fluctuations, as described in further detail herein above with respect to FIG. 2, in accordance with aspects of the present invention.

Referring now to FIG. 4, a diagram showing a high-level view of an exemplary environment 400, including an interconnected computing network 402 to which the present invention can be applied, is illustratively depicted in accordance with embodiments of the present invention. The elements shown relative to FIG. 4 are set forth for the sake of illustration. However, it is to be appreciated that the present invention can be applied to other network configurations as readily contemplated by one of ordinary skill in the art given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

In some embodiments, the computing network 402 can include at least a set of computer processing systems 404, 406, 408. The computer processing systems 404, 406, 408 can be any type of computer processing system including, but not limited to, servers, workstations, desktops, laptops, tablets, cellphones, smart phones, media playback devices, and so forth, in accordance with aspects of the present invention. For the sake of illustration, the computer processing systems 404, 406, 408 can each represent a computer server device.

In the embodiment shown in FIG. 4, the elements thereof are interconnected by a network 402. However, in other embodiments, other types of connections can also be used in accordance with aspects of the present invention. Additionally, one or more elements in FIG. 4 may be implemented by a variety of devices, which include but are not limited to, Digital Signal Processing (DSP) circuits, programmable processors, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), Complex Programmable Logic Devices (CPLDs), and so forth. These and other variations of the elements of environment 500 are readily determined by one of ordinary skill in the art, given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

Referring now to FIG. 5, a diagram showing a method 500 for increasing accuracy of video analytics tasks in real-time by reducing fluctuations in accuracy of video analytics applications across video frames by retraining a pretrained Deep Neural Network (DNN) using transfer learning and/or adaptive tuning of a confidence threshold, is illustratively depicted in accordance with embodiments of the present invention.

In various embodiments, a video/video stream can be acquired using one or more video cameras for use (e.g., real-time use) in video analytics applications (e.g., object/person detection) in block 502. Fluctuations in the accuracy of video analytics applications across consecutive frames of a video can be identified in block 504. In block 506, identified fluctuations can be quantified by determining an average relative difference between frames (e.g., consecutive frames) as follows: Equation F2 can quantify the observed fluctuations by



tp
⁡
(
i
)

-

tp
⁡
(

i
+
1

)



mean
(

gt
⁡
(
i
)

,

gt
⁡
(

i
+
1

)

)

for frame i, where tp(i), gt(i) are true positive object detection count and ground-truth object count respectively on frame i (on a moving window of 2 frames), and (2) Equation F10 can quantify the observed fluctuations by



max
⁢

(

tp
⁡
(
i
)

,
…

,

tp
⁡
(

i
+
9

)

)

-

min
⁡
(

tp
⁡
(
i
)

,
…

,

tp
⁡
(

i
+
9

)

)



mean
(

gt
⁡
(
i
)

,
…

,

gt
⁡
(

i
+
9

)

)

(on a moving window of 10 frames), in accordance with aspects of the present invention. It is to be appreciated that the above equations can be scaled to any amount of frames, and the above amounts are presented for illustrative purposes, in accordance with aspects of the present invention.

In some embodiments, fluctuations in accuracy can be reduced by applying transfer learning to a deep learning model initially trained on image analytics tasks, and re-training the deep learning model using video frames captured for plurality of different scenarios in block 508. In block 510, a quality of object detections can be determined based on an amount of track-ids assigned by a tracker across different video frames. An accuracy of video analytics tasks (e.g., object detections) can be increased by executing the video analytics tasks in real-time using the re-trained deep learning model and/or dynamically adaptively tuning a confidence threshold in block 512, in accordance with aspects of the present invention.

In various embodiments, the dynamic adaptive tuning of a confidence threshold in machine learning models in block 512 can significantly improve prediction accuracy, and can include adjusting a confidence threshold, analyzing a difficulty level, and filtering predictions, in accordance with aspects of the present invention. The confidence threshold adjusting can set an initial confidence threshold value for a machine learning model, and this threshold value can be used to distinguish between comparatively high-confidence and low-confidence predictions made by the model.

A difficulty level analysis can be performed to assess the comparative difficulty level associated with detection in a certain frame. This could be based on various factors, including, for example, the complexity of the data, the amount of data, the variability of the data, or any other factor that may affect the model's ability to make accurate predictions. Based on the assessed difficulty level, the confidence threshold can dynamically adjust the confidence threshold value. For frames with a comparatively higher difficulty level, the confidence threshold value may be increased, thereby reducing the likelihood of accepting low-confidence mispredictions. Conversely, for frames with a comparatively lower difficulty level, the confidence threshold value may be decreased, thereby increasing the likelihood of accepting high-confidence predictions.

Prediction filtering can be performed in block 512, which can filter out particular predictions made by the machine learning model based on the adapted confidence threshold value. For example, predictions with a confidence level below the adapted threshold value can be filtered out as low-confidence mispredictions, while predictions with a confidence level above the adapted threshold value can be accepted as high-confidence predictions, in accordance with aspects of the present invention. In this way, the present invention can further improve the prediction accuracy (e.g., of a transfer learning retrained model) and overall performance of the original machine learning model and the transfer learning retrained model by adjusting the confidence threshold in machine learning models based on the difficulty level associated with detection in a certain frame, thereby providing more reliable and accurate results, in accordance with aspects of the present invention.

In block 514, the reduction of fluctuations can be optimized by iteratively repeating the steps of identifying fluctuations, determining and quantifying the identified fluctuations, reducing fluctuations in accuracy, and determining a quality of object detections until a threshold is reached. Object and person detection and tracking with increased speed and accuracy can be performed in block 516 using the re-trained model for one or more applications (e.g., video surveillance, autonomous driving, biometric locking, etc.), in accordance with aspects of the present invention

Referring now to FIG. 6, a diagram showing a method 600 for reducing fluctuations in accuracy of video analytics applications across video frames by retraining a pretrained Deep Neural Network (DNN) using transfer learning, is illustratively depicted in accordance with embodiments of the present invention.

Transfer learning is a machine learning technique where a pre-trained model can be adapted for a different but related problem. In various embodiments, a deep learning model (e.g., YOLOv5 COCO model), initially trained on an image dataset (e.g., COCO), can be retrained using video frames to enhance its performance on video analytics tasks. This approach can leverage the model's existing knowledge gained from the image dataset and can apply it to the video domain, thereby reducing the computational resources and time required for training from scratch as opposed to retraining, in accordance with aspects of the present invention. It is to be appreciated that although the specific YOLOv5 COCO model is discussed herein for illustrative purposes, the present principles can be applied to any sort of deep learning model in accordance with various embodiments of the present invention.

In various embodiments, improved video analytics performance (as quantifiably compared to conventional systems and methods) can be achieved by retraining models pretrained on images (e.g., YOLOv5 COCO model) using video frames for the retraining by applying transfer learning to adapt the model to the specific characteristics of video data, thereby enhancing its accuracy and consistency in video analytics tasks, in accordance with aspects of the present invention. The retraining process can involve several steps, including, for example, extracting video frames from the video dataset, and utilizing these frames as the new training data for the YOLOv5 COCO model. The extraction can be performed at a fixed frame rate or adaptively based on the content changes in the video. The extracted frames can be pre-processed to match the input requirements of the YOLOv5 COCO model, and this pre-processing can include one or more of resizing the frames to required/specified dimensions, normalizing the pixel values, and augmenting the data to increase the diversity of the training set, in accordance with aspects of the present invention.

In some embodiments, the YOLOv5 COCO model can then be retrained using the pre-processed frames in block 610. The retraining process can include fine-tuning the model's parameters to minimize the difference between the model's predictions and the actual labels of the video frames, which can be achieved using a backpropagation algorithm and optimization method, such as stochastic gradient descent (SGD) or Adam. During retraining, the model can learn (e.g., using a neural network) to recognize patterns specific to video data, and which are not present in the image data the model was initially trained on. This learning can include capturing and accurately accounting for the temporal dependencies between consecutive frames and the subtle variations caused by the dynamic adjustments of video camera parameters.

In some embodiments, after retraining, the, adapted retrained YOLOv5 COCO model can perform any of a plurality of video analytics tasks with improved accuracy (as compared with conventional systems and methods), as the retrained model can effectively handle the fluctuations in pixel values between consecutive frames and provide consistent results across different camera models and vendors for any of a plurality of video analytics tasks, in accordance with aspects of the present invention.

In various embodiments, a sequence of video frames captured using a camera can be monitored, captured, and/or received (e.g., by a server 206 with reference to FIG. 2) in block 602. Each frame can be passed through a convolutional neural network (CNN), or other type of DNN, trained on a large-scale image dataset, with the CNN acting as a feature extractor, transforming each frame into a high-dimensional feature vector and generating output of a sequence of feature vectors corresponding to the sequence of frames in the video, in block 604. In block 606, the sequence of feature vectors can be passed through a temporal feature aggregation layer to capture temporal dependencies between consecutive frames, and outputting a second sequence of feature vectors, and a second sequence of feature vectors can be passed through a classifier to capture temporal dependencies between consecutive frames and output a prediction for the selected task in block 608.

In block 610, the entire model (including the feature extractor, the temporal feature aggregation layer, and the classifier) can be trained using, for example, end-to-end using backpropagation and an optimization algorithm (e.g., stochastic gradient descent), and the training can include fine-tuning pre-trained weights of the CNN during the training. In block 612, a generated model prediction for each frame in the video can be output, taking into account both the visual content of the frames (e.g., captured by the CNN) and the temporal dependencies between them (e.g., captured by the temporal feature aggregation layer), in accordance with aspects of the present invention

Referring now to FIG. 7, a diagram showing a system 700 for increasing accuracy of video analytics tasks in real-time by reducing fluctuations in accuracy of video analytics applications across video frames by retraining a pretrained Deep Neural Network (DNN) using transfer learning and/or adaptive tuning of a confidence threshold, is illustratively depicted in accordance with embodiments of the present invention.

In various embodiments, one or more video cameras 702 can acquire video data, and video data streams from the camera 702 can be transmitted over a computing network 704 (e.g., WiFi, wireless, 4G, 5G, CAN bus, LAN, WAN, wired, etc.), and can be analyzed and processed (e.g., for performing video analytics tasks) using one or more processor devices 706 and/or a neural network