104 that is at the left side of the patient. An actuator 17 may include one or more motors that are controlled so that they drive the rotation of a joint of the arm 104, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical tool 107 that is attached to that arm. Motion of several actuators 17 in the same arm 104 can be controlled by the spatial state signals generated from a particular UID 14. The UIDs 14 can also control motion of respective surgical tool graspers. For example, each UID 14 can generate a respective grip signal to control motion of an actuator, e.g., a linear actuator that opens or closes jaws of the grasper at a distal end of surgical tool 107 to grip tissue within patient 106.

In some aspects, the communication between the surgical robotic table 105 and the user console 102 may be through a control tower 103, which may translate user commands that are received from the user console 102 (and more particularly from the console computer system 16) into robotic control commands that transmitted to the arms 104 on the surgical table 105. The control tower 103 may also transmit status and feedback from the surgical table 105 back to the user console 102. The communication connections between the surgical table 5, the user console 102, and the control tower 103 may be via wired (e.g., optical fiber) and/or wireless links, using any suitable one of a variety of wireless data communication protocols, such as BLUETOOTH protocol. Any wired connections may be optionally built into the floor and/or walls or ceiling of the operating room. The system 101 may provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system.

FIG. 2 is a block diagram of the surgical system 101 in which a hub (or hub device) 20 of the system communicates with several devices according to one aspect. In some aspects, the hub 20 may be (include or be a part of) the control tower 103, or may be (e.g., a device separate from the control tower 103 and) configured to perform at least some of the operations of the control tower 103 as described herein, such as receiving user commands from a user device (e.g., UID 14) and translating the commands into robotic control commands to be routed to one or more devices of the surgical system. In addition, the hub 20 may be configured to receive surgical data streams from one or more surgical devices (and/or sensors) and produce (and/or pass through) one or more video streams (which include surgical data from at least some of the received surgical data streams, for example) for display on one or more displays. More about the operations of the hub 20 is described herein.

As shown, the surgical system 101 includes the hub 20, a (optional) switch 21, a network 22, one or more cameras 23, one or more display (screens) 24, a (first) surgical device (e.g., an ultrasonic generator) 26a, a (second) surgical device (e.g., a robotic arm) 26b, a (first) sensor (e.g., a proximity sensor) 27a, a (second) sensor (e.g., a temperature sensor) 27b, and a control device 25. In one aspect, the system may include fewer or more components. For instance, the system may include one or more sensors (and/or one or more surgical devices), and/or may include more than one control device. In one aspect, at least some of these elements may be arranged (disposed) within an operating room, such that they are configured to communicate (e.g., with one another) during a surgical procedure that is conducted within the operating room.

The hub 20 may be any type of electronic device that is able to communicate with one or more other devices in order to exchange data, and to produce (and/or pass through) one or more video streams, each video stream having at least some data received from one or more of the other devices, for display on one or more displays 24. In one aspect, the hub 20 may include electronic components (e.g., one or more processors, memory, etc.) for conducting at least some of the operations described herein. For example, the hub 20 may be a desktop computer, a laptop computer, etc. In another aspect, the hub 20 may be one or more (remote or cloud-based) servers. In some aspects, the hub 20 may be a handheld electronic device, such as a tablet computer or a smartphone.

Each of the surgical devices 26a and 26b may be any type of surgical device that is configured to perform (e.g., automatically, without user intervention, and/or through manual (or user) manipulation) one or more surgical tasks and/or to produce at least one surgical data stream (which may include data/information related to its associated surgical task(s)). For instance, the first surgical device 26a may be an energy (or ultrasonic) generator which may be configured to operate an ultrasonic tool for use during a procedure (e.g., in order to cut and cauterize tissue). In one aspect, the surgical data stream produced by the first device may include data related to the ultrasonic tool, such as its operating state (whether the tool is actively cutting tissue), and/or parameters associated with the ultrasonic instrument, such as a resonant frequency. The second surgical device 26b may be a robotic arm, such as arm 104 shown in FIG. 1. In one aspect, the second surgical device 26b may provide surgical data that indicates its orientation (pose) within the operating room.

Each of the sensors 27a and 27b may be any type of electronic device that is configured to detect (or sense) the environment (e.g., an operating room) and produce sensor data (e.g., as one or more surgical data streams) based on the environment. For example, the first sensor 27a may be a proximity sensor that produces sensor data that indicates a position of one or more objects with respect to the sensor. The second sensor 27b may be a temperature sensor that detects an ambient temperature (e.g., within the operating room). In another aspect, the surgical system may include other sensors, such as a microphone that is arranged to capture sounds within the ambient environment (e.g., operating room) and produce one or more audio streams (e.g., having digital audio data) that include the captured sounds (e.g., as one or more audio frames). In one aspect, the sensors and surgical devices may each be separate electronic devices (with respect to one another). In another aspect, at least some of the sensors may be a part of (e.g., housed within) one or more of the surgical devices (and/or other electronic devices that are capable of being communicatively coupled with the hub 20). For example, the proximity sensor 27a may be a part of the robotic arm 26b, and may allow the system to determine the position of the arm with respect to other objects (e.g., in order to avoid collisions).

The control device 25 may be any type of electronic device that is configured to interact with one or more other electronic devices of the surgical system 101. For example, the control device may be a handheld UID allows a user to interact with the surgical system (e.g., manipulating the robotic arm 26b). In another aspect, the control device 25 may be any type of computing device, such as a tablet computer that is arranged to provide control commands to the surgical system and receive surgical data for presentation (e.g., display on a touch-sensitive display screen) on the device 25.

In one aspect, the hub 20 may be able to communicatively couple to the surgical devices/sensors and/or the control device through various methods. For example, as shown, the hub 20 is communicatively coupled (e.g., establishing a communication connection) with the second surgical device 26b, the second sensor 27b, and the control device 25, via a network 22, in order to exchange data. In one aspect, devices may communicate via any (computer) network, such as a wide area network (WAN) (e.g., the Internet), a local area network (LAN), etc., through which the devices may exchange data between one another and/or may exchange data with one or more other electronic devices, such as a remote electronic server. In another aspect, the network may be a wireless network such as a wirelessly local area network (WLAN), a cellular network, etc., in order to exchange digital data. With respect to the cellular network, the hub (e.g., via a network interface) may be configured to establish a wireless (e.g., cellular) call, in which the cellular network may include one or more cell towers, which may be part of a communication network (e.g., a 4G Long Term Evolution (LTE) network) that supports data transmission (and/or voice calls) for electronic devices, such as mobile devices (e.g., smartphones). In another aspect, the devices may be configured to wirelessly exchange data via other networks, such as a Wireless Personal Area Network (WPAN) connection. For instance, the hub 20 may be configured to establish a wireless communication link (connection) with an element (e.g., the second surgical device 26b) via a wireless communication protocol (e.g., BLUETOOTH protocol or any other wireless communication protocol). During the established wireless connection, the electronic device may transmit data, such as sensor data as data packets (e.g., Internet Protocol (IP) packets) to the hub 20.

In another aspect, the hub 20 may be communicatively coupled via a wired connection, such as to the first surgical device 26a and the first sensor 27a. For example, the hub 20 may be connected to either of the devices through any type of wired connection, such as a fiber optic cable, an Ethernet cable, High-Definition Multimedia Interface (HDMI) cable, Digital Video Interface (DVI), Serial Digital Interface (SDI), etc. In some aspects, the hub device 20 may communicate with any of the devices using any type of communication protocol (e.g., Transmission Control Protocol (TCP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Data Distribution Service (DDS). In one aspect, the hub may communicate with one or more devices via an Application Programming Interface (API). For example, to communicate with the second surgical device 26b, via the Internet (through the network 22), the hub 20 may use Representational State Transfer (RESTful) APIs to exchange surgical data.

Each of the cameras 23 may be any type of camera (e.g., a complementary metal-oxide-semiconductor (CMOS image sensor) that is configured to capture video (and/or image) data as a video stream (e.g., that includes a series of still images, each image showing a scene within a field of view of the camera). In one aspect, the camera may be an endoscope that is designed to capture video of a surgical site within a body of a patient during a surgical procedure. In one aspect, the camera may be a monocular camera that (e.g., has a single camera sensor that) captures one digital image (e.g., a still image) at a time (e.g., to produce an endoscopic video stream). In another aspect, the camera may be a stereoscopic (stereo) camera with two (or more) lenses, each with a separate camera sensor for capturing individual still images (e.g., for producing separate video streams) in order to create 3D video. In one aspect, the camera may be a stand-alone camera (e.g., an electronic device with a primary function of capturing video data). In another aspect, the camera may be a part of an electronic device, such as a smartphone. In which case, the smartphone may be communicatively coupled (e.g., via a wireless network, such as a BLUETOOTH network), in order to transmit a captured video stream to the controller 30. In some aspects, video stream(s) received from the camera(s) 23 and/or the video stream(s) produced by the hub 20 may be high definition (HD) video that may include 10-bit 4K video such as, for example, of resolution 3840×2160 pixels (which is also referred to as 2160p), 1920×1080 pixels (also referred to as 1080p video), and 1280×720 pixels (also referred to as 720p video) with a frame rate of 59.94 and/or 60 image frames per second (fps).

The switch 21 is coupled to each of the cameras 23 to receive one or more video (and/or audio) streams produced by each camera, and is coupled to the hub 20 to receive one or more video (and/or audio) streams produced by the hub 20 (e.g., each stream having surgical data, as described herein). In one aspect, the switch may pass through one or more video streams produced by the cameras 23 to the hub 20. The switch is also coupled to the displays 24, and is configured to pass through one or more video streams from one or more cameras 23 and/or the hub 20 to one or more of the displays for display. In one aspect, the switch 21 may include one or more input devices (e.g., a physical button), which allow a user to manually define which streams pass through (into and/or out of) the switch 21. In another aspect, the switch 21 may be configured by an electronic device, such as the hub 20 to pass through one or more streams based on control signals received from the hub 20. In one aspect, the switch may be an optional device. In which case, the camera(s) and the display(s) may be coupled (e.g., wired and/or wirelessly) to the hub, which may be configured to receive video data from the cameras and then produce one or more video streams (which may at least partially include the video data captured by the cameras) for display on the displays 24.

In one aspect, the operations performed by the surgical system may be in real-time (e.g., during a surgical procedure). In which case, the hub 20 may exchange surgical data with one or more surgical devices, sensors, and/or control devices during a procedure. For example, the hub 20 may receive control signals from the control device 25 to manipulate the robotic arm 26b. As another example, the hub 20 may be configured to produce one or more video streams which may be presented on one or more displays 24. In another aspect, the hub 20 may present surgical data as being overlaid on top of (or superimposed above) video data captured by one or more cameras 23. For example, the hub 20 may receive video streams from the cameras 23 and surgical data produced by a surgical device and/or sensor. The hub 20 may superimpose above (overlay on top of) a video stream from a camera at least a portion of the surgical data (e.g., temperature readings from the temperature sensor 27b). In one aspect, to superimpose surgical data, the hub 20 may alpha blend (e.g., portions) of surgical data to a video stream. The alpha blended video stream may then be displayed on a display 24 to provide a user with a of a camera 23 along with desired surgical information (e.g., in real-time). In another aspect, the hub 20 may receive video streams and/or surgical data streams, and store the data (e.g., in local and/or remote storage locations) for later viewing.

As described herein, each of the devices (e.g., cameras 23, surgical devices 26a and 26b, sensors 27a and 27b, and/or control device 25) may produce one or more surgical data streams using their own individual system (internal) clocks. In which case, data received by the hub 20 from various devices may not be time synchronized accurately. Unsynchronized surgical data may result in the hub device 20 combining data inaccurately. For example, when the sensor 27a is a microphone, an audio stream produced by the microphone may have a time offset with respect to a video stream produced by a camera 23. As a result, when the audio stream is combined with the video stream, the audio may lag (or lead) the video, which is undesirable. Therefore, the surgical system 101 of the present disclosure is configured to time-synchronize surgical data streams received from the devices of the surgical system based on event detection. More about the operations performed by the surgical system is described herein.

FIG. 3 is a block diagram of the surgical system 101 that time synchronizes surgical data streams received from several devices based on event detection according to one aspect. The system includes one or more (e.g., electrical) components (or elements), such as the display 24, a speaker 31, a microphone 32, the camera 23, one or more surgical devices (and/or sensors), and a controller 30. In one aspect, the surgical device(s) (or sensor(s)) 28 may include one or more surgical devices and/or sensors, as described herein (e.g., devices 26a and 26b and/or sensors 27a and 27b). The system also includes two electronic devices 29a and 29b. In one aspect, at least one of the electronic devices 29a and 29b may be a part of the surgical system (e.g., device 29a being one of the displays 24). In another aspect, at least one of the electronic devices 29a and 29b may be a device that is separate from the surgical system. In particular, this separate device may not be communicatively coupled (e.g., via a wired and/or wireless connection) with one or more devices (e.g., the hub 20) of the system. In another aspect, the electronic devices 29a and 29b may be from different third-party manufacturers. In some aspects, the system may include fewer or more devices, such as having only one electronic device 29a or having many (e.g., “N”) surgical devices 28. In another aspect, the system 101 shown in this figure may include other elements that are not shown, such as having a control device 25.

The controller 30 may be any type of electronic component that is configurable to perform one or more computational operations. For example, the controller may be a special-purpose processor such as an application-specific integrated circuit (ASIC), a general purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller 30 is configured to perform time synchronization operations upon one or more surgical data streams based on event detection, as described herein. In one aspect, the controller 30 may be a part of an electronic device, such as the hub 20 of FIG. 2 (and/or the control tower 103 of FIG. 1).

The controller 30 is configured to receive one or more surgical data streams produced by one or more (e.g., surgical) devices (and/or sensors). For example, the controller is configured to receive (at least one) audio stream produced by the microphone 32, where the audio stream may include one or more (ambient) sounds captured by the microphone 32 from within the operating room. The controller may also be configured to receive (at least one) video stream produced by the camera 23 (e.g., inside an operating room, such as an endoscope, or a camera that is arranged to have a field of view of at least a portion of the operating room). In another aspect, the controller may be configured to receive one or more surgical data streams (N data streams) from one or more surgical devices and/or sensors 28. In one aspect, the controller may receive at least some of the data streams in real-time (e.g., receiving the video stream from the camera 23 as the camera is being used to perform an endoscopic surgical procedure). In another aspect, the controller may receive at least some of the surgical data streams from memory (e.g., memory 38) of the surgical system. Thus, at least some of the operations performed by the controller may be performed in real-time and/or performed “off-line”, meaning after the data streams were produced by their respective electronic devices and stored in memory. In which case, the surgical system may be configured to receive the one or more surgical data streams (e.g., from memory and/or from one or more devices), and store the streams in (e.g., local) memory. The (e.g., controller 30 of the) surgical system may be configured to perform at least some of the operations described herein upon at least some of the (locally) stored data streams (e.g., to time-synchronize the data streams). In another aspect, one or more of the surgical data streams may be received from a remote memory device, such as a remote server that is communicatively coupled (e.g., via the internet) to the controller 30.

The controller 30 includes several operational blocks that enable the controller to time-synchronize at least some data streams, as described herein. For example, the controller includes an event detector 33, a (e.g., internal) clock 35, a timestamp generator 36, and a time synchronizer 37. The controller 30 also includes memory 38. Examples of memory (e.g., non-transitory machine-readable storage medium) may include read-only memory, random-access memory, CD-ROMS, DVDs, magnetic tape, optical data storage devices, flash memory devices, and phase change memory. In one aspect, the memory may be a part of the controller. In another aspect, the memory may be separate from the controller (e.g., a part of another electronic device). In which case, the controller may be configured to communicate with the memory in order to perform operations, such as a read operation and a write operation.

As described herein, the controller 30 may be configured to time-synchronize surgical data streams based on event detection. In one aspect, the controller may perform synchronization operations responsive to detecting audio (or acoustic) events, which may occur within an operating room during a surgical procedure. In particular, the event detector 33 may be configured to receive one or more audio streams produced by the microphone 32, each of which may include ambient sounds captured within an ambient environment (e.g., the operating room), in which the microphone is located, and may be configured to detect an audio event (e.g., that occurs within the operating room) by performing an acoustic analysis upon the audio stream.

In one aspect, an audio event may be an occurrence in which an electronic device within the operating room produces one or more sounds. For example, as shown in this figure, the electronic device 29b may be a surgical monitor that is producing a sound through one or more (e.g., built-in) speakers (shown by waved lines being emitted from the device), such as a beep or a non-verbal sound. As another example, when the electronic device 29b is a heart monitor, the sound produced by the machine may be a reproduction of a patient's heartbeat. In another aspect, an audio event may be a non-speaker (e.g., mechanical) sound produced by a device within the room. In particular, the sound may be a (e.g., pre-defined or pre-known) sound produced by a piece of electronics within the operating room. For example, an audio event may be a sound of an actuator of a robotic arm actuating to move the robotic arm from one pose to another. As another example, a mechanical sound may be a sound of an ultrasonic instrument vibrating while in use by a surgeon. As yet another example, the sound may be a ticking sound of an analog clock that is hanging on an operating room's wall. In another aspect, the audio event may be a naturally occurring sound that occurs within the operating room. In particular, the sound may be an ambient sound that is present (or is known to be present) within an operating room. Examples of naturally occurring sounds may include sounds of movements, such as a sound of a door opening or closing, a sound of a gurney's wheels moving while the gurney is being moved about the operating room, a sound of a person walking or shuffling about, etc. Other naturally occurring sounds may include background (or ambient) sounds, such as a sound of traffic or a train passing by. In another aspect, an audio event may be any type of sound that may be captured by the microphone 32, contained within one or more audio streams produced by the microphone.

As described herein, the event detector 33 may be configured to detect an audio event by performing an acoustic analysis upon a received audio stream (e.g., produced by microphone 32). For example, the detector may be configured to compare at least a portion of the audio stream (or signal) with one or more predefined signals, each predefined signal associated with a particular audio event. In one embodiment, the predefined signals may be retrieved by the event detector from memory 38. In particular, each predefined signal may be different for each audio event, and thus, upon the received audio signal (at least partially) matching a predefined signal (e.g., up to at least a threshold), the event detector may select the associated audio event associated with the predefined signal.

In another aspect, the event detector may detect an audio event based on one or more spectrograms. For example, the event detector may transform one or more sounds contained within the audio stream into one or more spectrograms. In particular, the detector may produce (or generate) one or more spectrograms based on the audio stream, each of which represents one or more sounds from (at least a portion of) the stream as frequency content, in the frequency domain (e.g., which may be a function of time). In one aspect, to produce a spectrogram, the detector may perform a split sliding window (e.g., Fast Fourier Transform (FFT) spectral analysis) function, in which the detector performs the spectral analysis upon one or more audio (digital) samples (e.g., the number of samples being defined based on a window size) to produce a spectrogram of the samples. In one aspect, each spectrogram may spectrally represent the audio stream across one or more slices (or samples). In some aspects, a spectrogram may indicate a magnitude (e.g., a numerical value) of each frequency bin across one or more frequency bins. In some aspects, the spectrogram may be a Mel-frequency cepstrum, which is represented by one or more Mel-frequency cepstral coefficients (MFCCs).

In one aspect, the event detector may use one or more produced spectrograms associated with the audio stream to detect an audio event. In particular, the event detector 33 may apply one or more produced spectrograms (e.g., as input) into a (e.g., predefined) model, which may indicate an audio event (or more specifically an identification of a captured sound that represents an audio event) associated with the one or more spectrograms (as output). For example, the event detector may perform a table lookup into a data structure (e.g., stored in memory 38) that associates each of one or more spectrograms with an identification of (e.g., one or more sounds of) a respective (predefined) audio event. Upon identifying a matching (or at least partially matching, up to a threshold value) spectrogram in the data structure, the event detector identifies an audio event (represented by the produced spectrogram) associated with the matching spectrogram. In another aspect, the model may be an acoustic model that was produced in a controlled environment (e.g., a laboratory).

In another aspect, the event detector 33 may detect the audio event by applying at least a portion of the audio stream (e.g., one or more representative spectrograms) as input into a machine learning (ML) model, which produce an identification of the audio event associated with the spectrograms as output. In some aspects, the event detector may receive a ML model trained using training data that associates audio events that occur (e.g., within an operating room) with respective spectrograms. In particular, the ML model may be a (e.g., pre-trained) deep neural network (e.g., a convolution neural network) which uses one or more spectrograms as input (e.g., where the spectrogram matches an input size of the network), and classifies one or more sounds represented by the input spectrogram(s) as one or more audio events (as output). In another aspect, the ML model may be any type of pre-trained model for identifying audio events, such as a (e.g., linear) regression model, a decision tree, a random forest, etc. Thus, the event detector may receive the audio event as output from the ML model responsive to using a produced spectrogram based on at least a portion of a received audio stream as input into the MP model.

In some aspects, the event detector 33 may perform one or more audio signal processing operations upon the audio stream. For example, the captured audio stream may include ambient noise (e.g., a signal-to-noise ratio (SNR) that is above a threshold), and as a result, the event detector may produce a clean (less or no noise) signal, which includes one or more sounds. For instance, the detector may apply the audio stream to a noise suppression algorithm, which outputs a clean audio signal (stream) with little (e.g., below a SNR threshold) to no noise. As another example, the detector may apply one or more audio filters (e.g., a low-pass filter, etc.) to the audio stream (e.g., when the audio stream has wind noise (e.g., low-frequency noise), the detector may apply a high-pass filter).

In one aspect, sounds captured by the microphone may at least partially overlap one another (e.g., which may result in at least two sounds occurring at approximately the same time). In which case, the event detector may be configured to extract both sounds and determine which (either or both) of the sounds may be used to detect an audio event. For example, when two sounds occur, one sound may at least partially mask another sound. In which case, the event detector may determine that the masking sound be used, as opposed to the masked sound. To make this determination, the event detector may identify one or more sounds within the received audio stream. To do this, the event detector may perform a spectral analysis, such as blind source separation (BSS) upon the audio stream to identify one or more sound sources (e.g., as one or more audio signals) within the audio stream. The event detector may determine which of the sources to use based on one or more criteria. For instance, the detector may use the sound source with a lower (e.g., below a threshold) SNR. In another aspect, the detector may use a sound source that has a consistent (e.g., staying above a threshold) signal level (e.g., whether a sound fades in the background, which may be the case when a sound source is moving away from the microphone).

In another aspect, the event detector may determine which of one or more spectrograms to use to determine whether an audio event has occurred. For instance, for one or more sounds identified within the operating room, the event detector may produce at least one spectrogram. In which case, the event detector may choose a spectrogram (from several spectrograms), for detecting whether an audio event associated with the chosen spectrogram has occurred. In one aspect, the event detector may make this choice based on a confidence score produced for each spectrogram. In some aspects, the confidence score may be determined based on a spectral comparison between spectrograms (e.g., based on whether at least one frequency bin within one or more spectrograms has a magnitude above a threshold).

In some aspects, the event detector may detect multiple (one or more) events, and may determine which of the multiple detected events are used to optimize the surgical data streams (e.g., to time-synchronize the streams). For instance, the event detector may produce two separate spectrograms (e.g., a first spectrogram associated with a first portion of the audio stream and a second spectrogram associated with a second portion of the audio stream), and may use the spectrograms to detect separate audio events. As an example, the event detector may detect an audio event of a heart monitor beeping, and another audio event of an energy device being used by an operator during a surgical procedure. The event detector may determine which of the audio events to use based on a confidence score. Specifically, the event detector may determine a first confidence score for the first audio event and a second confidence score for the second audio event. In one aspect, a confidence score may be based on one or more characteristics of the audio events. For instance, a confidence score may be based on what type of audio event is detected, such as whether the event is a unique sound (e.g., a sound of a particular energy device), or whether the sound is an ambient background sound (e.g., a door closing, wind noise, etc.). In this case, the more the unique the audio event is, the higher the confidence score may be. In another aspect, the confidence score may be a running average across a period of time. For example, the event detector may determine that the sound is consistently present within the operating room, and therefore can be used to time-synchronize the streams with a higher level of accuracy. In one aspect, the audio event with the higher confidence score (e.g., over a period of time) may be used to time synchronize streams, as described herein. Returning to the previous example, the event detector may use the first audio event to time synchronize streams (e.g., to produce a timestamp) instead of the second audio event based on the first confidence score being greater than the second confidence score.

As described thus far, the (event detector 33) may be configured to detect events, such as audio events based on an analysis of one or more data streams. In another aspect, the event detector may be able to detect other types of events. For example, the event detector may be configured to detect an image event, based on an image analysis of one or more images (or image data) of one or more video streams captured by one or more cameras 23 of the surgical system. For example, the event detector may perform an object recognition algorithm upon a video stream to detect an object captured within a field of view of the camera 23. In one aspect, the object recognition algorithm may analyze the video stream to detect characteristics of an object, such as geometric shapes, color, etc. Once identified, the algorithm may apply the characteristics to a model that produces an identification of an image event as output. In another aspect, the detector may perform a table lookup into a data structure that associates image characteristics with an image event. As a result, once one or more matching characteristics are identified, the detector may select an associated image event from the table. Thus, in one example, when the electronic device 29a is a heart monitor that is displaying a visual representation of a user's heartbeat and the field of view of the camera 23 includes the display screen of the device 29a, as shown in this figure, the event detector 33 may detect an image event as the visual representation of the heartbeat shown on the device 29a.

In one aspect, image events may be based on a detection of objects within an operating room, such as medical equipment, a clock, etc. In another aspect, image events may be based on a detection of objects that are known to reoccurring within an operating room. Objects that periodically (and continuously) reoccur may be used to perform operations, such as time-synchronization operations. Examples may include lights (e.g., LEDs) that periodically flash on surgical equipment (to provide visual information and/or warnings to operators), images or graphical objects that are displayed on displays, etc. In some aspects, detectable objects may not be viewable by operators. For instance, the surgical system may include one or more infrared emitters that emit an infrared signal. Such a signal may be detected by an infrared detector (not shown) of the surgical device, which the event detector 33 may use to detect an image event.

As described herein, the event detector may use confidence scores between detected events in order to determine whether event is used to perform one or more operations, such as time-synchronization. In one aspect, the event detector may be configured to use surgical data from one or more surgical devices to validate a confidence score (which may thereby increase the confidence score). In particular, the controller 30 may be configured to validate that a particular audio event is occurring, which may then be used to perform one or more operations. For example, the event detector may detect an audio event of a surgical device outputting a sound, such as beeping sounds of a heart monitor. The event detector may be configured to determine whether an image event of a visual representation of the beeping sounds is occurring (e.g., contemporaneously with the beeping sounds) by monitoring a video stream captured by the camera 23. If so, the event detector may apply a higher confidence score to the audio event of the beeping sounds, and may use such an event to time-synchronize surgical data streams, as opposed to using another detected audio event (e.g., of a door closing), which may have a lower confidence score (e.g., since the event detector may not be able to validate that sound).

In one aspect, the ML model may be trained using a set of (e.g., predefined) training data, which may be derived in a controlled environment (e.g., laboratory). In particular, the training data may be created based on one or more audio (target) signals that are emitted by one or more (e.g., medical) electronic devices (e.g., during operation while use in a surgical procedure). For example, the audio signal (which may not vary over time) may be used by the medical device to convey surgical information and/or warning sounds (e.g., beeping noises of a heart monitor). In some aspects, the training data may include (e.g., spectrograms of) the audio signals that include known sounds (e.g., that occur within an operating room), and an indication of what the sounds are, and the data may be used (e.g., through supervised learning) to train the model.

In one aspect, the one or more target signals may be processed to enhance performance of the trained ML model. For example, signals may be modified (augmented) through various signal processing operations, such as adding noise (e.g., white noise, pink noise, brown noise, etc.), adding realistic ambient background noise from a surgical operating room, applying various signal-to-noise (SNR) ratios, adjusting signal levels (sound output level) by applying one or more scalar gains, applying one or more audio filters (e.g., low-pass filter, high-pass filter, etc.) to mimic acoustic room characteristics that could occur during an audio recording (e.g., while the microphone 32 captures sounds of) an operating room. In one aspect, different signal processing operations may be performed upon one target signal (e.g., sound of a heart monitor), to produce one or more versions of that signal, each version being a (e.g., slightly) different variation of the original signal. As a result, the training data can be expanded in a realistic manner (e.g., such that the sounds of the target signals are as if they were captured within an operating room, during a surgical procedure) in order to improve real-world application of the trained model. In one aspect, the training data may include (or may be derived from) the one or more audio target signals (e.g., as one or more audio files) have a same (or similar) track length (e.g., one or more seconds), ensuring that the targeted sound is fully contained from the start of the track to the end of the track.

In one aspect, the ML model may be pre-trained, such as by an electronic device that is remote to the surgical system 101 (e.g., by a cloud-based server). In another aspect, the ML model may be trained by one or more electronic devices (e.g., the hub 20) of the surgical system 101.

The clock 35 may be an internal clock of (e.g., a clock based on an electronic oscillator circuit integrated within the controller 30 of) the (e.g., hub 20 shown in FIG. 2 of the) surgical system 101 that is configured to produce a clock signal (data), which the controller 30 uses to perform timely signal processing operations. In one aspect, the clock 35 may be defined by the controller based on clock data retrieved from an external source, such as being based on an atomic time provided by a GPS satellite network). In another aspect, the clock may be defined according to timing messages received from one or more other devices, such as being set based on time data received from a Network Time Protocol (NTP) server.

The timestamp generator 36 generates (or produces) a timestamp based on a detected event by the event detector 33 and based on the internal clock 35 (of the controller 30). In particular, the generator may receive time data from the clock 35 and receive an indication from the event detector 33 that an event, such as an audio event, has been detected, and produce a timestamp, in response. For example, upon determining that an audio event has occurred, such as a beep from a heart monitor, the timestamp generator may produce a timestamp. In one aspect, the timestamp may be a time at which the audio event is detected by the surgical system. In another aspect, the timestamp may indicate a time associated with the clock 35 at which the audio event is captured by the surgical system. For example, the time may be a time at which the microphone 32 captured the sound of the audio event from within the ambient environment. This determination may be based on the spectral analysis performed by the event detector. For instance, as described herein, the event detector may detect audio events based on spectrograms that are produced using one or more audio samples of the audio stream. In this case, the timestamp produced by the generator may be associated (or correspond) to a time of the one or more audio samples (e.g., a time at which the audio samples are received from the microphone). Thus, a timestamp may indicate a moment in time along an internal clock 35 at which an audio event within an audio stream is received by the (e.g., controller 30 of the) surgical system 101.

The time synchronizer 37 receives the audio stream (produced by the microphone 32), the video stream (produced by the camera 23), and/or the N surgical data streams from the one or more surgical devices and/or sensors 28. The synchronizer receives the timestamp produced by the generator 36 and uses the timestamp to synchronize at least some of the received data streams. In particular, the synchronizer tags the timestamp (e.g., by adding the timestamp to metadata) to the N surgical data streams, the audio stream, and the video stream. In one aspect, the synchronizer may add the timestamp to a portion of each of the data streams that is associated with the time indicated by the timestamp. For example, if the timestamp is a time, such as 00:00:00, the synchronizer may add the timestamp to portions of a data stream (e.g., one or more samples of the audio stream) that is received by the surgical system at 00:00:00. In another aspect, the portions to which the timestamp are tagged may not be received at a time of the timestamp. As a result, the time synchronizer 37 may produce synchronized streams, which the controller 30 stores in memory 38.

In one aspect, in addition to (or in lieu of) storing the synchronized data streams, the controller 30 may present at least some of the surgical data through one or more output devices. For example, the controller may produce one or more video streams, each having at least some surgical data from one or more synchronized streams, and may display the produced video streams on the display 24. As another example, the controller may produce one or more audio streams, each stream including at least some surgical data for driving the speaker 31. In another aspect, the controller may “pass through” at least some of the data streams to the output device(s). For instance, the controller may display the synchronized video stream produced by camera 23 on the display 24. Thus, the controller may time-synchronize data streams, while displaying at least some of the surgical data received within the streams (e.g., during a surgical procedure).

As described thus far, the surgical system is able to detect events that occur within an operating room by analyzing one or more data streams, which may include surgical data associated with the operating room, such as an audio stream produced by the microphone 32 within the room. Such events may occur naturally within the operating room, such as being associated with mechanical sounds (e.g., actuator sounds while robotic arms move), or being associated with natural sounds, such as a sound of a door opening or closing. In one aspect, events may be the result of an affirmative action by the surgical system, such that a stimulus is produced by the system such that the stimulus is captured by one or more surgical data streams and is used as a detected event. In some aspects, the surgical system may include one or more output electronic devices, which may be configured by the system to produce an output that may be captured by the system (e.g., an image captured by the camera 23, a sound captured by the microphone 32, etc.). For example, the system may be communicatively coupled with a light emitting diode (LED), which may be a part of a surgical device 28. The system may transmit a control signal to the LED, such that it emits an optical signal, which may be captured by the camera 23, and used by the controller 30 to detect an image event. Such an event may be used to time-synchronize the surgical data streams, as described herein. As a result, the system may periodically cause the LED to produce the optical signal, such that new (updated) timestamps based on newly detected image events are continuously added to the surgical data streams in order to time-synchronize the streams over a period of time.

In one aspect, the surgical system may use event detection for time-synchronizing surgical data streams. In another aspect, the surgical system may perform other operations based on event detection. For example, the surgical system may use event detection to ensure that the hub 20 is in proper communication with surgical devices (e.g., during a procedure). As described herein, the hub may communicate with various devices in order to receive surgical data and/or to provide one or more commands, during a procedure. In some instances, communication between the hub and a device may be interrupted (e.g., the device being inadvertently unplugged, when both devices are connected via a wire). In which case, the surgical system may use event detection to confirm that a device and the hub are exchanging data. For example, the controller 30 may provide a control signal to a device to emit a sound, and the controller may determine whether an audio event occurs in which a speaker of the device produces the sound. If an audio event does not occur (e.g., within a period of time), the controller may present a notification (e.g., displaying a pop-up notification on the display 24), indicating that communication has been lost with the device.

In another aspect, event detection may be used to track progress of a surgical procedure, and to output a notification based on the progress. For example, the controller 30 may be configured to determine a surgical procedure of which surgical data is being received through one or more surgical data streams by the controller (e.g., based on user input, or based on the surgical data), and may be configured to determine whether a detected event corresponds to the surgical procedure. For instance, upon detecting a particular event (e.g., an audio event indicating that an operator is using an ultrasonic instrument), the controller may determine whether the surgical procedure requires the use of such an instrument and/or whether use of the instrument is to occur within a predefined schedule of the procedure. If not, the controller may be configured to output a notification indicating that an event has been detected that does not correspond with the surgical procedure.

FIGS. 4-7 are flowchart of processes for aspects of performing one or more digital signal operations (e.g., time-synchronization) upon one or more surgical data streams that include surgical data (e.g., captured from within an operating room during a surgical procedure) based on event detection. In one aspect, at least some of the operations may be performed in real-time, such as detecting audio events during a surgical procedure within an operating room using an audio stream captured by a microphone within the room. In another aspect, at least some of the operations may be performed postoperatively (e.g., based on audio and/or video captured during a surgical procedure, but after the procedure has been performed). In some aspects, at least some of the operations of at least some of the processes described herein may be performed by (e.g., the controller 30) of (e.g., the hub 20 of) the surgical system 101, described herein.

Turning to FIG. 4, this figure is a flowchart of a process 40 for an aspect of time synchronizing surgical data streams using detected events within an operating room. The process 40 begins by the controller 30 receiving (e.g., during a surgical procedure) one or more (e.g., audio, video, etc.) data streams from (e.g., produced during the surgical procedure by) one or more surgical devices of the surgical system within an operating room (at block 41). In one aspect, at least one of the data streams may include an audio stream. For example, the data streams may be received from one or more (e.g., surgical devices and/or sensors 28 of FIG. 3, one or more microphones 32, and/or one or more cameras 23. In one aspect, data streams may be received once the device is activated (or powered on). In another aspect, devices may provide data streams based on user input (e.g., which may be received through an input device that is communicatively coupled to the devices).

The controller 30 detects that an (e.g., audio) event has occurred during the surgical procedure and within the operating room based on one or more data streams (at block 42). For example, the controller 30 may perform an acoustic analysis of one or more audio streams to detect an audio event detected by one or more microphones of the system. The controller performs one or more digital signal operations based on the detected event (at block 43). For example, the controller may time-synchronize one or more data streams based on a detected event, such as a detected audio event. More about time-synchronizing streams based on a detected audio event is described herein.

In another aspect, the controller may perform a classification operation to classify data within one or more data streams based on a detected event. For example, upon detecting an audio event, such as a sound of a door closing, the controller may add a classifier (e.g., as metadata) into one or more data streams (e.g., an associated audio stream), in order to classify the sound. As another example, when the controller detects an object within a video stream (e.g., a blinking LED of a surgical device), the controller may add an object classifier to the video stream. In which case, the controller may add the classifier as metadata and/or may add a visual representation of the classifier. For example, the controller may overlay a visual representation of the object onto one or more video streams in which the object is identified. In another aspect, the controller may modify one or more data streams based on a detected event. For example, upon detecting speech within an audio stream, the controller may perform one or more audio signal processing operations upon the stream. As an example, the controller may apply one or more audio operations in order to mask (or remove) the speech from the audio stream (e.g., by applying one or more audio filters). Alternatively, the controller may perform one or more audio signal processing operations to enhance sounds of one or more audio events, such as enhancing sounds of a particular surgical device, such that a user may monitor the device during a postoperative review of a surgical procedure.

In another aspect, the controller may derive one or more statistics relating to surgical data of the data streams based on the event detection. For example, the controller may detect when an energy surgical device has been activated by detecting sounds caused by the device while active, and may output statistics relating to the activation of the device. For instance, the controller may output a duration during which the device was active. As another example, the controller may indicate periods of time during which the device is active and/or may indicate an average duration during which the device is active (e.g., across a duration of a surgical procedure). In other aspects, the controller may be configured to derive other statistics. In some aspects, the controller may add statistics to one or more data streams, as metadata. In another aspect, the controller may overlay at least some statistics on top of one or more video streams, such that a user may view the statistics along with video content within the stream.

The controller 30 stores the one or more data streams in memory 38 (at block 44). For example, as described herein, the controller may perform time-synchronizing operations upon one or more data streams, based on the detected events. Once synchronized, the controller may store the streams in memory (e.g., for later review by a user). In one embodiment, the stored streams may include at least some metadata determined by the controller 30, such as classifiers and/or statistics. The controller may optionally present at least one of the (e.g., time-synchronized) data streams during the surgical procedure (at block 45). As described herein, at least some of these operations may be performed during a surgical procedure. Thus, to present the streams, the controller may display one or more of the data streams on one or more displays 24.

FIG. 5 is a flowchart of a process 50 for an aspect of detecting an audio event that occurs within an operating room for performing time synchronization. In one aspect, at least some of the operations in this process may be performed by the controller