The ELR systems and methods may use laser encoded or laser spectral imaging for white light imaging as it is smaller and provides high depth of visibility. The ELR systems may include X-ray probes or capsules. For example, an X-ray probe may include a source and a receiver on the tip of the probe and may use reflectance to perform the imaging. The ELR systems may use laser-driven ultrasound (US) without contact and without using the photoacoustic effect. The ELR systems may be configured to perform hyperspectral imaging. The ELR systems may obtain color gradient information from white light imaging and translate the color gradient information into depth perception information in a manner similar to how nurses check for veins before making an intravenous (IV) connection.

Endoluminally suturing may be challenging and may have associated risks. The ELR systems and methods of this disclosure address these potential challenges and risks by projecting a needle path and alerting surgeons when there is a needle slip through haptic feedback. The ELR systems and methods may project a needle path to assist surgeons, for example, with manipulating needles or performing suturing. The needle entry or exit point may be projected to guide surgeons to adjust as tissue moves during a procedure, for instance, when tissue is grasped. For example, when suturing, the needle projection can direct the position of the needle to a desired location. Electromagnetic (EM) tracking may be used to track the needle in real-time in a manner similar to catheter navigation.

In aspects, the ELR systems and methods may account for needle slip in real-time and may adjust the needle trajectory accordingly. The location and orientation of the needle in the driver tool determines where the needle will pass through the patient anatomy. A poorly aimed suture needle may not produce effective sutures and may pass through a critical structure or snare objects into the suture that should not be snared, possibly causing future patient complications. The ELR methods of this disclosure may include 3D imaging of the suture needle and driver tool, which simplifies and/or provides an additional method for slip detection. Optical detection of needle slippage may be performed using one or more imaging devices, e.g., white light cameras, and image processing software. The goal of the image processing software is to detect independent motion of the suture needle or the driver tool. The suture needle and driver tool are expected to move as one unit when no slippage occurs. Additionally, the suture needle and driver tool should move with the same angular motion. Using an image processing software with object identification, such as a convolutional neural network (CNN), the driver tool and the suture needle can be detected in each frame of video. Points on the tool and the needle are selected for motion measurement. When no slippage is occurring, all points should move with the same direction and velocity.

Provision of additional cameras allows an increase in detail and detection of slippage. Tissue tension data, which may be obtained through imaging, machine learning or other suitable tissue sensing techniques, may be used to predict, alert, and/or compensate for needle slip. The ELR system may provide haptic feedback, e.g., resistance, in response to detected needle slip, to guide the surgeon to the desired path, but still allow the surgeon to have control. The ELR system may provide vibrations when needle slip is detected.

Visualization and tracking in endoluminal procedures may be a challenge. The ELR systems and methods of this disclosure may use indocyanine green (ICG) for high-definition imaging to improve visualization and tracking. According to aspects of this disclosure, Near-Infrared (NIR) imaging with markers such as ICG (as illustrated in FIGS. 5A-5C) or cancer detection markers, such as Surgilab markers, along with white-light imaging may be used to guide ELR procedures. Electromagnetic tracking may also be used to better guide endoluminal procedures. The ELR systems and methods of this disclosure may use NIR imaging along with white light imaging in ELR to: track the direction of blood flow, combine laparoscopic and endoluminal tracking of ELR catheters, perform chip-based NIR for cancer detection using markers such as the Surgilab molecule (TR).

The ELR systems and methods of this disclosure may incorporate ELR catheter navigation including electromagnetic (EM) tracking and retraction. The catheter navigation may include 3D tracking of instruments relative to certain point (e.g., tracker on the catheter). The ELR catheter or instrument tip path may be tracked using electromagnetic sensing. The electromagnetic sensing may be performed by a subsystem that uses a drape or sheet with transmitters at different locations and a receiver or sensor disposed at the tip of the ELR catheter or instrument. Based on a signal received by the sensor, the tip of the ELR catheter or instrument may be tracked as a function of the patient's bed.

Using the information about the path and optionally using pre-operative or peri-operative imaging with or without machine learning, the ELR system may predict and automate driving catheter forward or retracting the ELR catheter or instrument back. The ELR controls may be similar to a gas pedal, brake pedal, or reverse gear features in a typical car.

Anatomical parts may be marked with temporary metal clips for tracking functions, e.g., for tracking a catheter in a ureter. Specific distances or margins may be detected by suitable sensors. The ELR system may issue an alarm when a robotic arm or instrument is within proximity of the markers, which may be associated with critical structures.

The ELR systems and methods of this disclosure may use transient images or video to reconstruct a 3D image or model of anatomy. Imaging from one or more imaging modalities may be used to reconstruct 3D anatomy and track the ELR catheter or instrument for helping surgeons visualize real-time navigation. Virtual colonoscopy or bronchoscopy images may be taken and computer vision may be used to reconstruct a 3D model of the anatomy, e.g., an organ. The reconstructed 3D model may be fit with what the clinician sees in the camera. The reconstructed 3D model may be updated as the procedure is progressing. In aspects, CT or fluoroscopic imaging may be used for 3D reconstruction to see where the ELR catheter, tool, or instrument is located and/or as an overlay for real-time guidance.

The ELR systems and methods may use multiple cameras views that are stitched together to give a wider variety of views.

The ELR systems and methods of this disclosure may incorporate visualization features and functions various types of cameras at various locations to improve white light image quality or usefulness. Cameras may be placed on the front tip, the side, and/or facing the back of the face of the ELR catheter or instrument to capture more views. The views may be reconstructed to get a better field of view or to get a 3D map with depth perception. Cameras may be attached on the catheter or instrument tip. The cameras may pop-up or be located on the side of the catheter or instrument. The cameras may be implemented by a chip-on-tip, which may incorporate near infrared (NIR) imaging capabilities; a fiber optic component, or any other lens system suitable for placement on the catheter or instrument tip. Micro-Electro-Mechanical Systems (MEMS) technology may be used to build multiple lenses into the ELR system that can be turned on or off to get different fields of view.

The ELR system may include a capsule endoscopy device, e.g., a PillCam, which looks forward and backward using a front camera and a rear camera, respectively. The two cameras may provide proximal and distal views of the pathology. This may be used for retrograde resection of the pathology. Alternatively, a front-facing camera and a back-facing camera may be coupled to the endoscope or the catheter to obtain images including forward and backward views. Images including the two camera views may be obtained during a preoperative endoluminal survey. Then, images including the two camera views may be obtained at a later time (e.g., intraoperatively) and compared to the images including the two camera views of the preoperative endoluminal survey. In one aspect, the images of the two intraoperative camera views may be overlaid on the two preoperative camera views, respectively, and displayed to a user on a display.

In aspects, a 3D model may be reconstructed using images from the capsule endoscopy device. Landmarks may be identified and frame-by-frame stitching may be performed. The stitching may include stitching progressive views captured by an optical sensor together into a continuous view of the anatomy as the capsule endoscopy device passes by.

Magnets may be placed outside a patient's body to fix cameras or LEDs along an anatomical wall, e.g., the abdominal wall, during procedures. The LEDs may have different wavelengths to provide differentiated visibility.

The cameras may provide an orthogonal side view with articulation. In aspects, a wider field of view may be mapped with two imaging devices, e.g., cameras, coupled to the endoscope. For example, one of the imaging devices may be coupled to the side of the endoscope. A 2D view, a 3D view, or some other adaptation may be presented. The ELR systems and methods may include a Google street view-like feature.

The ELR systems may include different catheters. The different catheters may include a catheter for white light and a catheter for a camera. A camera may be placed on each instrument, e.g., in a multi-articulated system. Structure of light, time of flight, and stereoscopic views may be used for 3D mapping.

The ELR system may incorporate endomicroscopy. This may be used for biopsy. For example, all or a portion of an endomicroscope may be inserted or otherwise incorporated into the biopsy needle.

The ELR system may include multiple cameras splitting a common fiber optic channel connecting to different sensors at the distal end portion.

In aspects, suction may be used to pull tissue or provide tissue tension. The suction may suck tissue into the ELR system to isolate the tissue or organ of interest. This feature may be used in conjunction with endomicroscopy. The ELR system may include a suction channel, which sucks tissue into a tip before using other arms (in a multi-arm articulated ELR system for large bore applications) to perform tasks, such as suturing.

The ELR systems and methods of this disclosure may stabilize cameras, visibility, and/or instruments using, for example, air pressure or insufflation. Localized pressure may provide for easier advancement of a catheter through a lumen. The ELR systems may include a balloon catheter, which uses balloons for locomotion or articulation of the catheter. A proximal balloon and a distal balloon may be used to stretch out the area of tissue. The ELR systems and methods may insufflate between the proximal and distal balloons to open up a lumen.

The distal tip of the catheter may not necessarily be used to provide stabilization. One or more instruments may come out of the side of the catheter at a more proximal location and, in this configuration, may provide stabilization. Balloons proximal and/or distal to the catheter or instrument opening may be used to provide stabilization. The ELR systems and methods may provide image stabilization functionality. Vibrations of the ELR catheter or instrument tip may be sensed by piezo sensors or by image movement detection. Image movements may be compensated for by adding mechanical damper to the ELR catheter or instrument tip or by video processing such as smoothing. The ELR systems may provide the ability to lock cameras or instruments at a desired location. This may be especially applicable to catheters or instruments that are out of view. The ELR systems may issue an alert to the surgeon if the cameras or instruments are forced to move.

The ELR systems may include a Nitinol (or other memory active materials that can change properties based on Electromagnetic or thermal fields) frame with barbed-like fixation (e.g., a barbed stent-like fixation) to come out of the working channel to hold tissue in tension while it is resected or operated on, as illustrated in FIG. 6. The ELR systems may include a stiffening endoscope, which may be implemented by a pully wire system to pull components together. The stiffening endoscope may use material that changes with different electric or thermal fields (e.g., memory-active materials that change in the presence of magnetic, electric, or thermal fields). The stiffening endoscope may use a garden hose-like design that stiffens when filled with a fluid, e.g., a saline solution. An ELR channel may be filled once the endoscope reaches target anatomy so that the endoscope can stiffen up to provide better stabilization. The endoscope may retract when the fluid leaves the ELR channel. The ELR systems and methods may use magnets outside the body to stabilize and/or drive the distal tip of an ELR catheter or instrument. The ELR systems may include a rolling sleeve to open up or advance along a lumen (similar to water tube toys, snakes, or water wigglies) to roll up to drive the catheter forward. This may also provide stabilization.

The ELR systems and methods may include manometry or a balloon catheter. A balloon catheter with saline may be used to get a pressure mapping. Manometry may be used to detect peristalsis. When peristalsis is detected, the procedure may be paused until a wave has passed. Manometry may be used to detect stricture or dilation. A balloon catheter may be used to open up a lumen. Manometry may be used to check for suture line defects. Manometry may be performed along the length of the endoscope. This may be used to help with navigation.

The ELR systems and methods of the disclosure may provide for platform compatibility. The ELR systems may integrate with robotic-assisted surgery (RAS) systems. The ELR systems may use RAS system design and components except for a single port (multi-arm multi-articulated) system that goes in the place of a trocar holder.

In aspects, the ELR systems and methods may be used in a Combined Endoscopic Laparoscopic Surgery (CELS) application. The combined endoluminal robotic approach allows for the use of an endoluminal and general surgery robot. Tracking where all instruments are located using both ELR and RAS systems may lead to key surgical steps.

The combined ELR and RAS system may allow for switching between laparoscopic and endoscopic views. In other words, the combined ELR and RAS approach may allow for switching views between ELR and RAS. The views could be changed both manually, e.g., via a toggle switch at a common surgeon console, or automatically, e.g., via tracking the stage in the procedure and automatically changing the view based on the stage in the procedure. For example, the combined system may switch between ELR and RAS views during a suturing procedure. In one implementation, while doing inside to outside suturing, the needle tip may be tracked and the system may automatically switch views based on the position of the needle tip, e.g., based on whether the needle tip is inside or outside. In aspects, whenever ELR and RAS tools need to interact, e.g., while performing full thickness suturing or applying tension to tissue, the tools may be tracked to perform this functionality.

The coordination of tools during a CELS procedure may involve the following features. During CELS, for example, a team of surgeons may coordinate use of laparoscopic instruments used in visually-aided, minimally-invasive surgery with endoluminal tools for polyp and cancer resection. The combination of the tools allows the surgical team to better identify the location of the surgical site while providing alternate options for preceding with the procedure based on what is learned in surveying the surgical site. For example, a surgeon may recognize that a polyp is not removable via a colonoscope due to local anatomy such that a laparoscope is required to complete the procedure.

This disclosure describes methods to recognize and relay the positional orientation between endoscopic and laparoscopic tools in real-time. Electro-navigation is a method used to tracking devices in the body and is common practice for Electro-Navigation Bronchoscopy (ENB). The bronchoscope or biopsy tools can have an electromagnetic sensor attached to the distal tip while the patient lays on an electromagnetic field generator. Detection of distortions in the field correlate to a position of the sensor in 3D space. By placing such sensors on each tool used in the CELS procedure, one may track the positional orientation of each tool. With a reference sensor placed in a known anatomic location or using a calibration procedure based on known anatomic structures, the 3D positions may be referenced and overlayed on the patient's CT or fluoroscopic image to show relation to anatomy.

An alternative to electro-navigation is use of radio signals. In this embodiment, the sensor board is replaced with a phase array antenna and each surgical device sensor is replaced with a low power transmitter. Each device generates a uniquely coded signal that is specially located by measurement of phase and angle by the antenna array. The transmitter may be split axially such that two halves of the instrument distal tip produce opposite phase signals to allow recognition of rotational orientation. An additional transmitter further proximal on the instrument would afford recognition of instrument angle.

Another aspect is placement of an infrared (IR) laser source in the endoluminal distal tip projecting out axial to the body of the endoluminal catheter. With the catheter distal tip pointed at the desired surgical site, IR penetration through the colon would be sufficient to allow detection of the IR light on the abdominal side of the colon wall, allowing a laparoscopic tool with IR detection to identify the IR source and ‘home in’ on the location. This provides the surgeon with directional information in locating the intended surgical site. Using the diffusion pattern of the IR light in the colon wall would allow the laparoscopic tool to identify the brightest location, correlating to the center of the IR beam.

Additionally, using a method to detect orientation of the distal tip of the endoluminal catheter in relation to the patient may inform the surgeon as to the location of the surgical site in relation to the patient (e.g., it is supine or prone, left or right). Methods to detect orientation of the distal tip of the endoluminal catheter include using two electromagnetic sensors spaced sufficiently apart to provide distinct 3D coordinates, or using an inertial measurement unit to detect distal tip orientation in relation to gravity. If laparoscopic tool rotation and angle information is available, the one or more tools may be presented in the view with navigation cues.

FIG. 7 is a block diagram that illustrates a robotic surgical system 700 in accordance with aspects of this disclosure. The robotic surgical system 700 includes a first robotic arm 702 and a second robotic arm 704 attached to robotic arm bases 706 and 708, respectively. The first robotic arm 702 and the second robotic arm 704 include a first end effector 716 and a second end effector 718, respectively. The end effectors 716, 718 may include robotic manipulators or grippers suitable for operating the endoscopic catheters and tools of this disclosure. The first end effector 716 operates one or more tools 712, including a suture needle driver tool, a grasping tool, and/or a flexible endoscope (not shown). The second end effector 718 operates a sheath or catheter 710, which may include one or more channels for receiving and guiding the one or more tools 712. The robotic surgical system 700 may further include an electromagnetic (EM) generator 714, which is configured to generate an EM field, which is sensed by an EM sensor incorporated into or disposed on the suture needle. Additionally, or alternatively, the EM sensor may be incorporated into or disposed on a distal end portion of the driver tool. The EM sensor may output sensor measurement data, which may be used to determine the position and orientation of the suture needle. In aspects, the EM generator 714 may be embedded in the operating table 715 or may be incorporated into a pad that may be placed between the operating table 715 and the patient 711.

The first and second robotic arms 702, 704 may be controlled to align the end effectors 716 and 718 such that proximal end portion of the catheter 710 is distal to the proximal end portions of the one or more tools 712, and such that the one or more tools 712 remain axially aligned with catheter 710.

In one aspect, the first robotic arm 702 inserts the catheter 710 through, for example, a tracheal tube (not shown) in the mouth of the patient 711, and into the bronchial system of the patient 711. Then, the second robotic arm 704 inserts the one or more tools 712 through the catheter 110 to reach a target within the bronchial system of the patient 711. The first and second robotic arms 702, 704 may move the catheter 710 and one or more tools 712 axially relative to each other and into or out of the patient 711 under the control of a surgeon (not shown) at a control console (not shown).

A navigation phase may include advancing catheter 710 along with the one or more tools 712 into the patient 711, and then advancing the one or more tools 712 beyond the distal end of the catheter 710 to reach a desired destination such as a target. Other modes of navigation may be used, such as by using a guide wire through a working channel of the catheter 710. The surgeon may use a visual guidance modality or a combination of visual guidance modalities to aid in navigation and performing the suturing procedures, such as fluoroscopy, video, computed tomography (CT), or magnetic resonance imaging (MRI). In aspects, the one or more tools 712 are deployed through longitudinally-aligned working channels within the catheter 710 to perform a suturing procedure and any other desired procedures. In aspects, the robotic arms 702, 704 include three joints 701 and three arm segments 705. In other aspect, the robotic arms 702, 704 may include greater than or less than three joints 701 and three arm segments 705.

FIG. 8 is a block diagram that illustrates a robotic control system 800 for controlling the robotic surgical system 700 of FIG. 7. The robotic control system 800 includes a control system 810, which controls the robotic surgical system 700. For example, the control system 810 may execute the method 1000 of FIG. 10. The control system 810 may interface with a display 822, a user controller 825, an endoscopic camera 826, and an insufflation system 828. The control system 810 may be coupled to the robotic surgical system 700, directly or indirectly, e.g., by wireless communication. The control system 810 includes a processor 812, a memory 814 coupled to the processor 812, a random access memory (RAM) 816 coupled to the processor 812, and a communications interface 818 coupled to the processor 812. The processor 812 may include one or more hardware processors. The control system