>FIG. 2. The beacon 400 may take the form of a staff-worn, retractable lanyard, a patient wristband, an embedded module in a device, or a mobile device running an application and configured to broadcast. Each beacon 400 has a BEACON_ID that is readable by the beacon detector 108, which may be associated with a person (e.g., a particular healthcare worker), a role (e.g., nurse), a device (e.g., a spirometer), or an object (e.g., an infusion pump). The beacons 400 also broadcast information other than the BEACON_ID to the beacon detector 108 and ultimately, the microprocessor 402. Such additional information may include accelerometry, button presses, specific function activation, hear rate, heart rate interval, galvanic skin responses, and breathing rate.

The base unit 100 continuously scans for advertising beacons 400 and publishes received packets with “BEACON_ID”, “BEACON_INDICATOR”, and “BEACON_ACTIVITY” to the feature collector 120. In one embodiment, the beacons 400 and the beacon detector 108 use Bluetooth Low Energy (BLE) and the base unit 100 is connected to a directional antenna aimed at a patient bed area 300 to detect BLE beacons 400, as shown in FIG. 4. In that embodiment, the BEACON_INDICATOR is the Received Signal Strength Indicator (RSSI) of the broadcasted packet and the BEACON_ACTIVITY is the motion state determined from the accelerometer of the beacon 400, which is encoded in the packet. In alternative embodiments, the beacon detector 108 and the beacons 400 use a different type of communication, such as passive RFID, Zigbee, ultra-wideband, infrared (e.g., IRDA), ultrasound, and light (e.g., with beacons as fiducial markers).

Referring now to FIGS. 5-6, there is shown a side view of the camera 106 field-of-view with pixel rows in circles (1-8) in FIG. 5 and a top view of the camera 106 field-of-view with pixel rows (1-8) and columns (A-P) in FIG. 6. In the depicted embodiments, the camera 106 of the base unit 100 is shown on the wall above a patient's bed 300. The camera 106 is angled downward such that the head of the bed to an area past the foot of the bed is in the field-of-view of the camera 106. As subjects and non-subjects enter and exit the field of view of the camera 106, the camera 106 captures images, processes the raw sensor data, and publishes the data to the feature collector 120. In one embodiment, the camera 106 is an array of four Grid-Eye far-field infrared sensors that capture 16 by 16 pixels of temperature information over a 120 degree by 120 degree field of view, up to 10 frames per second. In the same embodiment, 10 frames per second is averaged down to 2 frames per second with output being: 16×16 average, 16×16 variance. In alternative embodiments, the camera 106 is a sensor, such as a stereoscopic IR camera (e.g., Intel RealSense), LIDAR, time of flight, radar, and passive infrared (e.g., motion detector).

Referring back to FIG. 1, the feature collector 120 aggregates and analyzes data received from the detection components of the base unit 100, such as the camera 116, the beacon detector 108, the sensors 112, and the call bell detector 118, for example. The feature collector 120 derives informative, non-redundant values (features) from the data and transmits the features to a scene classifier 202 on the cloud platform 200 and the feature database 504. Referring now to FIGS. 16-19, there are shown flowcharts for the method for detecting and determining the positioning of objects in a room. The depicted embodiment of the method utilizes a beacon (BLE location) algorithm and a camera objection detection (camera location) algorithm. The BLE location algorithm is based on exceeding a threshold to trigger the “enter” (entrance into a room) and a running average falling below the same threshold to trigger the exit (exit from a room). In one embodiment, the camera location algorithm uses the known position and orientation of the camera's field-of-view to infer the radial distance of clusters of pixels from the camera. However, other embodiments and techniques are contemplated, such as those from open source libraries like OpenCV, for example. The embodiment depicted in FIGS. 16-19 is described in terms of input and outputs for the BLE location algorithm and the camera location algorithm.

At a first stage shown in FIG. 16, there is a low level classifier per base unit 100, which may be embedded in the base unit 100 for optimization. For each time step, a BLE location algorithm inputs an array of [Time, BEACON_ID, BEACON_INDICATOR, BEACON_ACTIVITY] and outputs an array of [Time, BEACON_ID, BEACON_INDICATOR, BEACON_ACTIVITY, State: In or Out]. The camera location algorithm inputs an array of [Time, CAMERA_AVGVAL_MATRIX, CAMERA_VARIANCE_MATRIX] and outputs an array of [Time, CAMERA_OBJ_ID, CAMERA_OBJ_COORD {Row 1-8/Column A-P}, CAMERA_OBJ_VARIANCE]. In one embodiment, the algorithm uses the knowledge of the base unit 100 height and camera 106 field-of-view orientation to determine the object coordinates based on the assumption that the objects are people touching the floor if the person is staff member, or on a raised bed if the person is a patient in center of the field-of-view.

At a second stage shown in FIG. 17A-17B, there is a high level classifier per base unit 100, which may also be embedded in the base unit. In one embodiment, shown in FIG. 17A, the BLE location algorithm is constrained by information from the camera location algorithm. First, the BLE location algorithm outputs enter/exits based on transition from out-to-in and in-to-out states. Second, the BLE location algorithm prevents out-to-in transitions when camera location algorithm does not detect a change in the number of objects present. The BLE location algorithm also prevents in-to-out transitions when camera location algorithm does not detect a change in the number of objects present.

In an alternative embodiment shown in FIG. 17B, the low level classifiers are fused using graph matching, i.e., the BEACON_ID is matched to the CAMERA_OBJ_ID for each time slice. First, a bipartite graph with BEACON_ID nodes and CAMERA_OBJ_ID nodes is created. Next, a source connects to each BEACON_ID node and a drain connects to each CAMERA_OBJ_ID node. Then, the edges between BEACON_ID and CAMERA_OBJ_ID nodes are weighted based on the “distance” between {BEACON_INDICATOR, BEACON_ACTIVITY} and {CAMERA_OBJ_COORD, CAMERA_OBJ_VARIANCE}. For example, high RSSI and low object row coordinates would be close, where low RSSI and high row coordinates would be close. Also, a “moving” beacon 400 activity state would be close to high camera object variance, and a “stationary” beacon 400 would be close to low camera object variance. Thereafter, all the edges with a weight below an uncertainty cutoff point are pruned. Next, the max-flow algorithm is solved to get a mapping of BEACON_ID to CAMERA_OBJ_ID. An exemplary bipartite graph formulated for the max-flow algorithm is shown in FIG. 18. Finally, for each time step per base unit 100, there is an output array of [Time, BEACON_ID, BEACON_INDICATOR, BEACON_ACTIVITY, CAMERA_OBJ_ID, CAMBERA_OBJ_COORD, CAMERA_OBJ_VARIANCE, CONFIDENCE_VALUE], where the CONFIDENCE_VALUE is derived from the “distance” of the match.

According to another embodiment, the low level classifiers are fused using probabilistic graphical model. A neural network is used to perform multi-modal sensor fusion with an input of [Beacon, Camera] and an output of [known coordinates as measured from reference system]. At the conclusion of the second stage, in all of the embodiments described above, the system 10 outputs an array of [Time, BEACON_ID, State: In or Out, Activity: Moving or Stationary, Coordinate, Confidence Value 0-1] for each base unit 100 (shown in FIG. 19).

At the third stage, the cloud platform 200 connected to the base unit 100 disambiguates input from the high level classifier for each base unit 100, as shown in FIG. 19. Here, a current state for each BEACON_ID across all base units 100 is determined. In one embodiment, the system 10 may transition a BEACON_ID to new “in” states if they occur at a configurable number of seconds. In another embodiment, the system 10 may choose a base unit 100 with a higher confidence value, if available, for “in” state. In another embodiment, the system may use a probabilistic model such as a Hidden Markov model to determine the current in or out state of a BEACON_ID at each base unit using a configurable number of historical states with the model with the maximum posterior likelihood across all base units being determined to be correct. Next, the output is an array of [Time, Base Unit ID, BEACON_ID, State: In or Out, Activity: Moving or Stationary, Coordinate, Confidence Value 0-1 and the event stream of [Time, Base Station ID, BEACON_ID, Event: Enter or Exit].

At the final stage, there is activity identification, as shown in FIG. 20. The system 10 inputs the disambiguation output from the cloud platform 200. The system 10 then classifies and outputs activities based on the mix of people and devices used, which is based on a lookup from BEACON_ID, and the path, which is based on coordinates over time through the base unit 100's field of view for the same BEACON_ID. Such activities include staff activity, patient activity, and patient with staff activity. For example, staff activity may include hourly rounding, bedside reporting, and grand rounds. Patient activity may include activities such as using a spirometer, ambulating, and lying in bed. In another example, patient with staff activity may include instances where healthcare providers are interacting with patients, such as rotating the patient for pressure ulcers, assisting patients with restroom use, and feeding the patient.

Referring back to FIG. 1, the feedback engine 122 subscribes to the scene classifier 202 and publishes commands to the display 114 or speaker 116 when location events and/or activities occur (or do not occur). Such location events and/or activities may correspond to feedback rules 124, which signal the feedback engine 122 to publish commands. The location events and/or activities may occur when the patient has not been visited recently, a rounding goal has not been satisfied, a bedside reporting goal has not been met, a patient is in danger of falling, a patient has requested help, staff is present after a patient requested help, and a patient has not been rotated or turned, for example. When a command publishes from the feedback engine 122 to the display 114, the command may manifest as a color pattern. For example, if the patient has not been visited recently, the display 114 may show an orange light pattern or if a rounding goal has not been satisfied, the display 114 may show a green light pattern. As the display 114 is integrated into the front housing of the base unit 100, the commands (i.e., light patterns) may be seen by healthcare providers, alerting them of a patient's current status. As previously stated, the commands may also be published to a speaker 116.

From the scene classifier 202, the activities are analyzed in an analytics module 206 using algorithms stored in the analytics database 208. Exemplary key metrics that are calculated in the analytics module include the frequency of visits, the duration of visits, an hourly rounding rate, a bedside reporting rate, a patient turning rate, and patient ambulation rate. These key metrics can be broken down and calculated for each staff role, patient, unit, department, day of the week, or time of day. After analysis, the resulting data is transmitted to the reporting API module 210, which uses a publish-subscribe mechanism such as MQTT, for example, where data from a reporting database 212 determines how the data will be reported to a mobile application 214 or web application 216. Once the data is transmitted to a mobile application 214 or web application 216, such as a restful HTTP interface, a user can view the data at the interface.

Still referring to FIG. 1, the cloud platform 200 also comprises a scene classifier trainer 500. The scene classifier trainer 500 comprises a label database 502 and a feature database 504. Feature data is recorded to the feature database 504 in a more highly compressed state than raw signal data to remove certain privacy concerns. Labeled data stored in the label database 502 includes manually observed information 600b, such as system trainer entries and patient feedback entries. Labeled data also comprises electronic record data 600a, such as electronic medical records (EMR), electronic healthcare records (EHR), hospital information systems (HIS), unstructured notes made by staff, workflow data entered by staff, and coding done by hospital billing specialists. Finally, labeled data may also include any device-generated data, such as data from medical devices.

The scene classifier trainer 500 then uses data from the label database 502 and the feature database 504 in model training 506 to create and store scene models 508. A scene model may include topic models, such as hierarchical latent Dirichlet allocation (LDA), Hidden Markov models, and neural networks. Model training 506 includes machine learning methods, such as back propagation, feed forward propagation, gradient descent, and deep learning. Therefore, future features from the feature extractor 120 may be more quickly classified into the appropriate scene using a scene model 508 created using the same or similar past features.

With reference to FIGS. 7A-11D, perspective views of the base unit 100 and skeletal models derived from the positioning system 10 are shown. Referring first to FIGS. 7A-7B, there is shown a front perspective view of the base unit and a side perspective view of the base unit 100, respectively. In the depicted embodiment, the base unit 100 is stadium-shaped with a curved first surface 2 and a flat second surface 4. The first surface 2 comprises a crescent shaped aperture 6. A camera, such as a Panasonic Grid-Eye, is connected within the front-bottom area of the base unit 100, such that the field-of-view extends outward from the base unit 100 to the environment. Referring now to