Investigation into the specific aims of some embodiments can generate numerous short-term to long-term benefits, including: (1) in the clinic, via investigating tools for nutritionists to better tracking participant intake, (2) in the home, as a method of users to track macronutrients for personalized health/medicine, and (3) in the lab, via support of bio-sensing technologies that can provide long-term biological read-out from implantable and wearable technologies. In the long-term, the platform can be integrated with additional biosensors to co-track pill or supplement ingestion, hormonal biomarkers (such as cortisol), and more. Such technologies can combine with other wearable devices (e.g., Fitbits and more) to better understand and optimize bodily response to diet and disease, and provide quantitative, closed loop information on the input-output response of the body.

d. Testing Data

Successful demonstration of implementations of each sensor class is made in-vivo on human subjects. These include an interlayer-RFid that can be utilized to detect certain nutrients from within the mouth, and a biocompatible strain gauge that operates within the working physiological motion ranges of the human body. In addition, initial studies are conducted on utilizing machine learning to fuse data from disparate sensor inputs along the throat, in-vivo.

d.1 Interlayer-RFid

An underlying technology for some embodiments is a bio-functional sensor termed an interlayer-RFid. Within the Testing Data here, a strategy to improve the biosensing characteristics of dielectric sensors is demonstrated by utilizing stacked and resonant-coupled split-ring resonators (SRRs) (FIG. 5, broadside-coupled SRRs). Of note, a functionalized “smart” interlayer that absorbs and swells in response to biomolecules intercedes these two split rings. Due to this structure, the sensor quality factor and resonant frequency are highly dependent on this interlayer material. Such a construct allows: (a) detection of changes in interlayer thickness, (b) temporal resonant behavior due to the interaction of biomolecules with the interlayer material, and (c) increased sensitivity due to the concentration of electric field through the interlayer. When this interlayer is composed of a biopolymer—in this case fibroin (a hygroscopic material that swells to variable thicknesses in different solutions), this structure displays unusual sensitivity to a variety of macronutrients. This construct is used to discriminate nutrients/foods such as glucose (apple juice), salt (soup), water, and alcohol (mouthwash) in-situ from within the mouth (FIG. 5).

d.1.1 Sensor Performance

Bio-interlayer-RFids display responses to nutrients that allow discrimination of intake. While other dielectric sensors exhibit constrained frequency behavior when switched from water to biofluids, this is not the case for the interlayer-RFid. This is because properties of the interlayer material (such as charge, hydrophilicity, or oleophilicity) exhibit interactions with biomolecules by swelling or squeezing in response (FIG. 6). Within the presence of salt, charged salt molecules penetrate the silk fibroin interlayer and act as shielding for the charged fibroin proteins. This results in the reduction of resonant frequency and reduced amplitude (due to more charged particles). In the presence of oil or alcohol, such molecules interact with the hydrophobic portion of silk fibroin to swell the construct. This leads to the increase in resonant frequency. With the addition of glucose, the sensor exhibits a temporal response—the resonant frequency reduces before increasing to its final value (above the initial resonant frequency). It is believed this is due to small molecule osmosis and initial rejection of the molecule by silk. With this behavior, saliva, fatty food, alcoholic drinks, sugary drinks, water, and salty foods exhibit distinct spectral responses (or signatures). While many foods exhibit a dominant nutrient characteristic (of salt, fat, sugar, and so forth), this sensor can distinguish combinations of foods. This can be partially seen in the in-vivo data of sensor response to soup (a combination of salt and fat).

Bio-interlayer-RFids exhibit sensitivities to the nutrients of real-world foods. In addition to the distinct spectral responses of the sensors to various nutrients, the sensors display concentration-dependent shifts in response to such biomolecules. The glucose sensitivity of a base trilayer construct is about 0.6 MHz per 1 g/L of glucose. This is well within the sensitivity specified for modern food products that possess exceptionally high sugar content. As an example, apple juice possesses a sugar concentration of about 200 g/L. The largest signal response occurs at 1× saline solution—this would be corresponding to a high salt-content drink. Notably, the sensor retains signal fidelity and sensitivity throughout this increase in salt. Typical dielectric sensors of this size would lose signal at this content of salt; however the interlayer material constrains penetrated salt, and thus the dielectric loss of the construct. The sensitivity to ethanol is within the detection sensitivity specified for alcohol drinks (about 12%) or mouthwash.

Given that biomolecules penetrate the interlayer material, the sensors display a concentration-dependent temporal response to the presence of nutrients. Generally, it is found that at room temperature, sensors take about 15 minutes to plateau or saturate. Within about 3 minutes sensor response will hit close to about 70% of the final response. It is found that temporal shifts occur much faster in-vivo due to the higher temperature and great fluidic agitation. Initial shifts are noticeable within about 30 seconds to about 1 minute in-situ (and in-vivo, FIG. 7). Such sensors can track nutrients in both liquids (as demonstrated) and solid foods. The average person chews 10 to 20 times on a piece of food, and a single bite is in the mouth for about 20 to about 30 s. Flavors are reinforced and amplified during subsequent chews over the course of a meal (about 20 minutes). The tongue and human perception of nutrients give an estimate of desired device sensitivity for solid foods. When drinking a beverage, the flavor/sugar is perceptible in the mouth for minutes after drinking (at a similar taste to the drink itself). After chewing and eating a fatty piece of meat, fat or salt lingers between taking subsequent bites. Nutrients such as fat, salt, and sugar are naturally meant to extract and mix rapidly during chewing so that the tongue is able to taste the foods people are eating. Although the amount of dissolved nutrient is low, the volume of saliva lingering in the mouth is similarly low. Nutrients that remain after swallowing food do not disappear instantaneously but should be renormalized with fresh-produced saliva (this can take significant time).

d.1.2 Design Issues and Resolutions

Certain issues for the interlayer-RFid can be resolved with engineering development. These include sensor specificity and response time.

Faster response time can be addressed via modulating a porosity of a conductive split-ring metal and/or the interlayer (as shown in FIG. 8). Such a strategy can reduce the diffusion length of the sensor by over an order of magnitude and reduce the response time of the sensor by several orders of magnitude.

Greater specificity/selectivity of the sensor can be addressed via several strategies, including utilizing sensor arraying and machine learning techniques to parse array data. A second strategy is to modify or switch the interlayer with materials that selectively absorb or swell in response to select nutrients. For example, various hydrophobic sponges can be fabricated that are oleophilic and hydrophobic. Such materials can selectively quantify and integrate the consumption of fats.

d.2 Large Gauge Factor, Biocompatible Strain Gauges.

Another underlying technology of some embodiments is a biocompatible strain gauge that operates within the working physiological range of various motions of the human body (FIG. 9). Ultrasensitive, conformal piezo-resistive strain sensors are capable of measuring large mechanical strains of the chest during breathing. The sensors have a form factor of a disposable BandAid© and are attached to the subject similarly. The strain sensors are comprised of hierarchical wrinkled platinum (wPT) nanostructures embedded into skin-like silicone rubber. Over a large and physiologically relevant range, these wPT sensors achieve a high sensitivity among metal thin film strain sensors. In comparison to other strain gauges, the wPT strain gauges conform and stretch sufficiently to monitor large physiological movements.

d.2.1 In-Vivo Tracking of Respiration

The sensors are developed into respiration sensors. In-human/early feasibility trial is conducted, which successfully correlated data obtained from the respiration sensors to the gold standard: a medical-grade spirometer (CareFusion, Yorba Linda, CA). Importantly, these preliminary studies demonstrate a high correlation with respiratory rates, as well as respiratory volumes compared to continuous measurements from the spirometer (FIG. 10a). The respiration sensors are first applied to the abdomen and ribcage to measure the change in strain at their respective locations. The breath-by-breath resistance change of the sensors are then linearized using a power model before being fitted to a multiple linear regression model. Across a pilot cohort of 7 subjects, the multiple linear regression model (MRL) provided an excellent fit with adjusted R²values ranging from about 0.92-0.97 across all subjects. While 2 sensors are used for this study, the fit was still very good (R²values range from about 0.83-0.95) when using just the ribcage sensor data. Moreover, when the MRL model was applied to calculate respiration volume in a subsequent test set, the sensors showed excellent predictability with a Lin's concordance correlation coefficient of about 0.962 for all the subjects combined (FIG. 10b). Volume is calculated accurately, under stationary conditions, as compared to the standard. The sensors also accurately measured different respiration rates compared to the measured rates from the spirometer.

With respect to sensor fabrication and manufacturability, processes are developed to scale manufacturing and reduce per unit cost. The sensors are characterized and demonstrated to be linear and robust, with cyclic fatigue testing of over 15 thousand of cycles.

d.2.2 Joint Sensor

The sensors are mounted onto the index finger to measure joint angles of the proximal interphalangeal (PIP) joint. Using an anchor setup, the sensor is positioned between the PIP and knuckle joints (FIG. 11). The distal anchor was located between the distal interphalangeal (DIP) and PIP joints while the proximal anchor was located between the PIP and knuckle joints (but closer to the knuckle). The anchor is comprised of an FDA-approved double-sided skin adhesive on a non-stretchable but flexible thin substrate. The flexible substrate is positioned over the PIP joint to absorb the bending strain from finger flexion and decouple the bending and strain of finger flexion, allowing the sensor to selectively measure planar displacement. Using a finger goniometer for verification, the finger was flexed and extended from 0-90-0° at 5° (±1° for human error) increments. Ten cycles of flexion and extension are recorded and normalized with each increment lasting for about 5 sec. The maximum observed linear displacement is about 113%. Each of these about 5 sec periods are averaged and plotted against their respective joint angle value. The sensor data exhibited high signal-to-noise ratio with clear differentiation between each 5° incremental step (see FIG. 11 data subset) and is found to respond linearly and precisely to changing PIP joint angles. Notably, this resolution is significantly better than other methods of finger joint angle tracking.

d.3 Neck Module

In some embodiments, the neck module includes a piezoelectric sensor, an acoustic sensor, and the strain gauge sensor all connected to a microcontroller board including a microcontroller (including a processor and an associated memory storing processor-executable instructions) and a wireless communication unit. An initial prototype of the neck module is designed in solid works and printed using 3-D printers. FIG. 12 shows the initial prototype. Following are the details of each of the components:

Piezoelectric Sensor: The sensor (LDT0-028K) boasts a sensitivity of about 50 mV/g to about 800 mV/g. This sensitivity is sizable enough to detect even relatively small perturbations without amplification.

Acoustic Sensor: The sensor (CEM-C9745JAD462P2.54R) is an electret microphone. Its operating voltage range of about 1.0 Vdc to about 10.0 Vdc allows for the sensor to be powered in parallel with the microcontroller using a CR 2032-coin cell battery.

Microcontroller Unit (MCU): RFduino is a microcontroller board that utilizes the RFD22301 MCU. The RFduino is capable of Bluetooth Low Energy (BLE) communication without extraneous RF modules. This device also has the capability of setting up a communication network between a master RFduino and up to 7 slave RFduinos—a network available from Nordic Semiconductor entitled Gazell (GZLL). RFduino is Arduino integrated development environment (IDE) compatible and the libraries for BLE and GZLL are open sourced. In practice, analog voltages can be sampled at predetermined general-purpose input/output (GPIO) pins and after the 0-3.3 V analog readings are converted to discrete values between 0-1023 inclusive.

d.4 Embedded Systems

A machine learning-based image recognition software is developed for running in a central node. The central node includes a) an ARM-based microcontroller, b) a tensor processing unit (TPU), and c) a smart glass with a built-in camera and a battery as shown in FIG. 13. As a demonstration of the working capabilities of these systems, implementation is made of image processing/analysis capabilities. The image analysis is divided into food category analysis and food volumetric analysis in parallel.

First, the image from the camera is categorized into one of 101 pre-trained foods through MobileNetV1, which is an efficient convolutional neural network (CNN) model for platforms such as mobile and embedded vision systems. To maximize image classification performance and overcome performance constraints in mobile platforms, this prototype utilizes a TPU accelerator. Unlike a microprocessor with vector extension, which can perform constrained general-purpose operations, the TPU can perform millions of special-purpose operations in a single cycle, showing 83× performance per watt compared to the microprocessor.

Second, volumetric analysis for the image is performed through the following procedure. The image processing procedure is applied to divide the segment of the image first. The procedure is k-means cluster, using the R, G, B colors and X, Y positions (total 5 numbers) as the features for cluster. Plates of pre-measured sizes are used, and the height and area of the food is calculated based on the shape of the six predetermined foods such as hexahedral, cylindrical, conical, flat, spherical, and irregular shape. As an example, FIG. 14 shows the segmentation result of the morning sandwich and shows the process of obtaining the volume of food through the procedure.

d.5 Machine Learning

Machine learning procedures are used to detect various eating events as well as classify foods based on the collected data from various sensor modalities. As part of a preliminary study, 3 sensors are used (piezoelectric, acoustic, and strain) to collect data from 5 subjects to classify among 5 activities—i) Talking, ii) Mouth opening, iii) Chewing, iv) Swallowing, and v) Others. Here are the stages followed:

Data Collection and Labeling: A significant stage to train a machine learning procedure is accurate data collection and labeling. A data collection interface is developed that shows the collected data in real time. It also allows labeling the data while collecting that helps in the training phase.

Data Processing: As data is collected from real subjects in an in-vivo set up, the data is quite noisy. To filter out the noises, a sliding window mechanism is used. Sliding window takes the mean or standard deviation of all the data points within the window and puts the value in the middle index of that window. In this way, the data collected is processed from each modality. In this implementation, a sliding window of 33 is used.

Feature Extraction and Fusion: Once the data is processed, next extraction is made of the time domain features from the data. For this implementation, mean and standard deviation are used as features for each of the collected signals from the 3 modalities. Then all the features are fused from each signal and fed to various machine learning procedures.

Procedures: Implementation is made of 6 state-of-the-art machine learning procedures—AdaBoost, Decision Trees, Naïve Bayes, Nearest Neighbors, Random Forest, and Support Vector Machine (SVM). K-fold cross validation method is used to train the procedures. The value of K is 50 for all procedures except the SVM which is 3. In addition to the cross validation method mode filter is used to improve the accuracy. After getting the predicted results from the machine learning procedures, the mode filter is used for eliminating the noises and improving the performance of the estimator. The filter is based on a sliding window of length 51. In each window, the mode of the labels are calculated and set as the value of the center of the window. Table 1 shows the accuracy of different procedures in classifying the 5 activities. It can be seen from the table that Random Forest procedure performs better compared to any other procedures. The performance of the procedures can improve if more features are evaluated to train the procedures. Also tuning different parameters like sliding window size, the value of K in cross validation—can lead to improved results.

TABLE 1

Accuracy of various procedures.

Algorithm
Accuracy

Random Forest
76.64%

K Nearest Neighbor
68.82%

Decision Trees
69.29%

Naïve Bayes
55.26%

AdaBoost
59.09%

SVM
23.16%

Event Detection: The ultimate output of the machine learning procedure will be different eating events predicted from the collected data. FIG. 15 shows the actual eating events against the predicted events using Random Forest procedure for the piezoelectric data subset.

e. Development

Prototype development can be made, and validation of sensor modules can be performed either in-vitro or in-vivo. A base embodiment of the wearable platform can be composed of two modules: an in-mouth module composed of a compact, edible device to track and broadcast nutrient data from within the mouth (FIG. 16); and a neck module composed of a flexible analog read-out device tagged to the throat to track muscle motion and swallowing (FIG. 16). A central processing node can be integrated either within the neck module or into an extended embodiment of smart glasses (FIG. 17) that can double as image capture (for feedback) and the nexus of electronics for the embedded system. Data can be transmitted from and to this node via BluetoothLE. Motion from the jaw and throat maps broadly to food density and texture, while salivary data maps to nutrients. In effect, a fusion of sensor data is proposed that will each partially capture the nutrients, density, and texture of food (in addition to an individual's distinct response to a food)—this biometric data can be analyzed in combination with machine learning.

e1. Biosensor Development and Validation

Synthesis and development of interlayer-RFid biosensors with improved selectivity/specificity and structural stability. While base interlayer-RFid bio sensors can perform sufficiently for the wearable platform proposed herein, these sensors can be further improved in their ability to categorize nutrients. Development on the interlayer-RFids can be made to improve their ability to categorize and identify nutrients. A sensor (based on silk fibroin) is able to cleanly identify single nutrients (fat/alcohol, water, glucose, and salt); however, categorizing complex combinations of nutrients may present some challenges. As noted in testing data, there are several solutions to this issue including the creation of sensor arrays, the integration of machine learning, and the development of improved interlayer materials.

This section will focus on the development of improved materials to integrate as the interlayer of interlayer-RFids to enhance nutrient selectivity (FIG. 18). This can occur via first mixing in dopant materials with different surface properties, and secondly direct modification of side-chains on a fibroin interlayer. Investigation can be made into selectively-absorbing interlayers. This can primarily focus around oleophilic/water-rejection interlayers. Preliminary investigations can focus on modifying commercially available oil-absorbent sheets (Uline, Light sorbent pads); however, hydrophobic aerogels can be directly synthesized that can be sized and shaped to design specifications. These sensors can be useful in the scope of nutrient sensing due to their permanent absorption and integration of fats.

Interlayer bulk properties can be modified via mixing of alternative insoluble biopolymers. Tests can be performed on chitosan and gelatin that possess different isoelectric point (pI), hydrophilicity, and oleophilicity than silk. Additionally, investigation can be made on approaches to directly modify silk fibroin side-chains. Investigation can be made on glutaraldehyde and diazonium reactions to bind various side-chains to silk, include hydrophobic, charged, and hydrophilic groups. Such modifications can change the selectivity of the interlayer material.

The behavior of synthesized interlayer-RFids can be characterized for response to mixtures of nutrients dissolved in artificial saliva including salt, various fats, alcohol, and glucose. Spectral response can be verified using a bench-top network analyzer.

Objectives Include:

- Synthesis of biopolymer-doped interlayer-RFids.
- Synthesis of side-chain modified fibroin-interlayer-RFids.
- Fabrication of fat-absorbing interlayer-RFids.
- Characterization of selective interlayer-RFids bio sensors.
- Optimization and process development of selective interlayer-RFid bio sensors.

Flexible, multi-sensor printed circuit board (PCB) design and development. Practical usability and form factor are issues that can hamper wearable devices. At a base level, a 2-component system is proposed with two modules that can be tagged straightforwardly to the body (the throat and a tooth). The neck module should be low-weight and flexible, so as to be able to conform to the body while not impeding the motion of the jaw or throat. The multi-sensor neck module can be adapted to a flexible PCB in order to improve the module usability. Two formats of a flexible PCB board can be utilized. The first involves boards that can be custom-designed and synthesized, which can be made by masking and etching copper-polyimide-copper substrates. The second involves flexible PCBs with silk screens. In addition, this module can be designed to support the addition and subtraction of sensor sub-types. Demonstrated sensors include acoustic, piezoelectric, and strain gauge sub-types; however, additional sub-types (such as EMG sensor) can provide expanded biometric information, obviate other sensors, or reduce processing demands. This can include a “plug-and-play” board design to evaluate and implement the sensor fusion strategy. This type of design can be generated by fabricating sensor nodes that can be turned on or off as specified by the platform.

Objectives Include:

- Preliminary (base sensors and microcontroller) and advanced (including ARM-based SOC) design of flexible neck module PCB.
- Fabrication and pick-and-place of neck module electronics on flexible PCB.
- Plug-and-play biosensor design.
- In-vivo testing of flexible, low form-factor neck module.
- Size and weight optimization of neck module.

Alternatives: While flexible PCBs may possess improved form factor, lower weight, and reduced user burden, mechanical flexing and movement by the user may produce noticeable noise in the system. Two solutions can address this issue: a first involving signal filtration and classification strategies to identify and remove noise induced via mechanical movement, and a second utilizing either a single connected stiff PCB board, or connected islands of stiff PCB sensor/filtration nodes.

e2. Electronics Design and Implementation

Base embedded system: Development of ARM SOC to monitor and catalog biosensor data. The ARM-based embedded system, which can be regarded as a central node, can wirelessly collect data from different sensors, and store them into a local relational database. Other microcontroller boards can be used that are similar in capabilities, and have similar architectures, input/output (I/O) interface, and run Linux-like operating systems. These can be programmed using tools available to system programmers such as compilers and debuggers.

For the sensors, collecting the sensor data can be implemented as a continuous collection, or can be programmed to occur with user input—such as a button press. Sensor data is retrieved via BluetoothLE in order, and timestamped for analytical techniques. A relational database, SQLite, is used as a local database for storing sensor values with partitioned relations (tables), and the database schema is suitably configured. An Event table stores each event such as sensor reading, taking picture, and corresponding time and detail. A SeriesofEvents table stores a series of events referring to the Events table, which is continuous or is initiated with user input. The Sensor table has a pre-defined sensor type. This data can either be stored in a local memory or transmitted to the cloud. This data can be processed in parts within this node or in the cloud as specified by the application.

Objectives Include:

- Development and encoding of an ARM-based small form factor board to store, catalog, and analyze biometric data.
- Development of a computing, communicating, and data processing architecture for optimization of memory, power, and accuracy of data analysis.
- Design and implementation within the neck module

Extended embedded system: “Smart” glasses, central node capable of image processing and biosensor data processing. FIG. 19 shows a flow chart of an entire process of image analysis that can be implemented to run on an integrated smart glass platform. The food type and volume are stored as a single event and corresponding details in the local database described above.

Embedded systems such as the smart glass platform have particular specifications, including cost, size, area, and energy constraints, that may constrain accuracy of nutrition analysis when processing in-situ to meet real-time constraints, and improvement on such accuracy can be made while keeping energy efficiency high. Thus, a concern is to ensure that the smart glass system consumes low power while achieving high accuracy of nutrition analysis. To mitigate these conflicting challenges, consideration is made of modality scaling and virtualization,

Modality scaling increases the accuracy of the analysis using additional sensors. Generally, added sensors consume much less power than adding a processing unit. For example, the accuracy of two images from two different angles instead of a single image analyzed can be much higher. Alternatively, or in conjunction, adding data from bio sensors and/or neck sensors can improve confidence in one candidate food classification versus others. This fusion of sensing modalities is further discussed below.

Objectives Include:

- Integration of image processing capabilities within sensor suite within ARM-based SOC.
- Development of an energy efficient, image-based food classification system.
- Integration of smart glasses into biometric data stream.

Alternatives: Virtualization is another way of improving accuracy. Currently, a TPU can process a small size convolutional neural network (CNN), which results in lower accuracy. In order to overcome such constraint, ways of partitioning the CNN into smaller components can be made so that the TPU can process one component at a time in a way transparent to the user. While such virtualization can increase the processing time, it can result in improved accuracy. This can be referred to as local virtualization. Remote virtualization works in a different way, tiering the computation such that a portion of the processing (sometimes referred to as lightweight processing) is done locally while another portion is done in the cloud. The advantage of this method is that the data being transmitted can sometimes be reduced, thereby reducing transmission energy and if properly partitioned, can result in smaller energy consumption compared to local virtualization.

Neck module: Design of analog and analog/digital (A/D) circuit microcontroller system. The electronics of the flexible neck module is adapted to control, filter, and amplify signals from various sensor sub-types. Broadly, this neck module is adapted to perform two primary tasks: (a) Extract, filter and amplify signals from acoustic, piezoelectric, and strain gauge sensors, and (b) Provide wireless power to the in-mouth, interlayer-RFid based module (attached to a tooth).

Signals extracted from throat biosensors can be prone to noise (particularly due to flexible substrates and perturbations introduced by human motion), and should be filtered to extract a biological response. The response of the resistive strain gauge can be extracted via a Wheatstone bridge circuit, whereby the voltage at the output of the strain gauge (in series with another resistor) is compared to the output across a constant comparator resistor (also in series with another resistor). This signal can be amplified and filtered using low footprint, analog op-amps (INA series, Texas Instruments). Breakout board designs for acoustic and piezoelectric sensors (included various filtration and amplification components) utilized in the Preliminary Data can be similarly translated to a custom PCB board design in order to be integrated onto flexible substrates. Biosensor data (composed of analog and digital signals) can then be fed into low footprint, ATtiny20 microcontrollers, which can transmit the data to the central, ARM SOC processing node via BluetoothLE.

The in-mouth module can be powered via near-field, inductive coupling originating from a source signal generated by the microcontroller or the ARM SOC. This signal can be fed through a power amplifier chip and finally a planar coil tuned to about 900 MHz. This signal can be turned on and off as specified to probe the sensor.

Objectives Include:

- Design of strain gauge read-out circuit (including Wheatstone bridge and analog filters).
- Simplification and adaptation of piezoelectric sensor break-out board design.
- Simplification and adaption of acoustic sensor break-out board design.
- Design and implementation of wireless power link between neck module and in-mouth module.

In-mouth module: Fabrication of radio frequency (RF) power and read-out circuit for interlayer-RFid. Development is proposed of a low-form factor, wearable solution to read out data wirelessly from interlayer-RFid sensors. Structurally, this module is composed of two layers: (a) a mouth-environment exposed sensing layer composed of a biocompatible interlayer-RFid, and (b) a sealed read-out circuit that is remotely powered via near-field inductive powering (generated from the neck module). This module can contain minimal sealed electronics (a small RF integrated circuit (IC) chip), and can be biocompatible and edible while retaining a small enough footprint to fit on a tooth (FIG. 20). In additional embodiments, a set of eating event sensors (e.g., a mechano-acoustic sensor) and a temperature sensor are also included in the in-mouth module.

Electrochemical impedance spectroscopy (EIS) techniques based on precise capacitance measurements can be used for the read-out circuit. For example, a technique can be used to measure the dielectric constant of a material based on connecting the material under test in parallel with an LC tank used in a voltage-controlled oscillator (VCO), which is embedded in a phase-locked loop (PLL). By measuring the change in the PLL output frequency, the capacitance of the material can be measured. As another example, a sinusoidal voltage with frequency up to about 50 MHz is applied across the material under test; the resulting sinusoidal current is down-converted to direct current (dc) using a pair of mixers in quadrature. The resulting dc output voltages from the mixers thus give the real and imaginary parts of the admittance. A similar technique can be used, but with a high-frequency design. The sinusoidal current conducted in the material under test is applied to the input of a distributed low-noise amplifier (LNA) and then down-converted to an intermediate frequency of about 100 MHz.

An embodiment of the read-out circuit for directly measuring the magnitude of an impedance of a split-ring resonator is shown in FIG. 21. Input frequency f_inis swept linearly over a certain time period and applied to the input of a phase-locked loop (PLL). The PLL output, a voltage with frequency f_o=Nf_in, is applied to the input of a transconductance block g_m, which provides a current i_xwith the same frequency that is magnetically coupled to the resonator. The resulting voltage v_x, which has an amplitude proportional to the magnitude of the resonator's impedance, is applied to an amplitude detector, which is then digitized by an analog-to digital converter (ADC). This digitized data is fed to a microcontroller for transmission to a central node. Other strategies wherein relevant data is transmitted within reflected backscatter generated by the in-mouth module can be considered.

Preliminary implementation can be via a PCB with chip RF components. Such components are available in small chip format. Further implementations of these circuits can be made on flexible PCB boards. Upon completion of prototype boards, the capability of such devices will be tested against both benchtop network analyzers and portable vector network analyzers (VNAs) (miniVNA brand) on bio-interlayer-RFids. The sensitivity and accuracy of the output resonant frequency and amplitude from the portable VNA and custom PCB can be compared against the benchmark results of the benchtop network analyzer. The goal is that the custom boards will possess similar accuracy with portable VNAs, and optimization of circuits can be made to achieve this goal (via noise reducing schemes, chip layout and component design, and more). Further, fabrication of these components into a custom-designed RF IC chip can be made.

Objectives Include:

- Design and implementation of the RF read-out circuit with available components (PLL, PFD, ADC, and amplitude detector).
- Benchtop validation of the read-out capabilities of implemented circuit.
- RF IC design, fabrication, and test loop.
- Integration with interlayer-RFid bio sensors, and benchtop validation of combined in-mouth module.

Alternatives: Design considerations include the ability to reduce power demands and reduce the input energy to power the in-mouth module. Resonant read-out of the sensor may prove less stable in a complex, moving environment of the mouth. Another strategy to read-out the sensor is direct capacitive readout of the permittivity of the interlayer. Such a strategy is more power hungry and slightly more cost ineffective than a resonant read-out; however, this strategy is likely more stable to mechanical movement.

e.3 Data Engineering and Machine Learning

Design of event detection method: detection of bodily actions from wearable sensors. For the detection of bodily actions, the ‘Python’ language can be used to implement various machine learning procedures. Machine learning procedures can be implemented in ‘TensorFlow’. ‘TensorFlow’ is an open source Python-friendly library for faster and easier numerical computation of machine learning procedures. Another reason for choosing Python language for the implementation of the event detection framework is that it can be deployed to the proposed ARM-based SoC. The ARM-based SoC is adapted to perform local data processing and is portable.

Details on the data engineering and machine learning framework are given below. The process of detecting the events can broadly be classified into 2 parts—(1) Clustering of input data, and (2) Event detection.

Clustering of input data: To develop a robust food intake monitoring system, various modalities are used to collect data from multiple individuals belonging to different classes of age, gender, and ethnicity. Different individuals will have different eating behavior like chewing frequency or swallowing rate. Processing this disparate classes of data can be troublesome and the event detection may be less accurate. Thus, it is desirable to cluster data before processing for event detection. If data is clustered based on the class variation, insight can be gained of the eating behavior and food habits of a particular class of people. This will also allow developing a class-specific robust food intake monitoring system with higher accuracy. FIG. 22 shows a stepwise procedure to cluster data before event detection. The first stage involves the collection and labelling of data from different individuals using various state-of-the-art modalities. The second stage is dedicated to the synchronization of data. The third stage can be to apply clustering procedures to cluster the data collected from various individuals (for example, this can be individuals of a particular age or gender). In the fourth stage, data corresponding to individuals in a particular cluster can be sent to the ‘Event Detection Block’ (FIG. 23).

The final output of this block is the detection of eating activities along with their temporal information. This block can extract and fuse features from gathered data to be fed into the machine learning procedures, such as AdaBoost, Random Forest Classifier, Naïve Bias Classifier, Nearest Neighbors, Support Vector Machine, Decision Trees, and so forth. Deep Neural Network and Reinforcement learning also can be used.

Objectives Include:

- Implementation and development of stepwise procedure for input data clustering.
- Collection and supervised labelling of multi-sensor, training data using wired sensors (strain gauge, piezoelectric, and acoustic) during facial actions.
- Collection and supervised labelling of multi-sensor, training data from wireless neck module.
- Training of machine learning procedures on collected data.

Development of dietary and nutrition classification procedures. This section describes the proposed sequential stages in a ‘Food Detection Block’ (implemented in Python). The final output of this block is the classification of intake food type. FIG. 24 shows the proposed stepwise procedure for food type classification. The first stage of food detection can start with the collection of nutrient data. Then it passes the data to the next stage for synchronization. The second stage synchronizes data based on their starting points. The third stage can merge the temporal information of the event data obtained from the stage 6 of event detection with the synchronized nutrient data obtained from stage 2. The reason for this merging of event and nutrient data is that the temporal information of eating events can add more information along with the nutrient data to classify the type of food. The fourth stage can apply the food classification procedure, which can make use of both the nutrient and temporal information of the event data to determine the type of food via fused data points. This data can be displayed in a simple-to-read format as shown in FIG. 25.

Within this aspect of development, in-vitro validation of the in-mouth sensors/module can be performed to single and multiple nutrients. Initially testing of devices can be performed in artificial saliva with changing concentrations of a single nutrient. Next, sensors can be subjected to specified mixtures of nutrients. Specifically, nutrient-1 concentration can be held constant, and nutrient-2 concentration can be modulated over time. Base nutrient concentration can be: salt—50% phosphate-buffered saline (PBS), alcohol/vegetable oil—20%, and glucose—50 g/L. As an example of one multiplexing experiment, salt can be held at 50% PBS, and successive addition is made of increments oil to the solution with mixing. For the characterization experiments, the spectral response of sensors can be verified via a bench-top network analyzer (up to about 40 GHz).

Objectives Include:

- Implementation and development of stepwise procedure for food detection.
- Collection of spectral data generated by interlayer-RFid bio sensors and/or completed in-mouth module to single and multi-nutrient liquids.
- Training of machine learning procedures on interlayer-RFid bio sensor and in-mouth module data.
- Validation of 2-component prototype, and collection of multi-modal data from an in-vitro model.

Example Embodiments

In some embodiments according to a first aspect, a system for tracking diet and nutrition includes: (1) an oral module configured to be affixed within a mouth of a user; and (2) a body module configured to be affixed adjacent to a body part of the user, such as a neck of the user.

In some embodiments according to the first aspect, the oral module includes a set of salivary sensors responsive to a level of at least one nutrient. In some embodiments, the set of salivary sensors includes multiple salivary sensors responsive to levels of different nutrients. In some embodiments, each of the set of salivary sensors includes a pair of split-ring resonators and a sensing interlayer disposed between the pair of split-ring resonators.

In some embodiments according to the first aspect, the oral module further includes a probe antenna and a read-out circuit connected to the probe antenna, and the read-out circuit is configured to probe the set of salivary sensors through the probe antenna, and generate an output signal corresponding to the level of the at least one nutrient. In some embodiments, the read-out circuit includes a phase-locked loop and a transconductance block, which are configured to generate a probe signal applied to the set of salivary sensors through the probe antenna, and the read-out circuit includes an amplitude detector to detect a response signal from the set of salivary sensors through the probe antenna.

In some embodiments according to the first aspect, the oral module further includes a wireless communication unit and a microcontroller connected to the read-out circuit and the wireless communication unit, and the microcontroller is configured to derive information regarding the level of the at least one nutrient from the output signal of the read-out circuit. In some embodiments, the microcontroller is configured to direct the wireless communication unit to convey the information regarding the level of the at least one nutrient.

In some embodiments according to the first aspect, the oral module is configured as a tooth patch, tooth tattoo, or tooth cap to be affixed to a tooth of the user.

In some embodiments according to the first aspect, the body module includes a set of eating event sensors responsive to an eating event. In some embodiments, the set of eating event sensors is responsive to either of, or both, jaw movement and throat swallowing. In some embodiments, the body module includes a first set of eating event sensors responsive to jaw movement, and a second set of eating event sensors responsive to throat swallowing. In some embodiments, the body module includes a strain gauge responsive to jaw movement. In some embodiments, the body module includes either of, or both, a piezoelectric sensor and an acoustic sensor responsive to throat swallowing.

In some embodiments according to the first aspect, the body module further includes a wireless communication unit and a microcontroller connected to the set of eating event sensors and the wireless communication unit, and the microcontroller is configured to derive information regarding the eating event from output signals of the set of eating event sensors. In some embodiments, the microcontroller of the body module is configured to direct the wireless communication unit of the body module to convey information regarding the eating event. In some embodiments, the microcontroller of the body module is configured to direct the wireless communication unit of the body module to supply power to the oral module.

In some embodiments according to the first aspect, the body module is configured as a neck patch to be affixed to a neck of the user.

In some embodiments according to the first aspect, the set of eating event sensors includes multiple eating event sensors, and the microcontroller of the body module is configured to detect eating activities from output signals of the multiple eating event sensors. In some embodiments, the microcontroller of the body module is configured to detect the eating activities by extracting features from respective ones of the output signals, fusing or merging the extracted features, and detecting the eating activities from the fused or merged features. In some embodiments, the microcontroller of the body module is configured to detect the eating activities by executing a machine learning procedure. In some embodiments, the microcontroller of the body module is configured to identify temporal information associated with the eating activities.

In some embodiments according to the first aspect, the microcontroller of the body module is configured to identify a food composition or consistency by fusing or merging the information regarding the level of the at least one nutrient and the temporal information associated with the eating activities.

In some embodiments according to the first aspect, the system further includes a visual module, and the visual module includes a camera configured to acquire a set of images of a food intake.

In some embodiments according to the first aspect, the visual module further includes a wireless communication unit and a microcontroller connected to the camera and the wireless communication unit, and the microcontroller is configured to derive information regarding the food intake from the set of images. In some embodiments, the microcontroller of the visual module is configured to identify the food intake from the set of images. In some embodiments, the microcontroller of the visual module is configured to derive an extent of the food intake from the set of images. In some embodiments, the microcontroller of the visual module is configured to direct the wireless communication unit of the visual module to convey the information regarding the food intake. In some embodiments, the microcontroller of the visual module is configured to identify a food composition or consistency by fusing or merging the information regarding the food intake with at least one of: (a) the information regarding the level of the at least one nutrient or (b) the information regarding the eating event.

In some embodiments according to the first aspect, the visual module is configured as smart glasses.

In additional embodiments, certain components explained above with respect to an oral module and a body module can be integrated within a single or common module, such as the oral module. In particular, in some embodiments according to a second aspect, a system for tracking diet and nutrition includes an oral module configured to be affixed within a mouth of a user.

In some embodiments according to the second aspect, the oral module includes a set of salivary sensors responsive to a level of at least one nutrient. In some embodiments, the set of salivary sensors includes multiple salivary sensors responsive to levels of different nutrients. In some embodiments, each of the set of salivary sensors includes a pair of split-ring resonators and a sensing interlayer disposed between the pair of split-ring resonators.

In some embodiments according to the second aspect, the oral module further includes a probe antenna and a read-out circuit connected to the probe antenna, and the read-out circuit is configured to probe the set of salivary sensors through the probe antenna, and generate an output signal corresponding to the level of the at least one nutrient.

In some embodiments according to the second aspect, the oral module further includes a set of eating event sensors responsive to an eating event. In some embodiments, the set of eating event sensors is responsive to either of, or both, jaw movement and throat swallowing. In some embodiments, the body module includes a first set of eating event sensors responsive to jaw movement, and a second set of eating event sensors responsive to throat swallowing. In some embodiments, the set of eating event sensors includes a mechano-acoustic sensor, such as an accelerometer.

In some embodiments according to the second aspect, the oral module further includes a wireless communication unit and a microcontroller connected to the set of salivary sensors, the set of eating event sensors, and the wireless communication unit, and the microcontroller is configured to derive information regarding the level of the at least one nutrient from the output signal of the read-out circuit and derive information regarding the eating event from output signals of the set of eating event sensors. In some embodiments, the microcontroller is configured to direct the wireless communication unit to convey the information regarding the level of the at least one nutrient and convey the information regarding the eating event.