Four levels of in-sensor computing in bionic olfaction: from discrete components to multi-modal integrations

Abstract

Sensing and computing are two important ways in which humans attempt to perceive and understand the analog world through digital devices. Analog-to-digital converters (ADCs) discretize analog signals while the data bus transmits digital data between the components of a computer. With the increase in sensor nodes and the application of deep neural networks, the energy and time consumption limit the increment of data throughput. In-sensor computing is a computing paradigm that integrates sensing, storage, and processing in one device without ADCs and data transfer. According to the integration degree, herein, we summarize four levels of in-sensor computing in the field of artificial olfactory. In the first level, we show that different functions are conducted by using discrete components. Next, the data conversion and transfer are exempt within the in-memory computing architecture with necessary data encoding. Subsequently, in-sensor computing is integrated into a single device. Finally, multi-modal in-sensor computing is proposed to improve the quality and reliability of the classification results. At the end of this minireview, we provide an outlook on the use of metal nanoparticle devices to achieve such in-sensor computing for bionic olfaction.

---

1. Introduction

The data collected from the analog world can’t be directly processed using the von Neumann architecture¹ in which analog-to-digital converters are usually required. This architecture separates the processor and memory (cf. the well-known memory wall) which is friendly to engineers, however, the time consumption required for data access shouldn’t be ignored² with the outbreak of data in the era of the Internet of Things (IoT). In addition, the energy and area consumption due to the presence of ADCs³ also limits the further development of edge computing. To address these problems, in-memory computing^4–7 and in-sensor computing^8,9 were recently proposed. Given that the human brain¹⁰ is the very paradigm that integrates sensing, storage, and calculating with high efficiency and low power consumption, bio-inspired electronic devices are constructed to mimic the functions of synapses and neurons.¹¹ In the biological olfactory system, odorant molecules are sensed by olfactory receptor cells. Electric signals are generated by olfactory receptor cells and transmitted through the glomeruli to the olfactory bulb, in which signals are preprocessed and then transmitted to the brain. Inspired by the biological olfactory system in which sensing, storage, and processing are not separate, in-memory computing and in-sensor computing are proposed in the olfactory field. In-memory computing is a computing architecture aiming to alleviate the data transfer between the memory and processor.¹² Usually, data representing synaptic weights are stored in a memory array and computing is performed in the same place, thereby greatly reducing the need for data transmission. In-sensor computing is the further step of in-memory computing.⁹ In this case, designers attempt to utilize the analog response from sensors directly rather than driving ADCs and DACs together to generate the voltages required by in-memory computing. Despite the great progress in visual^13–17 and tactile^18–22 in-sensor computing, the development of artificial olfactory is truly challenging mainly because of the involvement of material exchanges and sometimes chemical reactions. In this minireview, we focus on the different levels of in-sensor computing in the artificial olfactory. We discuss the requirements and difficulties in the integration of sensing, storage, and computing and also summarize the reported exploration of sensing mechanisms, processing algorithms, and integration methods. Finally, we propose a possible candidate—metal nanoparticles—for the future development of multi-modal in-sensor computing for bionic olfaction.

2. Four levels of in-sensor computing

The integration degree is an important metric for in-sensor computing. According to the integration degree, in-sensor computing can be divided into four levels (Fig. 1). Originally, sensing, storage, and computing are conducted by discrete components, corresponding to the lowest integration which can be designated as the first level of in-sensor computing. Second, the most representative components in the traditional edge computing system, ADCs, are discarded by directly using analog signals to participate in subsequent calculations. In the computing part, in-memory computing is conducted according to Ohm's law and Kirchhoff's law in an analog form. In the third level, sensing and computing are integrated into one device. The device responds to environmental signals while simultaneously carrying out computing. In the last level, multiple types of external signals can be detected by the same device, which can be called multi-modal in-sensor computing. The integration of sensors in different fields can capture information in different dimensions, which is helpful to conduct classification in an unknown environment.

Fig. 1 Four levels of in-sensor computing. Four levels are arranged from top to bottom according to their integration degree. Analog-to-digital convertor (ADC), digital-to-analog converter (DAC), trans-impedance amplifier (TIA), and the number of modals are used as metrics to compare different levels of in-sensor computing. In the first level, sensing, storage, and processing are carried out by discrete components. Sensory data are captured by ADCs and digital data are stored in memory with inevitable time consumption due to communication protocol. In the second level, sensory signals are encoded into spikes and utilized by in-memory computing directly without ADCs and data transfer. The method of encoding should be designed to bridge sensor and sequential computing parts. In the third level, in-sensor computing is conducted in a single array. Integration degrees are improved further. In the fourth level, multi-modal in-sensor computing. Different types of analog stimuli are perceived by single devices. With the cooperation of data from several dimensions, classification accuracy and reliability can be improved.

2.1. The first level: discrete components

In traditional edge computing systems, there are two main parts: data acquisition and data analysis. In the data acquisition part, sensors capture external analog signals and change their electrical properties accordingly. Usually, the resistance of gas sensors will change in different gas atmospheres. Some gas sensors also generate different voltages according to the concentration and types of the gas. These electrical properties of gas sensors will be converted into voltages and measured by ADCs under the control of microcontrollers (MCU). Although ADCs play a vital role in bridging the analog and digital worlds, several negative effects cannot be ignored with the increasing of sensor nodes. Due to the fixed amount of time required for ADCs to communicate with MCU according to the communication protocol, the latency of the system is inevitable. Usually, high-speed ADC needs multi-stage pipeline architecture to conduct analog-to-digital conversion which means the integration degree of ADCs needs to be a trade-off with converting speed and power consumption. The involvement of the ADCs adds a non-negligible burden to edge computing systems. Besides, the colorimetric sensor array is an important multi-modal identification method for volatile organic compounds (VOCs).^23–30 With the colorimetric sensor array being exposed to the VOCs, the difference in the color captured by the scanner can be used for gas classification by machine learning.

In the data analysis part, calculations are carried out in von Neumann architecture, and information is extracted into types that humans can understand directly. In this part, several statistical algorithms can be used. Principal component analysis^31–33 (PCA) is usually used to reduce the dimensionality of the data from multiple sensors which is also helpful for data visualization. The clustering algorithm is another commonly used data analysis method, which can extract categorical information from data in the form of unsupervised learning. Supported by multi-threaded computing devices such as graphics processing units (GPU), deep learning is widely used in complex data analysis scenarios. The powerful fitting ability of neural networks also brings a huge amount of calculation. In the state-of-the-art, hundreds of billions of synaptic weights participate in the classification together, which is a computing-intensive task. However, discrete memory and processors need to transfer data frequently. In von Neumann's architecture, data from ADCs are first stored in memory, which are fetched by processors when they need to be calculated. During the whole calculation, the intermediate results are frequently stored in and accessed from the memory because the basic calculation unit is an adder. All the calculations should be carried out obeying the communication protocol between processors and muti-level cache. A feasible and widely used optimization scheme is to utilize multi-thread processors, which is also a trade-off between speed and power consumption.³⁴

An artificial neural network (ANN) is a powerful and data-intensive method for classification. Liu et al.³⁵ proposed a tactile-olfactory bionic sensing array for object recognition (Fig. 2a). The resistance of silicon-based gas sensors decreases as the gas molecules are absorbed by the gas-sensitive materials (Fig. 2c). When sensors are exposed to the gas environment, the gas molecules will contact the surface of the sensitive semiconductor material and corresponding chemical reactions will occur. The electron transfer relative to the chemical reaction leads to resistance changes in the semiconductor material. The degree of the resistance change will be dependent on the types and concentrations of the tested gas. As different sensitive materials have various responses to specific gas molecules and the same sensor responds to multiple gases to different degrees, six gas sensors are integrated onto one glove to sample data during grasping. A multi-modal neural network (Fig. 2d) is carefully designed for combining the information from tactile and olfactory sensors to recognize different objects. Multi-modal analog signals can complement each other and improve recognition accuracy. The requirement that gas sensors need to meet is high sensitivity and designability to capture enough information from more dimensionality for subsequent recognition.

Fig. 2 Discrete sensor, memory, and processor system. (a) Silicon-based tactile sensors and olfactory sensors. Fourteen tactile sensors are placed on each finger while six different gas sensors are placed on the palm. (b) The schematic of PEN3 E-nose. The gas in the chamber will be reacted with ten different gas sensors while sensory data are sampled by a data acquisition system and processed by computer. (c) Sensory response of tactile and olfactory sensors. Due to the piezoelectric effect, the voltage generated by tactile sensors increased with the increase of pressure. The resistance change of gas sensors is sensitive to the concentration of the gas. (d) Structure of neural networks for recognition. The tactile and olfactory information is processed and fused in carefully designed neural networks. (e) The transient response curve of ten sensors to the test gas. The sensory data are represented as the resistance of gas sensors. (f) The structure of CNN used for beer classification. The temporal information is organized in a format like pictures, which is compatible with CNNs for classification. Reproduced with permission. Copyright 2019, Elsevier.

Using deep learning to solve the classification problem in the gas sensing field, sensory data should be converted into the form^36–39 required by the neural network appropriately. Jirayupat et al. conducted individual authentication based on breath odor sensing with a 16-channel chemiresistive sensor array for the first time.³⁴ Sixteen sensors with different sensing materials are utilized together for capturing olfactory information. A fully connected neural network with 16 input neurons is used to implement the classification task, in which each input neuron corresponds to an olfactory sensor in turn.

Shi et al.⁴⁰ proposed a data mining method of an electronic nose for identifying beer olfactory information (Fig. 2b). A PEN3 E-nose used in this article consists of 10 metal oxide sensors. The redox reaction between sensors and gases will change the conductivity of sensor-active materials (Fig. 2e). In the computing part, a hybrid model consisting of convolutional neural networks (CNN) and support vector machine (SVM) is constructed (Fig. 2f). The CNN containing convolutional layers, pooling layers, and fully connected layers is trained first. A total of 900 points of data from ten sensors are organized as an input feature map with a size of 30-by-30. The data containing timing information are encoded into a format like pixels in pictures, which is suitable for CNNs. The targets are 5 classes of gas with one-hot encoding. After the CNN is trained, fully connected layers are replaced by a trained SVM, which shows a higher classification accuracy.

In the first level of in-sensor computing, more methods can be tested for classification conveniently. Different data processing methods have their advantages and disadvantages. PCA is a kind of data preprocessing method, which can be a metric for measuring the difficulty of classification tasks. Usually, PCA is used to reduce the dimensionality for visualization. If the sample points are linearly separable in the reduced dimension, the classification seems to be relatively simple. In this case, clustering algorithms or artificial neural networks with fewer layers can be enough to implement the classification tasks. If the sample points are linearly divisible, the classification task is complicated, indicating the necessity of deep neural networks or other powerful algorithms. From the point of view of sensor construction, ANN is mostly used in the field of in-sensor computing at a higher level. The core computing tasks of ANN are multiply accumulation, which is much easier to be implemented on devices than other algorithms. The methods for integrating sensor data into a format compatible with neural networks or other in-memory computing algorithms will be valuable for the next levels of in-sensor computing. In addition, at this level, sensors, storage, and processors are discrete. It is a proven solution in “relatively low speed” or “relatively small amount of data”. A lot of demonstration is reported at this level. More attention can be paid to the co-design of sensors and data analysis methods, and it can be a foundation at other in-sensor computing levels.

2.2. The second level: sensors and in-memory computing

Given that the data transfer in von Neumann architecture limits the further development of edge computing, several attempts are conducted to alleviate its negative effects. One of these attempts is reducing the distance between processors and memory. However, the essential computing architecture is still discrete, leading to limited improvements. What if data transfer is eliminated? In-memory computing is the very computing architecture in which memory and processing are carried out in the same place. The computing process will be mainly conducted in analog format. Take the neural network as an example, multiply accumulation (MAC) is the basic and data-intensive operator. The computing task consists of the multiplication between input data and synaptic weights and the accumulation of previous multiplication. Basically, the synaptic weights can be represented by the conductance of non-volatile memory units while the input data can be represented by the voltage amplitude. The memristor, as an example of non-volatile units, can be arranged as a crossbar to conduct MAC in parallel. In this case, the voltage representing input data in MAC can be applied to the memristor crossbar array, and multiplication is finished according to Ohm's law. Since the memristors in the crossbar are connected in parallel, the accumulation is finished according to Kirchhoff's law. Without a complex communication protocol, computing tasks are carried out by applying voltages and the results of MACs are represented by the amplitude of the current. With the approach above, in-memory computing alleviates data transfer while ADCs are still indispensable.

In the second level of in-senor computing, the input data of the first MAC can come from sensors directly without ADCs. The input data of the neural network includes information about the analog signals to be classified. Traditionally, the information is converted by ADCs from analog type to digital type. In the case of in-memory computing, the information subsequently needs to be converted into voltage for memristor arrays. What if ADCs are eliminated? The analog signals from sensors can be converted into the appropriate format and applied to memristor arrays directly. Some of the gas sensors change their resistance corresponding to different gas.⁴¹ An additional divider resistor can be used to convert information from resistance to voltage. As for the gas sensors^42–45 featuring a gas-controlled drain current,⁴⁶ the signals can be encoded into spike form.

Some electrical characteristics of devices can be used for bridging the sense and in-memory computing. Han et al.⁴⁷ proposed an artificial olfactory neuron for an in-sensor electronic nose (Fig. 3a). A chemiresistive gas sensor made of semiconductor metal oxide is used for capturing gas information. The ambient oxygen will be ionosorbed on the surface of the sensing material while free electrons will be extracted, leading to an electron-depleted region generated on the surface of the metal oxide. When the gas reacts with the sensing material, the electron-depleted region will change, resulting in a resistance change of the metal oxide. In this way, the information is presented as the resistance change of the gas sensor. In the computing part, the neuronal leaky integrate-and-fire (LIF) function can be mimicked based on the 1T-neuron by using the single transistor latch (STL) phenomenon (Fig. 3b). A silicon-based MOSFET with a channel width of 200 nm and a gate length of 1900 nm is fabricated as a 1T-neuron. When a gate voltage of 0 V was applied, the MOSFET should have been turned off. When a voltage beyond the threshold was applied to the drain electrode, which is called the latch-up voltage, a large current will abruptly flow through the device because of the STL phenomenon. To mimic the LIF function, 1T-neuron is constructed with a MOSFET and a capacitor connected in parallel (Fig. 3c). With the capacitor being charged under a constant current source, the voltage applied on the drain is close to the latch-up voltage. When the STL occurs, the MOSFET turns on and the capacitor will be discharged. With the capacitor being charged and discharged periodically, the 1T-neuron converts the information from current to voltage pulses (Fig. 3d). The frequency of the pulses is dependent on the capacitance and the current source, which is relative to the resistance of gas sensors connected in series. The type and concentration of test gas are encoded into the pulse frequency, which can be used for building a spike neural network for classification. Some effects like the latch-up can also be utilized to combine the sensors and in-memory computing under design.

Fig. 3 Sensors with in-memory computing. (a) The schematic of an artificial olfactory neuron. The model is composed of a semiconductor oxide gas sensor and a MOSFET-based 1T-neuron. (b) The single transistor latch phenomenon of 1T-neuron. When the drain voltage reaches the latch-up voltage, a large current was abruptly flown due to the single transistor latch. (c) A schematic of the current-to-spike converter circuit. The current was applied to the drain electrode and voltage was output at the same drain electrode. The capacitor was expected to be charged and discharged periodically. (d) The output voltage of the capacitor. The frequency of voltage spikes is dependent on the drain current, which is affected by sensors connected in series. Reproduced with permission. Copyright 2022, Wiley. (e) The schematic of reservoir computing conducted by memristive devices. The input is the voltages applied on the devices while the output is the conductance representing the status of nodes. (f) Voltage spike trains are converted from the sensory information of response speed and response amplitude. Reproduced with permission. Copyright 2021, Wiley. (g) Gas measurement system. The signals applied to H control the micro-heater. The occurrence of reaction can be controlled by the micro-heater. The sensing layer of the FET-type gas sensor is n-type In₂O₃. The voltage applied on the gate and drain electrodes allows the measurement of the drain current representing the sensory data. (h) The transient response of sensors as parameters of gas types and concentration. Since the NO₂ gas is an oxidizing gas, electrons are taken from the In₂O₃ films, resulting in the increase of majority carriers in the FET-type sensors and the increasing of drain current. (i) The schematic diagram of the SNN and FET-type sensor array for the artificial olfactory system. The sensory data are encoded into spike trains by the encoder circuits. Reproduced with permission. Copyright 2021, Elsevier.

Reservoir computing is an attractive data processing method related to temporal signals, which is consistent with the characteristics of gas sensing. Wang et. al.⁴⁸ proposed an olfactory inference system (Fig. 3e). The data from sensor arrays are processed in RC computing first and the extracted features are classified by a 3-layer neural network (ANN). RC computing is a power-efficient network, which is suitable for working with temporal information. The sensory data comes from the dataset “Twin gas sensor arrays Data Set”, which includes the resistance changes of gas sensors under different ambient conditions. The responses of gas sensors are transformed into spike trains compatible with reservoir computing (Fig. 3f). The information about response speed is represented as the resistance difference between two neighboring sample points and the spike trains will be generated if the resistance difference is beyond a threshold. The information about the static response is represented as the frequency of another spike train generated obeying the Poisson model. A volatile memristor (W/WO3/PEDOT:PSS:/Pt) based reservoir is utilized to extract features from spike trains while the conductance states of the memristor are output features of the reservoir. A non-volatile memristor (Pd/W/WO3/Pd) array is used for building the 3-layer ANN classifier. Although the sensory data comes from the asynchronously collected dataset, attention should be paid to the method that converts the information to a compatible format for computing. The number of devices required for RC computing is small, leading to a reduction in energy consumption further. Based on the previous work, a bio-inspired neuromorphic sensory system is proposed.⁴⁹ In the system, leaky integrate-and-fire (LIF) neurons realized with volatile memristive devices Pt/Ag/TaOx/Pt are the basic computing units. The sensory neurons based on LIF neurons and a gas sensor array convert the chemical information of gases into neural spikes. Synaptic weights realized with non-volatile memristive devices Pt/Ta/TaOx/Pt conduct MAC calculations in parallel. Relay neurons based on LIF neurons classify the signals into classes finally. The resistive sensory information can be encoded into the frequency of voltage spikes by the LIF neurons, which is based on the periodic discharge of the capacitor with the help of a volatile memristor.

Spiking neural network (SNN) is an algorithm that utilizes spike coding of information with the potential improvement of energy efficiency. Kwon et al.⁵⁰ proposed a fusion of SNN and FET-type gas sensors (Fig. 3g). The sensitive material in the FET-type sensors reacted with gas at an appropriate temperature, resulting in the drain current (I_D) dependent on the type and concentration of the gas (Fig. 3h). The amplitude of current related to sensory information is encoded into the spike frequency for SNN implemented with the integrate-and-fire (IF) neuron (Fig. 3i). During the encoding process, the membrane capacitor is charged and discharged periodically, generating voltage spikes with frequency dependent on the sensory current. Non-volatile synaptic arrays consisting of the gated Schottky diode (GSD) are used for implementing the SNN, in which the synaptic weights are trained in a fully connected neural network first. Benefiting from the IF neuron, the sensor current can be converted into spike rates without using ADCs or DACs. With the help of a non-volatile memory array, in-memory computing is conducted without additional memory to store many weights. The spike encoding of information in SNN also leads to low power consumption processing.

Integration technology is also important for in-sensor computing. Shulaker et al.⁵¹ reported a three-dimensional in-sensor computing system for gas classification (Fig. 4a). In their system, each sensor is fabricated vertically above a memory cell, enabling each sensor to connect directly into its respective underlying memory cell with an inter-layer via (Fig. 4c). The gas sensor consisting of CNT-FET will change its resistance to a different gas (Fig. 4d). The test gas will affect the threshold voltage of functionalized CNTFETs, making it a sensitive resistive gas sensor. Another CNT-FET plays the role of a “divider resistor” connected serially but unexposed to the test gas (Fig. 4b). It converts resistance to the voltage which will be applied to a non-volatile memristor. The conductance of the memristor is dependent on the voltage and indicates the information of the gas. In this way, the analog signals can be written into memory in parallel without ADCs, realizing a massive sensing-to-memory bandwidth. Because of the variance of the memristor, the appearance of SET is relative to the voltage applied statically. Thus, the distribution and locations of the memory cell that are set to a low resistance state include information about the test gas. In the computing part, a support vector machine (SVM) is used for gas classification (Fig. 4e). Half adders composed of CNT-FET add the bit-wise multiplication results of the feature vector and weight vector. The feature vector is fetched from the memory array sequentially while the weight vector is trained and stored off-chip. Because of the serial data reading, the data transfer between half adders and off-chip memory will limit the processing speed. An additional memristor array used for in-memory computing can be considered to accelerate the classification process further owing to the elimination of serial data access.

Fig. 4 Three-dimensional integration of in-sensor computing. (a) Illustration of the nano system. The gas sensors are fabricated with carbon nanotube field effect transistors. Resistive random-access memory is utilized to store sensory information. The remaining logic circuits conduct the classification of gas. (b) Schematic of the array. Two CNTFETs are connected in series. Only one of them is exposed to gas while the other is covered by dielectrics. The divided voltage is applied to RRAM and sensory data are written into the memory directly with a massive sensing-to-memory bandwidth. (c) The memory part of the system. The RRAM and CNTFET are connected to form a 1T1R structure, with a low resistance state and high resistance state representing logic “1” and logic “0”. (d) The sensor part of the system. The drain current of CNTFET exposed to the gas contains information on gas types. The RRAM can be set by a higher divided voltage with a higher probability. (e) The computation part of the system. A support vector machine is conducted by the system. A pre-trained vector of weights and a sampled vector of features are multiplied and added by half-adders. Reproduced with permission. Copyright 2017, Springer Nature.

In the second level of in-sensor computing, MACs in the computing part are mainly conducted in analog format. Intermediate results can be used to generate voltages representing the input data of the next MACs without data transfer between memory and processors. No matter how many times the input data need to be inferred, synaptic weights stored in memristor arrays will not be fetched or written, which is the time-consuming part of von Neumann architecture. Besides, ADCs are omitted. The problems with power consumption and speed are optimized to some extent resulting from reducing the usage of communication protocol. In the olfactory field, the development of sensors is attractive. The gas sensors designed can be integrated with the computing part with some extra effort. To bridge the sensors with an in-memory computing array, analog signals should be carefully processed for compatibility.

2.3. The third level: in-sensor computing in a single device

In the previous level, analog signals are used to participate in computing directly and synaptic weights are stored in the same place without fetching and writing. However, analog signals from sensors are usually too weak to be applied to crossbar arrays directly. External operational amplifiers introduce external power consumption which is inconsistent with the goal of in-sensor computing. In the third level, sense and in-memory computing are carried out in a single device. To play the role of the sensor,⁵² the device should respond to environmental signals. To conduct in-memory computing, the format of responses should be able to participate in computing. These requirements are difficult to meet in single device.

Only a few examples have demonstrated in-sensor computing in the olfactory field, which is related to some junior processing tasks. Song et al.⁵³ proposed a bionic memory system with a gas-controlled memristive effect based on a single organic field-effect transistor (OFET). The PCDTPT thin film used in OFET serves as the sensing and storage layer to generate the drain current (I_D). The test gas (NO₂) is physisorbed on the PCDTPT film, leading to carriers generated to increase I_D. The sensing process is conducted at room temperature, resulting in a slow increase of I_D. With the test gas being turned off, the I_D decreased gradually for the same reason. The response speed, which is the faster the better in other gas sensors, was designed to be slow for the gas-controlled memristive behavior. Paired-pulse facilitation can be mimicked with gas pulses rather than voltage pulses. The analog sources are sensed by the OFET gas sensor and the data are stored in a format of I_D with a volatile characteristic, which is processed with the bio-inspired PPF in a single device.

In the third level of in-sensor computing, sensing, and computing should be implemented in the same device. It is a challenge for researchers to integrate both sensing and computing into one device, especially in the olfactory field. The goal of in-sensor computing is sensing and processing signals without ADCs and data transfer. In the olfactory field, devices need to exchange materials with gas which results in the extra difficulty of integrating computing parts into sensors compared with other applications. Therefore, there are only a few reports that meet the requirement of the third level in-sensor computing. The difficulty comes from two aspects, which can be termed functional limitation and geometry mismatch. For the functional limitation, traditional in-memory computing requires devices to perform MACs using their conductance while a large proportion of gas sensors converting olfactory signals to their resistance. The single device in the third level of in-sensor computing should utilize more kinds of properties to balance sensing and computing. For the geometry mismatch, the requirements for the arrangement of devices are different for the role of the sensing unit and computing unit. Addressing the limitation mentioned above allows more sensors to implement high-level in-sensor computing.

2.4. The fourth level: multi-modal in-sensor computing

The high-level algorithm commonly related to in-sensor computing is an artificial neural network, which generates intelligence from a huge amount of data. Except for the smartly designed models and powerful computing devices, the high-quality dataset involving sufficient information is also important. To reach high recognition accuracy, several types of analog sources can be processed together.^54–56 Based on the in-sensor computing architecture, more attention should be paid to multi-modal in-sensor computing.

Several analog sources can be designed to affect the behavior of devices in different dimensions. Wang et al.⁵⁷ proposed an MXene-ZnO memristor for multimodal in-sensor computing (Fig. 5a). The crossbar array is a structured ITO/MXene-ZnO/Al sensory memristor on the flexible PDMS substrate, in which MXene-ZnO is the switching layer. Normally, non-volatile memristor features that the resistance states can be modulated by electrical pulses, which is similar to the MXene-ZnO memristor under dark conditions. The resistive switching comes from the formation and abruption of oxygen vacancy conductive filaments. Besides, MXene-ZnO memristors can be set by optical pulses and reset by electrical pulses (Fig. 5b). Under ultraviolet light, the photons with appropriate energy will be absorbed by ZnO and the excitons will be generated which will be separated at the interface of MXene-ZnO. The photo-electrons will be trapped by MXene, forming an internal electrical field. In other words, the photogate effect can also induce the formation of oxygen vacancy conductive filaments, which is called photo-mediated resistive switching. Except for optical responses, the switching behavior also varies in different humidity (Fig. 5c). The on/off ratio decreases with the increase of RH. Under the ambient high RH, water vapor will absorb onto the MXene-ZnO via the double hydrogen bond. During the formation of oxygen vacancy conductive filaments, the growth will be restricted due to the electrostatic attraction between proton and oxygen vacancy. Therefore, the MXene-ZnO can respond to both optics and humidity, making it possible to construct a multi-modal in-sensor computing system. In the computing part, in-sensor noise filtering is implemented (Fig. 5d). Raw data are converted into light pulses used for setting the MXene-ZnO memristor under different humidity conditions while the resistance states represent the filtered output data used for subsequent classification. The noise supression can reduce the training cycles with the same target classification accuracy. The entire process is an attempt for multi-modal in-sensor computing, and the device can be designed to respond to several analog sources in different manners, except for the resistance being changed directly.

Fig. 5 Multi-modal in-sensor computing with an MXene-ZnO memristor. (a) The resistive switching characteristics of MXene-ZnO memristor with electrical stimulus. The inset shows the distribution of the statistic ON/OFF ratio. (b) Non-volatile photon-mediated resistive switching characteristics. Under UV light, the high-resistance states are closed to the low-resistance states with increasing light intensity. With the appropriate light intensity, the devices can be set without electrical stimulus. (c) Humidity-mediated resistive switching characteristics. The SET voltage and RESET voltage increase with increasing humidity. (d) Schematic illustration of high-level in-sensor computing. The sensing memristor is used to mimic synapses to implement weight updating. Reproduced with permission. Copyright 2021, Wiley.

3. Outlook

In the different levels of data processing architecture, the integration degree is increasing gradually which raises more strict requirements on the basic devices and materials. In the first level, discrete components can be designed separately. A lot of choices of materials are available to build sensors, memory, and processors. To alleviate von Neumann's bottleneck, several in-sensor computing methods are proposed to conduct sensing, storage, and processing without ADCs and data transfer. Except for the energy saving due to the absence of ADCs, additional attention should be paid to the bridging methods in converting the information from sensors to the format suitable for subsequent computing. With the further increase of integration degree, in-sensor computing is designed in a single device. The material needs to possess several controllable characteristics to participate in sensing and computing in a single device, which is more difficult in the olfactory field due to the involvement of the gases.

To construct high level in-sensor computing, devices that can play a role of not only sensors but also processors are needed. In the past several years, we have been developing a form of electronics that is based not on semiconductors but, instead, on metal nanoparticles. Metal nanoparticles with various choices of functional groups^58–60 are capable of being designed for complex application scenarios (Fig. 6). The AuNP-based device shows promising sensing properties in many types of analog stimulus, which are attractive for building multi-modal in-sensor computing. AuNP devices⁶¹ functionalized with various ligands have been utilized to implement cation sensors, acid/base gas sensors, humidity sensors, and so on. The electron tunneling probability and/or hopping rate in AuNP devices are very sensitive to the small changes in inter-particle distance and the dielectric environment between them. It is reported that the sensitivity for the volatile organic compounds is dependent on the relative alkyl chain length of the mediating –(CH₂)_n– vs. the capping –(CH₂)_m– structures. This suggests that AuNP devices are suitable for gas sensors with high sensitivity. Except for sensors, AuNP devices can also be used for processing circuits. Logic gates assembled from all-AuNP components were demonstrated, showing great potential to construct circuits with metal nanoparticles being the building blocks of both the information processing as well as sensing units. The electric-field-injected metallic cations can affect the electron tunneling/hopping energy barriers of AuNP devices, which have been utilized to implement artificial synapses with analog modulation of conductance.⁶² With the structure of Ag//MUA-DT AuNPs//Au,⁶³ the conductance of AuNP devices can represent the synaptic weights and conduct in-memory computing. With the capability of both sensing and processing, metal nanoparticle devices are suitable for building in-sensor computing in a single device. In addition, the involvement of several sensory data can improve the accuracy and reliability of classifications. Looking forward, we believe the metal nanoparticle devices are one of the appropriate candidates for in-sensor computing in the olfactory field.

Fig. 6 Metal nanoparticle devices. (a) The performance characteristics of logic gates assembled from all-AuNP components (NP gas sensors and NP humidity sensors). Inset: schematic diagram of logic gates. (b) Photograph of the OR gate comprising two types of sensors and two diodes. Scale bar, 1 cm. Inset: TEM of AuNPs film. (c) Scheme of the AuNPs artificial synapse in which a layer of functionalized AuNPs is sandwiched between an active Ag bottom electrode and an inert Au top electrode, Ag//MUA-DT AuNPs//Au. The right part shows the zero bias transmission spectra of the molecular bridges between two adjacent AuNPs with and without Ag⁺. (d) Current–voltage characteristics of the Ag//MUA-DT AuNPs//Au device. The conductance of the AuNP artificial synapse can be modulated in analog format. Reproduced with permission. Copyright 2022, ACS.

Author contributions

L. L. wrote the first draft of this manuscript. L. L., Y. Z., and Y. Y. revised the manuscript. Y. Y. conceived and supervised the project.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (22071037) and the Strategic Priority Research Program of Chinese Academy of Sciences (XDB36000000).

References

1. G. Indiveri and S. C. Liu, Proc. IEEE, 2015, 103, 1379–1397.
2. D. Ielmini and H. S. P. Wong, Nat. Electron., 2018, 1, 333–343.
3. Z. C. Tan, C. H. Chen, Y. Chae and G. C. Temes, IEEE Trans. Circuits Syst., 2020, 67, 4161–4173.
4. A. Sebastian, M. Le Gallo, R. Khaddam-Aljameh and E. Eleftheriou, Nat. Nanotechnol., 2020, 15, 529–544.
5. Y. N. Zhong, J. S. Tang, X. Y. Li, X. P. Liang, Z. W. Liu, Y. J. Li, Y. Xi, P. Yao, Z. Q. Hao, B. Gao, H. Qian and H. Q. Wu, Nat. Electron., 2022, 5, 672–681.
6. X. P. Liang, Y. A. Zhong, J. S. Tang, Z. W. Liu, P. Yao, K. Y. Sun, Q. T. Zhang, B. Gao, H. Heidari, H. Qian and H. Q. Wu, Nat. Commun., 2022, 13, 1549.
7. Z. R. Wang, S. Joshi, S. Savel'ev, W. H. Song, R. Midya, Y. N. Li, M. Y. Rao, P. Yan, S. Asapu, Y. Zhuo, H. Jiang, P. Lin, C. Li, J. H. Yoon, N. K. Upadhyay, J. M. Zhang, M. Hu, J. P. Strachan, M. Barnell, Q. Wu, H. Q. Wu, R. S. Williams, Q. F. Xia and J. J. Yang, Nat. Electron., 2018, 1, 137–145.
8. T. Q. Wan, B. J. Shao, S. J. Ma, Y. Zhou, Q. Li and Y. Chai, Adv. Mater., 2022, 202203830.
9. F. C. Zhou and Y. Chai, Nat. Electron., 2020, 3, 664–671.
10. H. W. Tan, Y. F. Zhou, Q. Z. Tao, J. Rosen and S. van Dijken, Nat. Commun., 2021, 12, 1120.
11. K. Roy, A. Jaiswal and P. Panda, Nature, 2019, 575, 607–617.
12. Z. R. Wang, H. Q. Wu, G. W. Burr, C. S. Hwang, K. L. Wang, Q. F. Xia and J. J. Yang, Nat. Rev. Mater., 2020, 5, 173–195.
13. H. Jang, H. Hinton, W. B. Jung, M. H. Lee, C. Kim, M. Park, S. K. Lee, S. Park and D. Ham, Nat. Electron., 2022, 5, 519–525.
14. D. Lee, M. Park, Y. Baek, B. Bae, J. Heo and K. Lee, Nat. Commun., 2022, 13, 5223.
15. S. Lee, R. M. Peng, C. M. Wu and M. Li, Nat. Commun., 2022, 13, 1485.
16. F. C. Zhou, Z. Zhou, J. W. Chen, T. H. Choy, J. L. Wang, N. Zhang, Z. Y. Lin, S. M. Yu, J. F. Kang, H. S. P. Wong and Y. Chai, Nat. Nanotechnol., 2019, 14, 776–782.
17. L. L. Gu, S. Poddar, Y. J. Lin, Z. H. Long, D. Q. Zhang, Q. P. Zhang, L. Shu, X. Qiu, M. Kam, A. Javey and Z. Y. Fan, Nature, 2020, 581, 278–282.
18. A. Moin, A. Zhou, A. Rahimi, A. Menon, S. Benatti, G. Alexandrov, S. Tamakloe, J. N. Ting, N. Yamamoto, Y. Khan, F. Burghardt, L. Benini, A. C. Arias and J. M. Rabaey, Nat. Electron., 2021, 4, 54–63.
19. M. Wang, Z. Yan, T. Wang, P. Q. Cai, S. Y. Gao, Y. Zeng, C. J. Wan, H. Wang, L. Pan, J. C. Yu, S. W. Pan, K. He, J. Lu and X. D. Chen, Nat. Electron., 2020, 3, 563–570.
20. H. T. Yang, J. L. Li, X. Xiao, J. H. Wang, Y. F. Li, K. R. Li, Z. P. Li, H. C. Yang, Q. Wang, J. Yang, J. S. Ho, P. L. Yeh, K. Mouthaan, X. N. Wang, S. Shah and P. Y. Chen, Nat. Commun., 2022, 13, 5311.
21. Y. Kim, A. Chortos, W. T. Xu, Y. X. Liu, J. Y. Oh, D. Son, J. Kang, A. M. Foudeh, C. X. Zhu, Y. Lee, S. M. Niu, J. Liu, R. Pfattner, Z. N. Bao and T. W. Lee, Science, 2018, 360, 998–1003.
22. Y. P. Zang, H. G. Shen, D. Z. Huang, C. A. Di and D. B. Zhu, Adv. Mater., 2017, 29, 1606088.
23. H. Lin, M. Jang and K. S. Suslick, J. Am. Chem. Soc., 2011, 133, 16786–16789.
24. J. R. Askim, Z. Li, M. K. LaGasse, J. M. Rankin and K. S. Suslick, Chem. Sci., 2016, 7, 199–206.
25. Z. Li, W. P. Bassett, J. R. Askim and K. S. Suslick, Chem. Commun., 2015, 51, 15312–15315.
26. Z. Li, M. Fang, M. K. LaGasse, J. R. Askim and K. S. Suslick, Angew. Chem., Int. Ed., 2017, 56, 9860–9863.
27. M. C. Janzen, J. B. Ponder, D. P. Bailey, C. K. Ingison and K. S. Suslick, Anal. Chem., 2006, 78, 3591–3600.
28. H. Dong, X. Zheng, C. Cheng, L. Qian, Y. Cui, W. Wu, Q. Liu, X. Chen, Y. Lu, Q. Yang, F. Zhang and D. Wang, Adv. Sci., 2023, 10, 2206699.
29. Z. Li and K. S. Suslick, ACS Sens., 2016, 1, 1330–1335.
30. Y. Cui, X. Zheng, C. Shen, L. Qian, H. Dong, Q. Liu, X. Chen, Q. Yang, F. Zhang and D. Wang, ACS Sens., 2023, 8, 71–79.
31. O. S. Kwon, H. S. Song, S. J. Park, S. H. Lee, J. H. An, J. W. Park, H. Yang, H. Yoon, J. Bae, T. H. Park and J. Jang, Nano Lett., 2015, 15, 6559–6567.
32. G. F. Wang, Y. S. Li, Z. Z. Cai and X. C. Dou, Adv. Mater., 2020, 32, 1907043.
33. G. Zeng, C. Wu, Y. Chang, C. Zhou, B. B. Chen, M. L. Zhang, J. Y. Li, X. X. Duan, Q. R. Yang and W. Pang, ACS Sens., 2019, 4, 1524–1533.
34. C. Jirayupat, K. Nagashima, T. Hosomi, T. Takahashi, B. Samransuksamer, Y. Hanai, A. Nakao, M. Nakatani, J. Liu, G. Zhang, W. Tanaka, M. Kanai, T. Yasui, Y. Baba and T. Yanagida, Chem. Commun., 2022, 58, 6377–6380.
35. M. Liu, Y. Zhang, J. Wang, N. Qin, H. Yang, K. Sun, J. Hao, L. Shu, J. Liu, Q. Chen, P. Zhang and T. H. Tao, Nat. Commun., 2022, 13, 79.
36. W. W. Zhang, L. Wang, J. Chen, X. Bi, C. S. Chen, J. Zhang and V. Hans, IEEE Sens. J., 2021, 21, 18459–18468.
37. B. Wang, J. Zhang, W. Li, Y. Zhang, T. Wang, Q. Lu, H. Sun, L. Huang, X. Liang, F. Liu, P. Sun and G. Lu, Sens. Actuators, B, 2023, 377, 133049.
38. T. Zhang, W. Ren, F. Xiao, J. Li, B. Zu and X. Dou, Eng. Regener., 2022, 3, 427–439.
39. C. Qin, Y. Wang, J. Hu, T. Wang, D. Liu, J. Dong and Y. Lu, Adv. Sci., 2023, 10, 2204726.
40. Y. Shi, F. R. Gong, M. Y. Wang, J. J. Liv, Y. N. Wu and H. Men, J. Food Eng., 2019, 263, 437–445.
41. X. W. Chen, J. Shi, T. Wang, S. Y. Zheng, W. Lv, X. Y. Chen, J. H. Yang, M. Zeng, N. T. Hu, Y. J. Su, H. Wei, Z. H. Zhou and Z. Yang, ACS Sens., 2022, 7, 816–826.
42. S. H. Hou, J. S. Yu, X. M. Zhuang, D. F. Li, Y. M. Liu, Z. Gao, T. Y. Sun, F. Wang and X. E. Yu, ACS Appl. Mater. Interfaces, 2019, 11, 44521–44527.
43. K. S. Kim, C. H. Ahn, S. H. Jung, S. W. Cho and H. K. Cho, ACS Appl. Mater. Interfaces, 2018, 10, 10185–10193.
44. Z. Yuan, M. Bariya, H. M. Fahad, J. B. Wu, R. Han, N. Gupta and A. Javey, Adv. Mater., 2020, 32, 1908385.
45. Z. Wang, L. Z. Huang, X. F. Zhu, X. Zhou and L. F. Chi, Adv. Mater., 2017, 29, 1703192.
46. G. X. Duan, S. M. Huang, Z. H. Feng, P. Xie, F. Zhang, Y. Zhou and S. T. Han, Adv. Funct. Mater., 2023, 33, 202209969.
47. J. K. Han, M. Kang, J. Jeong, I. Cho, J. M. Yu, K. J. Yoon, I. Park and Y. K. Choi, Adv. Sci., 2022, 9, 2106017.
48. T. Wang, H. M. Huang, X. X. Wang and X. Guo, InfoMat, 2021, 3, 804–813.
49. T. Wang, X. X. Wang, J. Wen, Z. Y. Shao, H. M. Huang and X. Guo, Adv. Intell. Syst., 2022, 4, 2200047.
50. D. Kwon, G. Jung, W. Shin, Y. Jeong, S. Hong, S. Oh, J. Kim, J. H. Bae, B. G. Park and J. H. Lee, Sens. Actuators, B, 2021, 345, 130419.
51. M. M. Shulaker, G. Hills, R. S. Park, R. T. Howe, K. Saraswat, H. S. P. Wong and S. Mitra, Nature, 2017, 547, 74–78.
52. C. Qian, Y. Choi, Y. J. Choi, S. Kim, Y. Y. Choi, D. G. Roe, M. S. Kang, J. Sun and J. H. Cho, Adv. Mater., 2020, 32, 2002653.
53. Z. Q. Song, Y. H. Tong, X. L. Zhao, H. Ren, Q. X. Tang and Y. C. Liu, Mater. Horiz., 2019, 6, 717–726.
54. Q. Chen, J. Li, W. H. Fu, Y. H. Yang, W. Q. Zhu and J. H. Zhang, Sens. Actuators, B, 2020, 323, 128676.
55. E. X. Wu, Y. Xie, B. Yuan, H. Zhang, X. D. Hu, J. Liu and D. H. Zhang, ACS Sens., 2018, 3, 1719–1726.
56. Y. Hong, M. L. Wu, J. H. Bae, S. O. B. Hong, Y. E. O. Jeong, D. Jang, J. S. Kim, C. S. Hwang, B. G. Park and J. H. Lee, Sens. Actuators, B, 2020, 302, 127147.
57. Y. Wang, Y. Gong, L. Yang, Z. Y. Xiong, Z. Y. Lv, X. C. Xing, Y. Zhou, B. Zhang, C. L. Su, Q. F. Liao and S. T. Han, Adv. Funct. Mater., 2021, 31, 2100144.
58. X. Zhao, J. H. Guo, J. Y. Wang and Y. Yan, Small, 2022, 19, 2205136.
59. X. Zhao, B. Tu, M. Y. Li, X. J. Feng, Y. C. Zhang, Q. J. Fang, T. H. Li, B. A. Grzybowski and Y. Yan, Sci. Adv., 2018, 4, eaau3546.
60. X. Zhao, L. Yang, J. H. Guo, T. Xiao, Y. Zhou, Y. C. Zhang, B. Tu, T. H. Li, B. A. Grzybowski and Y. Yan, Nat. Electron., 2021, 4, 109–115.
61. Y. Yan, S. C. Warren, P. Fuller and B. A. Grzybowski, Nat. Nanotechnol., 2016, 11, 603–608.
62. J. H. Guo, L. Liu, B. A. Bian, J. Y. Wang, X. Zhao, Y. C. Zhang and Y. Yan, Adv. Funct. Mater., 2023, 202212666.
63. J. H. Guo, L. Liu, B. A. Bian, J. Y. Wang, X. Zhao, Y. C. Zhang and Y. Yan, Nano Lett., 2022, 22, 6794–6801.

---