Bingjie Guo‡
a,
Xiaolong Zhong‡b,
Zhe Yu*c,
Zhilong Hea,
Shuzhi Liua,
Zhixin Wub,
Sixian Liub,
Yanbo Guob,
Weilin Chenb,
Hongxiao Duanb,
Jianmin Zengb,
Pingqi Gaoc,
Bin Zhangd,
Qian Chende,
Haidong Hee,
Yu Chen*d and
Gang Liu*ab
aSchool of Chemistry and Chemical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China. E-mail: gang.liu@sjtu.edu.cn
bDepartment of Micro/Nano Electronics, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
cSchool of Materials, Sun Yat-Sen University, Guangzhou, Guangdong 510275, China
dSchool of Chemistry and Molecular Engineering, East China University of Science and Technology, Shanghai 200237, China
eMinhang Hospital, Fudan University, Shanghai 201199, China
First published on 25th June 2024
Affective computing, representing the forefront of human–machine interaction, is confronted with the pressing challenges of the execution speed and power consumption brought by the transmission of massive data. Herein, we introduce a bionic organic memristor inspired by the ligand-gated ion channels (LGICs) to facilitate near-sensor affective computing based on electroencephalography (EEG). It is constructed from a coordination polymer comprising Co ions and benzothiadiazole (Co–BTA), featuring multiple switching sites for redox reactions. Through advanced characterizations and theoretical calculations, we demonstrate that when subjected to a bias voltage, only the site where Co ions bind with N atoms from four BTA molecules becomes activated, while others remain inert. This remarkable phenomenon resembles the selective in situ activation of LGICs on the postsynaptic membrane for neural signal regulation. Consequently, the bionic organic memristor network exhibits outstanding reliability (200000 cycles), exceptional integration level (210 pixels), ultra-low energy consumption (4.05 pJ), and fast switching speed (94 ns). Moreover, the built near-sensor system based on it achieves emotion recognition with an accuracy exceeding 95%. This research substantively adds to the ambition of realizing empathetic interaction and presents an appealing bionic approach for the development of novel electronic devices.
New conceptsWe report a bionic organic memristor network for near-sensor affective computing and high precision emotion recognition. Existing studies show that affective computing significantly enhances human–machine interaction (HMI) by gathering physiological electroencephalogram (EEG) signals, while its substantial data processing imposes considerable challenges on the execution speed and power consumption of the system. In this work, we purposely utilize a coordination polymer comprising Co ions and benzothiadiazole (Co–BTA) with multiple switching sites for in situ redox reactions, inspired by the selective in situ activation of ligand-gated ion channels, to construct a bionic organic memristor network. Our results show that the bionic organic memristor network based on the polymer exhibits excellent resistive switching performances, including reliability (200000 cycles), exceptional integration level (210 pixels), ultra-low energy consumption (1.08 pJ), and ultra-fast switching speed (25 ns). An implementation utilizing this bionic organic memristor network for EEG-based emotion recognition achieves high accuracy (>95%). This work provides an attractive bio-inspired method for the development of novel electronic devices and an application of energy-efficient near-sensor affective computing. |
Memristors, emerging devices endowed with large-scale in-memory computing capabilities, offer a significant avenue for addressing the challenges encountered in EEG-based affective computing through near-sensor data processing.11 This approach has proven effective in mitigating the complexities associated with the transmission of voluminous data in diverse intelligent analyses. For instance, Wei D. Lu's research group and Heejun Yang's research group have employed the Pd/WOx/Au memristor network and Au/SnS/Cr memristor network, respectively, to achieve fast and low-power execution of handwritten character recognition tasks.12,13 Additionally, the team led by Ming Liu and Qi Liu integrated a Pt/Ti/Nb2O5−x/Pt memristor network with sensors to build a multimode-fused spiking neuron capable of recognizing pressure and temperature by near-sensors.14 It is a regret that the aforementioned memristors are constructed from rigid inorganic materials, creating an inherent chasm in mechanical properties separating them and the flexible devices employed for EEG signal collection. In contrast, organic memristors with the characteristic of intrinsic flexibility are more suitable for seamless integration with flexible sensors, facilitating the development of near-sensor EEG-based affective computing.15
With the data explosion in the big data era, memristors are limited in their ability to process large amounts of data of near-sensor EEG-based affective computing. As the primary inspiration, the synapse is renowned for its ability to transmit neural signals in a high-throughput manner, which is conferred by the selective in situ activation of ligand-gated ion channels (LGICs) located on the postsynaptic membrane, as elucidated in Scheme 1a. After binding to neurotransmitters released from presynaptic neurons, LGICs open in situ to allow free ions within the synaptic gap to enter the postsynaptic neuron, accompanied by the generation of action potentials on the cell membrane. Different from other ion channels, such as mechanically-gated ion channels present on the mechanoreceptor,16 only a portion of the LGICs that can form chemical coordination with neurotransmitters are selectively opened uncontrollably. Recently, it has been widely reported that bionic organic memristors exhibit impressive performance attributes.17 These advancements are due to their utilization of organic semiconductor materials, which offer distinct advantages in molecular design, synthesis methodologies, and large-scale production.18,19 It provides a feasible technical route to solving the dilemma of EEG-based affective computing, that is, to develop a bionic organic memristor grounded in the distinctive operational principle of LGICs.
In this study, we have developed a bionic organic memristor that mimics LGICs by choosing a coordination polymer composed of cobalt ions and benzothiadiazole (Co–BTA) as an active material (Scheme 1b). Because substantial electron delocalization occurs as the hybridization of π–d conjugation between inorganic metals and the organic ligand,20 the Co–BTA film possesses abundant Co–N coordination bonds to serve as active sites for resistive switching through in situ redox transformation. Meanwhile, there are differences in the number of BTA molecules that provide N atoms for Co ions in these sites, resulting in a significant dispersion of their highest occupied molecular orbitals (HOMOs) under the steric hindrance effect with different strengths. It lays the foundation for selective in situ activation of our bionic organic memristor, which has been verified by various characterization analyzes and theoretical calculations. Therefore, the bionic organic memristor network based on Co–BTA exhibits excellent resistive switching characteristics, including outstanding reliability (200000 cycles), high integration level (210 pixels), ultra-low energy consumption (4.05 pJ), and fast switching speed (94 ns). Finally, our bionic organic memristor network was used as a near-sensor system to perform the task of EEG-based emotional computing, and the results showed that it can complete complex emotion recognition with an accuracy of 95%. This research opens up new possibilities for EEG-based affective computing and provides valuable insights for the advancement of near-sensor systems in HIM.
The excellent processing characteristic of Co–BTA film lays the material foundation for the development of high-performance devices, so a bionic organic memristor with the structure of Au/Co–BTA/indium tin oxide (ITO) was successfully manufactured by us. As shown in Fig. 1c and d, it can be found that its active layer of Co–BTA exhibits extremely high uniformity, such as a thickness of about 100 nm tested by scanning electron microscopy (SEM) and a roughness of less than 4 nm tested by atomic force microscopy (AFM). Furthermore, the ITO bottom electrode and Au top electrode of the bionic organic memristor were transformed into a crossbar structure by magnetron sputtering, thereby expanding the device into a highly integrated memristor network with 32 × 32 pixels (Fig. 1e), whose linewidth is 150 μm as shown in Fig. S4 of the ESI.† The intrinsic flexibility allows it to be bent substantially to match the mechanical properties of flexible EEG electrodes (the inset of Fig. 1e). Please note that SiO2/Si was selected as the substrate for fabricating the memristor network, so it should be ultrasonically cleaned with deionized water, ethanol, and acetone in sequence before use, and each cleaning lasts for 30 minutes.
The current–voltage curves of our bionic organic memristor are recorded in Fig. 2a, and it shows that the memristor can switch from a high resistance state (HRS) to a low resistance state (LRS) at the voltage of 1.5 V (set voltage, Vs), exhibiting a sudden 20-fold increase in current. When applying a reverse voltage up to −0.75 V (reset voltage, Vr), the device will return to the HRS and remain stable for a long time. Repeating the above operation for 600 times, there is no significant change in the switching characteristics of our memristor, which can be demonstrated from the statistical results with Vs of 1.5 ± 0.50 V, Vr of −0.75 ± 0.25 V, HRS value of 3045.4 ± 1561.1 Ω, and LRS value of 190.4 ± 86.8 Ω (Fig. S5, the ESI†). Our bionic organic memristor also exhibits a stable non-volatile time retention (>10000 s) and an ultra-low energy consumption for switching (4.05 pJ), as shown in Fig. 2b and c. Note that depending on the experimental setups, e.g. metal probes, cablings, measuring systems, etc., the transient current responses of memristor devices may or may not show initial spikes upon the application of voltage stresses.23–29 In order to comprehensively reflect the responding rate of our memristor, we define the switching speed as the time interval between when the stressing voltage starts to apply and when the current response reaches its maximum. As depicted in Fig. 2c, the present organic memristor shows a fast switching speed of 94 ns. Meanwhile, we tested the effect of bending on the performance of the memristor based on Co–BTA (Fig. S6 of the ESI†). During 100 bending cycles with a bending radius of 1.83 cm, the average fluctuation of HRS/LRS values and switching voltages is lower than 6.75%, which means that the prepared bionic organic memristor can match a variety of wearable flexible sensors including EEG electrodes. To fully elaborate the reliability, we have tested the device-to-device (D2D) reliability and cycle-to-cycle (C2C) endurance of the Co–BTA organic memristor. The 49 pixels in memristor network were selected for D2D testing (Fig. S7 of the ESI†), and its statistical analyses are recorded in Fig. 2d, showing that the average fluctuation of the switching voltages is only 9.48%, yet that of the HRS/LRS resistance ratios reaches 28.7%. Nevertheless, the ON/OFF ratios are maintained for larger than 10 for all devices, which can be reliably distinguished by external CMOS read circuits. As described in the Experimental section and Fig. 2e, the Co–BTA memristor exhibits promising endurance performance, with the device resistances and ON/OFF ratios only varying for 3.77% during 200000 continuous operating steps.30 All these figures of merits make the reliability of our biomimetic memristor significantly superior to other reported organic memristors (Fig. 2f).31–45
Before verifying the bionic resistive switching behavior of the Co–BTA film, it is necessary to confirm the types of active sites formed by Co ions in it, so we employ a variety of characterization techniques and advanced theoretical calculations. From the test results of X-ray photoelectron spectroscopy (XPS) in Fig. S8 of the ESI,† it can be found that the composition of the film mainly consists of Co, N, and C elements, and the deconvolution of the N 1s peaks at 399.0 eV and 401.5 eV reveal the presence of anilinic amine (–N–H–) and quinoid imine (–N) groups, respectively. These peaks indicate the formation of conjugation and electronic delocalization within the π–d conjugated structure. The binding energy of the Co 2p3/2 peak and the accompanying satellite peak is 781.9 eV and 787.1 eV, respectively, consistent with Co2+ compounds rather than Co3+ compounds.30,46,47 Based on these observations, we believe that the N atoms coordinated with Co ions can originate from 2, 3, or 4 BTA molecules (Z = 2, 3, or 4), which means that there are three different types of active sites in the Co–BTA. It is further characterized and confirmed by density functional theory (DFT). In the active sites with Z of 2, or 3, the Co ions and their coordinated BTA molecules are in the coplanar state, while the site with Z of 4 is like butterflies flapping their wings in space, i.e. four non-coplanar coordination molecules, as shown in Fig. 3a–c. The more structural images of the three active sites under different visual angles can be found in Fig. S9 of the ESI,† and their structural parameters are detailed in Tables S1 and S2 of the ESI.† All tests show that the Co–BTA film has three types of active sites, laying a data foundation for analyzing its bionic resistive switching behavior.
According to the electrostatic potential calculated by DFT, there is a significant electron-deficient effect in the Co–N coordination bond across all active sites, and their energies of the highest occupied molecular orbit (HOMO) respectively are −3.35 eV (Z = 2), −3.38 eV (Z = 3), and −3.00 eV (Z = 4). This suggests that the resistive switching of the Co–BTA film is achieved through the redox transition of the Co–N bond at the active sites, where the active site with Z of 4 is the easiest among the three to lose electrons for transition under the bias voltage. Considering that the DFT calculation is ideal, an electrochemical Pt-tip/Co–BTA/ITO device (Fig. S11 of the ESI†) is designed for in situ characterization of Raman spectroscopy to explore the microscopic changes of an actual sample during resistance switching. The results of Raman spectral analysis of the film under different bias voltages are recorded in Fig. 3d. As we all know, the infrared absorption peak of chemical bonds with similar structures red-shifts with the increase of the steric hindrance. From the chemical structures of the three active sites, the steric hindrance of the site with Z of 2 and that of the site with Z of 4 is the smallest and largest, respectively, so the three absorption peaks of the Co–N bond (486.7 cm−1, 603.2 cm−1, and 694.4 cm−1) from left to right in the infrared spectrum respectively belong to the active sites with Z of 2, 3, and 4 under the bias voltage of 0 V. When the Co–BTA film is subjected to the bias voltages of 1 V, 2 V, and 3 V, there is a new absorption peak (740 cm−1) appearing on the leftmost absorption peak of the Co–N bond, which is attributed to the disconnection of the Co–N bond and the oxidation of Co ions in the active site with Z of 4. When applying the bias voltage of −1 V, and −2 V, the chemical environment of this active site returns to the original state with the gradual disappearance of the new peak. Importantly, the absorption peaks of other active sites (Z = 2, and 3) remain unchanged during this reversible reaction.
A similar phenomenon induced by chemical state transitions also occurs in the test of in-site ultraviolet-visible (UV-vis) spectroscopy. Initially, there are two main peaks at the wavelength of 320 nm (blue part) and 465 nm (green part) in Fig. 3e, which is influenced by the π–π* jumping of the conjugate structure and the d–π* metal-to-ligand charge transfer. The corroborative evidence for this part is described in Fig. S12 of the ESI,† including the UV-vis spectra of the individual components and their analyses. During oxidation under positive voltage, these two peaks gradually differentiate due to the destruction of the d–π conjugated structure of Co–N–C, yet the differentiated peaks recombine under a reverse voltage, indicating the restoration of the structure to its original state. These results can prove that our sample has a unique selective in situ redox transition for resistive switching, which is similar to the selective in situ activation of LGICs. Specifically, the active sites with Z of 2, 3, and 4 in the Co–BTA film can all be in situ activated through redox reactions, where the activation energy of the site with Z of 4 is the lowest, becoming the only one that is selectively in situ active under the stimulation of small voltage. This behavior is exactly the same as that of LGICs on the postsynaptic membrane, which is enough to show that bionic molecular design is a potential strategy for improving device performance.
In order to show the potential for affective computing, emotion recognition is chosen as the application demonstration of our biomimetic organic memristor network, but before that, we have tested the computing capabilities of the bionic organic memristor as a neuromorphic device, including spike rate dependent plasticity (SRDP), spike timing dependent plasticity (STDP), and multiple conductance state regulation. In the SRDP tests, ten consecutive pulses of 10 μs with 0.9 V were used to stimulate the device, and the pulse frequency was altered by adjusting the interval time (25 μs, 15 μs, 10 μs, 5 μs, 4 μs, 3 μs, 2 μs, and 1 μs) between pulses to observe its impact on the synaptic conductance. The raw data tested on the device and the normalized results are exhibited in Fig. S13 of the ESI† and Fig. 4a, respectively. The synaptic conductance difference (Δω) is calculated by subtracting the 1st pulse conductance from the 10th pulse conductance. It can be found that the response current of the device increases with the increase of the stimulation pulse number and interval time, exhibiting a synaptic feature-related enhancement mechanism for the conductance dependence of time intervals (Fig. 4b). Fig. 4c depicts the STDP characteristics of the present Co–BTA memristor, with the changes of device conductance presented in the second and fourth quadrants of the plot. Note that although the Hebbian rule48 describes four types of learning behaviors of biological nerve systems, it does not mean that all memristive synapses working on various switching and carrier transport mechanisms have to obey these traces exactly. Herein, the selective in situ activation and time-dependent oxidation of the Z = 4 redox center accounts for the unique memristive switching behaviors of the Co–BTA memristor devices, leading to anomalous STDP curves that cannot be fitted by any of the existing learning models of the biological Hebbian rule. Nevertheless, this new STDP curve may offer potential possibility to widen the circuit functionality of artificial memristive synapses. On the other hand, when a voltage from −0.4 V to 0.02 V is applied in the step of 0.02 V, as shown in Fig. S14 of the ESI,† the device is reset from HRS to LRS with the continuous current variation, showing an ability for multiple conductance state regulation. It has a total of 288 continuous and adjustable conductance states (Fig. 4d), of which 16 conductance states between 0.32 mA and 0.49 mA were used to regulate the weights for emotion recognition due to their good linear change and time stability (>104 s, Fig. 4e).
Fig. 5a describes the working principle of the near-sensing system, which performs EEG-based affective computing by using the fabricated bionic organic memristor network to simulate a broad learning system (BLS). As a very popular artificial intelligence algorithm for processing EEG signals, BLS has the characteristics of real-time processing, low time complexity, and few-shot learning, so that its requirements for computing power and storage capacity of hardware are much lower than traditional deep learning models.49–51 Our system is evaluated to recognize four target-emotions, that is happiness, excitement, sadness, and anger, which are divided into two categories of arousal dimension (ranging from negative to positive) and valence dimension (ranging from weak to strong). The detailed affective computing process is described in the Experimental section. After 26 times of iterative training (Fig. 5b), the system achieves exceptional recognition accuracy rates of 95.04% for arousal and 95.02% for valence dimensions on the testing datasets, which means the built system based on a bionic organic memristor can successfully achieve high-accuracy emotion recognition through affective computing. The confusion matrices for the arousal and valence emotional dimensions are presented in Fig. 5c and d, respectively. Interestingly, the number and distribution of samples with identification errors in the confusion matrices are very close (arousal: 59, and 68; valence: 60, and 68), which prompted us to conduct simulation on computer software. Fig. 5e shows the recognition results of the software-execution (arousal: 94.97% ± 0.61%; valence: 95.01% ± 0.71%), which is almost the same as that of the memristor-based hardware system, and their weights of the output layer closely follow a linear narrow distribution ranging from 0.002 to −0.002 (Fig. S15 of the ESI†). It reveals that there is a strong linear mapping between the software-trained weights and the linear conductance of our bionic organic memristor, allowing the emotion recognition system built on it to demonstrate the unique ability to realize high-precision affective computing, which is expected to become an important component in the next-generation HMI technology.
(1) |
(2) |
(3) |
The instances that underwent DEFS were fed into the BLS for training. To enhance the model's generalization performance and robustness, the final recognition performance was confirmed by averaging the accuracies obtained from 10-fold cross-validation experiments on both the valence and arousal emotional dimensions. The used number of feature nodes is 100 with 10 batches for feature nodes. The number of enhancement nodes is 100, and the sigmoid transfer function with a shrinkage factor of 0.9 for establishment. The weights randomly generated are drawn from the normal distribution on the interval [−1, 1], and the regularization parameter of ridge regression is 0.001.
Footnotes |
† Electronic supplementary information (ESI) available: Fig. S1–S15 and Tables S1 and S2. See DOI: https://doi.org/10.1039/d3mh01950k |
‡ These authors contributed equally to this work. |
This journal is © The Royal Society of Chemistry 2024 |