Machine learning-assisted flexible dual modal sensor for multi-sensing detection and target object recognition in the grasping process

Abstract

Multi-modal information data is important for the grasping process of robotic fingers. Simultaneous bimodal perceiving of non-contact proximity distances and contact pressure stimuli is widely desired for artificial intelligence electronics, such as electronic skin and health monitoring. It is a challenge to independently detect and process different signals for target recognition without cross-coupling. A machine learning-assisted flexible dual modal sensor (FDMS) was developed for robotic electronic skin application to simultaneously engage in proximity distance and contact pressure measurements to fully process perception during grasping. FDMSs with a multi-layer structure (polydimethylsiloxane film, conductive silver paste, silicone rubber, and hydrogel film in layers) were developed for robotic electronic skin application. FDMSs with conductive silver coils were designed for proximity perception due to the variable capacitance value. A single electrode mode triboelectric nanogenerator (TENG) sensor with frictional electric effect and electrostatic induction was applied for contact pressure measurements. The AlexNet neural network was adopted to target material and hardness recognition from FDMSs in the robot-grasping process, and it achieved a success recognition rate of 93.49% for different materials and 92.22% for different hardness values. Compared to other algorithms, the performance of the AlexNet neural network was superior for target material recognition, which would improve human–robot interaction ability. The robot electronic skin exhibited dual perception feedback capability in proximity and contact perception with excellent flexibility and stability, which has great potential for human–robot interactions, soft robotics, and biomedical applications.

---

1. Introduction

With the rapid development of robotics technology, it is increasingly important for safe interactions to occur between robots, humans, and other objects with the assistance of multi-data sensing during the operation process.^1–3 For intricate tasks performed by the robot in more diverse and complex environments, it is particularly crucial for robots to possess perceptual sensing abilities similar to those of human skin.^4,5 Some inherent limitations remain for traditional rigid sensors interacting with soft objects due to the lack of flexibility and adaptability.⁶

Flexible substrates based on polymer materials such as polydimethylsiloxane (PDMS), polyethylene terephthalate (PET), and polyimide (PI) have been adopted for flexible sensor systems for multi-sensing data collection.^4,7–9 Therefore, it is more demanding and challenging to develop a flexible multi-functional sensor to detect multi-sensing data to improve friendly human–machine interactions and full perception in the grasping process.

Flexible electronic skin (e-skin)^10,11 is a new type of sensor that mimics the multi-sensory functions of human skin, and has wide applications such as in human–robot interaction (HRI),^12,13 structural health monitoring,¹⁴ and smart wearable devices,^8,15 and can be applied to detect various sensing parameter measurements such as proximity, temperature, and humidity.^16–18 Flexible tactile sensors with thin-film thermistors have been developed to implement multimodal perceptions of pressure, temperature, matter thermal property, texture, and slippage.^19–21 A self-powered, skin adhesive, and flexible human–machine interface (SSFHMI) based on a frictional nanogenerator has been developed for multi-sensory data recording and multiple interactions with a microcontroller and a programmable touch operation platform.^9,22

We propose the construction of a capacitive proximity and pressure dual modal sensor designed with spiral electrodes, which uses cross-linked calcium alginate film as the dielectric layer and can achieve continuous human–machine interaction.²³ The bionic skin is used for droplet environment reconnaissance measurements by converting various dynamic droplet-sliding behaviors into electrical signals based on triboelectricity.⁴ A hybrid resistive-supercapacitor (HRSC) has been developed that can achieve high sensitivity over a wide linear range, and expand application scenarios in prosthetic sensing and electronic skin.²⁴ Biomimetic electronic skin (BES) has been applied to robots, and excellent dynamic/static perception and material cognition functions have been achieved.^5,13 The electronic skin achieved spatial mapping applications based on dual-mode sensing (proximity and pressure) through edge and ion electron effects, such as non-contact measurements of three-dimensional (3D) objects and contact recognition.^25,26

Flexible sensors with machine learning algorithms have been designed as intelligent cognitive systems for real-time recognition of target objects.^27–29 Machine learning-driven micropyramid array bimodal (MAB) electronic skin was developed to enhance multi-sensory capabilities.^8,30 With the assistance of multi-layer perceptrons and convolutional neural networks, the MAB e-skin was trained to learn proximity and pressure sensing modes with 6 materials and 10 surface shapes, thereby achieving precise perception of different objects within one proximity pressure cycle.^22,31

High-performance bimodal sensing with high tactile sensitivity and biocompatibility is required in robotic electronic skin for multi-sensory data recording in the operation process. Flexible dual modal sensors combined with machine learning technology should be designed and developed for multi-sensory data collection and full process perception to achieve target recognition in the grasping process of robot finger manipulation.

A machine learning-assisted flexible dual modal sensor was developed for robotic electronic skin application and applied to multi-sensory data measurement and target recognition in the grasping process. Different machine learning algorithms were adopted to target material and hardness recognition by flexible dual modal sensor. The main contributions of the work are:

(1) The flexible dual modal sensor (FDMS) with proximity distance and contact pressure sensing function abilities, which was designed to fully and simultaneously process perception in the grasping process of the robotic fingers. A flexible dual modal sensor with multi-layer structure (PDMS film, conductive silver paste, silicone rubber, and hydrogel film in layers) was developed for robotic electronic skin application. A flexible conductive silver coil integrated into the FDMS was designed for contact distance perception due to the variable capacitance value. The single electrode mode triboelectric nanogenerator (TENG) sensor integrated with the FDMS was used for contact pressure measurements with a frictional electric effect and electrostatic induction. A flexible dual modal sensor can provide stable and reliable tactile sensing signals in the grasping process.

(2) The AlexNet neural network, which was developed to target material and hardness recognition with dual channel sensing data from the FDMS in the robot-grasping process. Obstacle avoidance of the robotic fingers was achieved based on the proximity sensors with conductive silver coils in the FDMS. It achieved a success recognition rate of 93.49% for touching different materials and 92.22% for the different hardnesses of the grasped object by the proposed AlexNet neural network. The AlexNet neural network-assisted flexible dual modal sensor was developed for multi-sensory data measurement and target recognition in the grasping process, which would improve the HRI ability from robot information signal to human.

The main work is arranged as follows: the design and principle of robotic electronic skin based on the FDMS with proximity distance and contact pressure measurements are depicted in section 2. The material preparation and fabrication process of the FDMS are described in section 3. The electrical performance of the FDMS with proximity distance and contact pressure function ability has been illustrated in section 4. The experimental results demonstrated that the AlexNet neural network has been sufficiently developed to target material and hardness recognition with dual channel sensing data from the FDMS in the robot-grasping process. This article ends with a summary and conclusion.

2. Design and principle

Robot electronic skin is designed to simulate human skin's perception ability for proximity distance and contact pressure measurement in the robot-grasping process. Fig. 1(a) depicts a FDMS composed of PDMS film, conductive silver paste, silicone rubber, and hydrogel film. The FDMS was laminated onto a robot finger for proximity distance and contact pressure measurements. Fig. 1(b and c) shows the working principle of the FDMS for proximity distance and contact pressure measurements. Fig. 1(b) depicts the working principle of proximity perception of the FDMS with conductive silver coils due to the variable capacitance value. The potential φ_t of the electrode of the conductive silver paste is computed by conformal mapping methods due to the variable proximity distance:where t and r represent the charged particles; d denotes the distance between the object and the electrode; S denotes the electrode gap; l denotes the width of the electrode; θ denotes the angle between the electrode plane and the surface of the object; Q denotes the charge of the equivalent electrode; ε₀ denotes the vacuum dielectric constant; and Φ_t and Φ_r denote the initial potentials when no object is near the electrode.

Fig. 1 (a) Schematic diagram of FDMSs for robot grasping manipulation. (b) Working principle of proximity perception of the flexible dual modal sensor. (c) Working principle of contact pressure perception of flexible dual modal sensor. (d) Robot electronic skin based on flexible dual modal sensors achieves multifunctional perception through machine learning.

Therefore, the variable potential ΔV_tr would be changed by the two charged particles due to the approaching process of the object, and it is represented as:

where ΔV_tr represents the variable potential difference between two charged particles, and V_tr denotes the potential difference between two charged particles without an external object.

Let Q = CV and ΔVQ = ΔQV, and import them into eqn (2), and it is rewritten as:

where ΔC_tr denotes the variable capacitance of the electrode during the approaching process, and C_tr denotes the capacitance value of the electrode.

Due to the transformation relationship between the conductors and insulators, the variable capacitance ΔC_tr is rewritten as:

where ε₀ and ε_t denote the dielectric constants of vacuum and objects, respectively.

Eqn (4) represents the geometric parameters of the electrode related to the detection accuracy. When the gap s increases or the distance d decreases, the detection accuracy tends to increase. The proximity distance detection accuracy increases with the increasing dielectric parameter ε_t.

Fig. 1(c) depicts the principle of the single-electrode mode TENG sensor with a stretchable acrylamide (AAM)/NaCl hydrogel for contact pressure perception based on the frictional electric effect and electrostatic induction, which would induce charge transfer and generate periodic currents during the contact and separation process of the external objects. Therefore, it is possible to measure the contact pressure (between the electrodes with objects) by FDMS with the TENG sensor.

Fig. 1(d) shows the target object recognition ability of the robotic finger with the FDMS and machine learning algorithm when grasping different objects. Through a large amount of training data, the neural network recognizes the material, hardness, and other characteristics of the target objects with the FDMS system. In the field of HRI, it is necessary for robots to acquire greater intelligence capabilities with multi-sensing data to adapt to diverse environments and complicated tasks.

3. Material preparation and fabrication

The material selection and fabrication processes play an important role in the mechanical and electrical properties of FDMSs, mainly including the preparation of proximity sensors in capacitive mode, contact pressure sensors in TENG mode, and integration of FDMSs with multi-sensory function. Fig. 2(a) shows the material preparation and fabrication process of the proximity sensor. First, 15 g of PDMS solution was mixed with 1.5 g curing agent in a ratio of 10 : 1, and then, the solution was warmed on a heating plate at 75 °C for 15 min. The PDMS film was subsequently peeled off from the glass wafer.

Fig. 2 (a) Material preparation of FDMSs with the proximity sensor. (b) Fabrication process of the AAM/NaCl hydrogel for the TENG pressure sensor. (c) Schematic diagram of the assembly process of FDMSs with a multi-layer structure. (d) Optical image of an FDMS with proximity distance and contact pressure-sensing function ability. (e) Optical image of an FDMS in the bent format. (f) Optical image of the robot finger with FDMSs. (g) Optical image of the robot finger in the grasping process.

A mask with a conductive silver coil pattern was placed on the PDMS film, and screen printing technology was adopted to transfer the silver coil onto the PDMS film with a spiral pattern. After the conductive silver paste was curried in the air, it was covered with another PDMS film. The conductive silver paste acted as the electrode for the proximity sensor, and possesses the advantages of excellent conductivity, good processing performance, high stability, and strong wear resistance. The proximity sensor can be used for contactless distance measurement between the object and the sensor, and is used as a shielding layer to avoid capacitive coupling between the experimental object and the metal part of the robotic finger.

Fig. 2(b) depicts the fabrication process of the AAM/NaCl conductive hydrogel, which was used as the electrode layer of the TENG contact pressure sensor. First, 1.564 g of AAM powder and 1.602 g of NaCl were dissolved in 10 mL of deionized water, and 0.006 g of N,N′-methylenebisacrylamide (MBAA) and 0.017 g of ammonium persulfate (APS) were then added as crosslinking agents and photoinitiators for AAM, with subsequent addition of 0.25% tetramethylethylenediamine (TEMED) as a promoter relative to the weight of the AAM monomer. The AAM/NaCl conductive hydrogel was poured into a glass mold at room temperature, and then was subsequently used for the fabrication of the TENG pressure sensor.

Liquid Ecoflex silicone with components A and B (in a 1 : 1 ratio) was mixed and then cured at room temperature for two hours to create a silicone rubber film. The silicone rubber film was then cut into a 5 × 5 cm section (the same size as the hydrogel) that was used as the negative friction layer for the TENG. Liquid Ecoflex silicone was covered on the hydrogel and silicone rubber film to encapsulate the TENG contact pressure sensor. The TENG contact pressure sensor was created with a sandwich structure that consisted of silicone rubber, hydrogel film, and silicone rubber, which were assembled to make contact during pressure measurements in the robot finger-grasping process.

Based on the proximity sensor unit with a conductive silver coil pattern, the TENG contact pressure sensor with a sandwich structure, and the FDMSs with multi-layers: the PDMS film, conductive silver paste, silicone rubber, and hydrogel film were assembled layer by layer, and a schematic diagram of the FDMSs is shown in Fig. 2(c). The fabrication of the FDMSs with an overall size of 50 mm × 50 mm × 3 mm was completed at room temperature, and they were used for contactless distance and contact pressure sensing to improve the HRI performance. Fig. 2(d) shows an optical image of the FDMS with proximity distance and contact pressure-sensing function ability. Fig. 2(e) demonstrates the excellent flexibility of the FDMS in the bent format, which is attached to the surfaces of different objects for better achieving the multi-sensing capability of the electronic skin (Fig. 2(f)). Fig. 2(g) shows the robot finger with the FDMSs successfully grasping the objects, which could be applied to multi-sensory data for more accurate manipulation in the grasping process.

4. Results and discussion

4.1. Electrical performance of the FDMSs

FDMSs with the proximity sensor and TENG pressure sensor units were applied for dual sensor data recording, and the performances of the FDMSs were validated by different experiments. The proximity sensor based on the capacitance variance was used for proximity distance measurement, and the working principle for proximity distance detection is shown in Fig. 1(b). The electrode with a spiral coil structure was designed as a large device capacitor for proximity sensor measurement, which improves the performance of close range sensing in capacitive mode with a large coupling area. Fig. 3(a) shows the output capacitance performance of the proximity sensor with a human finger at different distances. The human finger moves to the proximity sensor from undisturbed to a perceptible proximity distance (approximately 10 cm), and then to full contact, resulting in a capacitance change of 0.2 pF (from 1.34 pF to 1.14 pF), which is 14.9% lower than the undisturbed value.

Fig. 3 (a) The proximity distance of fingers leads to variable capacitance values. (b) The rate of variable capacitance with the distance between fingers. (c) Output voltage signal and fitting curve during a calibration test with different applied pressures. (d) The output voltage response of the contact pressure-sensing unit during cyclic testing under different forces. (e) The output voltage response of the contact pressure-sensing unit during cyclic tests at different frequencies. (f) Repletion tests of the pressure sensor units with 1000 cycles (the illustration shows the voltage changes in cycle 400–425).

Fig. 3(b) shows a decrease in the variable capacitance rate ΔC/C₀ as the height of the human finger above the proximity sensor (the proximity distance) decreases, which could be applied to proximity distance detection and obstacle avoidance. The proximity sensor is highly sensitive to fingers at close distances (<1 cm), and the rate of change in capacitance slowly decreases when the finger moves away from the proximity sensor. It successfully detected the target (human finger) at a distance of 10 cm with 2.1% ΔC/C₀.

The contact pressure sensor based on TENG mode is designed for contact pressure detection during the grasping process of the robot finger, and the working principle of the TENG pressure sensor is shown in Fig. 1(c). Fig. 3(c) shows an increase in the output voltage from the TENG sensing unit with the increasing pressure. The sensor is capable of two-stage stable linear sensitivity for different pressure detection. At the range of 0–0.45 N, the linear sensitivity of the sensor is 1.33 V N⁻¹, and at the range of 0.45–5.2 N, the linear sensitivity of the sensor is 1.39 V N⁻¹. As the external pressure continues to increase, the linear sensitivity of the sensor gradually decreases due to the principle of resistive voltage division.

The difference in the response of the sensing units to different pressures is a key indicator for pressure detection. Cyclic pressure tests were implemented on TENG contact pressure-sensing units for different pressure sensors. The pressure gradually increased from 0.5 N to 3 N with the same frequency. Fig. 3(d) shows the change in output voltage of the TENG pressure sensor under different forces with 1.87 V at 0.5 N, 2.72 V at 1 N, 3.51 V at 2 N, and 5.53 V at 3 N. Therefore, excellent performance was exhibited by the contact pressure sensor for external pressure sensing.

Fig. 3(e) depicts the collection of output voltages by the TENG contact pressure sensor under different loading frequencies. The pressure remained constant, and the loading frequencies were set to 0.05 Hz, 0.1 Hz, 0.2 Hz, and 0.5 Hz, respectively. Within a certain frequency range, the maximum output voltage remained at approximately 3.57 V with the same external pressure. The output voltage of the TENG pressure sensor maintained a stable value under different loading frequencies at the same external pressure load.

To further investigate the repeatability and durability of the sensors under external forces, Fig. 3(f) depicts 1000 cyclic pressure tests that were implemented under the same force. The peak output voltage under different loads fluctuated within a small range, while the overall trend of the output voltage signal of TENG pressure sensor remained stable. The magnified local image in Fig. 3(f) shows the detailed situation of the output voltage. The initial output value of the sensor was nearly 0 V without an external applied pressure. When the applied force was 2 N, the voltage value fluctuated around 3.86 V. The results of the cyclic test are basically consistent, with good repeatability and durability.

4.2. Target object recognition

The perception and manipulation performance of the robotic finger with electronic skin based on FDMSs was determined, and it included obstacle avoidance analysis under proximity sensing and target object recognition with different materials and hardness in the grasping process using the AlexNet neural network. Fig. 4(a) shows the experimental setup for obstacle avoidance tests based on the proximity perception of the FDMSs. It is important to evaluate the safety performance of human–machine interaction for the robotic arm with obstacle avoidance ability, and FDMSs with proximity ability would provide an effective method to achieve obstacle avoidance ability.

Fig. 4 (a) Experimental setup of the robotic finger with electronic skin. Obstacle avoidance test of the electronic skin (b) in the front and (c) on the back. (d) Recording output voltage signals from robotic electronic skin by grasping objects composed of different materials. (e) Data visualizations for the different target materials produced by the training process of the AlexNet neural network. (f) Confusion matrices of different target materials. (g) Output signals produced by grasping objects with different hardnesses. (h) Visualization for target objects with different hardness properties. (i) Confusion matrix for identifying objects with different hardness properties. (j) Success recognition rate for different target materials by different algorithms. (k) Success recognition rate for objects with different hardnesses by different algorithms.

Fig. 4(b) shows an immediate decrease in the capacitance value of the proximity sensor when a human finger approached the robotic arm from the front. The robotic arm moved back 5 cm to avoid collision, and the capacitance returned to the initial level. Fig. 4(c) depicts a similar performance for the proximity sensor, and the robotic arm moved forward 5 cm. The proximity sensor and feedback information successfully activated the obstacle avoidance function of the robotic arm.

AlexNet neural network-assisted FDMSs are proposed to target objects and hardness recognition in the robot-grasping process with the help of robotic electronic skin. This study did not directly apply the standard AlexNet architecture, but implemented three key improvements to address the multi-physics coupling signal characteristics of the flexible sensors. First, by replacing the depth-separable convolutional layer, the number of parameters required by the convolutional layer was effectively reduced, and the computational efficiency was improved. Second, the innovative introduction of spatiotemporal attention modules enables the network to adaptively focus on the spatiotemporal correlation features of proximity contact events. Finally, a transfer learning strategy was adopted to achieve recognition accuracy of greater than 90% on small datasets with only 1200 samples.

An FDMS with a TENG contact pressure sensor would collect the contact pressure signal for grasping different objects with different materials and hardness values. The data acquisition card would collect the real-time output signal of the TENG sensor for visualization, with further processing during grasping. The dataset for different target materials and hardness with five objects was constructed by FDMSs. Fig. 4(d) shows the output signals of the robotic finger grasping five different materials: rubber dolls, glass cups, plastic bottles, paper rolls, and wooden tubes. The corresponding output voltage signal from the flexible dual modal sensor with the TENG sensor unit was recorded for feature extraction and imported into the machine learning algorithm for target object recognition.

For each material, 240 samples were collected, and of these, 192 samples were placed in the training set and 48 samples were included in the validation set. The total of 1200 samples, including 960 samples in the training set and 240 samples in the validation set, was adopted by the AlexNet neural network for target material recognition. In the experiment, a batch_size of 32 and a learning rate of 0.0001 were used, and the Adam optimizer was used for training to calculate the loss value using the cross-entropy function. After observation, when the training iterations reached approximately 100 times, the accuracy of the model tended to stabilize.

Each sample was converted into a two-dimensional image with a size of 224 × 224 × 3, which was then imported into the proposed AlexNet neural network for further data processing. The AlexNet neural network was adopted to train the data, with 93.49% successful recognition accuracy. Fig. 4(e) shows the classifying results for different target materials, and blue, orange, green, red, and purple points represent rubber dolls, glass cups, plastic bottles, paper rolls, and wooden tubes, respectively. Fig. 4(f) shows the confusion matrix for classifying results of different target materials, and different target objects were successfully recognized by FDMSs and the AlexNet neural network.

Robot electronic skin with the AlexNet neural network was applied to the softness and hardness recognition of different objects. Different volumes of water were injected into the bottle, which was adopted to imitate the softness and hardness of the different objects. Fig. 4(g) shows the output signals of the grasping of empty bottles, 1/4, 2/4, and 3/4 volumes of water, and filling bottles with water. For each object with different hardness, the grasping signal was recorded by FMDSs, and 240 samples were divided for further processing, of which 192 samples were placed in the training set, and 48 samples were placed in the validation set. A total of 1200 samples, including 960 samples in the training set and 240 samples in the validation set, was imported into the AlexNet neural network for hardness recognition of different objects.

In the experiment, a batch_size of 32 and a learning rate of 0.0001 were set, and the Adam optimizer was used for training to calculate the loss value using the cross-entropy function. After observation, when the training iterations reached approximately 100 times, the accuracy of the model tended to stabilize. The AlexNet neural network was adopted to train the data from the FDMSs with 92.22% successful recognition accuracy. Fig. 4(h) shows the data visualization results with empty bottles, 1/4, 2/4, and 3/4 volumes of water, and bottles filled with water. Fig. 4(i) shows the confusion matrix of the classifying results of different hardnesses, and objects with different hardnesses were successfully recognized by FDMSs and the AlexNet neural network.

Fig. 4(j) shows different algorithms (CNN, ResNet18, Vgg16, MobileNet, SVM, DenseNet, ConvNet, and AlexNet) that were adopted to recognize the different target materials (Fig. 4d), with a 0.679, 0.6733, 0.7656, 0.8338, 0.6971, 0.7969, 0.8759, 0.9349 success recognition rate, respectively. Fig. 4(k) shows that different intelligent algorithms (CNN, ResNet18, Vgg16, MobileNet, SVM, DenseNet, ConvNet, AlexNet) were adopted to recognize the target objects with different hardnesses (Fig. 4g), with 0.6247, 0.7482, 0.6668, 0.7939, 0.5953, 0.8286, 0.8941, and 0.9222 success recognition rates, respectively. Success recognition rates for touching different materials (93.49%) and for the different hardnesses (92.22%) of the grasped object were obtained by the proposed AlexNet neural network. Compared to other algorithms, the performance of the AlexNet neural network was superior for target material recognition, which would improve the HRI ability.

4.3. Full perception during the grasping process

FDMSs with proximity distance and contact pressure-sensing functions were applied to full process detection in the robot grasping process. Fig. 5 shows the results of real-time feedback on the capacitance and voltage change during approach and contact process of the robot electronic skin based on FDMSs in the grasping process. When the robotic finger with electronic skin approached an object, the capacitance variable rate of the capacitive proximity sensor significantly decreased. When the robotic finger contacted an object, the TENG contact pressure sensor would be activated and generate an output voltage.

Fig. 5 Robot electronic skin with proximity and contact perception function ability in the robot-grasping process. Rubber dolls, plastic bottles, and wooden tubes were selected as the target objects, with a real-time capacitance change rate and voltage feedback for the robotic electronic skin.

Initially, the robotic finger starts to grasp the rubber doll, and approaches the detectable distance of the rubber doll during the 4th second, at which time the capacitance variable rate decreases to 0.042. At the 8th second, the robotic finger contacts the rubber doll, and a maximum output voltage of 6.2 V is recorded during the grasping of the rubber doll during this period. Similarly, plastic bottles and wooden tubes were selected as separate objects to be tested, and real-time perception data with proximity and contact pressure function ability were recorded in the robot-grasping process. The full process data during the robot-grasping process with approach distance and contact pressure were recorded by the flexible dual modal sensor with proximity and contact perception function ability.

Conclusions

A machine learning-assisted flexible dual modal sensor was developed for robotic electronic skin application in the grasping process. Combined with machine learning, robotic electronic skin can be applied to target material and hardness recognition by the output signal from FDMSs.

Robotic electronic skin integrated with proximity sensors and TENG contact pressure sensors was designed to collect the dual modal sensor data during the grasping process. FDMSs with multi-layer structure (PDMS film, conductive silver paste, silicone rubber, and hydrogel film in layers) were developed for proximity distance and contact pressure measurement. FDMSs with conductive silver coils were designed for proximity perception due to the variable capacitance. A single electrode mode TENG sensor with a frictional electric effect and electrostatic induction was applied for contact pressure measurement. Cyclic experiments were implemented with different pressures and loading frequencies, and the contact pressure sensor exhibited good pressure and frequency response performance.

The AlexNet neural network with FDMSs was developed to target material and hardness recognition in the robot-grasping process. The AlexNet neural network was adopted to achieve 93.49% detection accuracy with five different materials: rubber dolls, glass cups, plastic bottles, paper rolls, and wooden tubes. Robotic fingers were applied to grasp empty bottles, and 1/4, 2/4, 3/4, and full volumes of water, and the AlexNet neural network achieved 92.22% detection accuracy. Compared to other algorithms, there was superior performance of the AlexNet neural network during target material recognition, which will improve HRIs.

The robot electronic skin exhibited dual perception feedback capability in proximity and contact perception with excellent flexibility and stability, which has great potential for HRI, soft robotics, and biomedical applications. The data collection and process system were designed with the connected wire and power supply units.

An intelligent system should be developed to expand the potential application of the flexible dual modal sensor, with the following two goals:

(1) The new sensing function ability should be integrated into the flexible electronics for more parameter sensing in the grasping process of the robot fingers and data collection. The microprocessor would be designed to be fully flexible in the electronic system for full process perception.

(2) Intelligent algorithms in the sensor system should be studied to increase the intelligence of the robot finger system with the flexible sensor. It would promote smart manipulation in the grasping process of the robot fingers.

Data availability

The data in this article cannot be provided due to legal or ethical confidentiality requirements. The authors are grateful for your understanding.

Author contributions

Wentao Dong: conceptualization, methodology, funding acquisition, supervision, writing – review and editing; Kaiqi Sheng: data curation, investigation, validation, writing – original draft; Chang Chen: software, validation, writing – original draft; Xiaopeng Qiu: software, validation, and visualization.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

The authors acknowledge support from the National Natural Science Foundation of China (52165069), the Natural Science Foundation of Jiangxi Province (20224BAB214051), and the China Postdoctoral Science Foundation (2024 M750895).

References

1. M. L. Hammock, A. Chortos, B. C. Tee, J. B. Tok and Z. Bao, 25th anniversary article: The evolution of electronic skin (e-skin): a brief history, design considerations, and recent progress, Adv. Mater., 2013, 25(42), 5997–6038.
2. W. Dong, L. Yang, R. Gravina and G. Fortino, Soft Wrist-Worn Multi-Functional Sensor Array for Real-Time Hand Gesture Recognition, IEEE Sens. J., 2022, 22(18), 17505–17514.
3. L. Li, Z. Lou, D. Chen, K. Jiang, W. Han and G. Shen, Recent Advances in Flexible/Stretchable Supercapacitors for Wearable Electronics, Small, 2018, 14(43), e1702829.
4. Y. Xu, Z. Sun, Z. Bai, H. Shen, R. Wen, F. Wang, G. Xu and C. Lee, Bionic e-skin with precise multi-directional droplet sliding sensing for enhanced robotic perception, Nat. Commun., 2024, 15(1), 6022.
5. H. Zhang, H. Li and Y. Li, Biomimetic Electronic Skin for Robots Aiming at Superior Dynamic-Static Perception and Material Cognition Based on Triboelectric-Piezoresistive Effects, Nano Lett., 2024, 24(13), 4002–4011.
6. W. Dong, Y. Wang, Y. Zhou, Y. Bai, Z. Ju, J. Guo, G. Gu, K. Bai, G. Ouyang, S. Chen, Q. Zhang and Y. Huang, Soft human–machine interfaces: design, sensing and stimulation, Int. J. Intell. Robot. Appl., 2018, 2(3), 313–338.
7. W. Dong, L. Yang, R. Gravina and G. Fortino, ANFIS fusion algorithm for eye movement recognition via soft multi-functional electronic skin, Inf. Fusion, 2021, 71, 99–108.
8. Z. Chen, Y. Wang, Z. Zhang, D. Mei and W. Liu, Flexible dual-mode sensor with accurate contact pressure sensing and contactless distance detection functions for robotic perception, J. Intell. Manuf., 2024, 36(2), 1–13.
9. W. Dong, K. Sheng, B. Huang, K. Xiong, K. Liu and X. Cheng, Stretchable Self-Powered TENG Sensor Array for Human-Robot Interaction Based on Conductive Ionic Gels and LSTM Neural Network, IEEE Sens. J., 2024, 24(22), 37962–37969.
10. J. W. Jeong, W. H. Yeo, A. Akhtar, J. J. Norton, Y. J. Kwack, S. Li, S. Y. Jung, Y. Su, W. Lee, J. Xia, H. Cheng, Y. Huang, W. S. Choi, T. Bretl and J. A. Rogers, Materials and optimized designs for human-machine interfaces via epidermal electronics, Adv. Mater., 2013, 25(47), 6839–6846.
11. J. Kim, M. Lee, H. J. Shim, R. Ghaffari, H. R. Cho, D. Son, Y. H. Jung, M. Soh, C. Choi, S. Jung, K. Chu, D. Jeon, S. T. Lee, J. H. Kim, S. H. Choi, T. Hyeon and D. H. Kim, Stretchable silicon nanoribbon electronics for skin prosthesis, Nat. Commun., 2014, 5, 5747.
12. L. Wang, Z. Lou, K. Jiang and G. Shen, Bio-Multifunctional Smart Wearable Sensors for Medical Devices, Adv. Intell. Syst., 2019, 1900040.
13. C. Zhang, S. Liu, X. Huang, W. Guo, Y. Li and H. Wu, A stretchable dual-mode sensor array for multifunctional robotic electronic skin, Nano Energy, 2019, 62, 164–170.
14. W. Xiong, C. Zhu, D. Guo, C. Hou, Z. Yang, Z. Xu, L. Qiu, H. Yang, K. Li and Y. Huang, Bio-inspired, intelligent flexible sensing skin for multifunctional flying perception, Nano Energy, 2021, 90, 106550.
15. W. Xiong, H. Feng, H. Liwang, D. Li, W. Yao, D. Duolikun, Y. Zhou and Y. Huang, Multifunctional Tactile Feedbacks Towards Compliant Robot Manipulations via 3D-Shaped Electronic Skin, IEEE Sens. J., 2022, 22(9), 9046–9056.
16. C. Dagdeviren, Y. Shi, P. Joe, R. Ghaffari, G. Balooch, K. Usgaonkar, O. Gur, P. L. Tran, J. R. Crosby, M. Meyer, Y. Su, R. Chad Webb, A. S. Tedesco, M. J. Slepian, Y. Huang and J. A. Rogers, Conformal piezoelectric systems for clinical and experimental characterization of soft tissue biomechanics, Nat. Mater., 2015, 14(7), 728–736.
17. Y. Zhang, X. Zhou, N. Zhang, J. Zhu, N. Bai, X. Hou, T. Sun, G. Li, L. Zhao, Y. Chen, L. Wang and C. F. Guo, Ultrafast piezocapacitive soft pressure sensors with over 10 kHz bandwidth via bonded microstructured interfaces, Nat. Commun., 2024, 15(1), 3048.
18. Z. Gong, W. Di, Y. Jiang, Z. Dong, Z. Yang, H. Ye, H. Zhang, H. Liu, Z. Wei, Z. Tu, D. Li, J. Xiang, X. Ding, D. Zhang and H. Chen, Flexible calorimetric flow sensor with unprecedented sensitivity and directional resolution for multiple flight parameter detection, Nat. Commun., 2024, 15(1), 3091.
19. Q. Mao, Z. Liao, J. Yuan and R. Zhu, Multimodal tactile sensing fused with vision for dexterous robotic housekeeping, Nat. Commun., 2024, 15(1), 6871.
20. H. Zhang, D. Zhang, B. Zhang, D. Wang and M. Tang, Wearable Pressure Sensor Array with Layer-by-Layer Assembled MXene Nanosheets/Ag Nanoflowers for Motion Monitoring and Human–Machine Interfaces, ACS Appl. Mater. Interfaces, 2022, 14(43), 48907–48916.
21. X. Zhang, L. Ke, X. Zhang, F. Xu, Y. Hu, H. Lin and J. Zhu, Breathable and Wearable Strain Sensors Based on Synergistic Conductive Carbon Nanotubes/Cotton Fabrics for Multi-directional Motion Detection, ACS Appl. Mater. Interfaces, 2022, 14(22), 25753–25762.
22. X. Wu, Z. Yang, Y. Dong, L. Teng, D. Li, H. Han, S. Zhu, X. Sun, Z. Zeng, X. Zeng and Q. Zheng, A Self-Powered, Skin Adhesive, and Flexible Human-Machine Interface Based on Triboelectric Nanogenerator, Nanomaterials, 2024, 14(16), 1365.
23. J. Huang, H. Wang, J.-a. Li, S. Zhang, H. Li, Z. Ma, M. Xin, K. Yan, W. Cheng, D. He, X. Wang, Y. Shi and L. Pan, High-Performance Flexible Capacitive Proximity and Pressure Sensors with Spiral Electrodes for Continuous Human–Machine Interaction, ACS Mater. Lett., 2022, 4(11), 2261–2272.
24. H. Kong, Z. Song, M. Ding, C. Shao, J. Yu, B. Wang, W. Li, C. Li and L. Niu, Multifunctional high-performance pressure/proximity/temperature sensors enabled by hybrid resistive-supercapacitive response, Nano Res., 2024, 17(6), 5604–5613.
25. K. Li, Z. Li, Z. Xiong, Y. Wang, H. Yang, W. Xu, L. Jing, M. Ding, J. Zhu and J. S. Ho, Thermal Camouflaging MXene Robotic Skin with Bio-Inspired Stimulus Sensation and Wireless Communication, Adv. Funct. Mater., 2022, 23, 32.
26. B. P. Nabar, Z. Celik-Butler and D. P. Butler, Self-Powered Tactile Pressure Sensors Using Ordered Crystalline ZnO Nanorods on Flexible Substrates Toward Robotic Skin and Garments, IEEE Sens. J., 2014, 15(1), 63–70.
27. Y. Li, G. Matsumura, Y. Xuan, S. Honda and K. Takei, Stretchable Electronic Skin using Laser-Induced Graphene and Liquid Metal with an Action Recognition System Powered by Machine Learning, Adv. Funct. Mater., 2024, 34(30), 2313824.
28. A. A. Jan, S. Kim and S. Kim, A skin-wearable and self-powered laminated pressure sensor based on triboelectric nanogenerator for monitoring human motion, Soft Sci., 2024, 4(1), 202354.
29. M. Zhang, W. Wang, G. Xia, L. Wang and K. Wang, Self-Powered Electronic Skin for Remote Human–Machine Synchronization, ACS Appl. Electron. Mater., 2023, 5(1), 498–508.
30. X. Yuan, T. Puchuan, L. Zhou and F. Yubo, Self-powered wearable IoT sensors as human-machine interfaces, Soft Sci., 2023, 3(3), 26.
31. H. Niu, X. Wei, H. Li, F. Yin, W. Wang, R. S. Seong, Y. K. Shin, Z. Yao, Y. Li, E. S. Kim and N. Y. Kim, Micropyramid Array Bimodal Electronic Skin for Intelligent Material and Surface Shape Perception Based on Capacitive Sensing, Adv. Sci., 2024, 11(3), e2305528.

---