André
Colliard-Granero
ab,
Mariah
Batool
c,
Jasna
Jankovic
c,
Jenia
Jitsev
d,
Michael H.
Eikerling
ae,
Kourosh
Malek
*af and
Mohammad J.
Eslamibidgoli
*a
aTheory and Computation of Energy Materials (IEK-13), Institute of Energy and Climate Research, Forschungszentrum Jülich GmbH, 52425 Jülich, Germany. E-mail: m.eslamibidgoli@fz-juelich.de; k.malek@fz-juelich.de
bDepartment of Chemistry, University of Cologne, Greinstr. 4-6, 50939 Cologne, Germany
cDepartment of Materials Science and Engineering, University of Connecticut, 97 North Eagleville Road, Unit 3136, Storrs, CT 06269-3136, USA
dJulich Supercomputing Center, Forschungszentrum Jülich, 52425 Jülich, Germany
eChair of Theory and Computation of Energy Materials, Faculty of Georesources and Materials Engineering, RWTH Aachen University, Aachen 52062, Germany
fCentre for Advanced Simulation and Analytics (CASA), Simulation and Data Science Lab for Energy Materials (SDL-EM), Forschungszentrum Jülich GmbH, 52425 Jülich, Germany
First published on 24th November 2021
The rapidly growing use of imaging infrastructure in the energy materials domain drives significant data accumulation in terms of their amount and complexity. The applications of routine techniques for image processing in materials research are often ad hoc, indiscriminate, and empirical, which renders the crucial task of obtaining reliable metrics for quantifications obscure. Moreover, these techniques are expensive, slow, and often involve several preprocessing steps. This paper presents a novel deep learning-based approach for the high-throughput analysis of the particle size distributions from transmission electron microscopy (TEM) images of carbon-supported catalysts for polymer electrolyte fuel cells. A dataset of 40 high-resolution TEM images at different magnification levels, from 10 to 100 nm scales, was annotated manually. This dataset was used to train the U-Net model, with the StarDist formulation for the loss function, for the nanoparticle segmentation task. StarDist reached a precision of 86%, recall of 85%, and an F1-score of 85% by training on datasets as small as thirty images. The segmentation maps outperform models reported in the literature for a similar problem, and the results on particle size analyses agree well with manual particle size measurements, albeit at a significantly lower cost.
In the past three decades, a plethora of research activities have focused on developing and evaluating novel alternatives to improve the performance of Pt-based CL through experimental methods, computational studies, and comprehensive characterization.12–22 High-resolution X-ray and electron microscopy (in situ/ex situ) characterization methods are increasingly used for the evaluation of the catalytic activity and stability of the catalyst layer. Examples include ultra-small angle X-ray scattering (u-SAXS), focussed ion beam scanning electron microscopy (FIB-SEM), transmission electron microscopy (TEM), scanning transmission electron microscopy (STEM), or scanning transmission X-ray microscopy (STXM).23–26
To unravel the connection between microstructure properties and performance, it becomes essential to quantify certain features of the imaging data correctly, e.g., the particle size distribution (PSD), pore size, pore network, or aggregate size – with less reliance on human intervention and interpretation.27,28 This quantification usually requires additional analyses, which can be done either manually by the operator or automatically via processing the images through a set of specialized algorithms to extract the pertinent structural features.29,30
The type of post-processing analysis of materials characterization data alluded to be beneficial as numerous studies suggested that the electrochemical performance of CL is strongly dependent on the surface area, size, shape, and distribution of Pt NPs.31–34 For example, an increase in mean Pt particle size in an aged cathode and subsequent reduction at anode, associated with Pt2+ ion crossover, could indicate degradation of the carbon support.35–37 In another study the larger particle size and agglomeration after potential loading was attributed to the dissolution and re-deposition of Pt in the catalyst layer.38 Smaller Pt particles are not favorable either because of the stronger adsorption of reaction intermediates at the edges and corners of NPs; the latter results in blockage of oxygen reduction reaction path and makes the Pt more susceptible to dissolution.39
The widely employed technique for particle size analysis entails measuring a set of manually drawn straight lines from the edges of individual particles in the image.40 This approach employs standard image processing tools like ImageJ41 and may require thresholding the image and applying filters for denoising, to segment the particles from the support material or background. Notwithstanding, this essential thresholding step becomes tricky in capturing small particles or when the contrast between particle and support material is not strong. In addition, in a one-at-a-time inspection of particles, the operator usually considers only a subset of the particles for each image, resulting in the loss of information. The long, tedious, and detailed manual procedure makes this approach ineffective for high-throughput or real-time analysis. Moreover, identifying the overlapping particles is strongly dependent on the operator's bias and judgment. These issues generate an unreliable metric for the quantification and can slow down or even falsify the analysis.
Semi-automated approaches for the analysis of well-separated and monodisperse NPs have been developed. For example, for magnetite NPs, classical approaches using Otsu's binarization with Canny edge detection and edge linking algorithms were employed.45 For the population analysis of Pt NPs on glassy Carbon, an earlier report has introduced the local adaptive threshold (LAT) in the image processing steps.46 This step improved the segmentation of particles by thresholding the smaller patches of the image, but at the cost of additional computing time.46 Ponti and co-workers developed PEBBLES,47 the software program for the size measurement and morphological analysis of nanoparticles in TEM micrographs. PEBBLES applies fitting 3D intensity models to relate the contrast of the image and the depth of the dip in the intensity surface to the NP size, making the automatic measurement and analysis of the distribution of morphological parameters passible. However, as pointed out by the authors, the software is limited in accurately fitting the overlapping NPs or the heterostructured NPs (e.g., core–shell NPs).47
Over the last decade, deep learning (DL) methods based on the convolutional neural networks (ConvNets) have transformed the field of computer vision by outperforming the conventional methods in various image and video recognition tasks such as in verification/identification, classification, object detection, and segmentation, as well in other applications including image reconstruction, denoising, image synthesis, colorization, style transfer and other tasks.48,49 To leverage the advancements in DL-based approaches in microscopy image processing, more recently, the ConvNets algorithms have been deployed to several platforms for various analysis tasks like segmentation, object detection, denoising, or super-resolution microscopy. The most exciting example is the recently developed ZeroCostDL4Mic50 platform that provides the advantage of free computational resources on GoogleColab for training the deep neural network models. Unlike the conventional procedures requiring several preprocessing steps for particle analysis to construct suitable features, DL models utilize the information content directly from the raw TEM micrographs;51 once a suitable model is trained on a representative dataset, they can predict swiftly and accurately without the necessity for the human-based alteration of parameters. Moreover, the robustness of DL models is validated with various metrics like precision, recall, or accuracy, which introduces well-defined quantification metrics for particle analysis.
Image segmentation is a topic of unfading interest in various domains. Table 1 summarizes a few use cases from the recent literature, with information about the dataset used (to the best of our knowledge).52–61 Specifically in materials science, more recently, Yao et al. implemented the popular U-Net model44 for real-time nanoparticle segmentation in liquid-phase TEM videos.62 This novel approach enabled the statistical analysis of the diffusion, reaction, and assembly kinetics of the rod-, prism- and cube-shaped colloidal NPs.62 Cole and co-workers introduced ImageDataExtractor,63,64 a novel tool to automate the extraction and analysis of microscopy imaging data from the scientific literature. The main feature of the ImageDataExtractor is its capability in segmentation and quantification of NPs via Bayesian DL for a diverse set of morphologies.64 However, as will be discussed, ImageDataExtractor is limited in the accurate localization of overlapping NPs, as is the case for Pt NPs on a carbon support material.
Ref. | Dataset size | Public availability | Application |
---|---|---|---|
Haugland Johansen et al. (2021)52 | 104 | No | Microscopic foraminifera segmentation |
Velesaca et al. (2020)53 | 523 | Yes | Corn kernel classification |
Gené-Mola et al. (2020)54 | 582 | Yes | Fruit detection |
Kim et al. (2021)55 | 2592 | No | Bubble detection |
Zhou et al. (2020)56 | 1109 | No | Cervical cell segmentation |
Jayakody et al. (2021)57 | 3065 | Yes | Stomata detection |
Chen et al. (2020)58 | 500 | No | Aluminum alloy microstructure segmentation |
Toda et al. (2020)59 | 1200 | Yes | Seed phenotyping segmentation |
Poletaev et al. (2020)60 | 30000 | Yes | Gas bubble size distribution |
Zhiming Cheng et al. (2020)61 | 871 | Yes | Cell nuclei segmentation |
This paper presents a novel DL-based approach to automate the PSD analysis from TEM micrographs of catalyst layers in PEMFCs. We trained the StarDist model43 on our annotated imaging dataset for the instance segmentation of Pt NPs on the high surface area Carbon support. The model largely resolved the localization of overlapping particles. Unlike the conventional procedure, this approach adopts definite quantification metrics for evaluating the segmentation.
Our approach for the PSD analysis involves the four steps shown in Fig. 1(a): (1) manual annotation of Pt NPs, (2) supervised learning for particle segmentation, (3) programmed diameter measurement on the predicted region-of-interest (ROI), and (4) statistics and visualization. The quality of the trained model largely depends on the quality of the labeled data; therefore, it is important to choose an efficient and accurate tool for the first step. For this purpose, we used QuPath,65 a user-friendly open-source software initially developed for quantitative pathology and bioimage analysis. Forty high-resolution TEM images (1024 × 1024 and 2048 × 2048 pixels) of Pt NPs on Carbon support were manually annotated. The image set comprised four different magnification levels, at 10, 20, 50, and 100 nm scales, providing a generalization of the scale range to train more robust models. An expert performed the annotations considering visual evidence for the overlapping particles while eluding the strongly crowded Pt aggregates. We will make this dataset publicly available.
Fig. 1 (a) Methodical pipeline for deep learning-based analysis of particle size distribution involving 1. Particle annotation, 2. Supervised learning for the instance segmentation of particles, 3. Setting scale for image calibration and programmed particle size extraction, using OpenCV library,42 4. Obtaining particle size statistics with visualization libraries. (b) The left panel shows a TEM image with overlapped Pt particles supported on a carbon agglomerate; the StarDist method used in this work predicts star-convex polygons parameterized by the radial distances ri,j, and object probabilities di,j. for the pixel i,j.43 The right panel shows the U-Net architecture.44 (c) Example TEM images and their corresponding predicted instance segmentation of Pt particles using U-Net with StarDist formulation for the loss function. |
Given the images and corresponding labeled masks as ground truth, for the pixel-level localization of individual Pt particles, we trained the StarDist model.43 StarDist employs the U-Net architecture44 and overcomes the typical segmentation errors for the dense prediction of merged bordering particles by localizing them as star-convex polygons (Fig. 1(b)). StarDist was initially developed for the segmentation of biological systems to differentiate the overlapping cells in crowded images. In this model, each pixel is parameterized by two values, the radial distances and the object probabilities. For the former, a star-convex polygon is fitted from the pixel position to the edges of the ROI. The latter is represented by the normalized shortest distance from the edge which increases with decreasing the distance from the particle center.
Standard architectures for image segmentation usually involve fully convolutional networks that first perform convolution and down-sampling operations to extract essential features from the image (encoding into a latent space), followed by up-sampling and transpose convolution operations that make the activation maps larger until it reaches the input image size (decoding from a latent space). In the U-Net architecture,44 in addition to this data stream, the up-sampled decoder features are concatenated with the exact corresponding resolution from encoder feature maps (as shown by the gray arrows in Fig. 1(b)). Concatenating encoder and decoder feature maps enables recognizing object boundaries or edges by combining features across all levels, from low to high, leading to more accurate output segmentation maps (Fig. 1(c)). The StarDist implementation of U-Net employs data augmentation to expand the size of the training set by generating modified versions of images in the dataset, where the images are cropped in patches and augmented by rotation, changing intensity, and Gaussian blurring. In this model, the non-maximum suppression (NMS) technique is further used to suppress the multiple detections of individual objects. This way, the predicted box with higher object probability suppresses the overlapped boxes with lower probabilities. For training, the number of epochs and the number of steps per epoch were chosen as 400 and 100, respectively.
The performance of the trained model was analyzed by plotting standard metrics concerning the Intersection over Union (IoU) threshold. The IoU measures the number of pixels common between the ground truth and prediction masks divided by the total number of existing pixels in both masks. Conventionally, reliable predictions correspond to an IoU value ≥ 0.5. In our case, a true-positive (TP) represents the case if a prediction-target mask pair for nanoparticles has an IoU score that exceeds 0.5; likewise, a true-negative (TN) is that for support or background. On the other hand, a false-positive (FP) indicates a predicted NP mask with no associated ground truth mask, and a false-negative (FN) indicates a ground truth NP mask with no associated predicted mask. Examples of TP, TN, FP, and FN are demonstrated in Fig. 2. This way, the metrics of accuracy, precision, recall, and F1 score are defined as,
Accuracy = (TP + TN)/(TP + FP + TN + FN); |
Precision = (TP)/(TP + FP); |
Recall = (TP)/(TP + FN); |
F1 = 2 × precision × recall/(precision + recall) |
For an image with many NPs, it is not a severe problem to have a few FN predictions because it does not significantly affect the statistical analysis. However, a model with high precision indicates that false particles are not introduced to the particle size distribution analysis. Therefore, precision is a reliable measure for the evaluation of the model performance. The accuracy, on the other hand, can be a misleading metric for the images with a small number of NPs as it becomes biased in mainly relating how well the model predicts the background.
Given the model's prediction in segmentation maps, the ROI for each particle was white-filled, cropped, and saved into a JPEG file. Next, using the contrasted contours, we used the OpenCV library42 to place a bounding box around the predicted masks and measured the number of pixels along with the X or Y directions (step 3 in Fig. 1(a)). Here, manual image calibration was used for a pixel-to-real-distance conversion. Lastly, the particle size can be estimated as the average values along with the X and Y directions, and the corresponding histogram can be generated for the PSD analysis (step 4 in Fig. 1(a)).
Table 2 summarizes the performance of the segmentation model in terms of precision, recall, and F1-score for varying split ratios between the training and validation sets at the IoU threshold of 0.5. The model's performance becomes saturated with training sets as small as ten annotated images. We obtained a precision, recall, and F1-score as high as 0.86%, 0.85%, 0.85%, respectively, at a 3:1 ratio of training to the validation set. Further in-depth error sensitivity analysis of the model in terms of the AUC–ROC metric is shown in Fig. 4. The model is robust against the Gaussian blur with standard deviations as large as 8. The performance was also not affected concerning rotation and zoom-in transformations to extract the largest area from the center of the image. Even though the performance of the model may be further improved by expanding the training set, the main origin of discrepancies is for the non-rounded shape particles. This can be explained by general difficulty to predict non-convex shapes that is rooted in StarDist radial model for contour representation. As can also be seen from the predicted map, most of the localized particles are round-shaped or have a convex contour. This would suggest further room for improvements in the correct shape predictions using for example algorithms that use a more flexible contour representation for learning like the SplineDist,66 Contour Proposal Networks,67 or PolarMask++.68
Fig. 4 Sensitivity of the model in terms of AUC–ROC to Gaussian blur with varying standard deviation values (upper panel), and rotation transformations (lower panel). |
Train:validation ratio | Precision | Recall | F1 |
---|---|---|---|
34:6 | 0.86 | 0.81 | 0.83 |
30:10 | 0.86 | 0.85 | 0.85 |
26:14 | 0.85 | 0.86 | 0.86 |
22:18 | 0.85 | 0.84 | 0.84 |
16:24 | 0.84 | 0.83 | 0.84 |
10:30 | 0.87 | 0.8 | 0.83 |
5:35 | 0.81 | 0.77 | 0.79 |
2:38 | 0.67 | 0.42 | 0.52 |
As shown in Fig. 5(a), the ROI data makes it possible to count the particles in the image. Likewise, the area of individual particles can be calculated. These abilities can be in particular helpful in estimating the Pt/C ratio or the Pt mass loading from the image. Fig. 5(b) compares the segmentation maps obtained in this study with those from the recently developed ImageDataExtractor 2.0 software.64 ImageDataExtractor also uses a deep learning approach for particle segmentation. On our dataset, the StarDist model significantly outperforms the software in both predicting the number of particles and localizing the overlapping particles. As shown in Fig. 5(b), for the test TEM images at 10, 20, 50, and 100 nm scales, the StarDist predicted 27, 144, 628, and 286 particles, respectively, while ImageDataExtractor 2.0 found 14, 67, 29, and 133 particles. Notably, StarDist's capability to separate the overlapping ROIs avoids the fusion of the near particles, increasing the total number of detections in comparison to the software.
Fig. 5 (a) Localization and counting the particles made available by the instance segmentation approach. (b) Comparison between the predicted segmentation maps of test images at different scales using the StarDist model employed in this work and the ImageDataExtractor 2.0.64 |
Lastly, with the computed particle sizes the histograms for particle size distributions can be generated using typical visualization packages available in various programming languages. An example is shown in Fig. 6. Here, we used a bin size of 0.5 nm and the kernel density estimation (KDE) function for fitting the histogram. Binning can also be accomplished by using conventional statistical methods and expressions including Sturge's rule, Doane's rule, Scott's rule, Rice's rule or Freedman and Diaconis's rule.69–71,72 Fitting for particle size distribution histogram can be performed using algorithms based on a variety of distribution functions such as polynomial distribution, Gaussian distribution, Weibull probability distribution, lognormal distribution, or Rayleigh distribution.73,74,75
Fig. 6 Particle size distribution was obtained using the deep learning-based approach of the instance segmentation of nanoparticles in the TEM images of Pt/C catalyst layers. |
This journal is © The Royal Society of Chemistry 2022 |