From the journal Digital Discovery Peer review history

24-May-2022

Dear Mr Forrest:

Manuscript ID: DD-ART-03-2022-000026
TITLE: Machine-learning improves understanding of glass formation in metallic systems

Thank you for your submission to Digital Discovery, published by the Royal Society of Chemistry. I sent your manuscript to reviewers and I have now received their reports which are copied below.

I have carefully evaluated your manuscript and the reviewers’ reports, and the reports indicate that major revisions are necessary.

Please submit a revised manuscript which addresses all of the reviewers’ comments. Further peer review of your revised manuscript may be needed. When you submit your revised manuscript please include a point by point response to the reviewers’ comments and highlight the changes you have made. Full details of the files you need to submit are listed at the end of this email.

Digital Discovery strongly encourages authors of research articles to include an ‘Author contributions’ section in their manuscript, for publication in the final article. This should appear immediately above the ‘Conflict of interest’ and ‘Acknowledgement’ sections. I strongly recommend you use CRediT (the Contributor Roles Taxonomy from CASRAI, https://casrai.org/credit/) for standardised contribution descriptions. All authors should have agreed to their individual contributions ahead of submission and these should accurately reflect contributions to the work. Please refer to our general author guidelines https://www.rsc.org/journals-books-databases/author-and-reviewer-hub/authors-information/responsibilities/ for more information.

Please submit your revised manuscript as soon as possible using this link:

*** PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm. ***

https://mc.manuscriptcentral.com/dd?link_removed

(This link goes straight to your account, without the need to log on to the system. For your account security you should not share this link with others.)

Alternatively, you can login to your account (https://mc.manuscriptcentral.com/dd) where you will need your case-sensitive USER ID and password.

You should submit your revised manuscript as soon as possible; please note you will receive a series of automatic reminders. If your revisions will take a significant length of time, please contact me. If I do not hear from you, I may withdraw your manuscript from consideration and you will have to resubmit. Any resubmission will receive a new submission date.

The Royal Society of Chemistry requires all submitting authors to provide their ORCID iD when they submit a revised manuscript. This is quick and easy to do as part of the revised manuscript submission process.   We will publish this information with the article, and you may choose to have your ORCID record updated automatically with details of the publication.

Please also encourage your co-authors to sign up for their own ORCID account and associate it with their account on our manuscript submission system. For further information see: https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/#attribution-id

I look forward to receiving your revised manuscript.

Yours sincerely,
Linda Hung
Associate Editor
Digital Discovery
Royal Society of Chemistry

************

Reviewer 1

In this manuscript, the authors addressed the problem of GFA by using machine learning technique. The model stressed the effect of Wigner-Seitz boundary election density, which deepened the understanding of Inoue’s empirical rules. I believe this work can be published after minor revision.
(1) In Table 2, some of element properties must be clearly defined to avoid confusion. For example, what is the difference between “atomic number” and “universal sequence number”, between “periodic number” and “period”?
(2) The authors should specify which elements were included/excluded in their model.
(3) I suggest Table 4 be revised to display which features survived.
(4) It would be interesting to compare the results with the model recently published on npj Computational Materials, 7, 138, 2021.
(5) It is expected to discover new MG compositions with higher GFA (D_max) by applying this machine learning model. In the manuscript, it is necessary to list several most promising systems or compositions based on the predictions from the model, which are “virgin territory” and may inspire the readers to design new BMGs.

Reviewer 2

The authors propose a neural network approach to predict compositions that have the potential to form bulk metallic glasses. They then use this approach to probe the underlying mechanism behind metallic glass formation. This is a well written manuscript with great potential. However, it is not ready for publication yet. Following are some of the comments that I believe will improve the manuscript:

The introduction to the manuscript needs significant further expansion. Multiple studies have applied machine learning and deep learning towards the discovery of new bulk metallic glasses in particular. Please see some of them cited below:

1. Jeon, J., Seo, N., Kim, H.-J., Lee, M.-H., Lim, H.-K., Son, S. B., & Lee, S.-J. (2021). Inverse Design of Fe-Based Bulk Metallic Glasses Using Machine Learning. Metals, 11(5), 729. https://doi.org/10.3390/met11050729
2. Dasgupta, A., Broderick, S. R., Mack, C., Kota, B. U., Subramanian, R., Setlur, S., Govindaraju, V., & Rajan, K. (2019). Probabilistic Assessment of Glass Forming Ability Rules for Metallic Glasses Aided by Automated Analysis of Phase Diagrams. Scientific Reports, 9(1), 357. https://doi.org/10.1038/s41598-018-36224-3
3. Peng, L., Long, Z., & Zhao, M. (2021). Determination of glass forming ability of bulk metallic glasses based on machine learning. Computational Materials Science, 195, 110480. https://doi.org/10.1016/j.commatsci.2021.110480
4. Reddy, G. J., Kandavalli, M., Saboo, T., & Rao, A. K. P. (2021). Prediction of Glass Forming Ability of Bulk Metallic Glasses Using Machine Learning. Integrating Materials and Manufacturing Innovation, 10(4), 610–626. https://doi.org/10.1007/s40192-021-00239-y
5. Samavatian, M., Gholamipour, R., & Samavatian, V. (2021). Discovery of novel quaternary bulk metallic glasses using a developed correlation-based neural network approach. Computational Materials Science, 186, 110025. https://doi.org/10.1016/j.commatsci.2020.110025
6. Tripathi, M. K., Chattopadhyay, P. P., & Ganguly, S. (2017). A predictable glass forming ability expression by statistical learning and evolutionary intelligence. Intermetallics, 90, 9–15. https://doi.org/10.1016/j.intermet.2017.06.008

These studies, in addition to others need to be mentioned in the introduction. Furthermore, the authors need to provide context on how they have contributed to this growing body of work and what makes their approach different/better than previous approaches? There is mention of other models in section 4.2.1 but their context is not mentioned.

Section 2 (Neural-network architecture) is too long. The readership of Digital Discovery should be already familiar with many of the terms that have been defined in the section including deep NN and the universal approximation theory. While this is certainly a good introduction to the topic, it can be moved to a supplementary section for interested readers with the bulk of the section describing the network architecture. I must also commend the authors on the great job they have done justifying their decisions for the neural network architecture.

Tables 2 and 4 can get confusing to the reader. I would suggest having a single table in the main manuscript which lists the features that were used in the neural network. The authors can decide if they would like to list all the features within a supplementary.

Section 4.1. The authors provide a longer explanation of what k-fold cross validation is. Additionally, they list the definitions and formulae of metrics that are used in classification problems but don’t go on to say exactly which metrics they choose for this problem and the justification for that. As mentioned above, many of the definitions can be moved to a supplementary. This will make the manuscript more concise.

Fig 10 and 11. This may be pedantic but the colors red and blue are used in both plots but they represent different things within the two plots. Perhaps it will be better to make this uniform?

Reviewer 3

The work is a comprehensive study of a large number of features and provides useful insight into the physics behind GFA. The training and testing of various models has also been carried out with transparency.

Dear Dr Hung,

Thank you for giving us the opportunity to submit a revised manuscript for publication. We thank the reviewers for their insightful and encouraging comments, and for the time and effort spent evaluating our submission. The comments were notably constructive in helping us to improve the manuscript.
We have incorporated the suggestions made by the reviewers into the revised manuscript, and we supply a version with the changes highlighted. In this response, we present and discuss the changes made. All mentions of page, table, and figure numbers refer to the revised manuscript.

There is a major change to the end of Section 5. During the revision, it emerged that a mistake in the calculation of adjusted radii had caused the erroneous over-estimation of radii deviations. Since the compositions whose GFA is boosted by this mechanism are a small subset of all the alloys, it is not usefully informative to consider averaged or population-wide analysis such as that we presented in Figures 10 & 11 and Table 7 of the original manuscript. Instead, we follow Reviewer 1's suggestion (see comment 5 below) to incorporate discussion of promising alloy systems, and consider those in which adjustment of radii leads to increased GFA.

Reviewer 1

1. In Table 2, some of element properties must be clearly defined to avoid confusion. For example, what is the difference between "atomic number" and "universal sequence number", between "periodic number" and "period"?

We agree that several of the mentioned elemental properties may not be well known, and have names similar enough to cause confusion. Following this comment, and similar feedback from Reviewer 2, we have replaced Tables 2 & 4 of the original manuscript with a new Table 2 listing the features actually used by the model. Where applicable, we have added references to further discussion of these properties in the literature — we do this rather than including discussion in our own text as we would merely be duplicating the cited works. References are included for features such as the “periodic number” and “universal sequence number” mentioned in the above query, which are quantities proposed by Villars et al. (10.30970/cma1.0007) and Allahyari et al. (10.1021/acs.jpcc.0c07857) respectively, with the aim of usefully ordering the elements.
Features for which we have not provided references are those that may be considered “general knowledge” by a majority of Digital Discovery readers, such as the “period” referring to the row of an element in the periodic table. The full list of elemental properties has been moved to Table S1 of the newly introduced Supplementary Material.

2. The authors should specify which elements were included/excluded in their model.

Thank you for this suggestion, this information would indeed be useful to the reader when considering the scope of the work. As such, we have included a new figure (Figure 2) illustrating the material classes observed in the training data for each element of the periodic table.

3. I suggest Table 4 be revised to display which features survived.

We agree with this feedback, and have addressed it in response to comment 1.

4. It would be interesting to compare the results with the model recently published on npj Computational Materials, 7, 138, 2021.

The cited publication is indeed interesting and relevant. It considers the modelling of GFA with a wide range of machine-learning techniques. We have extended Table 5 to include comparative metrics sourced from Figures 3 & 6 of the Supplementary Materials of this npj Comp. Mater. paper. As there are many different models discussed in the paper, for our own comparison we choose only the best from each category of model (ANN, GPR, SVR, & RF) that was discussed. While the models reported on in the referenced work return excellent performance metrics, our model remains competitive. We emphasise that the central goal of our work was not to prioritize accuracy in modelling, but rather to extract useful theoretical insights that transcend any individual machine-learning model. This point is highlighted in the revised Introduction to our manuscript.

5. It is expected to discover new MG compositions with higher GFA (Dmax) by applying this machine learning model. In the manuscript, it is necessary to list several most promising systems or compositions based on the predictions from the model, which are “virgin territory” and may inspire the readers to design new BMGs.

This is a very helpful suggestion. We agree that providing some examples of high-GFA alloy candidates would be a useful addition to the manuscript. The reworked end of Section 5 now focuses on the design of promising alloy systems, and we provide some examples in Table 7.


Reviewer 2

1. The introduction to the manuscript needs significant further expansion. Multiple studies have applied machine learning and deep learning towards the discovery of new bulk metallic glasses in particular. … Furthermore, the authors need to provide context on how they have contributed to this growing body of work and what makes their approach different/better than previous approaches? There is mention of other models in section 4.2.1 but their context is not mentioned.

We are very happy to agree that the introduction should be expanded, placing our work in the context of the wider literature in this area. We have added three new paragraphs at the end of the Introduction with the intention of laying the contextual foundations of our work, and detailing the novelty of our approach. We are grateful for the suggestions of relevant works to cite, and have incorporated these and others into this new discussion.

2. Section 2 (Neural-network architecture) is too long. The readership of Digital Discovery should be already familiar with many of the terms that have been defined in the section including deep NN and the universal approximation theory. While this is certainly a good introduction to the topic, it can be moved to a supplementary section for interested readers with the bulk of the section describing the network architecture. I must also commend the authors on the great job they have done justifying their decisions for the neural network architecture.

This is very useful feedback! We have relocated much of the neural-network background to a new section (S1) in the Supplementary Material. We are pleased that our design discussion was convincing.

3. Tables 2 and 4 can get confusing to the reader. I would suggest having a single table in the main manuscript which lists the features that were used in the neural network. The authors can decide if they would like to list all the features within a supplementary.

We agree with these points, and have addressed them in our response to Reviewer 1 comment 1.

4. Section 4.1. The authors provide a longer explanation of what k-fold cross validation is. Additionally, they list the definitions and formulae of metrics that are used in classification problems but don’t go on to say exactly which metrics they choose for this problem and the justification for that. As mentioned above, many of the definitions can be moved to a supplementary. This will make the manuscript more concise.

We accept that the discussion of k-folds cross-validation is too lengthy for the readership of Digital Discovery, and that the definitions of metrics clutter the main body of the text. We have reduced the discussion of k-folds cross-validation to a minimal definition followed by discussion specific to this work. Metric definitions have been relocated to Supplementary Material section S3. We have added a brief comment about the decision to use the F1 score in Section 4.1 paragraph 2, and a further comment about the importance of considering multiple classifier metrics in Section 4.2 paragraph 7.

5. Fig 10 and 11. This may be pedantic but the colors red and blue are used in both plots but they represent different things within the two plots. Perhaps it will be better to make this uniform?

This is a useful comment, as it is indeed the case that the meaning of the red and blue colouring is unhelpfully swapped when moving from Figure 11 to Figure 12. However, these figures are no longer present in the revised manuscript.


Reviewer 3

1. The work is a comprehensive study of a large number of features and provides useful insight into the physics behind GFA. The training and testing of various models has also been carried out with transparency.

We thank Reviewer 3 for these comments.

We hope that the revised manuscript will be considered acceptable for publication in Digital Discovery.

Yours sincerely,
Robert M. Forrest & A. Lindsay Greer

13-Jun-2022

Dear Mr Forrest:

Manuscript ID: DD-ART-03-2022-000026.R1
TITLE: Machine-learning improves understanding of glass formation in metallic systems

Thank you for submitting your revised manuscript to Digital Discovery. I am pleased to accept your manuscript for publication in its current form. I have copied any final comments from the reviewer(s) below.

You will shortly receive a separate email from us requesting you to submit a licence to publish for your article, so that we can proceed with the preparation and publication of your manuscript.

You can highlight your article and the work of your group on the back cover of Digital Discovery. If you are interested in this opportunity please contact the editorial office for more information.

Promote your research, accelerate its impact – find out more about our article promotion services here: https://rsc.li/promoteyourresearch.

If you would like us to promote your article on our Twitter account @digital_rsc please fill out this form: https://form.jotform.com/213544038469056.

By publishing your article in Digital Discovery, you are supporting the Royal Society of Chemistry to help the chemical science community make the world a better place.

With best wishes,

Linda Hung
Associate Editor
Digital Discovery
Royal Society of Chemistry

Reviewer 1

The authors well addressed the reviewers' comments. I suggest the acceptance of this paper.

Reviewer 2

The authors have addressed all my previous comments satisfactorily. I have no additional comments at this time.