From the journal Digital Discovery Peer review history

16-Feb-2024

Dear Dr Krishnan:

Manuscript ID: DD-ART-01-2024-000027
TITLE: EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations

Thank you for your submission to Digital Discovery, published by the Royal Society of Chemistry. I sent your manuscript to reviewers and I have now received their reports which are copied below.

After careful evaluation of your manuscript and the reviewers’ reports, I will be pleased to accept your manuscript for publication after revisions.

Please revise your manuscript to fully address the reviewers’ comments. When you submit your revised manuscript please include a point by point response to the reviewers’ comments and highlight the changes you have made. Full details of the files you need to submit are listed at the end of this email.

Digital Discovery strongly encourages authors of research articles to include an ‘Author contributions’ section in their manuscript, for publication in the final article. This should appear immediately above the ‘Conflict of interest’ and ‘Acknowledgement’ sections. I strongly recommend you use CRediT (the Contributor Roles Taxonomy, https://credit.niso.org/) for standardised contribution descriptions. All authors should have agreed to their individual contributions ahead of submission and these should accurately reflect contributions to the work. Please refer to our general author guidelines https://www.rsc.org/journals-books-databases/author-and-reviewer-hub/authors-information/responsibilities/ for more information.

Please submit your revised manuscript as soon as possible using this link :

*** PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm. ***

https://mc.manuscriptcentral.com/dd?link_removed

(This link goes straight to your account, without the need to log in to the system. For your account security you should not share this link with others.)

Alternatively, you can login to your account (https://mc.manuscriptcentral.com/dd) where you will need your case-sensitive USER ID and password.

You should submit your revised manuscript as soon as possible; please note you will receive a series of automatic reminders. If your revisions will take a significant length of time, please contact me. If I do not hear from you, I may withdraw your manuscript from consideration and you will have to resubmit. Any resubmission will receive a new submission date.

The Royal Society of Chemistry requires all submitting authors to provide their ORCID iD when they submit a revised manuscript. This is quick and easy to do as part of the revised manuscript submission process.   We will publish this information with the article, and you may choose to have your ORCID record updated automatically with details of the publication.

Please also encourage your co-authors to sign up for their own ORCID account and associate it with their account on our manuscript submission system. For further information see: https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/#attribution-id

I look forward to receiving your revised manuscript.

Yours sincerely,
Dr Joshua Schrier
Associate Editor, Digital Discovery

************
EDITOR'S COMMENT:
- Please archive data files to a long-term repository (e.g., Zenodo) as mentioned by the reviewer, to provide long-term availability of the data and code resources. Provide the zenodo DOI in the data resource (this can be in addition to other non-repository links as well).

************

Reviewer 1

This benchmark work almost contains all the ingredients to reproduce the results! I have a few minor comments:

1. Data split. In section 4.1 of the dataset, the description of the train/validation/test split is inconsistent across the datasets. Except for the GeTe dataset, no mention of a validation set is provided for the other datasets. For example, in the LIPS dataset, the authors stated: “The training and testing datasets have 19000 and 1000 configurations, respectively.” So, is the validation set part of the training set? I’d suggest making the validation set explicit. Also, how is the split obtained? As far as I know, some datasets, like LiPS by Batzner et al., 2022, do not provide an explicit split. Are the authors performed the split by themselves? If yes, what splitting method is used? For all the datasets in section 4.1, I’d suggest adding such split info.

2. Data source. The datasets are hosted on Google Drive. It might not be easy for researchers in some countries to access Google services. It would be great if the authors could host the dataset on some other repos, such as zenodo (https://zenodo.org) or figshare (https://figshare.com).

3. Code versions. I did not find the version information of the codes (e.g. MACE, TorchMDnet) used to generate the results in this work. Because they are third-party codes, it would be better to provide such info (e.g. commit number) for full reproducibility.

Response1: Thank you for highlighting the importance of data splitting details, which is a crucial aspect from a machine learning perspective. We have provided separate folders containing the test, train, validation data in the Zenodo folder for each of the datasets. The number of data points in each of these splits is included in Table S16 in supplementary documents.

Text Added in section 4.1
“The data splits are tabulated in Table S17 in ESI”

Response2: We appreciate the valuable suggestions from the reviewer. We have already uploaded all the datasets and code over GitHub which is an open source platform. However, for ease of convenience we have now uploaded it to Zenodo as well. Please find the link below: https://doi.org/10.5281/zenodo.10678029

Response3: Thank you for the valuable suggestion. We have now added a table in the supplementary materials with the details of code version for each model. See Table S15 of the supplementary documents.

Text Added in Section 3 Para 1
“Further hyperparameter details for all models are tabulated in ESI Section A.10 Table S1 to S11 along with model's code version information in table S16.”

26-Feb-2024

Dear Dr Krishnan:

Manuscript ID: DD-ART-01-2024-000027.R1
TITLE: EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations

Thank you for submitting your revised manuscript to Digital Discovery. I am pleased to accept your manuscript for publication in its current form. I have copied any final comments from the reviewer(s) below.

You will shortly receive a separate email from us requesting you to submit a licence to publish for your article, so that we can proceed with the preparation and publication of your manuscript.

You can highlight your article and the work of your group on the back cover of Digital Discovery. If you are interested in this opportunity please contact the editorial office for more information.

Promote your research, accelerate its impact – find out more about our article promotion services here: https://rsc.li/promoteyourresearch.

If you would like us to promote your article on our LinkedIn account [https://rsc.li/Digital_showcase] please fill out this form: https://form.jotform.com/213544038469056.

We are offering all corresponding authors on publications in gold open access RSC journals who are not already members of the Royal Society of Chemistry one year’s Affiliate membership. If you would like to find out more please email membership@rsc.org, including the promo code OA100 in your message. Learn all about our member benefits at https://www.rsc.org/membership-and-community/join/#benefit

By publishing your article in Digital Discovery, you are supporting the Royal Society of Chemistry to help the chemical science community make the world a better place.

With best wishes,

Dr Joshua Schrier
Associate Editor, Digital Discovery


EDITOR'S COMMENT:

Please be sure to address the typographical error noted by Referee #1 in your page-proof corrections.

Reviewer 1

I thank the authors for adding more information regarding data and training. The updated manuscript has addressed my concerns.

There is a small typo in the text added in section 4.1: “The data splits are tabulated in Table S17 in ESI”. There is no Table S17 in the ESI; it should be Table S16.