From the journal Digital Discovery Peer review history

07-Aug-2022

Dear Dr Gow:

Manuscript ID: DD-REV-05-2022-000047
TITLE: A Review of Reinforcement Learning in Chemistry

Thank you for your submission to Digital Discovery, published by the Royal Society of Chemistry. I sent your manuscript to reviewers and I have now received their reports which are copied below.

I have carefully evaluated your manuscript and the reviewers’ reports, and the reports indicate that major revisions are necessary, especially in reducing the length of the general introduction to reinforcement learning, which I agree with - this makes the paper very long and the more interesting sections appear 'too late' in the manuscript.

Please submit a revised manuscript which addresses all of the reviewers’ comments. Further peer review of your revised manuscript may be needed. When you submit your revised manuscript please include a point by point response to the reviewers’ comments and highlight the changes you have made. Full details of the files you need to submit are listed at the end of this email.

Digital Discovery strongly encourages authors of research articles to include an ‘Author contributions’ section in their manuscript, for publication in the final article. This should appear immediately above the ‘Conflict of interest’ and ‘Acknowledgement’ sections. I strongly recommend you use CRediT (the Contributor Roles Taxonomy from CASRAI, https://casrai.org/credit/) for standardised contribution descriptions. All authors should have agreed to their individual contributions ahead of submission and these should accurately reflect contributions to the work. Please refer to our general author guidelines https://www.rsc.org/journals-books-databases/author-and-reviewer-hub/authors-information/responsibilities/ for more information.

Please submit your revised manuscript as soon as possible using this link:

*** PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm. ***

https://mc.manuscriptcentral.com/dd?link_removed

(This link goes straight to your account, without the need to log on to the system. For your account security you should not share this link with others.)

Alternatively, you can login to your account (https://mc.manuscriptcentral.com/dd) where you will need your case-sensitive USER ID and password.

You should submit your revised manuscript as soon as possible; please note you will receive a series of automatic reminders. If your revisions will take a significant length of time, please contact me. If I do not hear from you, I may withdraw your manuscript from consideration and you will have to resubmit. Any resubmission will receive a new submission date.

The Royal Society of Chemistry requires all submitting authors to provide their ORCID iD when they submit a revised manuscript. This is quick and easy to do as part of the revised manuscript submission process.   We will publish this information with the article, and you may choose to have your ORCID record updated automatically with details of the publication.

Please also encourage your co-authors to sign up for their own ORCID account and associate it with their account on our manuscript submission system. For further information see: https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/#attribution-id

I look forward to receiving your revised manuscript.

Yours sincerely,
Dr Kedar Hippalgaonkar
Associate Editor, Digital Discovery
Royal Society of Chemistry

************

Reviewer 1

This manuscript provides a broad review of the application of reinforcement learning in chemistry. This paper starts with an introduction of the theory and algorithms of reinforcement learning, followed by a discussion of the applications of reinforcement learning in drug discovery, molecular design, geometry optimization, biochemical sequence design, and reaction planning. I think this review merits to be published after the authors consider the following suggestions:

1. The manuscript is so long that it is not easy for the reader to focus on the key content of this paper. I suggest that the author could delete some unimportant sections to improve the readability of the paper.
2. The references in this manuscript should be revised, especially in the introduction part (p. 1-3). Currently, there are no references in the introduction. The authors should also modify the citation format of this paper to meet the requirement of the journal.
3. p. 11-12, “This was addressed by Hochreiter and Schmidhuber [1997] through the creation of Long-Short-Term Memory (LSTM), a system in which information is stored in a cell and the flow of information to and from the cell is regulated by a series of gates. A network will typically include a large number of LSTM units within its layers. The Gated Recurrent Unit (GRU) is a more recent development… ” It would help the reader if the manuscript contained a figure describing the architecture of RNN and its varieties (LSTM and GRU).
4. p. 12, “LSTM and GRU architectures can struggle with certain tasks due to limitations of their structure.” Transformer architecture has achieved great success in natural language process fields with a dramatic improvement compared with RNNs in many tasks. Recent works have also demonstrated the possibility of using the transformer architecture in certain RL applications (see Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. arXiv November 28, 2021 and Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv June 24, 2021). The authors can consider mentioning the mechanism of the transformer or the self-attention layer after the discussion of RNN, since it has almost replaced RNN in the NLP domain.

Reviewer 2

The review summarizes the theories and the general frameworks of reinforcement learning and its recent applications in chemistry, which is a good fit to the journal. However, I could not recommend the review to be published due to the following reasons.

First, the review lacks proper citations. For example, I did not find any citations in the Introduction section. The famous AlphaGo work was mentioned in the last sentence of the first paragraph on page 3 with no citation. Section 2, Theory of reinforcement learning, first cites Sutton and Barto's review (Sutton and Barto [2018]), but very few citations were given after that. Since this is a review article, having proper citations not only acknowledges the contributions of the original authors but also provides references to the audience for further reading. Hence, the citations in this review needs careful examination before it is submitted again.

Second, given the review focuses on applications in chemistry, I suggest that the author should reduce the length of the general introductions of reinforcement learning and neural networks in section 2 and section 3. Currently, section 2 and 3 take approx. 12 pages, or about 50% of the review. Many of these general introductions are not relevant to chemistry.

We thank the reviewers for their insightful and constructive feedback. We have addressed
the reviewers’ comments in a point by point response below, indicating what actions have
been taken for each point.

Reviewer 1

The manuscript is so long that it is not easy for the reader to focus on the key content of this paper. I suggest that the author could delete some unimportant sections to improve the readability of the paper.
- We have reduced the length of the paper by rationalising Sections 2 and 3, including removing the subsection on combining RL with other forms of machine learning (a small residual portion of this content now being covered in Section 4) and removing coverage of activation functions altogether, alongside smaller changes to make the remaining sections more concise. We have also implemented a two-column article template so that the remaining content is presented in a more compact fashion, and added subsections to improve readability.

The references in this manuscript should be revised, especially in the introduction part (p. 1-3). Currently, there are no references in the introduction. The authors should also modify the citation format of this paper to meet the requirement of the journal.
- We have added several additional references, including to the Introduction but also to other sections, and revised the text of the article to reflect these references where necessary. We have also altered the citation format to the recommended style.

p. 11-12, “This was addressed by Hochreiter and Schmidhuber [1997] through the creation of Long-Short-Term Memory (LSTM), a system in which information is stored in a cell and the flow of information to and from the cell is regulated by a series of gates. A network will typically include a large number of LSTM units within its layers. The Gated Recurrent Unit (GRU) is a more recent development… ” It would help the reader if the manuscript contained a figure describing the architecture of RNN and its varieties (LSTM and GRU).
- We have added a figure depicting a simple RNN. We have chosen not to include additional figures describing LSTM and GRU in detail for reasons of length per comment 1 and Reviewer 2’s comments, as the technical details of these architectures are not the most important part of the paper and would be over-represented if multiple figures were included.

p. 12, “LSTM and GRU architectures can struggle with certain tasks due to limitations of their structure.” Transformer architecture has achieved great success in natural language process fields with a dramatic improvement compared with RNNs in many tasks. Recent works have also demonstrated the possibility of using the transformer architecture in certain RL applications (see Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. arXiv November 28, 2021 and Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv June 24, 2021). The authors can consider mentioning the mechanism of the transformer or the self-attention layer after the discussion of RNN, since it has almost replaced RNN in the NLP domain.
- We have added a paragraph to Section 3 discussing Transformer architecture and its relationship to reinforcement learning; we chose to discuss this under “Other networks” after the introduction of the variational autoencoder for content flow reasons, since VAEs and Transformers both employ an encoder-decoder structure. We have also added content to Section 4 noting the relationship between Transformer architecture and the networks used in the molecular geometry optimisation application of Ahuja et al. (2021).

Reviewer 2

First, the review lacks proper citations. For example, I did not find any citations in the Introduction section. The famous AlphaGo work was mentioned in the last sentence of the first paragraph on page 3 with no citation. Section 2, Theory of reinforcement learning, first cites Sutton and Barto's review (Sutton and Barto [2018]), but very few citations were given after that. Since this is a review article, having proper citations not only acknowledges the contributions of the original authors but also provides references to the audience for further reading. Hence, the citations in this review needs careful examination before it is submitted again.
- We have added a large number of new citations, in particular to the Introduction and Section 2, and have revised the text of the article to reflect these citations where necessary. The mention of AlphaGo in particular is now cited to the original paper.

Second, given the review focuses on applications in chemistry, I suggest that the author should reduce the length of the general introductions of reinforcement learning and neural networks in section 2 and section 3. Currently, section 2 and 3 take approx. 12 pages, or about 50% of the review. Many of these general introductions are not relevant to chemistry.
- We have reduced the length of Sections 2 and 3 to take up less than 40% of the body of the review (this would be lower but for the addition of content to Section 3 per Reviewer 1’s third and fourth comments). This has been achieved by removing less relevant content such as coverage of activation functions and combinations of RL with other types of machine learning, and making the remaining content more concise.

27-Aug-2022

Dear Dr Gow:

Manuscript ID: DD-REV-05-2022-000047.R1
TITLE: A Review of Reinforcement Learning in Chemistry

Thank you for submitting your revised manuscript to Digital Discovery. I am pleased to accept your manuscript for publication in its current form.

You will shortly receive a separate email from us requesting you to submit a licence to publish for your article, so that we can proceed with the preparation and publication of your manuscript.

You can highlight your article and the work of your group on the back cover of Digital Discovery. If you are interested in this opportunity please contact the editorial office for more information.

Promote your research, accelerate its impact – find out more about our article promotion services here: https://rsc.li/promoteyourresearch.

If you would like us to promote your article on our Twitter account @digital_rsc please fill out this form: https://form.jotform.com/213544038469056.

By publishing your article in Digital Discovery, you are supporting the Royal Society of Chemistry to help the chemical science community make the world a better place.

With best wishes,

Dr Kedar Hippalgaonkar
Associate Editor, Digital Discovery
Royal Society of Chemistry


******
******

Please contact the journal at digitaldiscovery@rsc.org

************************************

DISCLAIMER:

This communication is from The Royal Society of Chemistry, a company incorporated in England by Royal Charter (registered number RC000524) and a charity registered in England and Wales (charity number 207890). Registered office: Burlington House, Piccadilly, London W1J 0BA. Telephone: +44 (0) 20 7437 8656.

The content of this communication (including any attachments) is confidential, and may be privileged or contain copyright material. It may not be relied upon or disclosed to any person other than the intended recipient(s) without the consent of The Royal Society of Chemistry. If you are not the intended recipient(s), please (1) notify us immediately by replying to this email, (2) delete all copies from your system, and (3) note that disclosure, distribution, copying or use of this communication is strictly prohibited.

Any advice given by The Royal Society of Chemistry has been carefully formulated but is based on the information available to it. The Royal Society of Chemistry cannot be held responsible for accuracy or completeness of this communication or any attachment. Any views or opinions presented in this email are solely those of the author and do not represent those of The Royal Society of Chemistry. The views expressed in this communication are personal to the sender and unless specifically stated, this e-mail does not constitute any part of an offer or contract. The Royal Society of Chemistry shall not be liable for any resulting damage or loss as a result of the use of this email and/or attachments, or for the consequences of any actions taken on the basis of the information provided. The Royal Society of Chemistry does not warrant that its emails or attachments are Virus-free; The Royal Society of Chemistry has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. Please rely on your own screening of electronic communication.

More information on The Royal Society of Chemistry can be found on our website: www.rsc.org