From the journal Digital Discovery Peer review history

09-Jan-2024

Dear Dr Fang:

Manuscript ID: DD-ART-11-2023-000219
TITLE: Retro-BLEU: Quantifying Chemical Plausibility of Retrosynthesis Routes through Reaction Template Sequence Analysis

Thank you for your submission to Digital Discovery, published by the Royal Society of Chemistry. I sent your manuscript to reviewers and I have now received their reports which are copied below.

After careful evaluation of your manuscript and the reviewers’ reports, I will be pleased to accept your manuscript for publication after revisions.

Please revise your manuscript to fully address the reviewers’ comments. When you submit your revised manuscript please include a point by point response to the reviewers’ comments and highlight the changes you have made. Full details of the files you need to submit are listed at the end of this email.

Digital Discovery strongly encourages authors of research articles to include an ‘Author contributions’ section in their manuscript, for publication in the final article. This should appear immediately above the ‘Conflict of interest’ and ‘Acknowledgement’ sections. I strongly recommend you use CRediT (the Contributor Roles Taxonomy, https://credit.niso.org/) for standardised contribution descriptions. All authors should have agreed to their individual contributions ahead of submission and these should accurately reflect contributions to the work. Please refer to our general author guidelines https://www.rsc.org/journals-books-databases/author-and-reviewer-hub/authors-information/responsibilities/ for more information.

Please submit your revised manuscript as soon as possible using this link :

*** PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm. ***

https://mc.manuscriptcentral.com/dd?link_removed

(This link goes straight to your account, without the need to log in to the system. For your account security you should not share this link with others.)

Alternatively, you can login to your account (https://mc.manuscriptcentral.com/dd) where you will need your case-sensitive USER ID and password.

You should submit your revised manuscript as soon as possible; please note you will receive a series of automatic reminders. If your revisions will take a significant length of time, please contact me. If I do not hear from you, I may withdraw your manuscript from consideration and you will have to resubmit. Any resubmission will receive a new submission date.

The Royal Society of Chemistry requires all submitting authors to provide their ORCID iD when they submit a revised manuscript. This is quick and easy to do as part of the revised manuscript submission process.   We will publish this information with the article, and you may choose to have your ORCID record updated automatically with details of the publication.

Please also encourage your co-authors to sign up for their own ORCID account and associate it with their account on our manuscript submission system. For further information see: https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/#attribution-id

I look forward to receiving your revised manuscript.

Yours sincerely,
Linda Hung
Associate Editor
Digital Discovery
Royal Society of Chemistry

************

Reviewer 1

The manuscript presents a new metric for ranking retrosynthesis routes: An important and difficult topic which has the potential of reducing experimental work. The text is in general written with a clear language. Although the authors have resolved most comments from earlier submission, there is still a need to revise the manuscript. Specifically, the structure of Methods/results/discussion, some clarifications, and giving weight to the otherwise interesting results. I have listed a few comments which should be amended before acceptance to publication.

“Herein, we present the results for a radius of 1, results for other radii can be found in Supplementary Table 2, indicating that using a radius of 1 is an optimal choice for evaluating template sequences.” -> What in these results indicate that radius=1 is optimal?

Is there a better word for “patent routes” to emphasize the difference to “known routes” – perhaps “patent test routes”?

The structure of the text is slightly confusing. Now, there are results and conclusions presented in the “Data & Methods” section. It would be better to just present the data, fraction of n-gram overlap equation, Retro-BLEU score, evaluation metrics in the Methods section and then present the analysis of the reaction -> template n-gram analysis (related to Table 1), which also verifies the hypothesis that route length should be considered. The Methods section should just state how things were done. Analyses, results, and conclusions should be in the Results/Discussion/Conclusions sections.

“In practice, we might have an estimation of the number of required reactions for each target molecule to set this value.” -> By analyzing the model-generated reaction routes? Please specify in the text.

Eq. 5: Should it be the negative log-likelihood or the log-likelihood?

Figure 1 does not appear to be referenced in the text.

Fig. 2: Increase font size in axis labels and axis tick-labels.

It would be nice to have a file with the “known routes” bigrams in the supplementary material.

Fig. 4. These are only one example per case. It could be nice to see how these conclusions are overall, e.g. by randomly selecting (i.e. not cherry-picking) 5 or 10 targets where the top-ranked generated route has a higher Retro-BLEU score than the Patent route and evaluating (e.g. though visual inspection of the routes) among those how often Retro-BLEU suggests good alternatives to the Patent route. The routes could go into SI.

It looks like RXNmapper was used in the code to extract templates. If so, it should be referenced in the text. In general, all steps (e.g. needing to do atom mapping with RXNmapper etc.) should be outlined in the methods section.

Reviewer 2

The authors present Retro-BLEU which is a statistical metric to evaluate the plausibility of retrosynthesis routes. I was asked to perform a data and code review. While the code satisfies most of the criteria within the data review checklist (attached). My only minor comment is that the authors need to clarify which versions of the datasets and the code dependencies did they use for the study.

Reviewer 3

(Note: I used markdown to format this text)

The authors present a method, Retro-BLEU, for the assessment of the plausibility of retrosynthesis routes. They adapt the BLEU method commonly used to assess the quality of machine-translated text in natural language processing.

This topic is highly timely, as there is an urgent need for more thorough computational analysis and assessment of predicted retrosynthetic routes.

I would also like to thank the authors for the README.md in the associated GitHub repository, increasing the reproducibility and accessibility of their study.

Based on the previously reviews (attached to the manuscript), I believe this manuscript is almost ready for publication and I would like to congratulate the authors for their work so far. I have a few additional small comments which are detailed below.

### Introduction
No comments
### Dataset and Methods
#### n-gram overlap analysis
1. "feasibility of a synthesis route". In the abstract, you use plausibility instead of feasibility (which you use also in other places throughout the text). While plausibility, correctly, suggests that your method assesses whether the routes are plausible, feasibility suggests that the method also assesses a qualitative aspect (such as cost, yield, etc.). For this reason, I would continue to use plausibility instead of feasibility throughout the text.
2. "as the canonical SMILES of each molecule is unique". Did the authors consider potentially discarded stereochemistry, which could lead to identical SMILES for molecules just differing in stereochemistry.
3. In Equation 1, why did you choose to refer to the known n-grams as "recorded"? In my opinion, referring to them as "known", as you do later in the text would increase the readability of the text.
### Results and Discussions
#### Differentiating patent-extracted routes
4. "We believe that validated routes derived from patents are chemically feasible". It may be worth mentioning that the extraction of rules from patent text may add errors.

Response on Retro-BLEU

REVIEWER REPORT(S):
Referee: 1

Comments to the Author
The manuscript presents a new metric for ranking retrosynthesis routes: An important and difficult topic which has the potential of reducing experimental work. The text is in general written with a clear language. Although the authors have resolved most comments from earlier submission, there is still a need to revise the manuscript. Specifically, the structure of Methods/results/discussion, some clarifications, and giving weight to the otherwise interesting results. I have listed a few comments which should be amended before acceptance to publication.

“Herein, we present the results for a radius of 1, results for other radii can be found in Supplementary Table 2, indicating that using a radius of 1 is an optimal choice for evaluating template sequences.” -> What in these results indicate that radius=1 is optimal?

Thank you for your query regarding our choice of a radius of 1 in our analysis.

In the Supplementary Information, we have discussed the impact of the template radius on the sparsity of the reaction template space. The chosen radius for a template determines the extent of the chemical environment encapsulated around the reaction center, which in turn influences the sparsity of the chemical space.

A template with a radius of 0 focuses only on the atoms undergoing change at the reaction center. In contrast, a template with a radius r encompasses the surrounding r atoms. A radius of 0 proves to be insufficiently representative, as a single template might correspond to multiple reactions. Conversely, increasing the radius beyond 1 resulted in an excessively low bigram overlap in the patent test routes. At a radius of 2, the overlap template bigram ratio for patent routes drops to a mere 34.8\%, resulting in a chemical space that is too sparse for effective evaluation.

After considering these factors, we determined that a radius of 1 strikes the optimal balance for evaluating template sequences, offering a meaningful compromise between specificity and coverage. We've also added a brief explanation about this in the main text.

Is there a better word for “patent routes” to emphasize the difference to “known routes” – perhaps “patent test routes”?

Thank you for your suggestion regarding the terminology used in our manuscript.

To enhance clarity and distinguish more effectively between 'patent routes' and 'known routes,' we have adopted the term “patent test routes” throughout the manuscript. This change emphasizes their role as test cases in our analysis and aligns better with the context of our study.

The structure of the text is slightly confusing. Now, there are results and conclusions presented in the “Data & Methods” section. It would be better to just present the data, fraction of n-gram overlap equation, Retro-BLEU score, evaluation metrics in the Methods section and then present the analysis of the reaction -> template n-gram analysis (related to Table 1), which also verifies the hypothesis that route length should be considered. The Methods section should just state how things were done. Analyses, results, and conclusions should be in the Results/Discussion/Conclusions sections.

Thank you for your constructive feedback on the structure of our manuscript.

We understand your concerns regarding the separation of methods and results. In typical academic papers, the structure you propose would indeed be logical and well-received. However, in the context of our work with Retro-BLEU, the foundational ideas are derived from BLEU, where a natural analogy can be drawn with reactions serving as tokens. Our data analysis revealed a significant sparsity in reactions, indicating that reactions of the same type can effectively be categorized using templates. Therefore, we adapted the n-gram analysis from BLEU to focus on reaction templates, leading to the development of the Retro-BLEU metric. We found that our current structural arrangement more naturally aligns with the description and development of the Retro-BLEU concept. Hence, we have opted to retain the existing structure of our manuscript. We believe that in this specific context, maintaining the current structure aids in a smoother narrative and enhances the manuscript's overall clarity.

“In practice, we might have an estimation of the number of required reactions for each target molecule to set this value.” -> By analyzing the model-generated reaction routes? Please specify in the text.

We are grateful for your constructive advice regarding the methodology in the manuscript. In response to your comment, we have clarified in the manuscript that the estimation of the number of required reactions for each target molecule can be derived from the average length of the model-generated reaction routes. Alternatively, there are also some tools like DFRscore can be used to estimate the number of steps to synthesize a molecule. We have added supplementary explanations in this section for further clarity.

Eq. 5: Should it be the negative log-likelihood or the log-likelihood?

Thank you for pointing out this error, it should be the log-likelihood. We revised the manuscript accordingly.



Figure 1 does not appear to be referenced in the text.

Thank you for highlighting the omission of a reference to Figure 1 in our text. We have addressed this by including a mention of Figure 1 in the last paragraph of the Introduction section.

Fig. 2: Increase font size in axis labels and axis tick-labels.

Thank you for your suggestion regarding the readability of Fig. 2. In response to your feedback, we have increased the font size of both the axis labels and the axis tick-labels in Fig. 2 (also the similar figure in Supplementary Information).

It would be nice to have a file with the “known routes” bigrams in the supplementary material.

We acknowledge the importance of making these bigrams accessible for further research and reference. However, due to the substantial size of the "known bigrams" file, we have opted to host it externally. We have uploaded these files in our associated GitHub repository, ensuring that the data is readily available to all interested readers and researchers.

Fig. 4. These are only one example per case. It could be nice to see how these conclusions are overall, e.g. by randomly selecting (i.e. not cherry-picking) 5 or 10 targets where the top-ranked generated route has a higher Retro-BLEU score than the Patent route and evaluating (e.g. though visual inspection of the routes) among those how often Retro-BLEU suggests good alternatives to the Patent route. The routes could go into SI.

We agree that providing a broader range of examples would strengthen our conclusions. In response, we have included an additional set of five randomly selected routes in the Supplementary Information.

These routes have been subjected to human evaluation to demonstrate the efficacy of the Retro-BLEU score in suggesting viable alternatives to the patent test routes. We appreciate your guidance in improving our manuscript.

It looks like RXNmapper was used in the code to extract templates. If so, it should be referenced in the text. In general, all steps (e.g. needing to do atom mapping with RXNmapper etc.) should be outlined in the methods section.

You are correct in noting that RXNmapper was utilized for atom mapping in certain cases before template extraction. Specifically, while the patent reactions are already mapped, and the generated routes from PaRoutes have their template information with them (they are predicted via a template-based single-step retrosynthesis model), for the test routes on Retro*-190, we employed RXNMapper before extracting templates.

We have updated the Methods section of our manuscript to include this step and have duly referenced the corresponding research paper detailing RXNmapper.

We are grateful for your insightful feedback, which has helped in enhancing the accuracy and completeness of our paper.


Referee: 2

Comments to the Author
The authors present Retro-BLEU which is a statistical metric to evaluate the plausibility of retrosynthesis routes. I was asked to perform a data and code review. While the code satisfies most of the criteria within the data review checklist (attached). My only minor comment is that the authors need to clarify which versions of the datasets and the code dependencies did they use for the study.

We deeply appreciate your time and effort in conducting a thorough data and code review, and we are grateful for your positive feedback.

Regarding your valuable comment on specifying the versions of datasets and code dependencies used in our study, we have now included the versions of datasets and code dependencies in both the “Data Availability” section of our manuscript and the README.md file in our associated GitHub repository.

Referee: 3

Comments to the Author
(Note: I used markdown to format this text)

The authors present a method, Retro-BLEU, for the assessment of the plausibility of retrosynthesis routes. They adapt the BLEU method commonly used to assess the quality of machine-translated text in natural language processing.
This topic is highly timely, as there is an urgent need for more thorough computational analysis and assessment of predicted retrosynthetic routes.
I would also like to thank the authors for the README.md in the associated GitHub repository, increasing the reproducibility and accessibility of their study.
Based on the previously reviews (attached to the manuscript), I believe this manuscript is almost ready for publication and I would like to congratulate the authors for their work so far. I have a few additional small comments which are detailed below.

Thank you for your encouraging feedback and for acknowledging the timeliness and significance of our work in developing Retro-BLEU. We share your view on the urgency of accurately assessing multi-step retrosynthesis routes. Our aim with Retro-BLEU is to contribute meaningfully to this field, enhancing the quality and reliability of computational retrosynthesis. We are committed to advancing the field of AI-assisted synthesis planning and believe that tools like Retro-BLEU are a step towards achieving higher standards in retrosynthetic analysis.


### Introduction
No comments
### Dataset and Methods
#### n-gram overlap analysis
1. "feasibility of a synthesis route". In the abstract, you use plausibility instead of feasibility (which you use also in other places throughout the text). While plausibility, correctly, suggests that your method assesses whether the routes are plausible, feasibility suggests that the method also assesses a qualitative aspect (such as cost, yield, etc.). For this reason, I would continue to use plausibility instead of feasibility throughout the text.

Thank you for highlighting the important distinction between 'plausibility' and 'feasibility' in the context of our study.

Upon your suggestion, we have reviewed the manuscript and replaced instances of 'feasibility' with 'plausibility' throughout the text. We appreciate your contribution to the precision and clarity of our manuscript.

2. "as the canonical SMILES of each molecule is unique". Did the authors consider potentially discarded stereochemistry, which could lead to identical SMILES for molecules just differing in stereochemistry.

In our study, we conscientiously retained stereochemistry information during the extraction of templates. We recognize that stereochemical details, when documented in scientific literature, may play a crucial role in the nuances of chemical reactions.

By preserving this information, we aimed to ensure that our analysis captures these subtle but essential variations, reflecting more accurately the complexity of real-world chemical synthesis.
We hope this clarification addresses your concern and we value your attention to such important details in our work.

3. In Equation 1, why did you choose to refer to the known n-grams as "recorded"? In my opinion, referring to them as "known", as you do later in the text would increase the readability of the text.

Thank you for your recommendation regarding the terminology used in Equation 1. We concur that replacing 'recorded' with 'known' in the context of the n-grams will improve the consistency and readability of the text. This change aligns with our prior reference to 'known n-grams' and should facilitate a clearer understanding for our readers.

We have updated Equation 1 accordingly and reviewed the surrounding text to ensure this terminology is uniformly applied.

### Results and Discussions
#### Differentiating patent-extracted routes
4. "We believe that validated routes derived from patents are chemically feasible". It may be worth mentioning that the extraction of rules from patent text may add errors.

We agree with you that due to the possible errors occurred during the text extraction phase, those validated routes may also introduce errors. Acknowledging this, we have included a note in our manuscript about the possibility of such errors. This addition to our manuscript maintains its scientific rigor, while also acknowledging the real-world challenges and limitations inherent in data extraction from complex sources like patents.

We appreciate your attention to detail and your contribution towards enhancing the robustness of our study.

26-Jan-2024

Dear Dr Fang:

Manuscript ID: DD-ART-11-2023-000219.R1
TITLE: Retro-BLEU: Quantifying Chemical Plausibility of Retrosynthesis Routes through Reaction Template Sequence Analysis

Thank you for submitting your revised manuscript to Digital Discovery. I am pleased to accept your manuscript for publication in its current form. I have copied any final comments from the reviewer(s) below.

You will shortly receive a separate email from us requesting you to submit a licence to publish for your article, so that we can proceed with the preparation and publication of your manuscript.

You can highlight your article and the work of your group on the back cover of Digital Discovery. If you are interested in this opportunity please contact the editorial office for more information.

Promote your research, accelerate its impact – find out more about our article promotion services here: https://rsc.li/promoteyourresearch.

If you would like us to promote your article on our Twitter account @digital_rsc please fill out this form: https://form.jotform.com/213544038469056.

We are offering all corresponding authors on publications in gold open access RSC journals who are not already members of the Royal Society of Chemistry one year’s Affiliate membership. If you would like to find out more please email membership@rsc.org, including the promo code OA100 in your message. Learn all about our member benefits at https://www.rsc.org/membership-and-community/join/#benefit

By publishing your article in Digital Discovery, you are supporting the Royal Society of Chemistry to help the chemical science community make the world a better place.

With best wishes,

Linda Hung
Associate Editor
Digital Discovery
Royal Society of Chemistry

Reviewer 2

The authors have successfully answered all my queries as part of the data and code review.

Reviewer 1

I believe the authors have sufficiently well responded to the remarks by the reviewer and therefore I recommend it to be published as-is.