Comparison of LLMs in extracting synthesis conditions and generating Q&A datasets for metal–organic frameworks

Abstract

Artificial intelligence, represented by large language models (LLMs), has demonstrated tremendous capabilities in natural language recognition and extraction. To further evaluate the performance of various LLMs in extracting information from academic papers, this study explores the application of LLMs in reticular chemistry, focusing on their effectiveness in generating Q&A datasets and extracting synthesis conditions from scientific literature. The models evaluated include OpenAI's GPT-4 Turbo, Anthropic's Claude 3 Opus, and Google's Gemini 1.5 Pro. Key results indicate that Claude excelled in providing complete synthesis data, while Gemini outperformed others in accuracy, characterization-free compliance (obedience), and proactive structuring of responses. Although GPT-4 was less effective in quantitative metrics, it demonstrated strong logical reasoning and contextual inference capabilities. Overall, Gemini and Claude achieved the highest scores in accuracy, groundedness, and adherence to prompt requirements, making them suitable benchmarks for future studies. The findings reveal the potential of LLMs to aid in scientific research, particularly in the efficient construction of structured datasets, which can help train models, predict, and assist in the synthesis of new metal–organic frameworks (MOFs).

Graphical abstract: Comparison of LLMs in extracting synthesis conditions and generating Q&A datasets for metal–organic frameworks

Supplementary files

Article information

Article type
Paper
Submitted
02 Mar 2025
Accepted
13 May 2025
First published
29 May 2025
This article is Open Access
Creative Commons BY-NC license

Digital Discovery, 2025, Advance Article

Comparison of LLMs in extracting synthesis conditions and generating Q&A datasets for metal–organic frameworks

Y. Shi, N. Rampal, C. Zhao, D. J. Fu, C. Borgs, J. T. Chayes and O. M. Yaghi, Digital Discovery, 2025, Advance Article , DOI: 10.1039/D5DD00081E

This article is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported Licence. You can use material from this article in other publications, without requesting further permission from the RSC, provided that the correct acknowledgement is given and it is not used for commercial purposes.

To request permission to reproduce material from this article in a commercial publication, please go to the Copyright Clearance Center request page.

If you are an author contributing to an RSC publication, you do not need to request permission provided correct acknowledgement is given.

If you are the author of this article, you do not need to request permission to reproduce figures and diagrams provided correct acknowledgement is given. If you want to reproduce the whole article in a third-party commercial publication (excluding your thesis/dissertation for which permission is not required) please go to the Copyright Clearance Center request page.

Read more about how to correctly acknowledge RSC content.

Social activity

Spotlight

Advertisements