Students and Faculty Reflect on Large Language Models as Essential Tools for Reading Academic Papers

Key Takeaways

A study tested various large language models (LLMs) for their effectiveness in reading scientific literature.
While LLMs show promise, they struggle with data visualization and may utilize unreliable sources.
Experts urge students to critically engage with scientific literature despite the assistance of AI tools.

A Study on AI in Academic Research

Artificial intelligence (AI) is increasingly integrated into higher education, with students using it to enhance their learning and professors incorporating it into classes. A recent study by Cornell physicists and Google researchers evaluated the performance of six large language models (LLMs) — including ChatGPT, Claude, and Google Gemini — in understanding scientific literature at an expert level.

The study highlighted varying performance among the models, prompting researchers to identify areas for improvement in future AI systems. Eunah Ah-Kim, a professor of physics and study co-author, noted the significant time required to read and write scientific papers. She expressed that AI could simulate expert conversations to enhance comprehension. “AI models can be a more efficient way to engage with literature,” she stated.

The researchers compiled a database of 1,726 scientific papers about high-temperature cuprates, a group of superconducting materials. They generated 67 questions to assess the LLMs’ understanding, which were graded by 12 experts unfamiliar with the models’ identities. Findings revealed that ChatGPT, Claude, Perplexity, and Gemini excelled in sourcing data from the web, whereas NotebookLM and a retrieval-augmented generation model provided more reliable information by drawing from reputable sources.

However, Ah-Kim pointed out that while LLMs excel at processing text, they fall short in data visualization. “Models are good at handling texts, but they are not yet at an expert level,” she noted, emphasizing that experts depend on graphs and figures for deeper insights.

Students have also incorporated LLMs into their studies. Computer science and astronomy major Abrar Amin finds AI beneficial for exploring academic fields, enabling easier access to scientific papers. Though math student Parker Fuld acknowledges the tool’s utility, he regards AI primarily as an aid for understanding complex topics.

Despite their advantages, the current limitations of LLMs persist. Ah-Kim cautioned that these models sometimes reference non-peer-reviewed or unreliable sources. “Credibility vetted sources should only be used in scientific literature,” she asserted. Astrophysics student Anna Johnson voiced skepticism about relying on AI for paper comprehension, citing concerns about reliability.

Students like biological sciences major Natalie Deng also noted AI’s occasional errors, highlighting the need for cross-referencing with original articles, which can be time-consuming. Ultimately, Ah-Kim encourages students to engage deeply with the literature, affirming the importance of critical reading skills in research. “Learning how to read critically and measure how claims relate to evidence is vital,” she concluded.

The content above is a summary. For more details, see the source article.