“[The] analysis based on excess words usage suggests that at least 10% of 2024 abstracts were processed with large language models (LLMs).”
A recent paper titled “Delving into ChatGPT Usage in Academic Writing Through Excess Vocabulary” was published on the nonprofit site arXiv, which archives pre-publication papers in various fields such as physics and mathematics. The study quantifies the long-held suspicion that many researchers rely on LLMs like ChatGPT for writing abstracts, revealing that the proportion is estimated to be at least 10% or more.
The advancement of generative artificial intelligence (AI) has led researchers to increasingly rely on these tools instead of writing papers entirely on their own. This reliance has raised concerns about whether AI-generated papers might constitute plagiarism. However, there are also positive aspects, such as AI translation playing a key role in breaking down the “language barrier” and narrowing the academic gap between non-English and English-speaking scholars.
It is widely known that researchers use AI, but it was difficult to distinguish AI usage in papers without explicit disclosure by the authors. A survey conducted last September revealed that over 25% of 1,600 researchers used generative AI.
In their recent study of AI usage in abstracts, researchers from the University of Tübingen in Germany and Northwestern University in the United States analyzed the frequency of words favored by AI. They collected 14.2 million words from abstracts of papers published between January 2010 and March of this year from PubMed, a database for natural science journals, and compared this data with more recent papers. The study found a sharp increase in new words early this year, about a year after the release of ChatGPT in late 2022 and the widespread adoption of generative AI. There was a notable rise in the use of 329 words during the first three months of this year, with 280 of these being adverbs or adjectives that describe phenomena, such as “potential,” “intricate,” “meticulously,” “crucial,” and “significant.”
Researchers have found that the increased usage of specific words, commonly employed by AI, suggests a growing involvement of AI in writing research papers. They estimate that at least 10% of research abstracts published this year likely had AI assistance. The study revealed that AI involvement was especially notable in computer science papers, with a rate of 20% in papers typically authored by those proficient in AI. Around one-third of computer science papers from China, where English proficiency is lower, were estimated to have been written with AI assistance. Across various disciplines, non-English-speaking countries showed a higher proportion of papers suspected to be AI-assisted, with English-speaking countries like the U.K. and Australia estimated at below 4% and regions like China, South Korea, and Taiwan exceeding 15%.
A major concern is whether AI-assisted papers should be regarded as legitimate scholarly work. Issues raised include potential plagiarism and the limited range of expressions AI might use, which could perpetuate biases and stifle creativity. Although less than 10% of AI-utilized papers appeared in high-profile, peer-reviewed journals like Nature, Science, and Cell—which have stringent review processes—the frequency of AI use was notably higher in journals with less rigorous reviews.
Despite these concerns, there is ongoing debate about the role of AI in academic writing. AI has the potential to relieve researchers of the time-consuming burden of writing, allowing them to concentrate more on innovation and collaboration. Additionally, it can assist non-native English speakers in articulating their ideas more effectively. As noted by The Economist, AI could help level the playing field for non-native speakers in prestigious journals, which are often dominated by native English speakers. Moreover, AI facilitates the global dissemination of scientific research, enabling evaluation based on the quality and originality of ideas rather than linguistic proficiency.
There is also evidence that AI has a positive impact on education, potentially helping to narrow the educational gap between white and non-white students. According to the Walton Family Foundation, a higher percentage of Black (80%) and Hispanic (84%) parents in the U.S. use AI to support their children’s education compared to white parents (72%). This indicates that non-white parents are more actively engaging with AI tools for educational support, including providing feedback on learning and assignments.
As the perception that AI’s role in academia may not be entirely negative gains traction, the academic community’s stance is evolving. While Science initially prohibited the use of AI in papers, it now permits it as long as it is explicitly disclosed. Similarly, Nature and Cell have adopted comparable policies.
