See also Writing, LLMs in education.
Modesty forbids:
Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W. (2025). Do LLMs write like humans? Variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences, 122(8), e2422455122. doi:10.1073/pnas.2422455122
An analysis of LLM-generated writing in a parallel corpus, using grammatical and rhetorical features (not just vocabulary, word length, etc.). Finds that GPT-4o and Llama 3 have systematic differences in style, favoring more information density and a more academic style, even when prompted with fiction or TV scripts.DeLuca, L. S., Reinhart, A., Weinberg, G., Laudenbach, M., Miller, S., & Brown, D. W. (2025). Developing students’ statistical expertise through writing in the age of AI. Journal of Statistics and Data Science Education, 33(3), 266–278. doi:10.1080/26939169.2025.2497547
Comparisons to student writing (in intro stats courses) shows similar effects. Includes more detailed comparison of style with examples. Asks the question: if LLMs do not write like experts do, but students use them to learn to write, how will students learn to write?
Chang, T. A., & Bergen, B. K. (2024). Language model behavior: A comprehensive survey. Computational Linguistics, 50(1), 293–350. doi:10.1162/coli_a_00492
General review of LLM abilities in language understanding, grammar, bias, reasoning, etc.
Mizumoto, A., Yasuda, S., & Tamura, Y. (2024). Identifying ChatGPT-generated texts in EFL students’ writing: Through comparative analysis of linguistic fingerprints. Applied Corpus Linguistics, 4(3), 100106. doi:10.1016/j.acorp.2024.100106
Compares student-written essays to ChatGPT-written essays (GPT-3.5 Turbo) on the same prompt, looking at lexical and syntactic features. “ChatGPT-generated essays demonstrated greater lexical diversity, higher syntactic complexity, more nominalization, substantially fewer errors, and higher word counts compared to human-written essays. Conversely, human-written essays exhibited higher usage of modals, epistemic markers, and discourse markers, which was derived from the differences in writing styles and approaches between humans and AI.”
Goulart, L., Laı́sa Matte, M., Mendoza, A., Alvarado, L., & Velosa, I. (2024). AI or student writing? Analyzing the situational and linguistic characteristics of undergraduate student writing and AI-generated assignments. Journal of Second Language Writing, 66, 101160. doi:10.1016/j.jslw.2024.101160
Looks in more detail at Biber’s feature set, comparing student writing to GPT-3.5’s writing on the same prompts. There are lots of excerpts and a bunch of factor analysis (following Biber’s MDA scheme), ultimately “showing that AI-generated texts are more informationally dense, explicit, and less involved than student-authored texts. EFL Students tend to integrate more personal references and features of involvement, making their writing more nuanced and contextually rich.” Seems in line with our PNAS results on more recent GPT versions.
Dawkins, H., Fraser, K. C., & Kiritchenko, S. (2025). When detection fails: The power of fine-tuned models to generate human-like social media text. arXiv. https://arxiv.org/abs/2506.09975
Examines Biber features in human- and LLM-written tweets, showing systematic differences (though different than those in Reinhart et al above, probably because of the genre). More interestingly, uses a corpus of tweets to fine-tune the models, showing this dramatically reduces the differences. It appears the unusual style of instruction-tuned LLMs can be adapted to a genre, given sufficient fine-tuning data.
Milička, J., Marklová, A., & Cvrček, V. (2025). Benchmark of stylistic variation in LLM-generated texts. arXiv. https://arxiv.org/abs/2509.10179
Compares lots of different LLMs (different GPT versions, Gemini, Claude, etc.) to humans, using the same parallel prompting technique in our PNAS paper (above) in the BE21 corpus. Compares each text based on its position in Biber’s original six factors, including cool interactive plots. Finds results similar to ours: LLMs tend to shift on dimension 1 (the involved vs. informational dimension, towards more information-dense text), as well as on several other dimensions, but the shift varies a lot by model. Also reproduces the analysis on a Czech corpus to see how the models behave in other languages, finding they are much worse at matching Czech style.
Jiang, F., & Hyland, K. (2025). Rhetorical distinctions: Comparing metadiscourse in essays by ChatGPT and students. English for Specific Purposes, 79, 17–29. doi:10.1016/j.esp.2025.03.001
Jiang, F., & Hyland, K. (2025). Does ChatGPT write like a student? Engagement markers in argumentative essays. Written Communication. doi:10.1177/07410883251328311
Jiang, F., & Hyland, K. (2025). Does ChatGPT argue like students? Bundles in argumentative essays. Applied Linguistics, 46(3), 375–391. doi:10.1093/applin/amae052
A whole series examining a corpus of student essays versus GPT-4 writing to similar prompts, using various rhetorical and structural features rather than just vocabulary. Find style difference such as:
Kirilloff, G., Carroll, C., Daboul, Z., Frank, A., Khan, R., Hinrichs-Morrow, M., & Weingart, R. (2025). “Written in the style of”: ChatGPT and the literary canon. Harvard Data Science Review, 7(4). doi:10.1162/99608f92.6d5fb5ef
Has ChatGPT try to imitate 10 well-known authors from the 19th century. (I really wish Edward Bulwer-Lytton were included.) Compares GPT-4’s paragraphs to the real authors. A random forest easily distinguishes them, leading them to conclude “GPT is extremely poor at replicating the style of the authors in our corpus”, though in unusual ways: it overuses features that might be considered “literary”, writing longer and more complex sentences. But also it tends to write about female characters when prompted to write like a female author, and about male characters when prompted to write like a man, exceeding the difference from the actual authors, so there is evidently some kind of bias (in training, instruction tuning, or what?).
Liang, W. et al. (2024). Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. In Proceedings of the 41st International Conference on Machine Learning (Vol. 235, pp. 29575–29620). https://proceedings.mlr.press/v235/liang24b.html
“Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these [ML-focused] conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals.”
Leppänen, L., Aunimo, L., Hellas, A., Nurminen, J. K., & Mannila, L. (2025). How large language models are changing MOOC essay answers: A comparison of pre- and post-LLM responses. https://arxiv.org/abs/2504.13038
Changes in student writing in a MOOC AI ethics course from 2020 to 2024, comparing before Nov 2022 (release of ChatGPT) and after Dec 2023 (a year after, when it was widely available). Student answer lengths jumped around March 2023. Certain words (“delve”, “foster”, “crucial”) appear much more often post-ChatGPT and topics of discussion have changed. No detailed stylistic analysis, but shows that student writing has shifted in the ways we might expect with widespread use of ChatGPT.
Mak, M. H. C., & Walasek, L. (2025). Style, sentiment, and quality of undergraduate writing in the AI era: A cross-sectional and longitudinal analysis of 4,820 authentic empirical reports. Computers and Education: Artificial Intelligence, 9, 100507. doi:10.1016/j.caeai.2025.100507
Analyzes 4,820 reports written by psychology majors in the UK, looking mainly at lexical features (vocabulary, based on 8 words previously associated with ChatGPT), nominalizations, syntactic complexity, and readability. Also ran a sentiment analysis and obtained the grades assigned to the reports. Use of the ChatGPT-associated words grew sharply in 2023 and 2024, before declining in 2025. (Reports were submitted between May and July each year, so this isn’t due to GPT-5’s release in August 2025.) Nominalizations and complexity measures also jumped and sentiment grew more positive. But “grades and feedback remained stable, indicating that surface-level linguistic enhancements do not necessarily translate into higher academic performance.”
If LLMs write text in a particular style, it stands to reason they might write code in a particular style.
Ramachandra, A., Chaudhary, S., Tran, J., Desai, R., Pang, A., & Salloum, M. (2026). Detecting AI-generated code in introductory programming courses. In SIGCSE TS (Vol. 1, pp. 894–900). doi:10.1145/3770762.3772522
Uses token counts and CodeBERT embeddings to train classifiers that can distinguish between student- and AI-written code for introductory assignments. They claim very high accuracy (98% or better), so the style is quite distinct. They note students who over-rely on AI often perform poorly in the course, though this is mentioned in passing without analysis; I would love to see them apply the classifier to student submissions and then track their later performance on exams or projects.