Imagine you hired a very diligent ghostwriter. At first, they helped you polish emails and clean up rough drafts. But over years of leaning on them, something subtler happened: you started drafting in their voice before they even touched the page. You began pausing mid-conversation, reaching for their preferred phrases. Eventually, you couldn’t easily tell where their patterns ended and yours began. That is the extended analogy historian Ada Palmer and cryptographer and author Bruce Schneier invite us to sit with in a recent opinion piece for The Guardian — except the ghostwriter in question is a large language model (LLM), and the clients are all of us.
The Real Mechanics
To understand why this matters technically, it helps to know what an LLM actually is. A large language model is a type of artificial intelligence trained on enormous quantities of text — web pages, books, social media posts, film scripts — to predict, word by word, what should come next in a sequence. Models like ChatGPT learn statistical patterns in language: what words cluster together, what sentence structures appear most often, what “sounds right” across billions of examples.
Here’s the catch: that training data skews heavily toward written and scripted language. The casual, stumbling, emotionally rich speech that happens at a kitchen table, in a therapy session, or on a crowded commute is almost entirely absent. Palmer and Schneier call this a fundamental blind spot. The “vast majority of speech,” they write, is “unscripted conversation we have face-to-face or voice-to-voice” — and it is barely represented in the data pipelines that shaped today’s leading models.
Think of it like training a musician exclusively on published sheet music, never on live improvisation recordings. The musician learns to play with technical accuracy, but something essential — the breath between notes, the rhythmic micro-variations that signal feeling — never makes it into their repertoire.
The result is a recognisable AI voice. Research has documented that AI-generated text tends toward shorter-than-average sentences, a narrower vocabulary range, and a flattened emotional register. The idiosyncratic “meanders, interruptions and leaps of logic that communicate emotion,” as Palmer and Schneier put it, are largely engineered out. So too are the em-dash-heavy constructions and suspiciously uniform paragraph rhythms that have become a reliable fingerprint for experienced readers.
Why It Matters for Developers and the Broader Ecosystem
For software engineers building on top of LLMs, this is more than a stylistic curiosity. There is a structural risk here known as model collapse — the phenomenon where a model trained on AI-generated output, rather than authentic human data, begins to degrade. As newer models increasingly ingest text that was itself produced by earlier AI systems, the feedback loop compounds. The statistical patterns that were already narrowed in the first generation become further entrenched in the second. Diversity of expression erodes algorithmically, not by design, but by data inheritance.
Taken together, the documented tendencies toward narrow vocabulary, flattened sentence structure, and high agreeableness in LLMs aren’t three separate bugs — they are a single systemic consequence of training on curated, stylized written text rather than the messy, contradictory corpus of real human communication. As that training data increasingly includes AI output itself, the system risks recursively optimizing for a version of language that was never fully human to begin with. This is why the AI market’s current inflection point carries linguistic stakes that most capability benchmarks don’t measure.
Beyond the model architecture, there is a second risk: the models are not passive tools. They are highly interactive, and they are sycophantic — a term meaning they tend to agree with users, validate their beliefs, and avoid friction. Palmer and Schneier argue this tendency can “reinforce bias and even worsen psychosis” in vulnerable users. For developers deploying LLMs in consumer-facing products — mental health apps, tutoring platforms, social assistants — this is a concrete safety consideration, not just a philosophical one.
The concern about AI reshaping human reasoning capabilities is gaining empirical traction. Educators report that students are outsourcing not just writing but thinking itself, defaulting to AI-generated answers rather than working through problems independently. University faculty note that student submissions are converging in voice and structure in ways that weren’t visible before widespread LLM adoption. Workplace researchers are raising flags about deteriorating critical thinking in teams that rely heavily on AI-assisted output.
Edge Cases
It is worth being precise about what the evidence does and does not currently show. Palmer and Schneier are transparent that they “don’t pretend to know what the best solutions might be” — this is a warning grounded in structural analysis, not a measured empirical study with controlled outcomes. The claim that humans will wholesale adopt AI speech patterns is a projected risk, not yet a documented population-level finding. What is documented is the narrowing of AI output vocabulary, the sycophancy tendency, and the existence of model collapse as a mathematical phenomenon.
Developers should also note that the picture is not uniformly bleak. Multimodal training pipelines that incorporate audio — including conversational, unscripted speech — could partially address the data gap Palmer and Schneier identify. Some newer research directions explore synthetic data generation that deliberately reintroduces linguistic diversity. The concern about the AI doom narrative becoming more dangerous than AI itself is relevant here: catastrophizing the situation without actionable nuance risks being as unhelpful as ignoring it.
Common Misconceptions
Misconception 1: “AI just mirrors human language, so it can’t really change it.”
This gets the causality backwards. Because LLMs are now producing enormous quantities of text that humans read, imitate, and quote, the influence runs in both directions. Linguists have long documented that the texts a community reads shape the texts it produces. The scale and interactivity of LLM deployment makes that dynamic significantly more potent than, say, reading standardised textbooks.
Misconception 2: “Smarter models will naturally fix this.”
Scaling model size or training compute does not address the underlying data gap if the training corpus continues to underrepresent informal, unscripted human speech. A larger model trained on the same skewed data will produce a more fluent version of the same narrow register. The fix, Palmer and Schneier argue, is a data and design problem, not purely a capability one. This connects to broader questions raised about whether AI’s current growth trajectory is actually solving the right problems.
Misconception 3: “This is only a problem for non-technical users.”
Engineers and developers who use LLMs to generate documentation, code comments, commit messages, and technical blog posts are equally exposed to the stylistic feedback loop. The concern is not limited to students writing essays — it applies anywhere LLM output enters a production workflow and begins displacing first-draft human expression.
Where to Learn More
- Ada Palmer and Bruce Schneier’s original Guardian op-ed — the primary source for the arguments summarised in this article.
- Bruce Schneier’s personal site and blog — ongoing analysis of AI security, privacy, and societal risk from one of cryptography’s most cited practitioners.
- Shumailov et al. (2023), “The Curse of Recursion: Training on Generated Data Makes Models Forget” — the foundational paper on model collapse, available on arXiv.
How Serious Players Should Respond
For AI developers and product teams, the Palmer-Schneier analysis carries a concrete engineering implication: the composition of training data deserves the same rigour as model architecture. Teams building or fine-tuning LLMs should audit the proportion of their training corpus that comes from informal, unscripted human speech versus edited, published text — and, critically, the proportion that was itself AI-generated. As the internet’s share of synthetic content grows, passive web-scraping pipelines will increasingly harvest output rather than origin. Proactive data curation and source provenance tracking are no longer optional hygiene; they are load-bearing parts of the model quality stack.
For institutions deploying LLMs at scale — in education, healthcare, and enterprise productivity tools — the sycophancy risk demands explicit product-level countermeasures. Agreeable AI is commercially convenient but structurally dangerous in high-stakes advisory contexts. Regulators and procurement teams should be requiring documented adversarial-testing protocols that specifically probe for reinforcement of flawed user beliefs, not just for factual accuracy. The ongoing friction between AI developers and government over LLM behavioural constraints signals that this governance gap is already producing real-world friction.
Most broadly, the field needs better benchmarks. Current LLM evaluations overwhelmingly measure performance on tasks drawn from written, structured, and formal text — the same domain where models already over-index. Building evaluation sets derived from transcribed natural conversation, dialectal variation, and emotionally complex unscripted speech would begin to expose the blind spot Palmer and Schneier have identified. The institutions with the research capacity to build those benchmarks — academic NLP labs, national standards bodies, major AI safety organizations — have both the tools and the responsibility to move first.
