First, a few words on where I am coming from. I am not anti-AI. I have written at length about my fascination for this technology. I think we should build more datacenters, I am excited about the scientific and medical progress humanity is about to make, and I enjoy using Claude Code to automate some of my own work. More generally, I believe that technology has been a force for good for humanity. Which is not to say it cannot have adverse effects worth thinking about carefully, especially if we want it to remain a force for good. Here, I want to describe one potential effect of AI: the end of human writing and reading if AI content continues to be indistinguishable from human writing.
My three assumptions
First, I assume that on average, the higher the probability a reader assigns to a block of text being AI-generated, the less likely he is to start reading it1. Whether this preference for human content is rational falls outside the scope of this essay: the fact that the preference exists is sufficient for my conclusions to hold. (I however discuss in this companion piece why I think such a preference is largely rational.)
Second, I do not assume AI-generated texts are mediocre. (I will not use the word "slop," which quickly loses relevance as models improve.) On the contrary, I assume AI-generated essays are (or will soon become) indistinguishable in form and substance from the work of talented human authors2.
The often-cited study3, where individuals were found to prefer AI writing to human content, but to dislike it when they knew it was AI, confirms both assumptions.
Finally, I assume that the hope of being read is a major motivation for human writers. Therefore, the lower someone estimates his probability to find an audience, the less likely he is to write. It is of course possible that some humans have the resources available to write purely for the pleasure of it, with no hope of social, professional or financial recognition, but I assume this is relatively marginal.
A market for lemons, with unlimited lemons
Let’s take a detour through George Akerlof’s famous 1970 paper, The Market for Lemons. Imagine a market for used cars. Most cars are good, worth say $10,000. They trade at that price; both buyers and sellers are happy. One day, the owners of bad cars ("lemons," worth close to nothing), find a way to make them indistinguishable, to the buyer, from good cars. They start selling.
Word spreads that some lemons are masquerading as good cars. Buyers now ask to be compensated for the risk of getting a lemon, and price every car at the expected value of a random car on the market, say $8,000. But at that price, some owners of good cars (who know what they hold) refuse to sell and withdraw from the market. The average quality of cars on sale falls. Buyers observe this and revise their offers down again, to reflect the new expected value of a car, say to $6,000. More good cars withdraw. The spiral continues until the market unravels: only lemons are left, and nobody buys.
Now consider the marketplace for texts on social media. The currency in which readers pay is attention (which, like money, is a finite resource). In 2019, before LLMs, the probability that a given block of text was a good car4 (a text whose words a human had chosen) was 100%. The reader applied no discount in the attention he allocated to the block of text. One day, LLMs appear, the probability drops slightly, say to 80%. Facing the same block of text as before, the reader becomes slightly less likely to read. Some writers, losing audience, stop expending effort to write. The share of human text falls to 70%. The reader applies a further discount in the attention he allocates, disincentivizing more potential writers. The spiral continues.
Note that the reader's discount falls on every text, human or machine, but a reduced reward only eliminates producers who have costs to cover. In Akerlof's model, the lemons never leave the market as their owners benefit from selling at any price, however low. The same happens here. Each drop in attention pushes some human writers below their break-even point, while no drop in attention can push an AI copy-paster below theirs. So, the composition of the pool keeps worsening5. At some point the reader understands that the probability of facing human text is close to zero, and he stops allocating attention to text altogether: the chance that he is reading ChatGPT's prose is simply too high. Without readers, the remaining writers quit. The market unravels.
The analogy underplays the unravelling. In Akerlof’s model the supply of lemons is bounded by the stock of existing bad cars. Here it is unbounded. The share of human text therefore falls through two channels at once: because human writers withdraw (Akerlof's channel), and because LLM text multiplies, as the technology spreads and models improve greatly. And there is a third effect: human texts are not only discounted (read with a lower probability) when the reader encounters them; they are encountered less and less, drowned in an ocean of AI content while the reader's navigation time stays fixed.
One source of inertia is writers with pre-AI reputations, who have proved they could write without LLMs. They will benefit from a presumption of authenticity, and their texts will not be discounted as much as the average text by readers (perhaps in the short-term, these writers will even benefit from an attention premium!). But unknown writers will face a massive suspicion of copy-pasting AI. Soon every aspiring author will think: “why spend weeks to do marginally better (perhaps worse) than an LLM, only to be suspected of having used one anyway, and to fail to reach an audience anyway?”. When legacy authors have retired, few will write.
In places where that inertia is absent by design, the unravelling is already visible. Geoffrey Litt from MIT reports that he takes no more pleasure in reviewing anonymous academic submissions in the peer-review context: he always wonders whether he is reading a colleague's thinking or Claude's, and fears he is spending more time reading than the author spent writing. He predicts anonymous peer review in its current form will not survive the AI. What has already happened to anonymous submissions will happen to every text by an author with no pre-existing reputation, meaning, eventually, to almost every text6.
In Akerlof’s model there is a simple way to stop the unravelling: make good cars and bad cars distinguishable from each other!
Why does anyone still mine diamonds?
For a long time, we had only natural diamonds. One day, synthetic diamonds appeared, for a fraction of the cost and the same results. So why do some people still spend millions to extract diamonds? Well because people have a higher willingness to pay for natural diamonds (whether that is rational or not is not the question) and because the two goods are still relatively distinguishable at the point of sale (paper trails, certificate of authenticity, laser inscriptions, imperfections in natural stones, etc.). So, people who prefer natural diamonds can still optimize for this preference and buy them at a premium, creating an incentive for producers to extract diamonds.
Now imagine if it was impossible to tell which was which. There would be a unique selling price for natural and synthetic stones, and the costliest production method would cease to exist. When there are two ways of producing the same output and buyers cannot distinguish between the methods, there is no reason for the costlier method to survive! The same will be true for human writing if those who prefer it are given no way to act on their preference7.
This is why watermarking would be a social good8: it simply lets humans optimize for their preference, allocate their attention in the way they wish to, and thereby maintain the incentive to produce the content they actually want to consume. Some of those who fight against watermarking resemble producers of synthetic diamonds fighting against natural-diamond producers' right to certify their stones are natural.
Then why do AI accounts currently thrive on social networks?
Hang in the Louvre a few magnificent AI-generated paintings without telling anyone they are AI-generated paintings. Visitors will crowd in front of them: the paintings are objectively beautiful, and everyone assumes a human has painted them. But build a museum openly dedicated to AI painting and say so on the door: it (probably) won’t attract many visitors, who know they can prompt these paintings at home. Social media today is the Louvre in the first case. Soon, it will be the second museum.
To see why the current moment is not an equilibrium, let’s go back to our diamond analogy.
Say the market contains nine sellers of natural diamonds (which buyers value at $10,000 each) and one newcomer selling a synthetic stone (which they value at $5,000). If diamonds cannot be told apart, there is a single price: the expected value of a random stone, $9,500. At that point, two things are worth noting. First, at $9,500, extraction is still profitable (say it costs $8,000 per stone), so natural producers keep extracting. Second, the synthetic seller earns $9,500 for a stone his buyers value at $5,000, making a $4,500 rent, due to the presumption that his diamond has a 9 out of 10 chances of being natural. But this situation is short-lived. Since the rent is massive, synthetic sellers multiply, the ratio of natural stones to synthetic stones falls, and the single price slides down, say to $8,500, then $7,500. Once the expected value of a random stone falls below the cost of extraction ($8,000), people stop mining and the natural producers exit. Prices keep falling and producers of synthetic stones lose their rent (because the thing that supported it – the presumption of likely being a natural diamond - disappears). For text on internet, we are at the $9,500 stage. Readers' attention has not yet fallen enough to drive out human writers9, so human text still coexists with AI text, and AI text benefits from a rent, due to the presumption of having a good chance of being human. But for the same reasons as in our diamond analogy, this situation will be short-lived…
Note, once more, that neither the market for diamonds nor the market for used cars unravels if both types of goods can be distinguished. With AI watermarks, readers who care only about the quality of output can continue reading AI content (just as some people are happy to buy synthetic diamonds); readers who value human authorship can disproportionately allocate their attention to human writing (just as some allocate a disproportionate share of their purchasing power to natural stones). This allows human authors to capture the attention premium required to justify the higher cost of their work (just as natural stone certifications allow diamond producers to capture the premium that makes mining economically viable). In the absence of watermarks, the two goods are grouped into a single pool that sustains only the cheaper production process and deprives those who want the costlier one from getting it.
Free-riding on the presumption of humanity
Another way to see all this is through the tragedy of the commons. The economist Garrett Hardin describes a situation in which shepherds share a pasture. Each shepherd has an interest in adding one more animal to the pasture, since he alone captures the benefit while the cost of overgrazing is spread across all shepherds. So they add animals, the pasture degrades, and eventually, the shepherds lose the resource they depended on and are collectively ruined.
AI writing puts us in an analogous situation. The shared resource is the trust (built through centuries of human writing) that we spontaneously grant to a text: by default, we still presume that it is likely that its words were chosen by a human. As long as that presumption exists, we still read. It is therefore individually rational to exploit that presumption by copy-pasting AI content: the texts get reads because people assume it has likely been written by a human. But those who do so erode the presumption they live off. Eventually, every block of text becomes assumed to be likely AI-generated, and even the AI copy-pasters lose their readers, just as even the shepherds who initially benefited from destroying the common pasture came to lose.
So, pointing to the fact that AI texts perform well today on social media is just like pointing to the fact that shepherds who added animals to the common pasture did well for some time. At the risk of repeating myself, the current situation is not an equilibrium. Those who fight watermarking are fighting for the right to keep free riding on a common (and degrading it) maintained by the people who exert effort to write.
What if AI text is not a lemon?
If my first assumption (the higher the probability a reader assigns to a block of text being AI-generated, the less likely he is to start reading it) was false, if people care only about the result not the production mechanism10, then two things would be worth noting. First, although one of my conclusion falls (humanity doesn’t stop reading), writing does cease (and watermarking wouldn’t change anything)11.
Second, if readers care only about the quality of the output, AI watermarks would not be detrimental to those who post AI-generated content. Of course, the vehemence with which they oppose watermarks suggests that they do not believe their own argument. They know that what they sell depends on the readers not knowing. They are fighting to prevent people from acting on their preferences. Again, one may find these preferences irrational, but they exist, nonetheless. I support free markets because they let people allocate their resources (here, their attention) to what they actually want, and because that allocation creates the incentives to produce what people want. Those who oppose watermarking do not want resources to be allocated optimally but to be allocated in their favor thanks to an information failure that stops people from acting on their preferences.
Am I not, as a writer, just a Luddite protesting the automation of my job?
First, I am not protesting the technology but the opacity. If, when goods are distinguishable, it turns out readers do not care about human-written text, then so be it, human authors will have lost fairly, and it would be more socially optimal for them to allocate their time doing something else than writing. I only want readers to have the choice.
More importantly, as argued, I believe that at equilibrium, there are no winners. I am sometimes compared to the taxi drivers who protested Uber. Uber did redistribute rents away from license holders, but it also generated massive surplus. Here, at equilibrium, readers lose (they stop reading, the probability of any text being machine-generated having become too high) and writers lose (suspected of being machines, deprived of an audience, unable to outcompete genius models who are capable of producing thousands of brilliant texts every second). Even the transitional winners (the accounts currently churning out viral essays from prompts) do not win, as the resource they depend on and are depleting (the assumption that words are most often chosen by a human) runs out.
For empirical validations, see for example this. This need not apply to every kind of text. Purely informational content will probably escape the rule (nobody cares if a weather report is written by a robot), as will text serving a coordination function in professional settings and, as will, of course, one's own conversations with chatbots.
In fact, I even believe a stronger version of the second assumption (which does not need to be true for my conclusions to hold, but that makes them hold more): AI-generated texts will soon be better than virtually all human texts. LLMs are already better at generating viral content that plays on readers’ emotions and partisan biases, they will probably soon produce content that is objectively better than that of the most intelligent humans. Furthermore, the LLM writing ticks (em-dashes; it’s not X it’s Y, etc.) that make them recognizable will probably be trained away. Today, they can probably be removed through careful prompting; we perhaps underestimate the amount of AI-generated content that we read. As Benjamin Todd from 80,000 hours put it on X: “Plastic surgery is most noticeable when it's bad, so it's more widespread and successful than it looks. Same with AI writing.”
Parshakov, Petr, et al. "Users Favor LLM-Generated Content--Until They Know It's AI."
In this model, “lemon” does not mean low-quality writing. It means writing possessing less of an attribute that readers value (human authorship). The analogy does not rely on AI writing being objectively bad (again, I do not think it is), just on readers having a preference against it. It is my first assumption.
The same logic operates within AI content. Some content producers probably spend hours prompting and iterating, while others copy-paste a lazy prompt. The careful prompter therefore has a break-even point too: as attention falls, he crosses it before the copy-paster (whose cost is zero). Human writers therefore exit first from the content producer pool, effortful prompters next, and at the equilibrium, there remains only the cheapest producible text. Of course, this assumes that the outputs of the two types of prompters are indistinguishable (otherwise the superior quality would help readers allocate attention to the careful prompter) which they increasingly are as models progress and can one-shot incredible content. Those who defend AI writing by pointing to the craft of prompting should keep in mind that these content producers will also be crowded out.
One possible exception: we may continue to read people whose standing comes from something other than writing (central bankers, politicians, CEOs, famous actors, etc.). Either because their public record makes them trustworthy (so we presume they chose their own words), or because their position makes their opinion worth knowing even if an LLM chose the actual words (for example, the president's essay is worth reading for what it reveals about how the country will be run; Prince Harry’s memoir is worth reading if it reveals factual information about his royal life. This is why ghostwritten books were read by the way. They were read despite being ghostwritten, but all things being equal, readers would have preferred them to be written by the celebrity). Note that this scenario implies that standing becomes a precondition for being read, and standing cannot be acquired by writing. Only the already established can write. This is important to keep in mind as an answer to the often-heard argument that "AI will democratize writing" (as the printing press ended the clerks' monopoly on the book, AI would end the literati's monopoly on the essay). In fact, here, in the long run, only the famous get read. The anonymous but talented young thinker has no chance.
Do not misunderstand the analogy. I am not claiming that the preference for natural stones is as rational as the preference for human writing (I don’t think it is). Nor am I saying that the end of diamond extraction would be a bad thing (it would probably be a positive evolution). The end of human writing is however different.
I am arguing here for the general principle of watermarking AI text. I understand the practical difficulties if only one LLM provider does so.
A second source of inertia that explains why attention doesn’t fall faster: readers discount texts according to their belief about the share of AI text in pool, not according to the actual share, and beliefs lag reality. The pool can already be massively AI while the average reader still presumes it is nearly all human, because his prior was formed over a lifetime in which a text could only have been written by a person. It is therefore no surprise that boomers (whose priors formed over more years) fall for AI content the most. But the lag is not eternal: expectations do eventually update.
By claiming that when it comes to writing, people do care about the production mechanism, I have been accused of defending the labor theory of value. But this accusation misunderstands why the labor theory is false. It is false because, Marxists notwithstanding, the value of a good is determined not by the effort invested in producing it, but by its value in the eyes of other humans. A good produced in a snap of the fingers can be worth more than one produced by a hundred men working day and night for ten years. From this, many conclude that it is fallacious to say the value of a text depends on the human effort behind it. But they forget that value is a function of how much other humans want something. For a text on social media, value is therefore not determined by objective quality (on this metric, I believe AI will soon do better than any human) but by readers' desire to read it. Which leaves an empirical question: on average, do people want to read a text on social media when they know it was written by an AI? Would they, if they could tell the two production processes apart, allocate a larger share of their reading time to texts whose words a human chose? Perhaps text is a good whose production process enters the value the final consumer attributes to it. Nothing in subjective value theory forbids that. If only the final product ever counted, how would we explain that natural diamonds sell for more than synthetic ones that are similar in every respect? That a perfect replica of the Mona Lisa is worth a minuscule fraction of the original? That a man is more moved by a love letter his wife wrote than by the same letter she had ChatGPT generate? One can of course dispute these preferences, argue that a reader is wrong to prefer a human-written argument over a comparable one from a machine, as one can argue a buyer is wrong to prefer a natural diamond over an identical lab-grown one. But preferences are what they are. The labor theory claims that effort creates value; I claim only that when it comes to writing, readers care about effort (I describe the shape of demand). Note also that if there existed a general rule that only the quality of the output counts, the future would be bleak: the day machines become better than us at everything, no human could bring value to another human.
Should we care? After all, nobody mourns the copyists that the printing press put out of work. If readers obtain better texts that they are eager to read, isn’t it a good thing if writers find another occupation that the market rewards more? Well perhaps, but there still would be social costs worth thinking about. The great mathematician Terrence Tao recently wrote about how it’s uncertain whether using AI to solve all open math problems would be a net positive. These problem’s value to society, he says, lies less in their answers than in what is achieved while humans struggle with them (the techniques discovered on the way, the obstacles mapped, the connections found, the mathematicians formed, etc.). Until recently, producing an answer and producing understanding were perfectly correlated, because the only way to solve a hard problem was to understand it. Tao observes that with AI, they are now negatively correlated: optimizing for the answer can reduce the understanding that working on the problem would have yielded. Perhaps, he argues, some problems should be declared off limits to AI, as one protects an archaeological site from excavators. Some of this perhaps transfers to writing. The value to society of a written text lies partly in the text, and partly in what its author had to do to produce it: the reading and the understanding of a field, the search for the right argument, the reasoning, etc. One learns to think by writing. Perhaps having more good texts but fewer capable minds wouldn’t be a net positive for society. In Terrence Tao’s words: “In most cases in pure mathematics, the problems are posed not because we desperately want the solution to these problems in and of themselves, but because we have seen from past experience that human-directed efforts to solve these problems tend to spur further development of the field through the efforts to solve such problems […]. Prematurely solving the problem by purely AI-powered methods […] can contaminate this process to the point where it actually becomes a net negative for the progress of mathematics as a whole.”