Why AI writing can't be great

Many of us have developed a sixth sense when it comes to written content, able to detect beyond a shadow of a doubt whether AI wrote something or a human did. That's starting to fade a bit for me, though, and I thought I'd explore why. This is important, not only from a general "future of work" perspective, but also for the business of marketing -- and that's where things are most troubling.

Lazy, but capable, writing

When ChatGPT first entered the scene, it led to an explosion of writing that was a level up for most people who don't write for a living. At the same time, it reduced the value of the entire collection of "good enough" AI slop that was to follow. Do a Google search for "In today's rapidly" in quotes, and you'll see the most obvious examples. There was a time when you might read anything after that phrase, like "In today's rapidly concrete…" or "in today's rapidly feather pillow," but search engines are better at showing you results you care about.

That said, search engines have simply hidden the problem. If you really want to see how bad it's become, pick a company -- any company -- and search for this: [site:<company url> "in today's rapidly"]. It's astonishing.

There are other clear indicators for AI writing:

• The antithesis: "This technology isn't just blah blah, it's yada yada."
• The pivot: "This shift represents a level up."
• The overdoing it: "The need for continuous reinvention is crucial."
• The used car salesman: "Here's the honest truth" or "I'll answer honestly because the topic genuinely deserves the real answer."

And so on.

Seeing a phrase like this is an instant turnoff for me. I usually stop reading, but if I do continue, I unconsciously switch to skimming around to look for the point buried under all those pillowy words. I see this even more outside of tech circles nowadays. Real estate, car ads, YouTube edutainment videos and more.

People in charge of content need to know this and stop posting things with these phrases. The solution, though, isn't heavy editing of largely AI-generated content. It's understanding why AI writes the way it does and how to best work with AI to generate content that is better than human or AI writing alone.

Why it's not getting better

If you think "Well, the models are getting better, so I'll be able to train it to sound like me," you're largely -- though understandably -- wrong. After all, we're several years into AI, and we still have many of the same problems we originally had. Most of the reasons content feels like AI stem from the following problems. While there are many other layers, I think these specific factors contribute the most to why today's models, which are light-years ahead of the initial AI models we started using, still can't quite get it right.

Problem 1: Models have inherent tendencies you can't prompt around

Plenty of people understand the need to avoid these trigger phrases, but they fail to understand that the problem is inherent to AI, not something you can just prompt around.

I'll remind you of the fable "The Scorpion and the Frog," where a scorpion asks a frog to carry it across a river, promising not to sting. Halfway across the river, the scorpion stings the frog anyway, dooming them both, because it's simply in its nature.

You can reach an agreement with the model, but, like the scorpion, it has inherent tendencies it can't avoid. Even if you say, "never use em dashes," it'll just write the piece as if it used em dashes, then replace the em dashes with commas or semicolons or something. So, a line that looks like this:

In today's rapidly evolving cheesecake landscape -- one dominated by certain factories near the mall -- it's crucial for people to consider the calories.

And turn it into this:

In today's rapidly evolving cheesecake landscape; one dominated by certain factories near the mall; it's crucial for people to consider the calories.

Problem (not) solved. You can try to prompt around that by saying things like "don't just replace em dashes; write sentences that don't need them," but then you'll get longer, frustrating output, such as:

In today's rapidly evolving cheesecake landscape, where the dominating factors involve factories located near the mall, it's crucial for people to consider the calories.

It's a solution, but maybe not the best one. It is, however, a great lead-in to the next problem: AI tries to write well, not communicate well.

Problem 2: AI flattens narrative

AI sees its job as generating probabilistically sound output that mimics the styles it was trained on. This results in a flattened, consistent tone across text, whether that's a prompt response or a full white paper. It also affects other output, such as images and videos. Have you ever noticed that AI-generated images have that evenly lit, programmatic variance quality to them that's hard to describe?

To understand why, it's worth taking a quick look at how AI processes prompts. Admittedly, I'm trying to boil an extremely complex concept into a broadly accessible explanation, so while there are likely some opportunities for smarter people than I am to say "Actually…," this should be directionally accurate.

Behind the curtain

AI input and output are broken down into tokens, which are roughly word-sized chunks of text. To the LLM, each token is a long list of numbers -- a vector -- that captures what the model "knows" about that word. This includes whether a word is a noun or a verb, concrete or abstract, positive or negative, and hundreds of other dimensions that only the model itself even knows (and are indescribable to humans). The dimensions used to process tokens are established by the LLM itself when the model is trained, and the actual probabilities are determined by weights.

Take the word bank, which, depending on context, could be where you keep money, the edge of a river, a verb meaning to save or the way a plane turns. When the model encounters bank, it doesn't just see the word, it reads the surrounding tokens and uses all those dimensions to figure out which bank this is. The words around it nudge the numbers in one direction or another until the meaning resolves. It does this for every single token.

The same process works in reverse to generate a response, write code, build a presentation or write the copy for a white paper. It's literally a probability chain from the moment you push Enter until the response is finished generating.

This is critical to understanding all the quirks of AI writing. It's not trying to be correct. It's not your writing; at best, it's an approximation of your writing. It's trying to write the most plausible sentence based on all the other tokens it has in context, influenced by token dimensions and weights determined when the model was trained. That's why saying, "write a story about a bad bank" might output a story about any kind of bad bank story (a plane turn gone wrong, a poor customer experience at a bank or a river bank that washed away), while "write a story about a bad bank robbery" will 100% write a story about bank robbery. But even then, it will be up to the model to decide whether it writes a story about a bank robbery that was bad for the robbers or one that was bad for the bank.

Why does this matter?

This matters because when AI writes, every token choice is a vote for the most statistically expected next word. And the most expected word is, by definition, the most ordinary one. Multiply that across thousands of tokens, and you get prose that's grammatically flawless, structurally sound and completely middle-of-the-road. It doesn't make bold choices because bold choices are statistically unlikely. It reaches for the same transitions, the same sentence rhythms and the same safe adjectives that were baked into its training data ages ago.

AI wants to write well, not communicate well. It's motel art. And when you ask it to be punchier, it does that in predictable ways as well, and for all the same reasons.

Problem 3: You can't train a frontier model to write like you

Telling AI to write like you and giving it examples just means it assesses how you write, what your sentence length is, how you start and end pieces, how many paragraphs you typically use, what person you write in, etc. The result is that it can kind of sound like you, but it does so mechanically and without regard for voice or cadence or any of the intangible things a writer might do to convey a point.

If I directed a model to all my articles, the assessment's output would be a combination of statistical (sentence length, paragraph count, how pieces open and close) and stylistic (first person for blogs, third person for customer content, leads with a personal anecdote, addresses the reader directly).

Following that assessment, every "write as Gabe" directive applies those learnings to the same underlying training discussed above. It approximates my voice by hitting the measurable targets (sentence length, paragraph count, person) from its learnings while still defaulting to its own instincts for everything it couldn't quantify. The result is like a wax museum. You say, "Wow, that's so realistic" aloud, while your inner voice adds, "for wax."

Avoiding the uncanny valley

I'm not trying to demonize using AI to write. AI helped me write this in a dozen ways, from vetting the token and vector description to writing a succinct, two-sentence version of "The Scorpion and the Frog." In both of those situations, the text was mine. I asked AI for help, then further refined it to make it my own, as it suggested one very long sentence. Zero parts of this article were cut and pasted from AI directly.

What's bad is relying entirely on AI writing, and assuming that just because it sounds like you at first pass, it will pass for you with a reader. I think the wax museum analogy holds true: it might be a heck of a good resemblance, but it's just not right. It's the uncanny valley that other technologies have run up against, from films (think "Polar Express") to video games to human-shaped robots.

We all need to learn how to use AI for what it does best and merge that with what we do best. The end result can be better than AI alone or even human alone. Using AI to do the vast majority of content creation, though, is off-putting to readers and, for reasons that are hopefully clear, inherently average.

Gabe Knuth is the principal analyst covering end-user computing for Omdia.

Omdia is a division of Informa TechTarget. Its analysts have business relationships with technology vendors.