Chatbots like ChatGPT can predict words with amazing precision. The danger lies in believing they’re capable of more.
The first problem with artificial intelligence is we have no agreed-upon definition for what it is or means. Some have reframed it as digital intelligence, or insist there is no AI, it’s just a set of tools. According to the US Federal Trade Commission, AI is just a marketing term. If that’s true, the name will be used by the world’s most sophisticated marketers to mean anything we as consumers believe. And they will likely employ AI to sell AI because few people understand AI.
As a professional marketer for big tech, the argument goes like this: convince me I have a problem and you have the solution, then access my emotions to make me want it. If it’s persuasion marketing we’re talking about—how we’re made to think and feel about a brand or product—then the biggest source of persuasion for AI is now coming from the chatbots themselves, how hundreds of millions of people are regularly using large language models (LLMs) like ChatGPT.
The biggest threat the technology poses is wrapped inside its key value proposition: automated, plausible results. The more plausible we perceive the machine’s outputs, the more likely we are to believe and trust the machine. The more we trust, the more we can be persuaded.
Through this sequence we risk conflating the machine’s ability to accurately predict with the belief it can produce accurate content. Worse, we confuse the ability to predict with the belief that these machines can reason. And we spiral into the realm of science fiction and emergent behavior, of digital sentience. The doomsday risk is less about the machines rising up and more about our misuse of the machines—the intentional misuse from bad actors, and the unintentional misuse from everyone else.
To extend the theory as analogy, imagine you are texting a friend on your smartphone and as you type, the phone suggests what words should come next. Maybe 90% of the time it’s accurate. But suppose you accepted the autocomplete every time—the output would likely stray from your original intent. It would get some of the words right but some of them wrong. Then imagine you just accepted all the words it suggested, and started to believe the phone actually knew what you wanted to say, and trusted it to communicate as well as you. It may sound far-fetched, but this is the kind of risk we are taking the more we allow LLMs to speak on our behalf, through written word.
Paradoxically, the better the models get at predicting text the more we’re inclined to accept it as truth. And the risk is we allow the models to unwittingly persuade us. In this way we haven’t created any new intelligence, we’ve only undermined our own.
Of the myriad biases humans suffer, automation bias inclines us to favor machine outputs over our own intuition. This fallacy has almost led me to follow my GPS into canals while driving in Amsterdam or through farmlands in southern Germany. We expect the thing to do the thinking for us and stop thinking as a result. The more convincing the thing, the more inclined we are to follow it. And this is the problem with the LLMs like ChatGPT: they’re quite convincing, but they’re not quite right.
These machines may predict a correct pattern of responses to a prompt but they do not produce correct content. It is the probability of what one might say in response to a prompt, accuracy not included. And there is a subtle but important difference that’s easy to overlook, the difference between plausibility and verifiable truth: what sounds like it could be true versus what can be proven as true.
My falling out with ChatGPT came when I asked it to help me write an essay and provide sources to help me back up an argument. The sources looked legitimate at first but soon proved to be false. When I called BS the bot apologized and suggested I go look myself. (“I’m just a language model, I can’t be expected to know these things.”)
But these shortcomings haven’t had any noticeable effect on the adoption of LLMs despite the lack of QA, and propensity to BS. Perhaps that is a compromise we’re willing to make, and helps us feel good as humans that we’ll still be needed to fact check. But can we humans be trusted to not fall for BS?
The ability to be persuaded is built on a premise of expertise: that I can farm out my decision making to someone (or something) who knows better than me what I need. In order to buy, we need some level of trust in the source of that persuasion before we’ll grant access to the emotions that activate our decision making.
The LLMs aren’t trying to sell us anything. The persuasion we’re susceptible to is rooted in our own cognitive shortcomings. The better the models get at predicting what we want to say the more prone we are to believing they’re right. The same logic holds true for intentional misuse of the models by bad actors: the better the models get at predicting what will persuade us, the more we’ll be persuaded by them.
Most of society will likely never agree on what constitutes real intelligence, human, machine, or otherwise. I hope we can at least keep our grasp on what is artificial and what is real. Because without that distinction, I don’t know what we’ll have. It will likely feel like many things, or nothing at all.
Words Matter 