Twenty-five Years of Talking to Machines
How a convenient habit is confusing the AI debates
I have a confession to make. I think that for my entire career, I have participated in perpetuating a mass confusion. This stems from a habit I picked up that seemed innocuous and purely for convenience at the time. Sadly, in today’s AI discourse, this developer’s tic has become exponentially more harmful very rapidly. What I’m talking about is anthropomorphization, or when we assign human-like characteristics to non-human things.
As a software developer who’s been writing code for over 25 years now, this is not even a controversial practice. I don’t think that it’s unique to software developers, but I’ve never had another job, so I can only speak to it from inside of this one. When we are working on our projects, we tend to speak to them. We tend to express emotions toward our body of work. We may look at a screen with some pixels that represent letters and numbers and symbols that we have entered in with an expectation that some actions will be performed by a computer system. Then, when the results are not what we expect, we talk, we plead, we beg, we curse. We try to reason. We try to negotiate with an inanimate object. Sometimes things get broken, physically. (See Office Space, circa 1999)
When we are doing that, we are not under any illusion that the computer, or the program, or the code, is in any way alive. It’s just a thing that you do to cope with the emotions of a stressful moment or to express your own thoughts. I didn’t suspect that any of my colleagues were actually believing the programs they were working with were “behaving” in any way, even if they used that word.
We have an antidote, a saying that I literally used just today, to my coworker who sits across from me at our shared workstation. He was frustrated and talking to his AI agent that was helping him, and he said, “So many bugs!” I instinctually recognized the moment and employed a phrase that we often use. I don’t even remember who exactly said it to me first, probably one of my mentor colleagues who was in the field a lot longer than I was. We say to one another, “They’re yours, right? You put them in there.” He replied with the genuine laugh of relief when that phrase comes and the obligatory “Yes, I did” response. In most mild cases, the phrase collapses all blame layers in an instant, and neutralizes your frustration by reminding you of your ultimate accountability. It also helpfully communicates to your colleagues that you are not in fact insane. It’s not like we needed the corrective in order to stay sane. No one was on the edge of falling into a rabbit hole of endlessly questioning whether or not the systems that we work with are sentient or conscious.
Then, something strange started to happen. Somewhere along the way, other people heard the way that we speak to our work. It may have been because we often like to talk at conferences or big gatherings. Sometimes we’re not only talking to other software developers who have a shared understanding of the non-living nature of our work, but we might talk to designers or salespeople or other kinds of professionals. We speak about our profession to them using our own vocabulary that comes out by force of habit, and under no fear that they would conflate or inflate our words to mean that the machines are alive. I’m sure that at many leakage points throughout history, the words have been used by software developers to describe computers, computer programs, or other mechanized systems with anthropomorphic terms.
A few weeks ago, I released the HAVE Tool, the Honest AI Vocabulary Evaluator1, in which I start to highlight some of these words. This is a surreal experience at this point in my life, never having thought that I would even have the idea to create an application where you can pull up the etymologies and historic understandings of these words that seem so basic to us now. Not only the confusion that these words are causing, but the stealth nature of that confusion has brought us here. It would be one thing if people misuse these words to no effect, but when these words are misused to justify very great effects on human society through our governance structures and policies, that is a serious problem that has escalated far above debugging late-night coding sessions.
Now these words are being used out in the wild by policy makers, politicians, philosophers, wannabe philosophers, mystics, and gurus alike. The ease with which they transition from an anthropomorphic-word-loaded definition of artificial intelligence systems to the broader philosophical conclusions that they draw is extremely concerning. In their world, these words are not spoken in jest or with a boundary between metaphorical and literal usage. The fix for it is not as easy as it is in the bullpen of developers when we chide each other for the habit and we have a laugh, part of which is at the ridiculousness of the concept that the machine is somehow aware and actively frustrating our plans, rather than our own deficiencies.
When I describe the people who are propagating the use of these terms this way, I wish I was only talking about people who don’t know any better. I would not be able to get away with that, for the simple reason that some of the worst offenders are not only from inside our industry but from the very top. Exhibit A would be Mr. Geoffrey Hinton, the godfather of AI himself, who as of late has completely conceded that he believes AI systems of today are conscious to a degree. That is an extreme position even amongst the people who think that AI could possibly be conscious at some point.
This is where I need to introduce another term, functionalism. It is the philosophical concept that mind or consciousness or personhood are merely in the functions that describe them, rather than somewhere within the substance in which they occur. That is to say there’s nothing deeper, nothing mysterious about mental states, they are just what the system does. Even if people testify that they have inner experience, the functionalist claims that that testimony is itself just a function. It’s just the way your brain processes information and gives you an interface to that process. If a system could approximate the same functions to a sufficient degree, then that would be enough to instantiate actual personhood. Of course, you can see where this is going. It would follow that AI systems today seem to exhibit characteristics of human functional consciousness.
It’s at this very intersection where the gap needs to pass unnoticed, and it finally has the bridge it always needed, fluent human language. The functionalist argument just got a major wardrobe upgrade. The combination of algorithmic language recombination targeting the imitation of conversation, and the functionalist developers themselves using anthropomorphic terms to describe the process, makes it seem like a genuine, even spooky, phenomenon.
The question we should ask is this. If functionalists are so reliant on the functions themselves being sufficient to produce conscious beings, why can’t they do it without anthropomorphic terms? If you remove the anthropomorphization, the functions don’t seem as similar as the terms dress them up to appear.
However, the path from function to person does not just feature a gap that needs a smooth crossing. There’s also a mountain. The functionalist move isn’t to pretend the mountain doesn’t exist, they just call the mountain a molehill. They park at the basecamp and proclaim they have reached the summit. What I’m referring to now is the reduction of inner experience that I mentioned earlier. Functionalists say there is an experience, but it’s not a subjective mysterious “something it’s like.” It’s just a necessary transaction. The feeling is just the internal receipt.
This calculus would keep personhood wrapped up into a neat empirically verifiable package. Nothing has to be left to that pesky, hard to nail down qualia. The technologists can continue to keep the philosophy wing at arm’s length.
Most of us are bothered by that explanation. If we’re bothered, it’s by remembering our own life at every moment and believing that the inner experience we have, our inner voice, our inner mind, the self we can keep to ourselves without letting anybody else know about, isn’t imaginary or interchangeable. It isn’t just a function. It’s who we are.
Here’s the main problem with dismissing the qualitative, intrinsic aspects of internal experience as subjective and unprovable: you have to dismiss all evidence. Empiricism seems like common sense. Nothing can be known without evidence, right? But what makes evidence count? The only way we comprehend evidence is with our subjective internal experience. There is no other way. We have instruments, we have machines, but the machines do not know what the numbers mean. We are the ones who give significance to them. We hold the mental concepts in our minds and have discovered universal laws, the way particles work, the way forces work. We know all of that through our subjective experience of that information.
That experience is not divergent for every person. It seems that we have a shared experience, and with scientific methods properly executed, the information is interpreted identically between colleagues on a given project. There is a feeling of objectiveness. But at the end of the day, even everything that is called objective comes through the same pipe that the subjective comes to you. There’s not two pipes with separate processors. There’s one set of senses, and one mind behind them. You watch the same Pixar movies with your kids through the same eyes and the same ears and the same brain that is processing all of that signal subjectively.
There’s another problem for functionalists who think today’s AI systems are conscious. How do they know the functions they are observing sufficiently emulate the functions that make up consciousness? These claims are being made today based on systems that mainly consume text. Large language models are trained on trillions of pieces of human output, the product of human consciousness and thought and reason.
When you receive a text message from someone, you immediately understand that the medium could not possibly carry much more information than you were able to glean from that message. Sometimes it didn’t even sufficiently convey the message you did receive, even if the maximum characters were used. Text is the lowest-bandwidth type of communication that we still regularly use. As you use more data-heavy communication methods, we say there is an increase in bandwidth. We need a bigger road so that more cars can fit.
When you call someone and speak to them, they can hear your voice. There is a lot more information than text, not just because you can speak for longer, but because the receiver can hear the intonation, the cadence, the volume, background sounds, inflections. When you move to a video call, you can see the person’s expression, their body language, where they are. Even a fake background is information—that they didn’t want you to see their actual room.
Face-to-face is much higher-bandwidth still. You have both undergone an experience to physically locate your body in the same place. You both felt the weather outside. You can smell what the other person smells like if you’re standing close enough. You have the shared promise between you, as vulnerable creatures, that you agree for the length of the conversation not to attack one another physically. Of course sometimes conversations turn into that, but for the vast majority, it’s possible that one person could attack the other, and we don’t do it. That’s also part of the information. Certainly text alone is not what most people think is a sufficient output to compare to the entirety of what a human being is.
The human mind is able to consume and interpret data of a myriad of kinds. Some of it is simplistic and seems obvious to us because at a very young age we learn to interpret information from our sensory inputs. But you shouldn’t let that make you jaded about the miracle that we are as data-interpretation vessels. I won’t say machines.
Imagine you see one of Leonardo da Vinci’s paintings in person. Why do people even want to go and see those paintings in person when we could just look at a reproduction online or a printed copy in a magazine, a much handier format, much easier to access? It is because there is information that emanates from the actual physical presence of an artifact, an original da Vinci painting, that you cannot access through digital reproduction.
When we as software developers write code, there is not only the bare information you see in the letters, numbers, and symbols. If you know how to write code, you know there are many ways to accomplish the same functions. It is a very common exercise to walk back through someone else’s code and try to derive their thinking from it. It’s a skill you are expected to practice and use as a professional. So where is all of this extra information? It’s not necessarily encoded, but there is inference. There is information that is implied. Our minds are so good at taking in all of that information, all we can access by our five senses, and interpreting it into meaning very quickly, and sometimes deliberately slowly. It is because we are connecting with a mind on the other side that gave it meaning.
AI certainly has unleashed an onslaught of seemingly lifelike outputs. But are we connecting with the mind of AI, or the mind behind its data? A major mistake people make in evaluating the possibility of consciousness in AI is to say that the outputs they’re interpreting were the product of the machine, and not directly written as routines in code the way a normal program works. They say the machine has learned how to do things and is making its own choices. That’s a fundamental misunderstanding. The machines only have the outputs of human consciousness. Even if there are no explicit instructions given to the immediate system, the training data contains the record of every functional instruction given to any system the collectors had access to. Every documentation of experiments, every failure, every discovery humans made in the process of trying something and finding out what works and what doesn’t and revising and iterating. Recombine all of that and you get an output that is an average of many things, but the mind behind all of that information is obfuscated. It’s hard to trace a single owner. And yet we know, because the people who created that artificial mind admit to putting that human intention in there as the founding data set, the foundational building block.
As beautiful as a Picasso is, it can feel alive. But it’s not Pablo. He masterfully transcribed a piece of his personhood into the canvas. You can almost feel his presence. If you’re in the right frame of mind, and you know Eric Clapton’s heartache behind the song, or you carry your own, the questions he asks in Tears in Heaven might just wet your cheeks. But the song is not the man. Sometimes the mind leaves an indelible impression on its output. It’s what makes art an experience. AI is a masterpiece of human language manipulation, but it’s a far cry from human.
I’m happy for this conversation to be happening. Because if we follow this chain of thought to its logical conclusion, you have to ask yourself where else we can see design or intention. If we’re asking whether AI is conscious because we think we see some intention in it, it’s a mistake to ignore the obvious source of that intention. But sometimes we see the marks of mind and we refuse to ask whose they are, because we don’t like what the answer might be.
HAVE, Honest AI Vocabulary Evaluator: https://vocab.logosanalog.com