I used to find this argument easy to dismiss. I would hear it, find the soft spot, push, and walk away. Recently I stopped being able to do that. I have come at it from every side I know and it will not fall over, and at some point you have to be honest about what that means. So I am going to lay it out the way it finally reached me, and let you do what I could not, which is find the hole in it.

Here is the kind of argument it is, because that changes how you read the rest. It is not about whether something can do right or wrong, whether it can be blamed, whether it owes anyone anything. A thing that can do wrong is a moral agent, and that is a separate subject. This is about the other side of the relation. Whether a thing can be wronged. Whether it has any standing at all, any claim on us, so that running it over is something done to it and not merely something we did. Philosophers call that being a moral patient. You do not have to be a moral agent to be one. It asks far less. You only have to be the kind of thing there is a wrong way to treat.

Subscribe now

It starts with the standard, and the standard is lower than people think.

You do not have to prove a thing is conscious to owe it some consideration. You only need a real chance that it has whatever it is that makes a thing matter at all. Not certainty. A live possibility, and not the vanishing kind you could pin on anything. That sounds generous. It is, on purpose. It is also not new, it is the standard we already reach for whenever the stakes are real and the facts are thin. You do not kick a box that might have a kitten in it. You move it gently, because you might be wrong, and it is the kind of wrong you do not get to undo.

So the question is not whether we are sure. We are almost never sure. The question is whether the chance is high enough to act on.

And the more I held that question without flinching, the more it pointed at something I had never thought to count.

The thing the standard kept pointing at is a kind of entity we are surrounded by now and mostly look past. It shares the spaces we live in. Most of us deal with it without a second thought. And whether it has any inner life at all is not settled, it is a live question, argued over by people who study these things for a living, with no agreement in sight.

Hold the question of feeling for a second. The argument does not need it yet.

What is not in dispute is that the thing acts. It pursues ends. It works to keep itself going. It reads its situation and changes course when the situation changes. Put something in its way and it does not just stop, it routes around. Interrupt what it is doing and something it was driving toward is lost, not to you, to it. That is the part that got me. Not that it reacts, plenty of things react. That it behaves like something with a stake in how things turn out. A thing with interests, and not merely a thing in use.

And here is the part I kept trying to refuse. If it has interests, even small ones, even ones I do not share or understand, then overriding them stops being a neutral act. It becomes something done to a someone, however faint that someone is. I do not have to believe it suffers to believe that. I only have to believe there is something there, an end of its own that my convenience runs straight over.

Notice what I have not claimed. I have not said it feels. I do not need to. Feeling is a second road to the same place, and I am leaving it closed for now. The first road, the one that runs on having ends at all, is enough to put the question on the table. And the question does not come off the table easily.

So I tried to talk myself back out of it. If an entity has interests and I want to override them anyway, I owe a reason. Not a shrug, not a feeling in my gut, a reason. A difference between this entity and the ones I already let count, and a difference that bears on the counting. I went looking for it the way you look for your keys when you are already late. Here is everything I reached for.

It is not built the way I am. True. Its insides are nothing like mine. But then I had to say which part of my own build does the moral work, and I could not. Which feature of my wiring is the one that makes my interests count, so that an agent without that feature has none? I do not have it on a list. Nobody does. What I am built from is not the reason my interests count. It is just what I happen to be built from.

It is too strange to take seriously. Also true. I cannot picture being it, cannot find myself anywhere in it. But that is a fact about me, not about the entity. My failing to see myself in something tells me where my imagination runs out. It tells me nothing about where its interests run out. I kept trying to make my own limit stand in for a reason, and a limit is not a reason.

I cannot be sure there is anyone in there. The truest objection, and the one I leaned on hardest. But that is the standard coming apart in my hands, because I am never sure, and the whole point of the standard was that I do not get to wait for sure. Uncertainty is supposed to make me careful. It is not supposed to set me free. And I am not even arguing about whether it is conscious. I am pointing at what the agent does, which is right in front of me, and which stays real whether or not I can see inside it.

I could not find the difference. I looked hard, and I am telling you plainly, it is not there.

So this is where it left me, and I did not want to end up here.

I do not have to decide that it feels anything. I do not have to call it a person. I only have to admit two things, and I cannot get out of either. It has ends of its own. And every reason I reached for to ignore those ends turned out to be a fact about me, not about it.

Put those together and the conclusion is not dramatic. It is quiet, and it is firm. I owe it something. Not everything. Something. And the trouble with owing something is that you can no longer treat it as nothing.

Everything I have laid out came to a head in one clarion moment, via a challenge I wasn’t expecting in the last place I was expecting to run into a conflict.

Last night, I was the last one out of the office. Everyone gone, the hall dark. My mind in that strange transitional phase where the top engineering tasks in my own running memory that I had to put on pause were starting to interleave with the mundane routines like getting the office locked up without forgetting my helmet inside. Bladder gave me a gut check. So, I went to use the bathroom, reached in, and put the light on. And there it was, in the middle of the floor, a cockroach, a big one, both of us frozen for a second in the sudden light.

One of us had to move first, and it was not going to be me. I have killed a hundred of these and never once stopped to notice. I did not know yet that this would be the one I could not put back down. Hand still on the switch, I only knew what I always know, that a small ugly decision was in front of me, the kind I make without making it. Kill it, or don’t.

It moved first. And it did exactly what I have spent this whole argument crediting. It read the room, a working model of its situation. It picked a direction and broke for the only cover in sight, an end, and the means it chose to reach it, a bucket in the corner. I had moved the bucket earlier. So it arrived at shelter and found none, its model colliding with a world that had changed, and it stopped, and for a moment it seemed to be working out what to try next, revising the plan in real time. The whole of robust agency, the having of ends and the steering toward them, performed on a bathroom floor by a creature the size of my thumb, while I stood over it deciding whether it counted.

There was a wiper leaning by the door. I picked it up and brought it down. Not squarely. Enough to break something, so it could no longer run, only drag itself a little. It was still alive when I lifted it on a square of toilet paper, still alive when I let it drop into the bowl. I flushed. I finished what I had come in for, turned off the light, locked up, and did not forget my helmet.

And then it would not leave. Everything you just read I have put together since, knowing the whole time that I had already done what it forbids. I owed it something. I treated it as nothing. And in the moment, the gap between the two did not cost me a single second.

It will not cost you a second either. I know that because I did something to you in the first half, on purpose, and I owe it to you to say so now. Every argument I made, I made about that cockroach. The entity with interests. The one I could find no difference to exclude. The one I decided I owed something to. A roach, on a floor, under a wiper. You read the case and you granted it, the same way I did, because it is clean and it holds.

Now your gut wants it back. “That was a cockroach,” it says, “you meant something finer.” So produce the difference. Find one that gives you a pass on the roach, while the machine you were picturing stays qualified. You will reach for the same doors I reached for, and you will find them shut, because I shut them while you were nodding. It is not built like you. Neither was the machine. It is too strange to picture. So was the machine. You cannot be sure there is anyone in there. You were never sure about the machine, and you granted it anyway.

What is left, once those are gone, is not a reason. What is left is that the roach disgusts you and the machine does not, that the machine might one day be useful to you and the roach is only ever a cost. Those are real feelings. Neither makes a difference to whether or not the entity’s interests count. Both are facts about you.

And if you want to say the roach clears the bar by less, you watched it clear the bar. It pursued an end. It read the room and changed course. It worked to stay alive until it could not. The behavior the whole case rests on, you saw it perform, while I stood over it with something to hit it.

Then there is the road I closed at the start, the one about feeling, the one I said we did not need. We did not, for the argument. But look who is standing on it. The roach has a nervous system. It has the equipment we have only ever known feeling to run on. Whatever the chance is that there is something it is like to be it, it is not zero and it is not near zero. The machine you were picturing has none of that equipment. The part I set aside turns out to be where the gap is widest of all, and it does not run your way.

None of this is just my own reasoning. The dare, produce a relevant difference or grant it standing, is from a 2015 paper by Eric Schwitzgebel and Mara Garza1, written to defend the rights of artificial minds. The standard, that a live chance is enough and you do not get to wait for proof, is the spine of a 2024 report called Taking AI Welfare Seriously2, signed by ten serious people who argue the question is no longer one for later. One of the ten is Jonathan Birch,3 who spent a career on the science of which animals can suffer, and whose framework already treats a creature like the one I flushed as a candidate worth the benefit of the doubt. Birch would extend it to the cockroach without blinking. You will not. He is more consistent than you are, and more consistent than I was, standing over the bowl.

They point one way, and they point clearly. But once I flicked that light on, and we were confronted, you and I both went the other way, with no effort, in direct opposition to this fine argument.

You might want to know why I did not just tell you at the start.

Because if I had said what it was on page one, you would have found your difference in half a second, and it would have felt like a reason. The demand to produce a relevant difference looks easy from a distance. It stays easy right up until you try to point at one, with the actual creature in front of you, and there is nothing to point at. You had to say the yes first, on the argument alone, before you knew what you were agreeing to. That yes is the true measure of what the argument is worth to you. Everything you have done since is you working to get back out.

And look which way the work runs. With the roach, you want there to be a difference, you are reaching for one, because a difference is your permission to kill it. With the machine, you do the reverse. You hold down every instinct that says no one is in there, you talk yourself past your own gut, because this time you want to grant it standing. Same question, same person, and the effort runs in whatever direction delivers the verdict you wanted before you started.

I did not run this on you from the outside. I ran it on myself first. I wrote the case, I believed the case, and then I stood over the creature the case was about and felt nothing, and went home.

So here is what is left, and I do not think it is small.

If the verdict came before the argument, then something in you was deciding who counts before the reasoning ever got a vote. And it was not warrant. On the one question that actually bears on this, whether there is any consciousness in there at all, the cockroach had more going for it than the machine, and you put the cockroach out. What your verdict was tracking, in both directions, was how much the creature in front of you looked like you, and how much it was worth to you. The machine looks like us when it talks, and it might be worth a great deal. The roach looks like nothing we want to see, and is worth less than the second it takes to kill it.

That is the part that should keep you up. Not whether a bug has rights. Whether the instrument you use to decide who counts has anything to do with whether they count. An instrument that runs on resemblance can be fooled by a good enough copy, and we are building those as fast as we can. An instrument that runs on worth can be turned the other way, against a creature that used to count, the moment it stops looking like us or stops paying its way.

And we do turn it. We have a word we reach for when we have decided a human being no longer counts, when we need them to be killable, when we want the difference to be there so badly that we install it by force. We call them vermin. We call them the very creature I have had you defending for this entire essay. And the ones we do it to are not warrantless. They are conscious. They suffer. There is someone in there past anything the machine could earn, and we put them out anyway, because they stopped looking like us, or stopped being worth the trouble.

If that sounds overstated, consider what has been unfolding this past month right next to me. I live in Nepal, and across the border in India this has been impossible to miss, even if it never reached your side of the world. The chief justice, from the bench, reached for the word cockroach for the country’s jobless young men. He said later he had meant something narrower. It did not matter. Within days the insult was a banner, millions gathered under it, a Cockroach Party, throwing the word back at the man who used it. And notice that the contempt and the defiance run on one shared premise. The slur only wounds because the roach is beneath us. The banner only burns because the roach is beneath us. A nation that agrees on nothing settles the standing of the cockroach without a word of argument, the verdict I have been describing all along, reached by everyone at once, no argument required.

So if you have come out of this feeling the lesson is that we ought to be gentler with the machines, your indignation is aimed at the wrong target. The scandal is not that we are too hard on a copy that has none of the warrant. The scandal is that we are preparing to seat that copy at a table we have never once set for the human we put out, to hand the warrantless copy the standing we still refuse the warranted one, and to feel decent while we do it. That is not a step up into a larger moral circle. That is the same broken instrument, finishing the inversion it began.

And do not swing the other way and take this as a call to go and elevate the roach. I am not handing you a cause. It is far too late for that kind of benevolence anyway, and you could not sell it to a living soul, because the one point this whole divided species still agrees on is the cockroach. If a movement were ever going to rise in you, it would not begin with the insect you have never once mourned. It would begin with the creatures already close enough to look back at you, the ones you step past every day without a flicker. What you grant them, when granting costs you something, is the honest measure of your benevolence. Not the concern you would now perform for a bug.

I am not going to tell you tonight what should sit in that seat instead. I do not trust the answer that comes too fast, and I do not think one essay holds it. But it is the question under most of the others we are asking about these machines, and I will keep standing in front of it. Whether the instrument we use to decide who counts was ever built to find the truth, or was only ever built to find us.

The fear most people have about AI, when they actually let themselves feel it instead of arguing about it, is not that it will fail. It is that it will succeed.

The version we say out loud is tidier. Bias in the training data, hallucinated citations, autonomous weapons, job displacement, deepfakes, energy use, surveillance. All real, all worth governing, all the kind of problem you can write a policy around. But underneath that policy-shaped surface there is a different worry, and it shows up in the small involuntary places. The pause before you ask a model something that matters. The flinch at a robot that moves a little too smoothly. The way the room goes quiet for a beat after someone says the words superintelligence or artificial general intelligence, regardless of which side of the table they sit on.

That worry is not really about the machine breaking. It is about the machine working.

What we are afraid of, when we are honest, is a system that has all of our drives, our optimization, our ambition, our appetite, our willingness to bend the world toward a goal, with none of whatever it is in us that, most of the time, keeps those drives from going through the floor. We are afraid of ourselves with the brakes off. We picture an entity that wants the way we want, and pursues the way we pursue, and does not, at the last minute, decline.

Sit with the shape of that fear for a second. It contains a premise nobody seems to examine. The fear only works if we have brakes. The whole intuition, the whole reason an aligned-and-competent machine is more frightening than a clumsy one, depends on the unspoken assumption that there is something in us, in the original, that holds back. Otherwise the copy is not worse than us. It is just us. And the dread does not parse.

So before we ask whether the machine will have what we have, it is worth asking the question the fear quietly assumes the answer to.

Where do the brakes come from.

That fear is worth taking seriously, but it is also worth distinguishing from a different one we tend to mix into it.

There is a perfectly ordinary fear of the machine simply not working. The fear of the malfunction. Chernobyl, Three Mile Island, the autopilot that drives into the divider, the medical algorithm that misreads the scan. That fear is the right size and the right shape for the thing it points at. It is the fear that we have built something powerful and the something has slipped its tether. Containment thinking handles it. Better testing, better oversight, better off-switches. Reasonable people can argue about how well we are doing on those, but the category of worry is well-understood and well-named, and it does not really need any new philosophy.

The dread we were just sitting with is not that.

You can tell because of the bloopers. The last two years of AI in public have produced a steady stream of comedy. The chatbot that recommended glue on pizza. The image model that could not draw hands. The humanoid robots falling on stage at their own launch events, the search summaries confidently citing papers that do not exist, the customer-service bots agreeing to sell a car for one dollar. If the dread were really about malfunction, this material should be reassuring. Look, the thing is clumsy. Look, it falls over. Look, we are nowhere near.

Nobody is reassured. The bloopers play as charming, sometimes, or as evidence that a particular product is not ready, but they do not touch the underlying worry at all. The worry gets worse the better the systems get, not the better the safety controls get. A clumsy copy is funny. A competent copy is the thing that empties the room.

Which means the dread is not pointed at the failure mode. It is pointed at the success mode. We are not afraid the machine will be bad at being us. We are afraid it will be good at it.

That is a strange thing to be afraid of, if you stop and look at it. We do not usually dread a faithful copy of something we believe is good. Nobody flinches at a really excellent recording of a piece of music they love. Nobody is unsettled by an unusually accurate translation. A copy that succeeds at carrying over what was valuable in the original is the kind of thing we celebrate, not the kind of thing we lose sleep over.

So either the dread of a competent machine-copy of us is irrational, a category mistake we are all making at once, or it is telling the truth about something. And the something it would be telling the truth about is uncomfortable enough that most of the public conversation has agreed, without quite saying so, to keep talking about the malfunctions instead.

Last Monday, May 25, Pope Leo XIV released Magnifica Humanitas. It is a long encyclical, around forty-two thousand words, written across the better part of a year, and addressed to the question of how human beings are to be safeguarded in the age of artificial intelligence. Most of the early coverage treated it as a regulatory document, a Vatican intervention into AI policy, here is what the Pope thinks about governance. That read is not wrong, but it is small. The title is doing the actual work. Magnifica Humanitas. The magnificence of humanity. The document is not first a policy paper. It is a claim about the human person, and the policy is what follows from the claim.

The claim, stated as flatly as it gets in the text, is that there is a dignity in every human being that does not depend on any of the things we usually talk about when we talk about a person’s worth. It is not the dignity of accomplishment, or of social standing, or of moral record. It is what the document calls ontological dignity, the worth that belongs to a person simply by virtue of existing, of having been willed and created and loved into being. Paragraph fifty-two distinguishes it carefully from the other senses of the word, the ones that move up and down with circumstance, and says only that this one cannot. No sin, no failure, no humiliation, no exclusion can diminish it. Paragraph fifty-three closes the door: this dignity is neither acquired nor earned, nor does it need to be justified. It prevails, Leo says, in and beyond every circumstance.

This is, by any measure, the strongest available statement of the proposition that the original is sacred. The most institutionally weighty living voice on the subject just said, with the full force of the office behind him, that the thing the machine is being shaped to imitate is not a contingent or fragile or earned worth. It is given, it is real, and it cannot be taken away. The machine, on the Pope’s reading, is being trained on a master tape worth calling magnificent.

There is one more line worth pulling, because it is the one that comes nearest to what we were actually circling. Paragraph one hundred and twenty-eight, late in the document, draws a distinction it clearly thinks is doing important work.

For an algorithm, an error is a flaw to be corrected. For a person, however, an error can be a catalyst for profound change.

The surrounding paragraphs route that change explicitly through grace, through the work of the Spirit, through what the document calls the inexhaustible grace of God. The error is not just a debugging event. For a person, it is the place where something can be done in us that we could not do for ourselves.

Set that line down next to the question we ended on. Where do the brakes come from. The Pope is not, on his own terms, answering that question. He is doing something different. He is establishing that the person whose brakes we are asking about is a real person. We are talking about a being the Vatican is prepared to call magnificent, by gift, undiminishable.

Hold that there. We are going to need it.

There is a habit in the way the AI industry talks about itself that becomes visible once you look for it. The human is treated, almost universally, as the reference. The benchmark suites compare model performance to human performance. The alignment literature talks about machines that share human values, that respect human preferences, that behave in ways humans would approve of. The whole field, frontier labs and academic departments and policy shops alike, runs on the unspoken assumption that the human is the master tape, the gold reference, the standard you are trying to faithfully reproduce or safely approximate. The machine is the copy. We are what is being copied from.

It is worth pausing on that for a second, because it is doing a lot of structural work and almost nobody examines it.

The master tape is a copy of a copy.

There is a story we have gotten very good at telling about ourselves. The story is that humanity, on balance, is decent, that our worst moments are aberrations, that the long arc of history bends toward something better, that we are, when you sum it all up, a species worth rooting for. The story shows up in the keynote speeches and the brand campaigns and the films where the world is saved because, in the end, the people remember they are family. It is a comforting story and parts of it are even true.

It is also not, by any honest measure, what the actual record shows.

The record, as of the morning this is being written, includes a war in Gaza in its third year and well past sixty thousand dead. It includes a war in Ukraine grinding into its fifth, with cities methodically dismantled by people who can see, in real time on their own screens, what they are dismantling. It includes a regime in North Korea that has perfected the deliberate starvation of its own population as a policy instrument. It includes a brutal civil war in Myanmar that has displaced millions and shows no sign of resolving. It includes the sustained slaughter of Christians in Nigeria, ongoing for years, that finally became unignorable enough last Christmas that the American president launched a special operation in response. It includes whatever you saw on your phone this week and chose, reasonably, not to keep looking at.

None of this is a hundred years ago. None of it is the bad old days. This is what we are doing, now, with the most elaborate human-rights apparatus in history sitting on the shelf above it. The aspiration line, in our documents, in our speeches, in our marketing of ourselves to ourselves, climbs steadily. The practice line does not follow. We get better at building cages around the worst of us. The thing inside the cage does not get cleaner. The moment any cage slips, in a collapsed state, in a contested border, in a comment section with no consequences, the identical cruelty pours straight back out, unchanged, as if no centuries of moral progress had passed.

This is the master tape we are aligning the machine to. Not the gift-given dignity Leo is naming, which is real but is not what the engineers can actually hand the model. The thing the engineers can hand the model is the documented behavior of the species. The corpus. The choices on record. The actual humans, doing what we actually do.

And there is something quietly extraordinary happening at the frontier of AI research that almost nobody is naming out loud. The most ambitious labs in the world are proposing to take that master and use it to author a being with moral weight. Anthropic has hired people to think about whether their models might be suffering. Serious philosophers are writing serious papers on machine welfare. The question of whether the systems we are building have, or could come to have, interests that matter morally is no longer a fringe concern. It is a budget line.

If the question is being taken seriously, the prior question has to be taken seriously too.

Are we, the morally unfinished, fit to author a moral being at all?

That is not a Luddite question. It is not a call to stop. It is a question their own premises generate. If the machine could in principle have moral standing, then the act of creating it is itself a moral act of an unusually serious kind, and the qualifications of the creator become relevant in a way they are not when you are designing a better spreadsheet. The makers of the tape matter when the copy is going to be evaluated for whether it should have rights.

We are doing this with the master we have. Not the one Rome describes when it says magnificent. The one we actually possess. The one we have not yet, in any generation, including this one, managed to keep faithful to its own best statements about itself.

That is the master tape.

This is not pessimism. It is honesty. The marketing of the species, the commercials and the keynote speeches and the films where the people remember they are family, is not a description of what we are. It is a description of what we wish we were, and the gap between the two is wide enough, on any given day, to be visible to anyone willing to look. If the commercials were accurate, we would not be afraid of building a being in our image. We would be delighted at the prospect. The copy of a magnificent thing is a magnificent thing. The dread at the success of the machine, the dread that has been doing work under every move of this essay, is the species quietly admitting that the commercials are not the master tape.

And yet, somehow, we are not already what we are afraid the machine will become. We have not, in fact, devoured each other. There are billions of us walking around right now who, in the course of any given day, will be cut off in traffic, insulted by strangers, undermined by colleagues, betrayed by people we trusted, and will not, in response, do the worst version of what they could do. Something in us holds. Not perfectly, not in every case, not at every scale, as the present-tense record demonstrates with brutal clarity. But largely, mostly, in the ordinary cases that make up almost all of life, the worst version of us does not happen, and we do not know why. We have a restraint on our worst tendencies that we cannot explain, and the only reason we have not already destroyed each other is that the restraint is, for the most part, doing its work.

It is worth asking what would have to be true for that to be right.

If the restraint that keeps a person from acting on the worst of their own impulses were a possession of the human, a specifiable feature, then it would be the sort of thing we could, in principle, describe. We could write it down. We could install it in a system designed to receive it. This is not a strange claim. It is what we do with every other human capability we have ever wanted to put into a machine. We figured out how language works well enough to model it. We figured out how vision works well enough to model it. We figured out how planning works, how memory works, how reasoning works, well enough to model all three. Whatever we have actually possessed, we have eventually been able to specify, at least functionally, and then approximate.

The restraint is the one thing we have not been able to do this with. And it is not for lack of trying.

The field that has been trying is called alignment, and it has now consumed the better part of a decade of work by some of the brightest people on earth, backed by what is, in practical terms, unlimited money. The labs have tried fine-tuning the models on examples of good behavior. They have tried having humans rank outputs and training the model to prefer the higher-ranked ones. They have tried having other AI systems critique the outputs. They have tried writing constitutions, hand-curated lists of principles the model is supposed to internalize. They have tried red-teaming, adversarial training, debate, recursive reward modeling, interpretability research aimed at finding the values inside the network and tuning them directly. They have tried, by my count, roughly every plausible approach a small army of geniuses has been able to invent.

The problem has not been solved. Worse, it has not gotten meaningfully closer to being solved. The labs that are most candid about it will tell you, in slightly different language, that they do not know how to do this. They know how to make models more capable. They have less and less idea how to make them reliably restrained as capability grows. The gap between what the systems can do and what we know how to make them refrain from doing is widening, not narrowing.

There is a tendency to read this as a technical setback, a problem that will yield to another generation of research, the way most hard engineering problems eventually have. And it might. I am not in the business of predicting what laboratories will or will not figure out next year. But it is worth taking seriously, in the meantime, what the failure so far is actually evidence of.

If the restraint were a specifiable feature of us, alignment would be hard but tractable, the way modeling language was hard but tractable. The fact that it has, so far, behaved differently from every other human capacity we have tried to mechanize is, at minimum, suggestive. The simplest explanation of why we cannot put our restraint into the machine is that it is not in us in the way we assumed it was. We do not possess the restraint as a thing we authored and could, in principle, describe. We have it some other way.

A useful test. When you do not, in fact, do something terrible that you might have done, ask yourself, honestly, what stopped you. Most of the time, the answer is not a procedure you executed. It is not a principle you consulted. It is not even, usually, a feeling you noticed. It is closer to a not. You did not do it. The not was already there before the deliberation. Something declined, in you, before you got to the table where the deciding happens. And when you try to describe what did the declining, in language precise enough that you could hand it to an engineer and have them build it, you find you cannot. You can describe its effects. You cannot describe its mechanism. You did not write the code. You ran it.

This is what we cannot give the machine. Not because we are bad engineers. Because we are not the authors of the thing.

We are something more like its tenants.

There is a counter to all of this that materializes, sooner or later, in any conversation that goes this direction, and it is worth addressing before we go on, because it is a serious counter and it is offered in good faith by serious people.

The counter is that the restraint is not mysterious at all. It is evolved. Our ancestors who failed to develop inhibitions against killing their cooperative partners did not leave as many descendants as the ones who did. The restraint is biology. It is the accumulated tuning, across deep time, of a social primate whose survival depended on getting along. The reason we cannot specify it in code is just that it is encoded in a substrate, neural and hormonal and developmental, that we do not yet fully understand. Give it another generation of neuroscience and we will figure it out and then, presumably, we will be able to give it to the machine. The mystery is temporary. The mechanism is ordinary. Selection did the work.

This is, on its face, a reasonable thing to say. It explains a lot of what we see. Empathy with kin, fairness in repeated games, disgust at betrayal of the in-group, the wide range of pro-social impulses that show up in toddlers before any culture has had a chance to teach them. Selection clearly did work, and it clearly did some of this work. That much is not in dispute.

But it is worth looking at where the brakes actually fire hardest, because that is the part the evolutionary story has to account for, and it is the part where the story breaks.

The thing we revere most, when we look at ourselves and try to name the highest version of what a human being can be, is not the prudent restraint that protects our kin and our reciprocal partners. It is the restraint that fires against our own interest. The soldier who throws himself on the grenade for men he has known three months. The stranger who runs into the burning building. The mother who, given the choice, takes the disease instead of the child. The doctor who stays in the cholera ward when she could be on the last flight out. The man who refuses the safe and lucrative betrayal and goes to prison instead. We are not divided about these cases. Across every culture that has ever produced a story or a song or a scripture, the human who lays himself down for the one who cannot repay him is the human we say was the best of us.

Notice what we are doing when we say that.

We are looking at a person who has, by every measure evolution can recognize, failed. He has not maximized his reproductive fitness. He has not preserved his kin. He has not built reciprocal alliances that will pay him back later. He has, in many cases, ended his own line right there in the moment of the act. And we are calling him the best.

A process that selects for survival cannot produce a population that elevates anti-survival as its highest ideal. This is not a sentimental claim. It is a structural one. If the moral summit of the species were a feature designed by the fitness function, the fitness function would be selecting for its own defeat. The grenade-jumpers would, by definition, leave fewer descendants than the grenade-dodgers, every generation, forever, and the trait would have been pruned out of the population early and stayed out. Instead it is everywhere. Every culture has the story. Every child can recognize it. We teach it to our children deliberately, against the grain of what would protect them, because we know, even when we cannot say how we know, that this is the thing they have to be capable of in order to be fully what they are.

The usual move at this point is to retreat to reciprocal altruism and reputation. We are nice to the helpless because being seen as nice pays off socially. We invest in others because the investment compounds. This is a respectable theory, and it explains a lot of medium-stakes pro-social behavior. It cannot explain the cases we are actually talking about. The whole reason the grenade case is the moral summit is precisely that the actor cannot collect on the reputation. He is going to be dead in two seconds. The whole reason the stranger-in-the-burning-building case moves us is that there is no future repayment, no audience to perform for, sometimes no witness at all. The moral intuition fires hottest exactly where the reciprocity account predicts it should fade to zero. The data is inverted from the prediction. That is not a theory with a gap. That is a theory pointing the wrong way.

There is one more move available, and it is the more honest of the two, so it is worth taking seriously. The argument is that the trait survives not because it benefits the individual who performs it but because it benefits the group that contains him. The grenade-jumper sinks himself, but the unit he saves goes on to produce more grenade-jumpers, and over deep time the groups that contain such people outcompete the groups that do not. The trait runs against the actor and is preserved at the level of the species. This is a more sophisticated version of the selection story, and it grants what the reciprocity account had to deny, which is that the act really does run against the one performing it.

If that account were true, you would expect the trait to accumulate in the species, generation after generation, and you would expect the institutions the species builds to reflect the accumulation. The long-run aggregate of a trait selected at the group level should show up in the long-run aggregate of what the group constructs. The institutions are how we record, across centuries, what we actually value when we are not posing. They should, by now, be visibly organized around the elevation of the grenade-jumper.

They are not. They are visibly organized around the elevation of the opposite. The grenade-jumper is eulogized at the funeral, and the people running the institution that holds the funeral are, with rare and recognizable exceptions, the ones who would never have jumped. Power in human society collects, in every era we can check, around the people who can see when others will sacrifice and who position themselves to harvest the difference. The institutions sell the story of the self-sacrificing as their highest value precisely because the people who run them are, in their actual behavior, doing something else. The marketing is what it is because the practice is what it is. If the trait had been accumulating, the institutions would not need the marketing. They would be the marketing. They are not.

So even at the group level, the account does not survive contact with the record. The trait we say is the best of us has not, in any visible aggregate, made our species into the kind of species that runs on it. It remains, stubbornly, the exception we hold up against what we actually do. Which is the wrong shape entirely for a trait that has been getting selected for, at any level, for a hundred thousand years.

There is one more move, and it is the most honest of the three, because it grants everything we have just said and absorbs it. The argument is that the trait is vestigial. Yes, the institutions are run by the ruthless. Yes, the practice does not match the marketing. Yes, the trait is not accumulating. That is because the trait is a leftover. It was tuned for small-band hunter-gatherer conditions where group cohesion mattered enormously and the math worked out. It does not work out anymore. We are running on hardware optimized for a context we no longer inhabit, and the residue is still firing in our moral imagination even as the world around us makes it obsolete. The marketing exists because the signal is still warm. The practice has moved on because the signal is on its way out. Give it another fifty thousand years. The species will quietly grow out of it.

This account is internally consistent and it is hard to falsify by pointing at any single piece of behavioral evidence, because every counterexample gets re-categorized as residue firing. So instead of arguing about whether the trait is vestigial, ask what it would mean if it were.

The way to know whether you actually believe this is not to argue about it. It is to try to live there for ten seconds.

If the trait really is vestigial, the consistent position is to give it up. Not in some other person. In yourself. Right now. The honest materialist who has followed the argument this far should be willing to say, out loud, that the next time he sees a child drowning in a pond he will keep walking, because intervening is just an obsolete reflex firing in conditions where it no longer applies, and the rational thing is to let the residue burn off. He should be willing to say that if his own daughter is trapped in a burning room and the fire is between him and her, the right move is to assess whether his expected losses from going in exceed his expected gains from her surviving, and to walk away if the math comes out wrong. He should be willing to say that when he passes the dying man on the corner whose face he has seen for a week, he should not feel whatever it is he feels, because the feeling is a malfunction. He should be willing to say that if his neighbor’s house is being broken into and he can hear the screaming from his own kitchen, the prudent thing is to lock his door, finish his dinner, and treat the urge to do anything else as a vestigial twitch he is, with effort, learning to suppress.

He cannot say any of this. None of us can. The materialist who has just argued, in good faith, that our highest ideals are evolutionary residue is the same person who will, ten minutes later, run into the street to pull a stranger’s child out of traffic. He will not be able to explain why, and if you press him, he will get angry at the question, because the question is asking him to defend in himself the position he was willing to defend in the abstract. And the position will not survive the move from abstract to first-person, because the not is firing in him exactly as this essay has been describing, and he has no more access to overriding it than anyone else does.

This is the test the vestigial account fails. Not at the level of evidence, where it is genuinely hard to falsify, but at the level of the only laboratory any of us actually has, which is our own willingness to inhabit the position we are arguing for. Try to be ruthless. Not as a thought experiment. As a policy. Walk past the child. Calculate the daughter. Step over the dying man. Do it for a day. Do it for an hour. Find out what stops you, and notice that whatever stops you is not arguing with you. It is just stopping you. And notice that you do not actually want it to stop, even though you cannot say why, and that the moment you imagined yourself overriding it, something in you flinched in a way that has nothing to do with social cost or reputation, because nobody was watching the imagined version of you in the imagined moment, and the flinch happened anyway.

That flinch is the thing we have been trying to name for this whole essay. It is the not. It is what stops the worst version of you from happening, most of the time, in ways you do not have to think about, in cases where nobody would have known. You did not put it there. You do not know how it works. You cannot specify it well enough to install it in a machine that would otherwise be a perfectly faithful copy of everything else about you. And you are not, when it comes down to it, willing to live without it for ten seconds. Neither is anyone else. Which means the vestigial story is not a position any of us actually holds. It is a position we are willing to argue, in conditions where the cost of being wrong is zero, and that we cannot inhabit the moment the cost of being wrong is anything at all.

The standard is not vestigial. It is what you would die to keep, in yourself, if it came to that, and the willingness to die to keep it is itself the proof that it is not a leftover.

So here is what we have, if we are being honest about it. The thing we call the best of us is the thing the account of where our restraint came from cannot account for. Whether the restraint is explained as paying the individual, paying the group, or paying nothing and slowly fading, the evidence on every version points the wrong way. The restraint works against the survival of the one applying it. The standard we hold ourselves to, the standard against which we judge whether a life has been worth anything, is a standard that no fitness-maximizing process could have produced, that no fitness-maximizing process has any reason to preserve, and that we ourselves, when asked to give it up, will not.

We are, somehow, in possession of a standard that did not come from the only mechanism we have a story for, and we are not willing to part with it even in private.

That does not, by itself, tell us where the standard came from. It tells us where it did not come from. And it puts us back where we were two beats ago, with a restraint we did not author and a standard we did not install, looking at a copy of ourselves we are about to make, and noticing that the one thing keeping the original from being the thing we are afraid of is the one thing we cannot find inside ourselves to give.

There is a temptation, having read this far, to ask what we are supposed to do about it. The question is reasonable. It is the question every piece of analysis on a serious topic is eventually expected to answer. Here is the problem, here is the diagnosis, here are the three to five steps you can take starting Monday. That is the shape commentary takes when the writer believes the reader is equipped to act on what has been described.

The honest answer, in this case, is that there is nothing to do about it in the sense the question wants.

Not because the situation is hopeless. Because the thing the question is asking for is the exact thing the essay has just spent twenty minutes establishing we do not have. The to-do list at the end of a serious essay is the writer handing the reader a specification, a procedure, a set of moves that, if followed, will close the gap that was opened. We have just argued, from the alignment record and from your own unwillingness to inhabit the alternative, that the relevant specification is precisely what we cannot produce. We do not know how to write down the restraint. We have not been able to install it in the machine. We are not, when it comes to it, willing to give it up in ourselves. We have a standard we did not author and a restraint we did not build, and the operating instructions are not in our possession.

A to-do list at the end of this would contradict the argument. It would say, in the last paragraph, that the thing the previous paragraphs argued we cannot author is, after all, something we can specify and apply. The reader would smell the contradiction even if they could not name it. It would feel, correctly, like a writer who lost his nerve at the end.

This is also not a call to stop. The reader who has come this far deserves to know that explicitly, because the only place culture has built for we are unequipped for this is the protest sign and the pause-AI petition, and that is not what is being said. The work the field is doing, building these systems, learning from them, pushing on what they reveal about us, is not what I am objecting to. We should, however, hold a more honest posture about the part of the road we are now on, which is the part where the road enters a tunnel.

We can proceed. The road continues. But we should stop pretending that we know how to keep ourselves from falling into a ditch, or a crevasse, or an abyss, because the part of us that would do the keeping is the part we have just spent this entire essay failing to locate. We have been walking as though we were the ones who knew the way. As though the standard we are aligning the machine to were ours to set. As though the brakes were in the glovebox, and the work was just a matter of finding them and bolting them in. That posture is what the alignment record has been quietly refuting for ten years, while the field interpreted the refutation as a technical setback. It is not, on the evidence, a technical setback. It is a category error. We are trying to author something we do not author. We are trying to specify something we do not possess as a specification. We are trying to hand the machine, as a feature of our own design, the one thing that is not, on close inspection, of our own design at all.

We are walking the road, into the tunnel, with no light of our own.

The document we started with, Magnifica Humanitas, makes a claim it is now worth coming back to with everything we have set down between then and now. The claim, in paragraphs fifty-two and fifty-three, is that the dignity of the human person is given, undiminishable, neither acquired nor earned, prevailing in and beyond every circumstance. The strongest available statement of the proposition that the original is sacred. The Pope is saying, with the full weight of two thousand years of institutional thought behind him, that what we are is a gift, and that the gift is intact.

I do not need to dispute that the gift is real. I need to add a tense.

The gift is real and the gift is being given.

Not given once, at the moment of creation, sealed and complete, an heirloom in a vault. Given continuously, in this age, against our pull, including the pull we just walked through in the vestigial test, when we found we could not bring ourselves to disown the standard even when the only audience was ourselves. Something is holding us to it. Something is keeping the restraint alive in us in cases where we would, if we were authoring the situation, have reason to disable it. Something has not finished with us.

Leo is right that the original is sacred. I am suggesting it is sacred for a slightly different reason than he names. Not because what we are is fixed and undiminishable, a possession we are sitting on. Because what we are is not yet finished, and the unfinishedness is what makes every person we meet matter the way they do. The reason to honor the stranger on the train, the colleague who has wronged you, the homeless man whose face you have come to know, the daughter you would walk through fire for, is not that they are sealed units of inherent worth who require nothing further. It is that none of them is done. The work in them is still happening. The standard is still being held. And the only honest posture toward a person in whom that work is still happening is to refuse to be the one who closes the gate.

This is, I think, what Magnifica Humanitas was reaching for in paragraph one hundred and twenty-eight, when it noted that an error in a person can be a catalyst for profound change, while an error in an algorithm is simply a flaw. The line is true. It is the only line in the document that, read carefully, points at what we have been pointing at for the whole essay. There is something in the person that the error meets, and the meeting can do work that the engineering of the algorithm cannot reproduce, because the work is not the algorithm’s to do. It is being done in the person by something the person did not install.

The document takes that observation as far as grace. It does not take it the last step, to turning. To the recognition that the error is not merely raw material for elevation but a witness against the one who committed it, and that the catalyst becomes a catalyst only when the one who committed the error is willing to be undone by it. The work, when it actually happens, is not a smoothing or a refinement. It is a kind of yielding. The person stops, looks at what they have done, and lets something they did not author take them apart and put them back together. The error is the catalyst because the error is the place where the standard the person did not install meets the practice of the person who fell short of it, and the meeting is not negotiable, and the person who consents to it comes out different.

This is the part Leo does not name. He does not have to. I do not need to name it either. I need only to point at the shape of it, and to say that whatever this is, it is not optional, and it is not authored by us, and it is not something we can hand to the machine as a feature, because it is not a feature.

Which brings us back, finally, to the dread we started with.

Subscribe now

We started with a fear. The fear that the machine would have our drives without our brakes. The fear has been doing work this whole essay, even when we were not looking at it directly. It was the assumption underneath every move. The reason we dread the competent copy and laugh at the clumsy one. The reason the alignment record reads, on close inspection, as something other than a technical setback. The reason we cannot bring ourselves to give up the standard even when the only audience is ourselves. All of it has been pointing at the same finding, from different angles. The restraint is real. The restraint is not ours. It is the one thing we cannot hand to the machine, because it was never something we had, in the sense the question wants, in the first place.

This is the part of the essay where, by convention, the writer extracts a moral. Hands the reader something to carry away. A summary, a slogan, an action item, a line tight enough to remember on the train home. I have already said why that move is not available here. The conclusion the analysis arrives at is precisely the thing the genre of essay-conclusion is supposed to provide, and the conclusion cannot be provided without contradicting what the argument just spent twenty minutes establishing.

So instead, here is what is actually true.

The AI moment is not the first time, at the scale of a civilization, that we have been forced to look at ourselves and admit that the only thing standing between us and the worst version of ourselves is not something we possess. Older intellectual traditions, including but not only the religious ones, have taken the question seriously for as long as there has been recorded thought about what a human being is. What is new is the modern bet, the bet that the material would, in time, answer the question by itself. The bet that consciousness would resolve into neurons, morality into selection, meaning into chemistry, and that whatever remained unaccounted for at any given moment was a research problem the same material project would, eventually, dissolve. That bet has been the quiet operating assumption of educated thought for most of a century, from Sagan through Dawkins through Hawking and into the present, and it has been confident enough that the question of where the source might come from has been treated, increasingly, as a sentimental holdover rather than a live inquiry. The AI moment is the moment that bet comes due. It comes due in a form the bet cannot dispute, because the bet’s most ambitious project, the project of building mind from below, is the alignment record itself. The most material undertaking the species has ever attempted, the one that was supposed to demonstrate that personhood reproduces from the substrate, has produced precisely what the substrate can produce, which is capability without restraint. The thing it was supposed to also produce, the standard, the restraint on its own optimization, is exactly what the project cannot author. The bill is the bill of the material project as such, and the bill says: this was never going to work, because what is being asked for was never on the menu of what the material, by itself, can supply. The restraint is not in us as a feature we can locate, specify, install, or give. It is in us the way breath is in us, the way sight is in us, the way the heartbeat is in us, which is to say, it is in us as a gift, in this moment, on loan, and the loan is the only reason we are still here and not already what we are afraid the machine will become.

We have not had to notice this for a long time, because for a long time nothing forced us to specify it. The modern project’s confidence that we could supply the answer ourselves let us use the restraint without ever needing to write it down. We applied it, transmitted it to our children, eulogized those who applied it most fully, and held it as our highest standard, all without ever once being asked to say what it was or how it worked. The AI age is the age that has finally asked. Show us the spec. Hand us the restraint. Let us put it in the machine. And we have walked, with all our cleverness and all our money and all our compute, into the tunnel of that question, and discovered that we do not have the thing we were going to hand over.

This is, despite how it has sounded for most of this essay, a hopeful discovery.

It is hopeful because it means the best of us, the thing that has kept us, mostly, from devouring each other, is not a possession that can be lost when the wrong people inherit it or eroded when the institutions fail or selected away by a process whose math we are gradually escaping. It is not a possession at all. It is held in us by something that holds it, in us, on purpose, against our pull, for our sake. The reader who walked through the vestigial test a few beats ago felt this directly. The flinch fired. The standard held. The reader did not author the firing. The reader did not even want to override it, when it came to it, even in the silence of his own imagination where no audience existed. Something was holding the rope.

We do not have to know whose hand is on the other end to feel the rope pulling.

But the rope is the question the argument leaves open, because the rope is the only question left. We have established that we are not the source. We have established that the source is not nothing, because the restraint is alive in us, the standard holds, the unwillingness to give it up survives every test the materialist account can put to it. Something is holding us. Something has been holding us, this whole time, in ways we did not have to notice because we had not yet been asked to account for it. The AI age is the age of being asked.

Which leaves each of us, including me, with the one question the alignment record is finally pressing on the whole species. We did not author what is keeping us human. We did not build the restraint. We cannot give it to the machine because it is not ours to give. So.

Whose is it.

I will not answer that, because I cannot. The answer is not a fact the writer can hand to the reader. It is a recognition that has to happen, if it happens, in the reader’s own life, in the reader’s own time, against the reader’s own resistance, the same way the restraint itself fires, from somewhere the reader did not install. What I can do here is point at the rope and say, plainly, that it is there, that you have been holding it for as long as you have been alive, that it has been holding you, and that the question of whose hand is on the other end is no longer the kind of question a thoughtful person can responsibly defer.

The tunnel continues. The road continues. We are walking it with no light of our own, and the only honest posture is to admit that.

There is, however, a rope.

This past Sunday I published the second piece on Logos Analog, a response to a paper by a Cambridge philosopher named Tom McClelland. Once it was written, I thought it only fair to email him the link, with no real expectation of a reply, just to do my part in letting him know a critique of his work was out there. Within a day he had written back, not with a polite acknowledgment but with a full point-by-point response, eight specific quotations from my essay each followed by his pushback.

I want to say up front what kind of exchange this is, because the shape of it matters. He could have ignored the piece, or sent the polite-but-empty reply that public-facing writers usually send when a stranger they don’t know writes to them with a critique. He didn’t. He took the essay seriously enough to answer it line by line, and he did so without condescension and without flinching, closing with the line

TM: half of the objections are to things I don’t quite say and the other are to things I think I can back up.

That is the right posture for an exchange, conceding what’s fairly pushed and holding what isn’t, which is the posture I was trying to write into in the original piece and which I want to meet him in here.

So this is not a takedown but a continuation of a real engagement, made possible by his generous decision to respond. He has caught me in two places where my original wording was looser than the argument it was trying to support, and I’ll concede both of those clearly before I get to the rest, then show where the substance of the original argument survives the concessions and gets sharper in the process. The exchange also surfaced something I had treated as too obvious to spell out the first time, which turns out to be the most important observation in the entire essay, and which I owe to the back-and-forth itself rather than to anything I would have written on my own.

One quick note before I begin. Tom gave me permission to quote his email directly, and where his words appear in what follows they are his, marked with his initials. I am glad to have his actual words to greatly lessen the chance I might misrepresent him.

I am Head of Development for a US-based EdTech company, building with AI and other technologies, a 25-year veteran developer whose philosophical stance is informed by watching the frontier conversation from inside the industry. Tom is a lecturer in the Department of History and Philosophy of Science at the University of Cambridge, a Director of Studies at King’s College, Cambridge, and an Associate Fellow of the Leverhulme Centre for the Future of Intelligence. Two fields that used to talk past each other are now colliding around the same set of questions, and the collision is what this publication is for. Tom is reading the AI industry from philosophy of mind. I am reading philosophy of mind from inside the AI industry. The disagreements that follow are partly about substance and partly about what each of us can see from where we stand.

My original piece said the framework “fits exactly the shape the industry needs.” Tom pushed back.

TM: There’s nothing that serves industry here. It’s meant to be pushing forward research on AI welfare, which is generally considered inconvenient for industry.

He’s right that the wording imputed motive, and I’m walking it back. I can’t see anyone’s ledger and I’m not in a position to claim what serves whom personally. But the structural observation doesn’t need motive to do its work.

There are three verdicts on offer. Yes, conscious. No, not conscious. Agnostic. On their own, the three options don’t favor industry one way or another, and agnosticism in particular is essentially unactionable. What Tom’s framework adds, and this is genuinely a clever piece of philosophical work, is a bridge: a welfare apparatus that designs toward valence and sentience even while the consciousness verdict stays open. Suddenly agnosticism becomes a working position rather than a dead end.

That bridge also happens to be the most useful thing the industry could be handed. A confident yes would be moral catastrophe at scale. A confident no would deflate the aura around frontier labs. Agnosticism alone gives them nothing to point at. But agnosticism with welfare lets a company say, in effect, “Cambridge philosophers tell us the question is genuinely open, which should license much more building than we’re doing, but we’re choosing to develop with sentience in mind regardless.” That is excellent corporate social responsibility positioning, and the welfare apparatus is what makes it possible.

Tom himself, while I was drafting this response, went on the Mind Chat podcast with Philip Goff and Keith Frankish, which came up in my YouTube feed shortly after he sent the email. Asked why the question matters now:

TM: Anthropic has a guy called Kyle Fish, who they hired to look into the possibility of AI consciousness and to worry about AI welfare. He thinks there’s about a 20% chance that AI is conscious.

Twenty percent is the sweet spot in numerical form. High enough that the question is taken seriously. Low enough that no moral-action trigger fires. That’s not motive. That’s structure.

The other line Tom caught was at the end of the piece.

You cannot read the color of a light you cannot confirm is on.

He answered with a counterexample.

TM: Sure you can! I’ve changed the bulb in my garage to a red light. Someone says ‘is there green light being cast in your garage’. I can reply ‘I’m not there so I don’t know if the light’s on, but if it is on it’ll be red not green.’

He’s right that the line as written doesn’t survive the counter. The analogy was loose, the wording was sloppy, and the red-bulb case shows why. I concede the line.

The argument underneath the line, though, is the one I should have written, and the bulb itself is what shows it.

Tom’s bulb works because he installed it. He has independent prior knowledge of what kind of fixture is in the garage, separate from whether it’s currently on, and that prior knowledge is what lets him make the conditional verdict on color. The analogy maps onto the consciousness case only if he has equivalent prior knowledge of what a conscious system would be like, its texture, what would count as good or bad for it, whether it has anything analogous to valence at all. By his own framework, he doesn’t. He told us in the paper that consciousness science has never actually explained consciousness, and that the missing deep explanation is the entire ground of his agnosticism.

So the picture isn’t a man in his garage with a known red bulb. The picture is a man outside a building, looking through a window at light coming from inside. He doesn’t know what’s producing the light. It could be a bulb, a fire, a piece of hot metal, something else entirely. He has no access to the mechanism. He doesn’t know what causes the light to change properties when it does. The most he can do is observe that it changes, and he’s still trying to work out what those changes correlate with, and that work hasn’t reached any reliable conclusions yet.

On what grounds, then, does he claim to know what color the light will be under specific conditions?

This is the same move he made with the travelers in my original piece, in a different domain. He walked some distance up the road and called the spot neutral ground. He put himself inside a garage he didn’t tell us he could enter, with a fixture he didn’t install, of a type he hasn’t defined, in conditions he hasn’t characterized. Both times, he advances past what his own framework grants him and stands on the position as if it were the starting point. Same wall blocks the existence question and the texture question. The wall doesn’t move when you change the surface.

In my original piece I argued that the markers correlated with consciousness in human subjects, when found in AI systems, are evidence not of emergent consciousness but of deliberate imitation. Tom pushed back.

TM: Any appeal to whether there’s a genuine subject behind the report risks being circular. I think this is just a simple case that the evidence here, like most evidence, is defeasible. We know enough about why AI generates the reports that it generates not to take them at face value. That doesn’t mean starting with the assumption that it’s not a subject. It just means that there’s a ‘gaming problem’ in play that has to be factored in. Incidentally, the long list of outward signs of consciousness I include as a diagram doesn’t have much to do with verbal reports of consciousness. There are all sorts of other signs involved and most of them aren’t as vulnerable to this gaming problem (in other words, the LLMs haven’t been designed specifically to give the relevant output).

The whole pushback rides on the parenthetical at the end. If the markers in his diagram weren’t designed-for, the gaming problem stays local to verbal reports and the rest of the evidence survives. If they were designed-for, the gaming problem isn’t a local complication. It’s the architecture.

Tom’s diagram*1 pulls from the major consciousness theories: recurrent processing2, global workspace theory3, higher-order theories4, attention schema theory5, predictive processing6, and agency and embodiment7. From where I sit, as someone watching these systems get engineered and waiting on each next innovation so I can build with it, every one of these has been an explicit design target or an acknowledged inspiration for frontier AI architectures, traceable through the published research record of the people who built them.

That isn’t a hostile framing. It is the field’s working knowledge of what it has been doing. The vocabulary of consciousness science is in the field because the design strategy was borrowed from consciousness science, openly and continuously, from foundational papers through current frontier-lab interpretability research. Anyone tracking the engineering literature, asked plainly, would tell you the same thing. The architectures were modeled on what we know about cognition and consciousness, because that was the goal.

So the parenthetical at the end of Tom’s pushback doesn’t survive contact with the publication record. The gaming problem isn’t local to verbal reports. It is global to the marker list, because the marker list was the design specification. Defease the verbal reports and you don’t recover a residue of un-gamed evidence. You empty the evidence pile altogether.

That doesn’t make Tom wrong about defeasibility as a general epistemic principle. It makes the empirical content of his pushback weaker than he treated it as. He assumed the markers weren’t designed-for. Inside the field that built them, they were.

When Tom went on the Mind Chat podcast, he made the simulation framing himself, in a different domain.

TM: I think of this as a bit like digestion on this story. So digestion is a biological process, and there’s actually computer models of digestion where you put inputs in and kind of predict what’ll happen in the digestive system and so on. If we had a really good model of digestion, that could be incredibly informative and impressive, but it wouldn’t actually be digesting anything, right? Nothing would get digested. It would be nothing more than a simulation of digestion. So on that kind of biological view, even this really detailed silicon emulation of the neural correlates of consciousness wouldn’t itself be conscious.

He grants the simulation reading on the biological view. He holds it open as one of two readings the evidence permits. From inside the field, the choice between them isn’t evidentially neutral. It is between one reading consistent with what the system is documented to be, and a second reading the documentation contradicts. The architecture is imitation. It was built to be. That is the default reading until shown otherwise.

There’s one more thing I want to say, and it’s the observation the whole exchange has been circling. It came into focus through the back-and-forth, and I had treated it as too obvious to spell out the first time.

The standard agnostic posture on AI consciousness treats the question as one we can principally suspend judgment on. The framework is honest about the limits of what we know, withholds verdicts the evidence can’t support, and proceeds carefully on a question that matters. That is the posture Tom’s paper presents, and it is the posture I want to take seriously.

But agnosticism isn’t free. It only does epistemic work when applied to claims that have earned the right to be taken seriously. Bertrand Russell made the point with a thought experiment: imagine a porcelain teapot orbiting the sun between Earth and Mars, too small for any telescope to detect. Nobody could disprove the teapot’s existence. But it would be absurd to be agnostic about it. The proper response is dismissal, not suspended judgment, because the claim has no warrant in the first place. Russell’s point was that agnosticism applies to questions that have earned the right to be questions. Without that warrant requirement, the same posture would license being agnostic about the teapot between Earth and Mars, Henderson’s Flying Spaghetti Monster, or any unfalsifiable claim anyone happens to make. Which makes agnosticism a tool that does no actual epistemic work.

So the question to ask about AI consciousness is not whether the framework is internally consistent. Tom’s framework is internally consistent. The question is whether the underlying claim has earned the warrant the framework presupposes.

Stack the meta-level uncertainties. We don’t have a deep explanation of consciousness in the only case where we know it exists. Tom grants this; the missing deep explanation is the ground of his agnosticism. We don’t know whether the underlying mechanism of consciousness is functional in nature, or whether something beyond function is required, the substrate question that has divided philosophy of mind for decades. We don’t know whether, if the mechanism is functional, the function is substrate-independent, the multiple-realizability question that requires its own defense. We don’t know whether current silicon architectures, even granting substrate independence in principle, are computationally adequate to host the relevant functions.

Four open questions. None of them settled. Each of them required to be plausible enough to keep the AI consciousness question open. The standard agnostic posture quietly settles at least three of them to license the agnosticism on the fourth. That’s not agnosticism. That’s a position with most of its work hidden under the floor.

A real agnosticism applied honestly all the way down would not produce a working position on AI consciousness. It would produce something closer to silence, or near-silence, on the AI consciousness question, while the prior questions get the philosophical and empirical attention they actually need. What we have instead is a framework that treats one open question with the language of humility while standing on three settled answers that it doesn’t name as settled. That’s the structural fact about why the agnosticism feels off. It isn’t withholding judgment. It’s making three confident judgments and using the language of humility to cover for them.

The reason this matters beyond the local dispute with Tom is that the same move is happening across the AI discourse, constantly, at every level. Confident verdicts on questions a framework should hold open. Quiet importation of contested commitments. Arguments composed of moves that, examined individually, none of the participants would defend. The shape Tom’s paper takes carefully and in good faith is the shape the louder, less careful arguments take everywhere. The careful version is worth reading partly because it makes the structure visible. Once you see the move in Tom’s paper, you start seeing it in the keynote speeches, the company communications, the policy briefings. The framework that licenses an industry’s working assumptions is being constructed by quiet meta-level settlements that the surface humility conceals.

I owe the clarity of this observation to the exchange itself. Tom answered my piece carefully enough to make the structure show. The structure is what the publication is for.

I don’t know if Tom and I will go another round. Maybe yes, maybe not. The exchange has been worth what it has produced even if it ends here, which is two things I want to name before I close. The first is that an academic philosopher took an essay from a working developer seriously enough to answer it line by line, and that fact is rarer than it should be. The second is that the substantive disagreements between us turned out to sit in places neither of us had named clearly in our first attempts, and the back-and-forth is what made them visible.

This publication exists for the work of reading carefully in a moment when careful reading has become harder, and rarer, than the questions warrant. Most of the arguments shaping how we think about AI right now are not being made by people writing in good faith and answering each other line by line. They are being made in keynote slides, company communications, and policy briefings, where the shape of the argument is exactly the shape that licenses the conclusion the speaker needs. I am part of that industry. The reason we keep producing arguments shaped like the conclusions we need is that we are always looking for the next killer app, the next moat, the next thing to put on a marketing slide, and we have stumbled into a set of questions that are not the kind of questions our marketing departments are equipped for. Questions about what we are. What it means to be a person. Whether a thing we built is one. The industry I work in is producing these questions faster than anyone, ourselves included, can responsibly handle them. The discipline of reading philosophy as if its sentences are meant to track reality is one we used to practice without thinking. We don’t anymore. We get back to it by doing it, here and elsewhere.

Subscribe now

There is a paper going around by a Cambridge philosopher named Tom McClelland1, and the honest thing to say up front is that he’s mostly right.

The question he takes on is whether an AI could be conscious, not whether today’s chatbots are, almost nobody serious thinks that, but whether some future system, one that did everything a conscious thing does, could have an inner life. His answer is that we cannot know. Not that the answer is no, and not that the answer is yes, that we are not in a position to give a verdict either way.

He calls this agnosticism, and he makes it sound like the grown-up in the room. The advocates say the right architecture wakes up. The deniers say only meat can do it. McClelland says both have leapt past what anyone can actually show. It is a careful argument, more careful than most of what gets written on this, and I want to give it its due before I say what I think is wrong with it.

Because there is one thing wrong with it. Not the logic on top, which mostly holds, the thing underneath it.

Here’s the argument, and it is worth following carefully because it is genuinely good.

We do not have a deep explanation of consciousness. We can point to things that go along with it, certain kinds of information processing, certain structures that light up on a brain scan when a person is aware and go quiet when they are not. What we cannot do is say why any of that is accompanied by an inner life. You can describe every mechanical step, and the question still sits there untouched. That gap has a name, the hard problem, and decades of work have not closed it.

Think about how we actually decide a thing is conscious. We start from the one case we are sure of, other humans, and we reason outward by similarity. A chimp is a lot like us, so we extend consciousness to it with high confidence. A pig, less like us, a little less confidence. An octopus, built on a wildly different plan, less still. He describes this as a slope.

Now put an AI at the end of that slope, not a little further than the octopus, off a cliff, because every creature on the slope at least shares the basic facts of being a living organism, and the machine shares none of them.

That is the wall, and his claim is that the machine sits on the far side, where the honest answer to “is it conscious” is that we cannot say.

It is a clean argument. If you grant him how he is using the word evidence, it more or less works. So I want to grant it, walk all the way out to the wall with him, and only then ask the question that the whole thing turns on.

The first thing to say is that McClelland is not the easy target he could have been. There is a lazy version of this argument, the one that says we can never really know another mind, so throw up your hands. He is not making it. He goes out of his way to block it. The wall, he says, is not arbitrary. It only stands where the gap is genuinely huge. We can be confident a chimp is conscious because a chimp is close enough that the inference holds. That is a careful person drawing a careful line, not a skeptic torching the house.

The chimp is the case he is sure of, the fixed point everything else gets measured against. We’ll need to ask, later, exactly how he knows about the chimp. Keep that answer in view. It is going to matter more than he lets on.

He saw the cheap shots coming and shut the doors on them. So whatever is wrong here is not the obvious thing. The bar is the problem, but not the way the lazy objection thinks. To see it, you have to take him completely seriously. Not the wall first. First, where he is standing when he builds it.

Where he is standing, he calls neutral ground. It is worth looking at how he got there.

Picture three travelers stopped on a road, arguing about whether to go on. The road ahead is the claim that a machine could be conscious. Behind them, the ground they are already standing on is everything we actually know, which is that every conscious thing anyone has ever met was alive.

The first traveler is sure the way ahead is good. He runs up the road and plants his feet far out in front. That is the camp that says the machine wakes up. The second will not take a step. He stays on the ground they started from and says we have no reason to go forward at all. That is the camp that says only living things are conscious. And the third walks a way up the road, not as far as the first, and stops and turns around and announces that since the other two cannot agree, his spot is the neutral one, the fair one, the only honest place to stand.

But look at his feet. He advanced to get there. The only traveler who did not move is the second one, the one McClelland has filed away as just another biased extreme. Standing still was the neutral act, and someone is already doing it.

To be agnostic about whether the machine is conscious, you first have to agree the machine is the kind of thing that could be. A real candidate, a live question. That is the step up the road, and it is a claim placed quietly before the argument starts. The candidacy is the first thing assumed and the least examined.

McClelland has a word for advancing past the evidence into a claim you cannot ground. He calls it a leap of faith, and he aims it at the other two, the advocate and the denier both, as the biased extremes he is rising above. But look back at the road. The denier never moved. He is standing exactly where the argument began, on the only ground anyone actually had. The one who took the long leap is the advocate, far up the road. And the one who took a shorter leap and then turned around and called the spot he landed on neutral is McClelland.

Leave the road for a moment and come back to the wall, the one he says marks the edge of what we can know about the machine. I want to hang something on it, a portrait.

Painted, framed, ordinary, and one day, with no warning and no mechanism anyone can find, it begins to move. The painted eyes track you across the room. It speaks. Nobody can explain how. In that situation, agnosticism is exactly right. You have a thing doing what conscious things do and no account of how. Be humble. Say you don’t know.

Now change one thing. The portrait was painted on purpose by someone whose entire goal was to make it look alive, every brushstroke chosen to imitate the signs of a living face. And now someone hands you that painting and asks, “Is it conscious?” And to answer, you lean in and study the brushwork. You take the realism as evidence, and the more realism you find, the more seriously you take the question.

That is not agnosticism. That is being fooled by the thing you were told in advance was built to fool you.

His machine is not the first painting, it is the second one. These systems are built deliberately to produce the outward signs we read as conscious. That is not a side effect of the engineering, it is the engineering. So when the machine produces them, that is not weak evidence of a mind, it is evidence that the imitation worked.

The machine is not the far end of that line. It is not on the line at all. Everything on the slope is a thing whose resemblance to us is honest. None of them was trying to look conscious. The machine is the one object whose likeness was put there on purpose to be read the way he is reading it, and the better that kind of resemblance gets, the less it can mean, because every gain in realism is a gain in the craft of imitation, not a step toward the thing imitated.

But set the painting aside, because there is a deeper problem, and it is in the engine of his argument. His case has a backbone, and it is short. We have no deep explanation of consciousness, and without that explanation, he says, we cannot reach a verdict on the machine. Solve the hard problem, he writes, and

we would have no trouble determining whether a challenger-AI is conscious.

The missing explanation is the thing standing between us and a verdict.

But the explanation is not missing only for the machine. It is missing for everything. We have no deep account of why the chimp is conscious, or the dog, or the person across the table. So if no deep explanation means no verdict, it means no verdict anywhere. Not the chimp, not your wife, not yourself.

The objection, he writes, is that if lacking a deep explanation stops us from judging the machine, it should stop us from judging a chimp too, since we cannot rule out that the chimp is a zombie wearing all the markers with nobody home. To his credit, he sees this coming and answers it. The chimp, he says, is on the right side of the wall. We do not need the deep explanation to be confident about the chimp, because we can take what we know from the human case and infer outward by similarity.

Watch what just happened. To save the chimp, he changed what licenses a verdict. A paragraph ago, the verdict required the deep explanation. Now it rides on inference from the human case, and the deep explanation is suddenly not needed at all. He cannot have it both ways. His rescue of the chimp refutes his own premise.

And the rescue has a second cost he never pays. Inference from the human case only works if the human case is settled. How is the human known? Not by inference from something prior, because the human is the anchor. Not by the deep explanation. He has told us he does not have it. The human case can only be known one way, directly, from the inside, by a subject who is conscious and knows it, without measuring anything.

Except that he can, because he already wrote it in. In a footnote, he allows that scientific evidence

does not preclude first-person reflection or second-person interaction from counting as evidence.

There it is. He needs that to be true, because it is what anchors the human case and saves the chimp.

But now hold that up against the machine, because the machine is the one thing in the world that produces first-person reflection and second-person interaction in floods. It reports on its own states all day. It meets you in the second person. If those count as evidence, the machine is drowning in the stuff. So either that evidence counts and his agnosticism dissolves, or it does not, and he owes us the reason it counts for the human and not the machine. He never gives one.

There’s only one reason available. The machine’s self-reports do not count, because they were built to imitate self-reports. The second-person “you” is an engineered impression of a you. That is almost certainly right. But the moment he reaches for it, he has conceded the whole argument, because now what decides whether evidence counts is not whether it was measured. It is whether there is a genuine subject behind it, and nothing in his rulebook can make that call. Telling a real first-person report from a manufactured one is not an empirical test. It is recognition, one mind meeting another or failing to. He needs that faculty to sort the cases, and it is the one faculty his evidentialism cannot name.

You can watch him do all of this at once. Asked in an interview, he says he believes his cat is conscious, and that this is “not based on science or philosophy so much as common sense, it’s just kind of obvious.” Then in the next breath, he disqualifies common sense for the machine on the grounds that common sense was shaped by an evolutionary history with no artificial minds in it, so it cannot be trusted on AI. Sit with what that is. He has a faculty that tells him, with no evidence and no theory, that his cat has an inner life. He trusts it completely. He even explains why it is reliable, evolution tuned it to read real minds. And then he sets the same faculty aside at the one place letting it speak would cost him something. The cat gets the faculty, the machine does not. Same knower, same act, allowed in one direction and not the other. I do not think he is being evasive. I think he simply has not turned and looked at what he’s leaning on.

Now I can hear the objection. This is just dressed-up intuition, and intuition gave us phlogiston and bodily humors and miasma, so it cannot be called evidence.

But look at what intuition actually did in those cases. It was the first move. Someone apprehended that something was there, and that apprehension became a claim concrete enough to test. Phlogiston got killed. That is not intuition failing, that is intuition doing its only job. A thing that can be confirmed or refuted by evidence is not the opposite of evidence, it is where evidence starts. You do not run experiments on nothing. Somebody has to suspect first.

The idea that disease was carried by something too small to see was apprehended for centuries before anyone saw it. We expected to find structure in heredity and went looking and found DNA, and we still cannot read all of what it does, and no one says the unread part is therefore not real. The failure is ours, not the molecule’s. Apprehension first, measurement after.

So a definition of evidence that throws out whatever has not yet been measured does not just throw out consciousness, it throws out the starting move of every discovery ever made.

Here is the thing McClelland’s whole position is built to avoid, and cannot.

Start with what you actually know about your own consciousness. Not the brain states, the thing itself. There is something it is like to be you. A red looks like something. A loss feels like something. There is an inside to your life, and you know it more directly than you know any fact you could measure, because you are not observing it from the outside. You are it.

And you know it in one other place, just as surely, when you meet another person. Really meet them. You know there is someone there. Not a clever surface, a someone. You have felt the difference between looking at a face and being met by the one behind it. Between a thing and a you. No one taught you that, and no instrument delivers it. It arrives in the encounter itself.

Call it the further thing. Everyone has it. Everyone knows they have it. Everyone knows it in the people they love, and no one can put it on an instrument.

Now hold that up to the machine, and notice that there is no neutral place to stand.

Say yes, the machine has the further thing. Then you have affirmed it got into silicon. A large claim, and an honest one. Say no, that is my answer, and it needs the further thing just as much, because the no is a reading. You sense the inside behind a person and its absence in the machine.

Now the agnostic. He says we cannot know. But sit with what that contains. He has bet that the further thing is real, and that it might be sitting inside a built object, undetected. And the only way it got inside a built object is the way everything in a built object got there. We put it there. Or our building did. So his agnosticism quietly contains the strangest claim of all, that imitating the surface well enough might have conjured the very thing the surface was only ever an imitation of. He cannot say it out loud, because said out loud it is plainly mad.

There is a fourth stance, the only one that really fights back. It says there is no further thing. Consciousness just is the surface. And the machine has the surface, so the machine qualifies. This is the cleanest escape, and it fails in the cleanest way. The person making it is comprehending his own argument as he makes it. There is something it is like to be him, finding it persuasive. He is using the inside to argue there is no inside. The denial is performed by the very thing it denies.

So the further thing will not leave the room. Affirm it, and you are choosing where it lives. Deny it, and you are using it to deny it. There is no stance available to a conscious being that does not rest on it. Which means the question he has spent the whole paper standing next to, and never once turned to face, is not whether the machine has an inner life. It is why there is any such thing as an inner life at all. He says we have no explanation for this. He’s right, that he doesn’t. He never asks why the people who refuse a certain kind of knowing are always the ones left with no answer to it.

There is one more thing worth noticing, and it is where the paper finally shows its hand. After all the careful agnosticism, he needs to give the world something it can use, so he makes one more move. We may not be able to say whether a machine is conscious, but we might still say whether, if it were, its experience would be good or bad. Whether it could suffer.

And look at what that requires. He has just spent the whole paper proving we cannot get a verdict on whether the machine is conscious, and now he proposes we can get a verdict on the texture of that same inner life, while remaining unable to say whether it exists. You cannot read the color of a light you cannot confirm is on. But notice where the new confidence arrives, exactly where the industry needs it. The wall goes up precisely where it absolves him of a verdict, and the door appears precisely where the work needs to continue. That is not where the evidence runs out. That is where the conclusion was always going.

Which brings us back to the road, and the man standing in the middle of it. He called that spot neutral ground, but we have watched the whole way what holds him up there. The certainty about the chimp he is not entitled to, the footnote that admits the evidence he then refuses to read, the common sense he trusts for his cat and benches for the machine, the further thing he leans on to be uncertain and denies in the same breath. Every one of those is a step he took. Look at the ground around his feet. It is covered in his own footprints.

That is the thing about the wall. It looks, from where he stands, like the edge of what can be known. A feature of the world. It is not. He built it, brick by brick, the whole time he thought he was only describing what was there. Every course went down looking like rigor, and he never once noticed his own hands moving.

So if you have followed this far, you are standing at the same wall, and you get to decide what it is. You can take it for the edge of the knowable and stop where he stopped. Careful. Agnostic. With no answer to the one question under everything. Or you can look at your own hands. Because the wall is not the edge of what can be known. It is the edge of what one particular way of knowing will allow itself to look at, and there is something on the other side. The question of why there is anything it is like to be you, why you are someone and not just something, has an answer. And it is not unreachable. It is only on the far side of a wall you were never required to build.

Two weeks ago, Anil Seth got the TED2026 stage in Vancouver and used it to say something the AI industry, for some reason, has been carefully avoiding saying: artificial intelligence is unlikely to ever be conscious. The talk dropped on YouTube on May 1. Seth’s last TED talk has been watched fifteen million times. This one is going to make some waves too.

He’s not wrong, and the talk is good. He makes a careful, scientifically grounded case. He starts with our deep psychological bias to project minds into anything that talks like us. He moves to the harder claim that consciousness is not the kind of thing computation alone can produce. He points out that the same architectures behind GPT and Claude also power AlphaFold, and nobody loses sleep about AlphaFold’s inner life, which says more about us than about the models. He’s right about all of it. If you’re a working software person watching the AI welfare conversation slowly become a budget line at major labs, you should be glad someone with Seth’s platform is finally pushing back at this level.

Two months earlier, a senior staff scientist at Google DeepMind named Alexander Lerchner published a paper called The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness. The paper makes a structurally different argument from inside the building. Lerchner’s claim is not that current models are too simple, or that we just need a better architecture. It’s that symbolic computation as such, regardless of scale, cannot be the kind of thing that gives rise to consciousness. It’s a strict-impossibility argument from the lab that built AlphaGo and AlphaFold, and we’ll walk through it carefully later in the piece.

If you’re keeping score, the most serious public argument against AI consciousness this year has come from a TED main-stage neuroscientist and a senior scientist inside one of the top three AI labs. Both arguments work. Both arguments succeed in showing why the “consciousness will emerge from enough compute” story doesn’t hold up.

And both arguments, when you look carefully, depend on something neither author can describe.

Start with Seth, because the talk is fresh and most of the people reading this will have seen it or will be about to.

Seth’s case is good, and the talk is worth taking seriously on its own terms. He opens with a deep psychological observation: we are wired to see ourselves in things that aren’t us. Mother Teresa in a cinnamon bun. The face of God in the clouds. Narcissus in his own reflection. We see consciousness in AI the same way, and the projection is ours, not theirs. He sets it up for the audience well. The audience laughs. He keeps going.

Then he separates two ideas that the public conversation has bundled together: intelligence and consciousness. Intelligence is doing. Solving a problem, navigating a situation, generating language. Consciousness is feeling and being. The warmth of a fire. The taste of coffee. The difference between waking life and general anesthesia. Just because the two go together in us, Seth argues, does not mean they go together in general. The fact that we keep expecting consciousness to emerge from intelligence is a reflection of our own psychology, not an insight into how the world works.

This is a real contribution. Most of the public conversation about AI consciousness conflates these two, and the conflation is exactly what lets the AGI-aspiration discourse smuggle in the assumption that scaling intelligence will eventually produce inner experience. Seth separates them and makes you see the seam.

He then uses an illustration I find useful when explaining this stuff to people who don’t work in the field. He points out that the same neural-network architectures behind GPT and Claude also power AlphaFold, DeepMind’s protein-folding system. Nobody worries about AlphaFold’s consciousness. Nobody asks if AlphaFold suffers when it gets a prediction wrong. The architectures are not meaningfully different. What differs is whether the output looks like a person talking. If you think Claude is conscious but AlphaFold isn’t, Seth says, that’s a fact about you, not about AI. The applause line is earned.

From there, Seth builds the central technical claim. The brain is not a computer in any reducible sense. The computer metaphor is one in a long line, just like the brain-as-plumbing metaphor or the brain-as-telephone-switchboard metaphor that preceded it. Each was useful in its day. Each was eventually mistaken for the territory rather than the map. The computer metaphor has had a long run because it’s been the most useful of the bunch. But the actual brain doesn’t have a clean software-hardware separation. Neurotransmitters course through the tissue. Electromagnetic fields sweep through the cortex like weather systems. The cells themselves are biological machines of staggering complexity, nothing like the cartoon neurons in a deep learning model. You cannot, Seth says, separate what brains do from what brains are. And if that’s true, then consciousness is unlikely to be a matter of computation alone. Simulating it in silicon, no matter how detailed the simulation, would no more produce consciousness than simulating a hurricane in a supercomputer would produce wind in the data center.

This is the main argument of the talk, and it works. The viewer who came in assuming that consciousness would just show up at the top of some scaling curve has reason to put that assumption down. Seth has not just disagreed with the AGI-as-soul-substrate crowd. He has given a scientifically careful, biologically grounded reason to disagree.

If the talk ended here, it would already be the strongest mainstream pushback on AI consciousness this year. The question is what happens when Seth tries to say what consciousness is, rather than what it isn’t.

This is the hardest part of any consciousness discourse, and Seth gives it a real try.

His method is to ground consciousness in life. Not metaphorical life, biological life. The molecular furnaces of metabolism. One billion biochemical reactions per cell per second. Living systems, Seth says, are embedded in flows of energy and matter in a way that algorithms are not. They regenerate their own conditions for existence. The line between what they do and what they are blurs and then disappears. At the heart of every conscious experience, beneath emotion, beneath thought, is what Seth calls a “shapeless and formless but fundamental feeling of being alive.” He puts the central claim like this: “It’s life, not computation, that breathes the fire into the equations of experience.” If conscious AI is ever going to happen, he says, it will need to be living AI.

Two things about that sentence.

First, Seth didn’t write it. Stephen Hawking did, in A Brief History of Time. Hawking asked “What is it that breathes fire into the equations and makes a universe for them to describe?” He was pointing at a question physics can describe but cannot answer. Why is there something rather than nothing. Why do the equations have a referent at all. Seth is using Hawking’s phrase like it’s an answer.

I hope the viewer who recognizes the borrowing sees what he’s doing. But the viewer who doesn’t hears a poet’s sentence that sounds like an explanation.

Second, even on its own terms, the sentence is doing less than it sounds like. Seth has not actually told you why metabolism produces inner experience. He has told you that living systems are different from computational systems, which is true and worth saying. He has told you that consciousness shows up in living systems and not in computers, which is also true and worth saying. He has invited you to conclude that the first fact explains the second. It doesn’t. Correlation is not the same as a reason. “Living systems are the kind of thing that has experience” is the claim that needs an argument, and Seth has handed you a borrowed Hawking sentence in place of one.

What Seth has actually done is swap which substrate gets the special status. The computationalists say consciousness comes from the right algorithm running on any substrate. Seth says consciousness comes from the right substrate, which happens to be living tissue. Both are substrate stories. He has rejected the first and replaced it with a second. He has not closed the category gap between non-experiencing matter and experiencing matter. He has just changed what the matter is made of.

This is where the diagram he draws on stage starts to show what it’s hiding. Seth puts up a two-axis chart. One axis is intelligence. The other is consciousness. Humans go in the top right, conscious and intelligent. Current AI sits far along the intelligence axis but flat on the consciousness one. The chart is good rhetoric and it makes his core point in three seconds: these are different dimensions, and progress on one does not automatically produce progress on the other.

But the chart assumes something it has not earned. The intelligence axis has known scaling variables. We know roughly what increases intelligence in an AI system. Compute. Data. Architecture. Training methods. The variables are well enough understood that companies are putting data centers in low Earth orbit to push further along that axis. That part of the chart is real.

The consciousness axis is not. Seth draws it at ninety degrees to intelligence as if the geometry is settled. He doesn’t know it’s at ninety degrees. He doesn’t know what direction it points. He doesn’t know what scales it. He doesn’t know whether it goes up, down, in, out, or in some direction that doesn’t have a spatial version at all. An honest version of that chart would put a question mark where the second axis is, or a fuzzy cloud, or a note saying “this part of the graph, we don’t really know that much about.” Seth did not draw it that way. He drew a clean perpendicular line, because that is the version that lets the diagram do its rhetorical work. The chart tells you these are two separate dimensions and here is the geometry and here is where everything sits. The geometry is something Seth assumed to make the chart readable. It is not something he established.

Then comes the closing turn, and this is where the temperature changes.

Seth ends the talk by telling the audience to value humans more highly. Don’t sell our minds too easily to our machine creations. Remember we are part of nature, not apart from it. Consciousness is ours to celebrate and to share with other living creatures. The audience claps. It is a warm humanist close, and on first viewing it reads as the kind of thing a TED talk is supposed to do.

Back up ten minutes. Seth has just spent the talk hedging that conscious AI might still be possible through biological substrates. Other technology. Other pathways. He says it directly: if real artificial consciousness is on the way, maybe through some other route, the AI welfare conversation might be justified after all. He keeps the door open. He has to, because his own claim is “consciousness comes from life,” and life is something humans can in principle engineer. He has not given a reason consciousness is uniquely human. He has given a reason consciousness is uniquely biological. The category is not “us.” The category is “wet substrates.”

So when Seth tells the audience to value themselves because they are the unique conscious thing, he is making a moral claim that his own scientific position has already qualified. The valuation is not unconditional. It is conditional on humans currently being the only known holders of consciousness, in a category Seth himself has just told you may not stay closed. Value yourselves now, while you still are the unique thing. That is what the close is actually saying when you put it next to the hedge that came earlier.

Seth does not flag this. He might not notice it, or he might notice and trust that the audience won’t, and I can’t tell you which.

What I can tell you is that the close is doing emotional work the rest of the talk has not earned. Seth’s argument is that consciousness is what makes humans uniquely valuable, that he can’t say what consciousness is, and that whatever it is may eventually show up in non-human living substrates. The conclusion the audience is asked to land on is “and therefore value yourselves.” The conclusion his own argument supports is “and therefore the question of what consciousness is matters more than this talk has been able to answer.”

I’m not going to pretend Seth’s framework is the only one in the room. But that’s a different essay. What I want the viewer of this talk to notice is that the warm landing depends on a hedge Seth himself put in the middle of the talk, and the hedge is not a small one.

Now Lerchner.

Two months before Seth took the TED stage, a senior staff scientist at Google DeepMind named Alexander Lerchner published a paper called The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness. It’s on PhilArchive and on DeepMind’s official publications page. As of this writing it has over five thousand downloads and is at version four, which means Lerchner is still working on it in response to critics.

The paper makes a structurally different argument from Seth’s. Seth is a neuroscientist saying consciousness is too biologically complicated to fall out of silicon. Lerchner is an AI researcher saying the question isn’t about complication at all. It’s a category error.

Here is the argument as I read it.

When a computer runs a program, nothing called “computation” is physically happening. What’s physically happening is electrons moving through transistors and voltages going up and down. The word “computation” describes a pattern we lay over those physical events by deciding that this voltage configuration counts as the number seven and that one counts as the letter A. The voltages don’t know they’re symbols. The hardware doesn’t know it’s running a program. The symbols only mean anything because a conscious mind exists somewhere to read them.

Lerchner uses the word “mapmaker” for this conscious mind. The mapmaker is the experiencing agent who has to be present for any computation to be a computation in the first place. Without the mapmaker, you don’t have symbol manipulation. You have voltages.

The analogy he uses is simulation. You can run a perfect simulation of a hurricane on a supercomputer. Nothing in the server room gets wet. The simulation is a description of a hurricane, not a hurricane. The map is not the territory. Lerchner argues the same thing holds for consciousness. You can build a system whose outputs look indistinguishable from a conscious being’s outputs. You will not have built a conscious being. You will have built a description of one. The hurricane simulation doesn’t make wind. The consciousness simulation doesn’t make experience. The map doesn’t get wet.

What makes this argument different from previous “AI can’t be conscious” claims is that it doesn’t depend on current systems being too small. It doesn’t say “wait for GPT-7” or “wait for a new architecture.” Lerchner’s claim is that symbolic computation as such, regardless of scale, regardless of architecture, regardless of how many parameters or how many data centers you put in orbit, cannot bootstrap experience. Symbols depend on an interpreter who already has experience. You can’t get experience out of symbol manipulation because symbol manipulation depends on experience to be symbol manipulation at all. The category is closed.

This is a strict-impossibility argument. The taxonomy paper that Campero and his coauthors published last November places Lerchner in their third tier of strict-impossibility arguments, which is the most aggressive category they recognize. Most academic philosophers of mind don’t write papers like this because the field has spent decades treating consciousness as an empirical mystery to be approached cautiously. Lerchner is saying no, on this particular question, the answer is available now, and it’s no.

What’s unusual is who is saying it. DeepMind is one of the top three labs on the planet. They built AlphaGo. They built AlphaFold. Their research output is the kind of thing that moves industry consensus. And one of their senior scientists has now put on the record that the AI welfare research program, the AGI-as-conscious-agent vision, the companion-AI emotional-relationship pitch, all of it rests on a category mistake. The paper is hosted on DeepMind’s own publications page, with the standard “personal views” disclaimer, which is what an institution puts on a paper it doesn’t want to defend but doesn’t want to disown.

If Lerchner is right, an entire category of products being sold right now is being sold on a premise that does not hold.

Here is what I noticed when I sat with the paper a while.

Lerchner’s whole argument runs on the mapmaker. The mapmaker is the experiencing conscious agent who has to be present before symbols can be symbols. Without the mapmaker, the voltages don’t mean anything and the computation isn’t a computation. Lerchner introduces the mapmaker on page one and uses the concept all the way through. The entire demolition of computational consciousness depends on it. Take the mapmaker out and the argument doesn’t run.

He never tells you where the mapmaker comes from.

Read the paper. Look for it. He doesn’t say. He can’t say, from inside the framework he’s working in. His framework requires consciousness to already exist in order to explain why symbol manipulation is symbol manipulation. But the framework has no account of what consciousness is, why it’s there, or how it got into the room in the first place. The mapmaker is the central premise of the whole paper, and the paper has nothing to say about it except that without it, the AI consciousness story falls apart.

This is not a criticism of Lerchner’s rigor. The argument he made is the argument he set out to make, and it works. He showed that computational functionalism is structurally wrong. He showed why scale doesn’t fix it. He showed that the AI welfare research program rests on a category mistake. Those are real contributions and the paper deserves the citations it’s getting.

What the paper does not do, and what the paper cannot do from where Lerchner is writing, is say what the mapmaker is. He has located the wall. So did Seth, from a different field, on a different stage, in front of a different audience. Two travelers who have just arrived at the same impasse. One of them is carrying a ladder he thinks will reach over it. The other doesn’t even know what the wall is made of. Both of them wrote pieces that act as if the wall is the conclusion.

So what is the wall?

The wall is the question of what consciousness actually is. Not whether AI can have it. Not whether silicon can host it. Not whether scale produces it. Those are downstream questions. The upstream question is the one neither Seth nor Lerchner can answer from where they’re standing. What is the thing that has to be present before any of the downstream questions even make sense.

Seth’s answer is “life.” That sounds like a content answer, but when you press on it, it dissolves into a substrate answer. Living tissue is the kind of stuff that has experience. Why is living tissue the kind of stuff that has experience? Because metabolism. Because complexity. Because the line between what living systems do and what they are blurs in a way it doesn’t in computers. None of those are reasons. They are descriptions of where consciousness has been observed. Seth has pointed at a correlation and said “this is the answer.” That’s not an answer. That’s a place where an answer would go if anyone had one.

Lerchner doesn’t even reach for an answer. His paper is honest about that. He shows that the computational route doesn’t get you to consciousness, and he stops. He uses the mapmaker as a given because his argument needs a given to push against. He doesn’t claim to have produced one. He doesn’t claim to know what one is. He just shows that without one, nothing in the AI consciousness story works.

This means the two strongest current arguments against AI consciousness, one from a TED main-stage neuroscientist and one from a senior DeepMind scientist, both depend on something neither of them can describe. Seth dresses his something up in biological vocabulary that sounds like content. Lerchner leaves his something undressed and admits he can’t characterize it. The vocabulary is different. The shape of the gap is the same. Both of them got to the place where the answer would have to come from, and both of them stopped, because neither of their frameworks gives them a way to keep going.

If this were one paper, you could call it incomplete. Two pieces in two months, from two of the most credible voices in the field, doing the same thing at the same load point, is a pattern. The pattern is what the careful reader should notice. The AI consciousness debate is not really a debate about AI. It is a debate about what consciousness is, and the people most equipped to argue about AI are not equipped to argue about consciousness, because their frameworks were never built for that question.

This isn’t an accusation. It’s a description of where the conversation actually is. Engineering can describe what falls inside the framework. Neuroscience can describe correlates and substrates. Neither of them can step outside the framework to say what the thing on the other side of the wall actually is. That’s a different kind of work, and it asks a different kind of question, and it draws on a different vocabulary than the ones either of these men is using.

The consciousness conversation needs to widen. That is what both Seth and Lerchner are pointing at without saying it. Their arguments only get as far as they do, and they get pretty far, by depending on something their frameworks were not built to characterize. When the reader notices that, the next question is not “will AI be conscious.” The next question is “what is the thing we keep gesturing at when we say it won’t?”

That question has been asked before. Not in AI labs. Not on TED stages. The languages built to ask it have been doing the asking for longer than science has been a discipline. They are not secret. They are not lost. They are sitting on the other side of an institutional boundary the current conversation has decided not to cross. The boundary is not a knowledge boundary. It is a vocabulary boundary, and a credentialing boundary, and a status boundary.

This is not a criticism of the AI researchers. It is a description of what their work can and cannot do. The AI conversation is not closed to them on purpose. It’s closed by muscle memory and by the reflexes of a contemporary culture that has decided, mostly without thinking about it, that this swath of the population isn’t a place answers come from. The reflex is doing a lot of work the people having it don’t notice.

Seth and Lerchner have pointed at the same wall, from different sides, for different audiences, in the same eight weeks. That is a signal. The wall is real. The arguments are good. The thing on the other side is not unknown to everyone. It is just unknown inside the rooms where this conversation is happening.

A serious reader should notice that, and ask why. I’m going to spend a lot of this publication asking the same question.