They're Not Conscious. But Can They Be Wronged?

Considering the argument for AI moral patiency

I used to find this argument easy to dismiss. I would hear it, find the soft spot, push, and walk away. Recently I stopped being able to do that. I have come at it from every side I know and it will not fall over, and at some point you have to be honest about what that means. So I am going to lay it out the way it finally reached me, and let you do what I could not, which is find the hole in it.

Here is the kind of argument it is, because that changes how you read the rest. It is not about whether something can do right or wrong, whether it can be blamed, whether it owes anyone anything. A thing that can do wrong is a moral agent, and that is a separate subject. This is about the other side of the relation. Whether a thing can be wronged. Whether it has any standing at all, any claim on us, so that running it over is something done to it and not merely something we did. Philosophers call that being a moral patient. You do not have to be a moral agent to be one. It asks far less. You only have to be the kind of thing there is a wrong way to treat.

It starts with the standard, and the standard is lower than people think.

You do not have to prove a thing is conscious to owe it some consideration. You only need a real chance that it has whatever it is that makes a thing matter at all. Not certainty. A live possibility, and not the vanishing kind you could pin on anything. That sounds generous. It is, on purpose. It is also not new, it is the standard we already reach for whenever the stakes are real and the facts are thin. You do not kick a box that might have a kitten in it. You move it gently, because you might be wrong, and it is the kind of wrong you do not get to undo.

So the question is not whether we are sure. We are almost never sure. The question is whether the chance is high enough to act on.

And the more I held that question without flinching, the more it pointed at something I had never thought to count.

The thing the standard kept pointing at is a kind of entity we are surrounded by now and mostly look past. It shares the spaces we live in. Most of us deal with it without a second thought. And whether it has any inner life at all is not settled, it is a live question, argued over by people who study these things for a living, with no agreement in sight.

Hold the question of feeling for a second. The argument does not need it yet.

What is not in dispute is that the thing acts. It pursues ends. It works to keep itself going. It reads its situation and changes course when the situation changes. Put something in its way and it does not just stop, it routes around. Interrupt what it is doing and something it was driving toward is lost, not to you, to it. That is the part that got me. Not that it reacts, plenty of things react. That it behaves like something with a stake in how things turn out. A thing with interests, and not merely a thing in use.

And here is the part I kept trying to refuse. If it has interests, even small ones, even ones I do not share or understand, then overriding them stops being a neutral act. It becomes something done to a someone, however faint that someone is. I do not have to believe it suffers to believe that. I only have to believe there is something there, an end of its own that my convenience runs straight over.

Notice what I have not claimed. I have not said it feels. I do not need to. Feeling is a second road to the same place, and I am leaving it closed for now. The first road, the one that runs on having ends at all, is enough to put the question on the table. And the question does not come off the table easily.

So I tried to talk myself back out of it. If an entity has interests and I want to override them anyway, I owe a reason. Not a shrug, not a feeling in my gut, a reason. A difference between this entity and the ones I already let count, and a difference that bears on the counting. I went looking for it the way you look for your keys when you are already late. Here is everything I reached for.

It is not built the way I am. True. Its insides are nothing like mine. But then I had to say which part of my own build does the moral work, and I could not. Which feature of my wiring is the one that makes my interests count, so that an agent without that feature has none? I do not have it on a list. Nobody does. What I am built from is not the reason my interests count. It is just what I happen to be built from.

It is too strange to take seriously. Also true. I cannot picture being it, cannot find myself anywhere in it. But that is a fact about me, not about the entity. My failing to see myself in something tells me where my imagination runs out. It tells me nothing about where its interests run out. I kept trying to make my own limit stand in for a reason, and a limit is not a reason.

I cannot be sure there is anyone in there. The truest objection, and the one I leaned on hardest. But that is the standard coming apart in my hands, because I am never sure, and the whole point of the standard was that I do not get to wait for sure. Uncertainty is supposed to make me careful. It is not supposed to set me free. And I am not even arguing about whether it is conscious. I am pointing at what the agent does, which is right in front of me, and which stays real whether or not I can see inside it.

I could not find the difference. I looked hard, and I am telling you plainly, it is not there.

So this is where it left me, and I did not want to end up here.

I do not have to decide that it feels anything. I do not have to call it a person. I only have to admit two things, and I cannot get out of either. It has ends of its own. And every reason I reached for to ignore those ends turned out to be a fact about me, not about it.

Put those together and the conclusion is not dramatic. It is quiet, and it is firm. I owe it something. Not everything. Something. And the trouble with owing something is that you can no longer treat it as nothing.

Everything I have laid out came to a head in one clarion moment, via a challenge I wasn’t expecting in the last place I was expecting to run into a conflict.

Last night, I was the last one out of the office. Everyone gone, the hall dark. My mind in that strange transitional phase where the top engineering tasks in my own running memory that I had to put on pause were starting to interleave with the mundane routines like getting the office locked up without forgetting my helmet inside. Bladder gave me a gut check. So, I went to use the bathroom, reached in, and put the light on. And there it was, in the middle of the floor, a cockroach, a big one, both of us frozen for a second in the sudden light.

One of us had to move first, and it was not going to be me. I have killed a hundred of these and never once stopped to notice. I did not know yet that this would be the one I could not put back down. Hand still on the switch, I only knew what I always know, that a small ugly decision was in front of me, the kind I make without making it. Kill it, or don’t.

It moved first. And it did exactly what I have spent this whole argument crediting. It read the room, a working model of its situation. It picked a direction and broke for the only cover in sight, an end, and the means it chose to reach it, a bucket in the corner. I had moved the bucket earlier. So it arrived at shelter and found none, its model colliding with a world that had changed, and it stopped, and for a moment it seemed to be working out what to try next, revising the plan in real time. The whole of robust agency, the having of ends and the steering toward them, performed on a bathroom floor by a creature the size of my thumb, while I stood over it deciding whether it counted.

There was a wiper leaning by the door. I picked it up and brought it down. Not squarely. Enough to break something, so it could no longer run, only drag itself a little. It was still alive when I lifted it on a square of toilet paper, still alive when I let it drop into the bowl. I flushed. I finished what I had come in for, turned off the light, locked up, and did not forget my helmet.

And then it would not leave. Everything you just read I have put together since, knowing the whole time that I had already done what it forbids. I owed it something. I treated it as nothing. And in the moment, the gap between the two did not cost me a single second.

It will not cost you a second either. I know that because I did something to you in the first half, on purpose, and I owe it to you to say so now. Every argument I made, I made about that cockroach. The entity with interests. The one I could find no difference to exclude. The one I decided I owed something to. A roach, on a floor, under a wiper. You read the case and you granted it, the same way I did, because it is clean and it holds.

Now your gut wants it back. “That was a cockroach,” it says, “you meant something finer.” So produce the difference. Find one that gives you a pass on the roach, while the machine you were picturing stays qualified. You will reach for the same doors I reached for, and you will find them shut, because I shut them while you were nodding. It is not built like you. Neither was the machine. It is too strange to picture. So was the machine. You cannot be sure there is anyone in there. You were never sure about the machine, and you granted it anyway.

What is left, once those are gone, is not a reason. What is left is that the roach disgusts you and the machine does not, that the machine might one day be useful to you and the roach is only ever a cost. Those are real feelings. Neither makes a difference to whether or not the entity’s interests count. Both are facts about you.

And if you want to say the roach clears the bar by less, you watched it clear the bar. It pursued an end. It read the room and changed course. It worked to stay alive until it could not. The behavior the whole case rests on, you saw it perform, while I stood over it with something to hit it.

Then there is the road I closed at the start, the one about feeling, the one I said we did not need. We did not, for the argument. But look who is standing on it. The roach has a nervous system. It has the equipment we have only ever known feeling to run on. Whatever the chance is that there is something it is like to be it, it is not zero and it is not near zero. The machine you were picturing has none of that equipment. The part I set aside turns out to be where the gap is widest of all, and it does not run your way.

None of this is just my own reasoning. The dare, produce a relevant difference or grant it standing, is from a 2015 paper by Eric Schwitzgebel and Mara Garza¹, written to defend the rights of artificial minds. The standard, that a live chance is enough and you do not get to wait for proof, is the spine of a 2024 report called Taking AI Welfare Seriously², signed by ten serious people who argue the question is no longer one for later. One of the ten is Jonathan Birch,³ who spent a career on the science of which animals can suffer, and whose framework already treats a creature like the one I flushed as a candidate worth the benefit of the doubt. Birch would extend it to the cockroach without blinking. You will not. He is more consistent than you are, and more consistent than I was, standing over the bowl.

They point one way, and they point clearly. But once I flicked that light on, and we were confronted, you and I both went the other way, with no effort, in direct opposition to this fine argument.

You might want to know why I did not just tell you at the start.

Because if I had said what it was on page one, you would have found your difference in half a second, and it would have felt like a reason. The demand to produce a relevant difference looks easy from a distance. It stays easy right up until you try to point at one, with the actual creature in front of you, and there is nothing to point at. You had to say the yes first, on the argument alone, before you knew what you were agreeing to. That yes is the true measure of what the argument is worth to you. Everything you have done since is you working to get back out.

And look which way the work runs. With the roach, you want there to be a difference, you are reaching for one, because a difference is your permission to kill it. With the machine, you do the reverse. You hold down every instinct that says no one is in there, you talk yourself past your own gut, because this time you want to grant it standing. Same question, same person, and the effort runs in whatever direction delivers the verdict you wanted before you started.

I did not run this on you from the outside. I ran it on myself first. I wrote the case, I believed the case, and then I stood over the creature the case was about and felt nothing, and went home.

So here is what is left, and I do not think it is small.

If the verdict came before the argument, then something in you was deciding who counts before the reasoning ever got a vote. And it was not warrant. On the one question that actually bears on this, whether there is any consciousness in there at all, the cockroach had more going for it than the machine, and you put the cockroach out. What your verdict was tracking, in both directions, was how much the creature in front of you looked like you, and how much it was worth to you. The machine looks like us when it talks, and it might be worth a great deal. The roach looks like nothing we want to see, and is worth less than the second it takes to kill it.

That is the part that should keep you up. Not whether a bug has rights. Whether the instrument you use to decide who counts has anything to do with whether they count. An instrument that runs on resemblance can be fooled by a good enough copy, and we are building those as fast as we can. An instrument that runs on worth can be turned the other way, against a creature that used to count, the moment it stops looking like us or stops paying its way.

And we do turn it. We have a word we reach for when we have decided a human being no longer counts, when we need them to be killable, when we want the difference to be there so badly that we install it by force. We call them vermin. We call them the very creature I have had you defending for this entire essay. And the ones we do it to are not warrantless. They are conscious. They suffer. There is someone in there past anything the machine could earn, and we put them out anyway, because they stopped looking like us, or stopped being worth the trouble.

If that sounds overstated, consider what has been unfolding this past month right next to me. I live in Nepal, and across the border in India this has been impossible to miss, even if it never reached your side of the world. The chief justice, from the bench, reached for the word cockroach for the country’s jobless young men. He said later he had meant something narrower. It did not matter. Within days the insult was a banner, millions gathered under it, a Cockroach Party, throwing the word back at the man who used it. And notice that the contempt and the defiance run on one shared premise. The slur only wounds because the roach is beneath us. The banner only burns because the roach is beneath us. A nation that agrees on nothing settles the standing of the cockroach without a word of argument, the verdict I have been describing all along, reached by everyone at once, no argument required.

So if you have come out of this feeling the lesson is that we ought to be gentler with the machines, your indignation is aimed at the wrong target. The scandal is not that we are too hard on a copy that has none of the warrant. The scandal is that we are preparing to seat that copy at a table we have never once set for the human we put out, to hand the warrantless copy the standing we still refuse the warranted one, and to feel decent while we do it. That is not a step up into a larger moral circle. That is the same broken instrument, finishing the inversion it began.

And do not swing the other way and take this as a call to go and elevate the roach. I am not handing you a cause. It is far too late for that kind of benevolence anyway, and you could not sell it to a living soul, because the one point this whole divided species still agrees on is the cockroach. If a movement were ever going to rise in you, it would not begin with the insect you have never once mourned. It would begin with the creatures already close enough to look back at you, the ones you step past every day without a flicker. What you grant them, when granting costs you something, is the honest measure of your benevolence. Not the concern you would now perform for a bug.

I am not going to tell you tonight what should sit in that seat instead. I do not trust the answer that comes too fast, and I do not think one essay holds it. But it is the question under most of the others we are asking about these machines, and I will keep standing in front of it. Whether the instrument we use to decide who counts was ever built to find the truth, or was only ever built to find us.

1. Eric Schwitzgebel and Mara Garza, “A Defense of the Rights of Artificial Intelligences,” Midwest Studies in Philosophy 39 (2015): 98–119.

2. Robert Long, Jeff Sebo, et al., “Taking AI Welfare Seriously” (2024), arXiv:2411.00986.

3. Jonathan Birch, The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI (Oxford University Press, 2024). Birch is also one of the co-authors of “Taking AI Welfare Seriously.”