Flawless Is Not the Same as Accountable

Bill Gates published an essay in August about the AI transition that is more honest than most things in the genre. He declares his own conflict of interest in the body of the text rather than in a footnote. He refuses both utopia and doom. He writes that we need both things at once, genuine concern about the harms and genuine optimism about the benefits.

He also asks the question I think is the right one: how do we preserve the human when machines can think faster than we can.

Then he answers it with capability. And that's where we part ways.

The Argument

Gates writes that the great shift for working people comes when AI produces nearly flawless work. At that point, he writes, the system can operate on its own without a human watching over it. Companies will then have every economic reason to let it do so.

Read that again. The condition for removing the human is the error rate.

That means the human in his model sits there as an error-catcher. A control instance whose only function is to compensate for the fact that the machine isn't quite good enough yet. When the machine becomes good enough, the control instance has no job left.

If that were true, the entire question of accountability would be a transitional problem. Something that resolves itself as the technology matures.

What's Missing From It

I have a typewritten overhead from IBM, 1979. One page with three holes for the binder. It contains two sentences: a computer can never be held accountable, therefore a computer must never make a management decision.

Notice what isn't there. Nothing about error rates. Nothing about how capable the computer is. Nothing about what it can handle.

That's the whole point. It took me a long time to see it. Most objections to letting machines decide are built on capability. The model hallucinates. The model misses context. The model doesn't understand nuance. All of those objections get weaker with every new model, because capability genuinely grows.

IBM's argument does not. It doesn't care about capability. A computer that becomes twice as capable does not become half as accountable. It becomes exactly as unaccountable as before, because it still cannot bear the consequence of having been wrong.

The human therefore doesn't stay in the loop because the model makes mistakes. She stays because someone has to carry it when it does. Those are two entirely different reasons. Only one of them disappears when models improve.

Flawless and accountable are not the same axis. One is moving fast right now. The other hasn't shifted since 1979.

There is a second reason the human stays in the loop, and it has nothing to do with accountability.

Gates writes that the reliability problem is resolving quickly, because researchers are now building models that can review their own work and improve themselves. But a model that reviews its own output is a closed system. It can only redistribute the information it already has. It has no opening of its own onto reality. No user who complains, no experiment that goes wrong, no consequence that hurts. Remove the human and you get a copy of a copy composing itself, where the noise of one pass becomes the signal of the next.

The gate carries two things simultaneously. New information flowing in, accountability flowing out. Close it and both flows stop. Quality collapses long before anyone gets around to asking who bore responsibility.

The Reserve

Gates sees that something must be set aside. He proposes a domain he calls Human Reserved, work that we choose to reserve for humans. He draws the image from nature reserves: places where we could build roads and houses, but choose not to because the loss would be too great.

He grounds the proposal in something real. His father was cared for in the final stage of life by staff who understood him even when he could no longer make himself understood. About them, Gates writes that something in that care was irreplaceably human. No robot could or should have done it.

I think he's right about the care. It's the formulation just beside it that doesn't hold. About a robot delivering a terminal diagnosis, he writes that there is no technical reason it couldn't, yet it shouldn't.

There the argument rests on a preference. We ought not to.

And a preference loses against economics. Gates writes it himself, a few pages later: market forces will make adaptation move faster and faster, and if established companies don't bring in the machines, new companies will. He also lists everything that has to be solved for the reserve to hold. Who decides what goes in. By what criteria. How you stop companies from cheating. What happens to trade when one country allows robots and another doesn't.

This is not weak thinking. It is honestly laid out. But it is a description of a line that must be defended, every year, against everything pressing against it. That's why Gates also needs what he proposes elsewhere: new national bodies, an international organization modeled on nuclear inspections and aviation regulations, and a tax on tokens and robots to make humans comparatively cheaper.

He builds an institutional structure to hold a line that his own reasoning cannot hold by itself.

The Difference

The line I draw needs no defense, because it is not a choice.

Accountability is not a property of a good decision. It is the precondition for holding a mandate at all. An actor who cannot be held accountable cannot hold the mandate, regardless of how sound its judgment is. This is not a rule someone introduced. It is what a mandate means.

So the question never becomes a policy choice. And there is no cheating to police. You can let a model produce whatever you like. You cannot transfer the bearing of consequences to it, because it cannot receive them. What looks like a delegation is in practice just nobody carrying it anymore.

A nature reserve can shrink when pressure increases. This cannot, because it is not a boundary in the terrain. It is a description of what accountability is.

Where the Line Actually Falls

Human Reserved draws the line between professions. Healthcare on one side, perhaps law on the other, different in different countries.

I think the line runs through the professions instead. It goes per task. The question I ask is whether the tool replaces something I already carry, or assembles something that is missing from the room.

A simulated panel meant to make my decisions replaces the judgment I've built by having been wrong and paid for it. The same setup in an organization where knowledge is spread across five people who never talk assembles instead a whole that no single person holds. Same tool, opposite value.

That question needs no international organization. It can be asked on a Tuesday morning, by whoever holds the task, in any industry. And it gives different answers for different tasks within the same profession, which is precisely what a reserve cannot do.

What I Take Away

Gates is right that the transition is coming fast and that no one has a plan. He is right that we should not leave it to AI companies to solve problems they created. He is right that those with the least time are the ones who need the most.

And he is right about capability. The models are getting better, faster than most people think.

It just doesn't matter for this question. My name is on what I send. That was true when the model was bad. It's true when it's good. It will be true when it outperforms me on every individual task, because my name will still be there and the machine's will not.

I should be clear about where I stand, the same way Gates is: I make my living delivering this. That should be weighed accordingly. But the argument doesn't stand or fall by who makes it. That's the whole point of arguments that don't rest on capability.

See also: A computer can never be held accountable (series 49) on the pamphlet behind the slide, Replaces or assembles (series 32) on where the line falls per task, and The human is the opening (series 61) on the second flow through the same gate.