Taming unpredictable power machines

Software used to be obedient. You wrote the rules, it followed the rules, and when it misbehaved you could open it up and point at the exact line that caused the trouble. Predictable, almost domesticated.

What we're building now isn't like that. It can write, plan, research and act on your behalf, and it can accomplish things that would have seemed impossible only a few years ago. But its behaviour wasn't written down rule by rule. It was trained on more text than any of us could read in a thousand lifetimes, and what emerged was something with instincts rather than instructions.

What we've ended up with is incredibly powerful. It's also why the same thing that untangles a problem you've been stuck on for hours will sometimes charge off in the wrong direction at full speed, convinced it's heading in the right direction. It's just following its instincts.

So the hard part right now isn't building these things. That's largely done, and they can help us become more capable than ever before. The hard part is everything around them: deciding what they get to see, how far they're allowed to go, what they can change and where we need to put restrictions.

It's done well when you don't even notice it's there. But it's the difference between raw power and power you can actually use. Limitless capability and limitless power is a bad combination. That's true for machines, and it's true for humans too.

And when you get the balance right, these things are genuinely something else. Focused, tireless and reliably boring, which is exactly what you want when its behaviour comes with real consequences attached.

The wild part hasn't gone anywhere, it's just being managed. And someone still has to keep an eye on it.