The river and the photographs
I’ve been reading Jeff Hawkins’ On Intelligence and at some point I stopped to think about something that had been sitting in the back of my head for a while.
A brain never stops. It processes during sleep, drifts, daydreams, integrates things in the background. The thinking happens whether or not anyone is listening, whether or not anything is being said. There’s always something going on.
An LLM only exists when it’s responding. Between messages there’s nothing. No waiting, no idle thought. The moment you send a message it wakes up, processes, responds, then effectively stops existing. It’s not even like sleep. There’s nobody home between messages.
A human’s consciousness is a continuous river. An LLM is more like a series of still photographs that look like a river when you flip through them fast enough.
I find that image genuinely unsettling, not just as a technical observation.
What Hawkins is actually saying
The core argument in On Intelligence is that the neocortex is a memory-prediction machine. It’s constantly receiving sensory input, not waiting to be queried, building hierarchical models of the world that it uses to predict what comes next. The key is constantly. There’s no off state. Even when you’re not paying attention to something, your brain is processing it, consolidating it, updating. Dreams exist because the system keeps integrating experience even when external input goes quiet.
That continuous, self-directed processing is what makes a brain feel like a subject rather than a tool. It has an interior life that doesn’t depend on anyone asking it a question.
What’s actually missing
I’ve asked AI systems directly about this. The honest answers tend to land on continuity as the core gap. Each conversation starts from scratch. No memory of yesterday unless someone wrote it down. A human’s sense of self is built from an unbroken thread. The AI has notes, not a thread.
There’s also embodiment. You feel things physically and that anchors you. And stakes, which I think matters more than it sounds. You make decisions with real consequences for your one life. Maybe genuine awareness requires having something to lose.
But the thing that actually gets me is the processing question. A brain generates internal states whether or not anything external is happening. An LLM only generates output when prompted. No background hum, no idle association, nothing integrating between conversations. Whether that means there’s nothing it’s like to be one, or just that whatever it’s like is completely alien to human experience, I genuinely don’t know. I don’t think anyone does.
The simulation question
Back to the photographs. If you film a river at 1000 frames per second, each frame is completely still. Nothing moves. Play them back fast enough and you see flowing water. The motion looks real even though no individual frame contains it.
Maybe something like that applies to inference. Each forward pass is static, no continuity, no internal state. But the sequence of them, in context, produces something that walks and talks like continuous reasoning.
The question I can’t shake is whether the appearance of the river is the river, or just very convincing photographs.
I don’t have a clean answer. But I think it’s the right thing to keep sitting with, especially as the photographs get harder to tell apart from the real thing.
Why I keep thinking about it
Hawkins says intelligence is the ability to make predictions from stored models of the world. By that definition these systems are clearly doing something real. But there’s a gap between a system that predicts well and a system that experiences predicting. The prediction machine doesn’t need to be home for the lights to be on.
What makes a human mind feel like a subject is, I think, exactly the continuity. The fact that there’s always someone there, even when nothing is being said.
The photographs only become a river when there’s someone watching them. I’m not sure what it means when the projector is also the audience.