A working intuition: good is not exhausted by computation

Does dinner with mom feel like a good use of time?

Not strategic, not productive, not efficient, nor optimal.

Just good.

Is it good enough to anchor whatever it is you’re doing instead against it?

Time is finite. The people we love won’t be here forever. This is an accessible truth we can (probably) agree on. 1

Here’s a small exercise:

(I stumbled on it while going through the first question in 36 Questions on the Way to Love.)

Think about what you’re choosing to do today instead of dinner with someone you love.

Now imagine that person lives in the next town. How much would you give up to have that dinner?

What if they live in another country?

Now imagine they’re gone entirely and you’re remembering the last dinner you didn’t have.

Something changes, doesn’t it? The dinner didn’t, but our ability to feel its worth did.

In this essay, I use good to name the choice I would still make while feeling the full weight of what I am giving up. The felt choice, not only the optimized one. When I am present at dinner, I am not calculating opportunity costs; I am attending to something that, once gone, cannot be recovered by rearranging a schedule.

Sometimes we don’t know what good feels like. It’s not uncommon for people to misread their internals.

We’re working on safety-relevant problems while time runs out on our human relationships. The formalization limits apply here too; we can optimize calendars (verifiable), we can work on important problems (validatable), but we can’t prove we allocated our finite attention to what will matter when we look back from an end-of-life point in time.

Can we build tools that help humans make some of these trade-offs better? If we had an AI that could predict E[remaining dinners with mom], would using it help us make more meaningful choices or make us more anxious?

AI has made idea generation cheap for many tasks, while evaluating those ideas remains costly. Technical checks can tell us whether something works as specified; they do not by themselves settle whether it is the right or good thing to build.

What if the AI’s predictions were wrong?

What if the predictions are right but we couldn’t verify them?

What if the predictions are right but we couldn’t really tell if that’s what’s good to build in the first place?

AI systems now perform some cognitive tasks at levels that change how I locate human intelligence in the world. Their competence is uneven: they fail in places I expect strength and surprise me in places I expect weakness. I take that as context for the questions below, not as a claim that human intelligence has been displaced in every relevant sense.

Jakob von Uexküll’s concept (1909): every organism has its own Umwelt, its subjective perceptual universe, shaped by its sensory apparatus and needs. Not “the world as it is,” but “the world as perceived through this body, these sensors, these purposes.”

Speculative analogy: mechanistic features might project something like an Umwelt onto a model’s representation space. If features fracture, one possibility is that our measurement categories stopped transferring, not that we discovered the model’s own subjective world.

Can we build tools that compare representations across different measurement systems? The stronger idea (that alignment requires a shared Umwelt, or that such sharing is structurally impossible) is a philosophical question here, not a result.

  1. Maybe this will not be true for all of human existence, but it seems to be a limitation of today’s experience. Until we reach whatever form of life expansion we settle on, I’ll try to deal with the feebleness of being a biological human. ↩