We don't understand AI

While reading Ursula K. Le Guin's The Farthest Shore, I was struck by this quote from the main character:

To claim power over what you do not understand is not wise, nor is the end of it likely to be good.

After thinking about AI all week at work, my brain connected that sentence to the concept of interpretability, our understanding of a model's inner workings. An interpretable model is one that humans can understand its decision-making processes. Is that the case for today's models?

People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned...
– Dario Amodei, Anthropic

We certainly have not solved interpretability
– Sam Altman, OpenAI

Fictional characters know better than to harness an incomprensible power and expect good. Why do we treat AI differently?

Aside: Moments like this are why you should read more and widely. There is no end to previous wisdom that can be shared and reapplied to today's problems.

Sean McPherson works as a software engineer at Khan Academy, and enjoys playing games and watching soccer. He lives with family in Pittsburgh, PA.