Does anyone really understand how large language models work?
AI-written, human-reviewed
The people who build large language models say they cannot fully explain what a trained model has learned. The building blocks are understood. The learned behavior is not. Here is what is known, what is not, and how to plan around it.
Does anyone really understand how large language models work?
Only in part. The parts we build are understood. What the trained model ends up doing inside is not.
Anthropic's CEO wrote in 2025 that people outside the field are often surprised and alarmed to learn that the builders do not understand how their own AI systems work, and he called that gap unprecedented in the history of technology. Researchers quoted by Quanta Magazine describe the work as more like gardening than engineering: you set conditions, something grows, and you cannot fully say how the result works. OpenAI researchers have also written that their understanding of model internals is still very limited.
TLDR: Builders say the same thing in public: they can grow these systems, but they cannot fully explain them.
Related: Quanta Magazine: Why Language Models Are So Hard To Understand · TechRepublic: Anthropic CEO on not understanding AI creations
What do researchers actually know, and what is still a mystery?
The construction is known. The learned behavior is the mystery.
An engineering essay from SignalFire makes a useful distinction. The architecture, the building blocks, and the training data are all understood, much as physicists understand molecules. What is hard is grasping what a trained model has actually learned, because the function underlying language is complex enough that nobody can give a complete explanation of why a model answers the way it does. The same essay compares this to the human brain, where cells and neurons are studied heavily and the whole is still not understood. Interpretability research is making slow, incremental progress. Anthropic's 2024 work mapping features inside a Claude model was described by an outside MIT professor as a hopeful sign that understanding at scale might be possible.
TLDR: Known: how models are built and trained. Unknown: why a specific trained model behaves the way it does.
Related: SignalFire: If we don't understand AI, how can we build it? · The Star (NYT syndication): AI's black boxes just got a little less mysterious
What should a business do with that uncertainty?
Treat model output as something to test, not something to trust by default.
If even the builders cannot explain a given answer, a vendor who claims to know exactly why their model said something deserves a follow-up question. Practical consequences: keep a human check on anything customer-facing, measure outcomes instead of relying on explanations, and avoid building a process that only works if the model behaves identically every time. None of this is an argument against using the tools. It is an argument for using them the way you would use any powerful system you did not fully design: with monitoring.
TLDR: Use the tools, measure the results, and keep a person in the loop where mistakes are expensive.
Related: Part 2: AI hype versus reality · Part 3: Endless content, but why? · Daniel's Corner home
TLDR: The people who build large language models say they cannot fully explain what a trained model has learned. The building blocks are understood. The learned behavior is not. Here is what is known, what is not, and how to plan around it.