How AI Models Work: A Builder's Guide

An AI language model works by repeatedly predicting the next chunk of text. It converts your input into numbers, runs those numbers through a large stack of learned weights that lets every chunk look at every other chunk, produces a probability score for every possible next chunk, picks one, appends it, and runs the whole thing again. There is no database lookup, no reasoning engine, and no fact checker in the loop. Everything a model appears to know is compressed into those weights, and everything it appears to do is a consequence of that one prediction step running over and over.

That description sounds reductive. It is also load-bearing: nearly every surprising behavior you hit when building on these models falls directly out of it.

This guide traces one real request all the way through, then uses that single path to explain why the models behave the way they do and which knobs actually change the outcome.

The request we will follow

Take a prompt you might genuinely send:

Summarize this support ticket in one sentence:
"App crashes when I tap Export on the reports page."

Everything below happens to that exact text.

Step 1: Your text becomes tokens

The model does not see characters or words. It sees tokens, which are common chunks of text, roughly the length of a syllable or a short word. Frequent words are usually a single token. Rare words split into pieces. Punctuation and whitespace are tokens too.

As a working estimate, Anthropic's documentation puts one token at approximately four characters or 0.75 words in English. Our prompt above is roughly 30 tokens.

Tokenization is not a fixed universal scheme. Each model family ships its own tokenizer, and changing it changes the arithmetic. The same documentation notes that Claude 4.7 and later use a newer tokenizer that produces roughly 30% more tokens for the same text than earlier models. The text did not change. The counting did.

This is why token counts, and therefore costs and context limits, are model-specific rather than a property of your prompt. If you want the mechanics in depth, including why non-English text and code tokenize less efficiently, tokens have their own explainer.

Everything traced in this guide runs on text tokens. Models that also take an image as input tokenize patches of the picture alongside the words and attend across both in the same pass, rather than running a separate image system on the side; what is a vision-language model covers that variant specifically.

Step 2: Tokens become vectors

Each token is converted into a vector, a long list of numbers, typically several thousand of them. This is the embedding.

The important property is that these vectors carry meaning as geometry. Tokens used in similar contexts during training end up near each other. "Crash" sits closer to "freeze" and "error" than to "export" or "Tuesday." Nobody hand-coded that. It fell out of training on enough text that the arrangement which best predicts the next token happens to be one where related concepts cluster.

Position information gets added here too, because the raw vectors carry no notion of order. Without it, "app crashes when I tap Export" and "Export crashes when I tap app" would look identical to the model.

Step 3: Attention decides what matters

This is the part that made modern models work, and the part worth actually understanding.

At each layer, every token computes a relevance score against every other token in the input, then pulls in a weighted blend of their information. That is attention. It runs many times in parallel, in stacks of dozens of layers.

Concretely, in our ticket, the token for "crashes" attends strongly to "App," to "tap," and to "Export," because those establish what crashed and when. It attends weakly to "one" and "sentence," which belong to the instruction rather than the incident. Meanwhile the tokens in "Summarize this support ticket in one sentence" attend to each other to establish the task.

Nobody labeled which words were the instruction and which were the content. The model learned that the pattern "instruction, then quoted material" implies a relationship, because that pattern appeared constantly in training data.

Two consequences follow immediately, and both matter in production.

First, attention is why the boundary between a system prompt and a user prompt is soft. Both end up as tokens in the same sequence attending to each other. The separation is a convention the model was trained to respect, not a wall it cannot cross. That is also the root of prompt injection: text that arrives as data can read like an instruction, because at the mechanical level there is no difference.

Second, attention cost grows sharply with input length, since every token relates to every other. That is the pressure behind context window limits and why long inputs cost more and can degrade in quality toward the middle.

Step 4: The model produces probabilities

After the final layer, the model outputs a score for every token in its vocabulary, often 100,000 or more entries. These become probabilities summing to one.

For our request, after the model has emitted "The app", the distribution over the next token might look roughly like this:

This is the model's entire output at each step. Not a sentence, not an answer, one probability distribution over one next token.

Step 5: Sampling picks a token

Something has to choose from that distribution, and the choice is not part of the model. It is a separate step you control.

Always taking the highest-probability token is greedy decoding: repeatable, and prone to flat, looping text. Temperature reshapes the distribution before sampling. Low temperature sharpens it toward the front-runner, making output more deterministic. High temperature flattens it, admitting less likely tokens and producing more variation along with more errors.

This is precisely why the same prompt gives different answers on different runs. The model was not inconsistent. The sampler rolled differently.

Sampling is also where you can impose hard guarantees, which is underused. Constrained decoding masks out any token that would break a required structure before sampling happens. OpenAI's structured outputs documentation describes this directly: with a strict JSON schema supplied, schema compliance is enforced at the generation level, and the model cannot emit a token that would violate the schema. The model is not being asked nicely to return valid JSON. Invalid tokens are removed from the menu.

Step 6: Append and repeat

The chosen token is appended to the sequence, and the entire process runs again on the new, one-token-longer input. And again, until the model emits a stop token or hits a length limit.

Every token you see was generated with full knowledge of all previous tokens and zero knowledge of the ones that come after. The model cannot plan a sentence and then write it. It commits to each word before knowing how the sentence ends.

This explains a genuinely odd behavior: a model that starts an answer badly often continues badly, because its own wrong opening is now context it is conditioned on. It also explains why asking a model to think step by step before answering helps. The intermediate tokens become context for the tokens that follow, so reasoning written out is reasoning the model can actually use.

Where training fits in

Everything above describes inference, what happens when you send a request. The weights that make it work come from two distinct phases.

Pretraining runs next-token prediction across an enormous text corpus. This is where the model acquires grammar, facts, code patterns, and the statistical structure of language. It is expensive, slow, and produces a model that continues text rather than one that follows instructions.

Post-training turns that into something useful. Supervised fine-tuning on demonstrations teaches the assistant format. Reinforcement learning from human feedback tunes the model toward responses people rate as better. This phase produces most of what you experience as the model's personality, helpfulness, and refusals.

The split matters practically because it determines what you can change. Fine-tuning adjusts behavior and format effectively. It is a poor tool for installing knowledge, which is why retrieval-augmented generation exists: put the facts in the context at request time instead of trying to bake them into weights. Choosing between those approaches has its own decision guide.

What the model does not have

Reading the pipeline back, notice what never appears.

There is no lookup step. When a model states a fact, it did not consult anything. It generated the statistically likely continuation. Correct facts and confident fabrications are produced by the identical mechanism, which is the mechanical root of hallucination and the reason the model's confidence carries no information about accuracy.

There is no memory between requests. Each API call starts cold. Everything that feels like memory is your application resending prior turns as input.

There is no truth check. Nothing scores an output for correctness before it reaches you. If you need that, you build it.

There is no execution. Unless you have wired up tools, a model writing code has not run it, and a model doing arithmetic is predicting plausible digits.

Size, and why bigger is not always the answer

Model capability scales with parameter count, training data, and compute, but not uniformly across tasks.

Breadth of knowledge and hard multi-step reasoning depend most on scale. Narrow, well-specified tasks depend on it far less. A model with a tenth the parameters, tuned for one job, often matches a frontier model at that job for a fraction of the cost, which is the entire premise behind small language models.

Architecture varies too. Mixture-of-experts designs hold many parameters but activate only a subset per token, which is why a model can be enormous by total parameters and still cheap to run. This is also why total parameter count is a weak comparison metric between model families, and why open-weight and closed models are hard to compare on specifications alone.

What this means when you build

A few things follow directly, and they are the practical payoff of the whole picture.

Put the important material near the edges of your prompt. Attention distributes unevenly across long inputs, and content buried in the middle of a very long context gets less of it.

Constrain the output rather than requesting it. If you need structured data, use schema enforcement. A prompt asking for JSON is a preference. Constrained decoding is a guarantee.

Set temperature to the job. Extraction, classification, and anything you will parse want low temperature. Brainstorming wants higher.

Assume no memory and no verification. Pass the context you need every time, and validate anything you act on.

Test on your own data. Since capability varies by task in ways aggregate scores hide, a small set of real examples beats any leaderboard. That is the same argument laid out in how to read an AI benchmark claim.

None of this requires understanding the mathematics. It requires holding one accurate mental picture: a very good next-token predictor, running in a loop, with no access to anything you did not put in front of it.

The model at the center of that walkthrough is itself a foundation model. See what is a foundation model for how that category differs from a narrow, task-specific one.

Frequently asked questions

Do AI models understand what they are saying?

They build rich internal representations that track meaning, context, and relationships well enough to produce genuinely useful work, and they have no beliefs, intentions, or awareness of truth. Both halves are true, and arguments about the word "understand" usually come down to definitions rather than facts about the system.

Why does the same prompt give different answers?

Sampling. At each step the model produces a probability distribution and something picks from it, usually with randomness controlled by temperature. Set temperature to zero and outputs become close to deterministic.

Does the model learn from my conversations?

Not during the conversation. Weights are fixed at inference time. Whether your data is later used to train future versions is a policy question that varies by provider and plan, not a property of how the model runs.

Why is it confidently wrong instead of saying it does not know?

Because the same process produces both. There is no separate step that checks a claim against a source, and "I do not know" is just another sequence of tokens that has to win the probability contest against a plausible-sounding answer.

Do I need to know this to build with AI?

Not to start. It becomes valuable the moment something behaves unexpectedly, because almost every strange behavior traces back to one of these mechanics, and knowing them turns debugging from guesswork into diagnosis.

One architecture pattern worth understanding on its own: what mixture of experts actually means