How an LLM works: tokens, attention, sampling and limits

When an engineer asks me how an LLM works, "it guesses the next word" is accurate, but it does not help them decide anything: why the bill explodes on Arabic text, why the model invents a reference, why a 128,000-token context does not cost "just" 16 times more than an 8,000-token one. This article starts from the general-audience version I originally wrote (neuron, network, most probable word) and goes one level down: the full mechanism, with the formulas that matter and orders of magnitude that I compute in front of you. I flag what I simplify. You do not need prior work on LLMs, only some comfort with vectors and matrices.

So what is AI, really?

A classic program follows rules written by a human: "if the amount exceeds 10,000, ask for approval". Machine learning reverses the approach. You provide examples and a parameterized function, and an optimization procedure looks for the parameters that best reproduce those examples. For a spam filter, you do not describe what spam is: you show thousands of labeled emails, and the parameters adjust until the function separates them correctly. It is like a child learning to recognize a cat: nobody recites a definition, you correct them, and one day they recognize a cat they have never seen. The comparison has limits (a child generalizes from very few examples, a model does not), but it gives the right intuition.

Four words to know

• Data: the examples shown to the machine (text, images, measurements).
• Training: the optimization that adjusts parameters to reduce an error measured on those examples. Long and expensive, but done once per model version.
• The model: the resulting function, that is, an architecture plus a set of frozen parameters.
• Inference: using the trained model, for example when you send a prompt. The parameters no longer move.

The artificial neuron, a tiny building block

An artificial neuron is a tiny function. It receives numbers, multiplies each by a weight (its importance), adds everything up with a bias, then passes the result through a nonlinear function:

output = f(w1*x1 + w2*x2 + w3*x3 + b)

Back to the umbrella example: the inputs are "cloudy sky", "the forecast says rain", "it is November", and the forecast weighs more than the color of the sky. The nonlinearity f (ReLU, GELU, sigmoid...) is not a detail: without it, stacking layers would be mathematically equivalent to a single matrix multiplication, and the network could only represent linear relationships.

Neural networks: strength in numbers

A single neuron does almost nothing. Organized in layers, with millions of connections, they become expressive: each layer transforms the representation produced by the previous one. On a photo, the first layers respond to edges, the next ones to textures and object parts, the last ones to the whole object. This hierarchy is not programmed, it emerges from training (a convenient description, not a law: in large models, the roles of layers are less clear-cut).

Learning fits in one loop. You feed in an example, measure the gap between the output and the right answer with a loss function, compute by backpropagation the gradient of that loss with respect to each weight, then move each weight by a small step in the direction that reduces the error:

w <- w - eta * dL/dw        (eta: learning rate)

The mixing console with millions of knobs from my original version is exactly this: each knob is turned very slightly after each batch of examples. When a network has many layers, we talk about deep learning. It took off about fifteen years ago thanks to two resources: more data and GPUs able to multiply matrices massively in parallel.

LLMs: predicting the next token

An LLM (Large Language Model) is a neural network with a Transformer architecture, trained on huge amounts of text and code with a single objective: given the previous tokens, predict a probability distribution over the next token. Nothing else during pre-training.

In this example the distribution covers four words. In reality it covers the model's whole vocabulary (tens of thousands of tokens), and what the model handles is not words but tokens. The complete path, from text to distribution, is as follows.

From text to tokens

A model reads neither characters nor words, but tokens, fragments produced by a learned segmentation. The most widespread algorithm is BPE (byte-pair encoding) and its variants: you start from bytes or characters, then iteratively merge the most frequent pairs in the corpus until a fixed vocabulary size is reached, roughly 30,000 to over 250,000 entries depending on the model. A common word becomes a single token, a rare word splits into several subwords.

Why subwords? It is a compromise. Characters give very long sequences (and the cost of attention grows with length). Whole words give a huge vocabulary and no way to handle a word never seen before. Subwords keep a reasonable vocabulary while covering any text.

Two very concrete consequences for an engineer:

• Cost and latency are counted in tokens, not words. The number of tokens per word depends on the tokenizer and the language. A vocabulary learned mostly on English splits less-represented languages more finely. Arabic, with its rich morphology (roots, agglutinated prefixes and suffixes), is often affected: the same content can consume noticeably more tokens than in English, so it costs more and fills more of the context window. The gap varies a lot from one tokenizer to another and it must be measured, not assumed.
• Numbers and code are sometimes split in surprising ways, which affects arithmetic (more on this in the limits section).

Measuring is easy:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")  # one tokenizer among others
for text in ["The sky is blue", "Le ciel est bleu", "السماء زرقاء"]:
    print(len(text.split()), "words ->", len(enc.encode(text)), "tokens")

Compare on your own texts, with the tokenizer of the model you actually use: each provider has its own.

Embeddings: from identifiers to vectors

An integer ID means nothing to a network. The first stage of the model is a lookup table, the embedding matrix, of size V x d (V vocabulary tokens, d dimensions), which maps each token to a learned vector. During training, tokens used in similar contexts end up with nearby vectors: this is what lets the model generalize from "cat" to "kitten" without an explicit rule.

An order of magnitude, calculated: with V = 128,000 and d = 4,096, this table alone weighs 128,000 x 4,096, about 524 million parameters, before the first layer.

One detail that matters: the attention mechanism described below does not account for token order by itself. Position information is therefore injected in addition (sinusoids in the original paper, rotary encodings in many recent models).

The Transformer and the attention mechanism

The Transformer architecture was introduced in 2017 in the paper Attention Is All You Need. An LLM-style model is a stack of L blocks with the same structure. Each block contains two sub-layers, each wrapped in a residual connection (the input is added to the output) and a normalization:

1. an attention layer, where each token fetches information from the other tokens;
2. a small feed-forward network (MLP), applied independently at each position, which transforms each vector.

The intuition. In "The bank refused the loan because it was too risky", what does "it" refer to? To represent that word correctly, the model must consult other positions in the sentence. Attention is a weighted lookup over the sequence: each token formulates a query, compares it to a key specific to every other token, then retrieves a mix of values, weighted more heavily where the key matches.

The formalism. From the matrix X of input vectors (one row per token), you compute three learned linear projections: Q = X·W_Q (queries), K = X·W_K (keys), V = X·W_V (values). Then:

Attention(Q, K, V) = softmax( Q·K^T / sqrt(d_k) ) · V

Let us read it for what it is. Q·K^T is an n x n matrix of dot products: the score of every pair (token i, token j), for n tokens. We divide by sqrt(d_k) (d_k: key dimension) because the variance of a dot product grows with dimension: without this scaling, scores get large and the softmax saturates, with near-zero gradients. The softmax, applied row by row, turns each row into positive weights summing to 1. Finally, multiplying by V produces, for each token, a weighted average of the values. It is nothing more than a weighted average whose weights are computed from content.

Two important clarifications:

• In a generative model, a causal mask is applied: token i may only look at positions j <= i, otherwise it would "see the future" during training.
• There is not one attention but several in parallel (multi-head), each with its own projections, in different subspaces. Their outputs are concatenated and projected back.

A useful estimate: a block holds about 12·d² parameters (about 4·d² for attention, 8·d² for the MLP). For a published configuration like GPT-3's (d = 12,288, 96 layers), 12 x 12,288² x 96 gives about 174 billion, very close to the announced 175 billion. The embedding matrices and architectural details explain the rest of the gap: it is an estimate, not a count.

Logits, softmax, and choosing the token

At the output of the last layer, the last token's vector is projected onto the entire vocabulary: you get one real-valued score per token, the logits. They are not probabilities (they can be negative and do not sum to 1). The softmax converts them:

p_i = exp(z_i) / sum_j exp(z_j)

Take illustrative logits for "The sky is": blue 3.0; grey 1.9; cloudy 1.3; beautiful 0.7. The exponentials are 20.1; 6.7; 3.7; 2.0, for a sum of 32.5. Hence the probabilities 61.9%; 20.6%; 11.3%; 6.2%, which is the opening histogram. The softmax is monotonic: the token with the highest logit is always the most probable, but the probability gap depends exponentially on the logit gap.

Generation is an autoregressive loop

The model produces only one token per pass. To write an answer, you repeat: compute the distribution, pick a token, append it to the input, start again, until an end-of-sequence token or a length limit.

Two engineering consequences. First, response time is roughly proportional to the number of generated tokens: this is why long answers are slow and why APIs bill input and output separately. Second, recomputing the keys and values of all previous tokens at every turn would be wasteful, hence the KV cache: you keep K and V of already-processed tokens and only compute those of the new token. This cache has a memory cost: for an illustrative configuration (32 layers, 8 key/value heads of dimension 128, 16-bit numbers, 8,192 tokens), you get 2 x 32 x 8,192 x 8 x 128 x 2 bytes, that is 1 GiB per active sequence. Multiply by the number of simultaneous requests: GPU memory is contested between the model weights and these caches.

Sampling: temperature, top-k, top-p

What remains is choosing the token from the distribution. Always picking the most probable one (greedy decoding) is deterministic but can give repetitive, flat text; drawing at random according to the probabilities gives more varied text. Three common settings:

• Temperature T divides the logits before the softmax. T < 1 sharpens the gaps (a more peaked distribution), T > 1 flattens them (more diversity, more risk), and T tending to 0 amounts to greedy decoding.
• Top-k keeps only the k most probable tokens before renormalizing.
• Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability reaches p. It adapts to the shape of the distribution: few candidates when the model is confident, more when it hesitates.

import math, random

def sample(logits, T=1.0, top_k=None, top_p=None):
    """logits: dict token -> score. Simplified version, no optimization."""
    scaled = {t: z / T for t, z in logits.items()}
    m = max(scaled.values())                      # numerical stability
    exp = {t: math.exp(z - m) for t, z in scaled.items()}
    total = sum(exp.values())
    probs = sorted(((t, e / total) for t, e in exp.items()),
                   key=lambda x: -x[1])
    if top_k:
        probs = probs[:top_k]
    if top_p:
        kept, cum = [], 0.0
        for t, p in probs:
            kept.append((t, p))
            cum += p
            if cum >= top_p:
                break
        probs = kept
    z = sum(p for _, p in probs)                  # renormalization
    r, acc = random.random() * z, 0.0
    for t, p in probs:
        acc += p
        if r <= acc:
            return t
    return probs[-1][0]

logits = {"blue": 3.0, "grey": 1.9, "cloudy": 1.3, "beautiful": 0.7}
print(sample(logits, T=0.5, top_p=0.9))

With the logits above, here is what the probability of "blue" becomes as a function of T (direct softmax computation):

A low temperature does not make the model "more truthful", only more deterministic. And even at T = 0, some APIs do not guarantee strictly identical outputs from one call to the next (floating-point operation order on GPUs, request batching): do not build a test on exact equality of generated text.

Context window and quadratic cost

The context window is the maximum number of tokens (input plus output) the model processes at once. In the naive version of attention, the Q·K^T matrix is n x n: the compute and memory of that term grow with the square of the length. At 8,000 tokens, that is 64 million scores per head and per layer; at 128,000 tokens, about 16.4 billion, which is 256 times more for a sequence 16 times longer.

Implementations such as FlashAttention avoid storing the whole matrix in memory, and various variants (sliding-window attention, sharing key/value heads) reduce the cost, but exact computation remains quadratic in time. Filling the window is therefore never free. And a large window does not mean the model uses everything you put in it equally well: several studies have observed that how well information is used depends on its position in the context. Above all, remember that the window is a budget to manage, not a bottomless disk.

How a model is built

The assistants you use are the product of several successive phases, which do not optimize the same thing.

Pre-training. The loss is cross-entropy: you minimize -log p(actual token | previous tokens) at every position. This phase accounts for almost all the cost. A common approximation of the compute required is 6 x N x D floating-point operations (N parameters, D training tokens). DeepMind's Chinchilla study concluded that, for a fixed compute budget, model size and amount of data should grow together: a 70-billion-parameter model trained on 1,400 billion tokens (20 tokens per parameter, the rule of thumb often retained) outperformed much bigger but less-fed models. For 70 billion parameters and 1.4 trillion tokens, 6 x 70e9 x 1.4e12 gives about 5.9e23 operations. Many recent models are trained on far more data than that ratio, because a smaller, longer-trained model is cheaper at every inference.

Supervised fine-tuning (SFT). The base model completes text, it does not answer a question. It is retrained on example conversations (instruction, good answer), with the loss computed on the answer tokens. What is learned here is a behavior and a format more than knowledge.

Alignment. Training starts from comparisons: a human, or sometimes another model, indicates which of two responses is preferable. Either you learn a reward model and then optimize the model with reinforcement learning (RLHF), or you optimize directly on the preference pairs (DPO and variants). More recently, reinforcement learning on verifiable tasks (code that passes tests, an exact math answer) is used for so-called reasoning models; I will not go into details, which vary by vendor and are not always public.

What "parameters" means

Parameters are the learned numbers: the weights of the embedding, attention and MLP matrices. A "70-billion-parameter" model is a file of 70 billion numbers. That translates directly into memory: at 2 bytes per parameter (16-bit precision), 70 billion is about 140 GB; quantized to 4 bits, about 35 GB, plus headroom for caches and activations. That decides the hardware needed to host it, a subject I cover in the article on local AI.

Orders of magnitude: a few billion to about ten billion for small models that run on a workstation; several tens of billions for mid-size models; 175 billion for GPT-3, which served as a reference; several hundred billion for the largest open-weight models. The size of recent proprietary models is generally not published. A higher parameter count does not automatically mean a better model: the quality and quantity of data and the tuning phases weigh just as much.

Why hallucinations are a consequence of the mechanism

A hallucination is a fluent, confident but wrong answer. It is not an isolated bug that you would fix line by line, it is the direct consequence of what the model optimizes.

• The objective is plausibility, not truth. During pre-training, no step distinguishes an accurate fact from a plausible sentence. The model learns the distribution of text, and false but well-formed text is probable.
• There is no "I don't know" state by default. At each turn, the softmax produces a distribution that sums to 1 and a token is chosen. Without knowledge, the distribution is flatter, but a token comes out anyway. Only training (SFT, alignment) teaches the model to abstain, imperfectly.
• Rare facts are poorly memorized. A bibliographic reference or a statute article number seen once in the corpus is compressed lossily into the parameters. The model reconstructs "something that looks like a reference": right format, right kinds of words, invented content.
• Errors feed themselves. A wrong token, once emitted, is part of the input of the following turns. The model continues coherently with its own mistake.
• Sampling adds risk. A high T or a wide top-p gives unlikely tokens a chance, but lowering T does not remove the problem, because the most probable token may already be wrong.

You can reduce hallucinations (put sources in the context, ask for verifiable citations, validate outputs with code, evaluate on a representative set of cases) but not eliminate them by construction. Design the application on the assumption that an answer can be wrong and that the system must be able to detect it or limit its impact.

Reasoning, calculation, dated knowledge

• Calculation. An LLM has no exact arithmetic module: it predicts digit tokens, with a number segmentation that depends on the tokenizer. It does well on common cases and fails unpredictably on long or rare operations. For any computation that must be right, call code or a tool.
• Reasoning. Having the model write intermediate steps (chain of thought) gives it more compute per answer and often improves results on multi-step problems, but those steps are neither a proof nor a faithful trace of what was computed. I treat them as a way to get better answers, not as a reliable explanation.
• Dated knowledge. Knowledge is frozen at the end date of the training corpus (the knowledge cutoff). A model knows neither the latest version of a library, nor your database, nor last week's event, unless it is supplied in the context.

Prompting, RAG or fine-tuning: what each one changes

These three levers act in different places, and they are often confused.

• Prompting means formulating the input better: clear instructions, the expected output format, a few examples (few-shot). The model learns nothing durably, it simply conditions its distribution on what you give it. It is the first lever to try because it costs nothing to change, but it consumes tokens on every call.
• RAG (retrieval-augmented generation) looks up, at question time, relevant passages in your documents, then places them in the prompt with the question. It addresses the problem of dated or private knowledge and lets you cite sources. Its quality is that of the search: if the right passages are not retrieved, the model will answer without them. I detail the architecture in the article on AI and your documents.
• Fine-tuning continues training on your examples, modifying the weights, often through low-rank adapters (LoRA) that train only a small fraction of the parameters. It suits fixing a style, an output format, or specializing the model on a narrow task. It is unreliable, however, for making it retain precise facts, and it requires building a dataset and an evaluation set. It is my last resort, not my first.

My usual order of work: prompting and evaluation first, RAG if the problem is missing knowledge, fine-tuning if the problem is behavior and the evaluation shows that the first two are not enough.

Thinking fast, thinking slow

A detour through psychology helps place the models. Daniel Kahneman, in Thinking, Fast and Slow, distinguishes fast, intuitive thinking (System 1) from slow, deliberate thinking (System 2). LLMs used as assistants, especially when they produce reasoning before answering, resemble System 2: slow, expensive, able to produce free text. Yet many decisions in an information system are System 1: is this email a complaint, which team should handle it, is this transaction suspicious. These are quick questions, repeated thousands of times, with a set of answers known in advance.

SOMs

This is the kind of decision targeted by a category announced in September 2026: SOMs (System One Models). The first representative, Jev, comes from the company TypeSafe. According to descriptions published by its vendor and relayed by the specialist press, it does not generate text: you give it a state (an email, a case file) and questions whose possible answers are fixed by a schema (choice from a list, score on a scale, yes/no), and in a single pass it returns an answer and a confidence per question. The vendor announces very large gains in speed and cost compared with LLMs; I found only vendor figures and second-hand write-ups, and no independent evaluation. I therefore take them as promises to verify on your data, not as established facts. As with any model, how often it picks the wrong box cannot be deduced from a brochure.

The underlying principle is verifiable by any engineer: a decision accompanied by a probability lets you choose a threshold. At 98% confidence, the software acts alone; at 55%, the case goes to a human. This only makes sense if the confidence is calibrated, meaning that among answers given with 90% confidence, about 90% are right. This is measured on a validation set, and it is the first thing I would check before plugging such a model into a real flow. You can also approximate this logic with a standard LLM, by constraining the output to a schema and reading the probabilities of the answer tokens when the API exposes them, but calibration is not guaranteed and must be measured the same way.

The right tool for the right task

The useful lesson is not which model is "the best", but how to compose them:

1. A classification model (a SOM or another) sorts. It routes and quickly flags unusual cases.
2. Classic software applies the rules. Calculations, dates, validations: everything that must be 100% exact.
3. The LLM handles open-ended cases. Writing, analyzing a document, explaining, with sources in the context when accuracy matters.
4. A human decides when confidence is low or the stakes are high.

What to remember

• An LLM is a function that, at each turn, converts a sequence of tokens into a probability distribution over the next token, through embeddings, attention and MLP layers, logits and softmax.
• Cost is played out in tokens (language included), in context length (quadratic attention in the naive version, KV cache in memory) and in answer length (one pass per token).
• Training happens in phases: pre-training (competence), supervised fine-tuning (format), alignment (preferences).
• Hallucinations follow from the plausibility objective: they are reduced with sources, tools and evaluation, not eliminated.
• Prompting, RAG and fine-tuning do not solve the same problems: missing instructions, missing knowledge, missing behavior.

If you had to start with just one thing, I suggest measuring the token count of your own texts (in your languages) and building a small evaluation set from your real cases. These two measurements inform more decisions than a public benchmark. And if a point seems oversimplified, it probably is: the mechanism is stable, but the details (position encodings, attention variants, alignment recipes) change quickly from one model to the next.

References

• Vaswani et al., Attention Is All You Need (2017): https://arxiv.org/abs/1706.03762
• Brown et al., Language Models are Few-Shot Learners (GPT-3, 2020): https://arxiv.org/abs/2005.14165
• Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, 2022): https://arxiv.org/abs/2203.15556
• Holtzman et al., The Curious Case of Neural Text Degeneration (nucleus sampling, 2019): https://arxiv.org/abs/1904.09751
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020): https://arxiv.org/abs/2005.11401
• DataCamp, article on Jev and System One Models (secondary source, vendor figures): https://www.datacamp.com/blog/system-one-models-jev