What Large Language Models Actually Do When You Type a Prompt

From Words to Tokens: The First Step

Before an LLM processes a single word you've typed, it converts your input into tokens — small fragments of text that serve as the model's basic unit of analysis. A token might be a full word like cloud, a partial word like un from unclear, or a punctuation mark. A typical English sentence breaks into roughly 1.3 tokens per word.

This tokenization step matters because the model never actually reads letters or words the way you do. It operates on numerical representations of these fragments. Each token is mapped to a point in a high-dimensional mathematical space — called an embedding — where similar concepts cluster together. The word king and the word queen end up geometrically close in that space, reflecting their semantic relationship. For a fuller glossary of these terms, the AI Glossary every non-expert needs covers embeddings, tokens, and more in plain language.

The Prediction Engine: How the Model Generates a Response

Once your prompt is tokenized and embedded, the model's core job begins: predicting what token should come next. This isn't a lookup in a knowledge database — it's a probability calculation. Given every token in your prompt (and the conversation history so far), the model assigns probabilities to thousands of possible next tokens and selects among the highest-ranked ones.

This process repeats token by token until the response is complete. The architecture that makes this possible — called a transformer — uses a mechanism called attention, which lets the model weigh how relevant each earlier token is when predicting the next one. That's what allows LLMs to keep track of the subject across a long paragraph rather than treating each word in isolation.

“These models are not doing anything like what we do when we understand language. They're doing something that produces similar-looking outputs through a very different process — one that is impressive precisely because it shows how much of what we call understanding can be replicated by pattern prediction at scale.”

— Emily M. Bender, Professor of Linguistics, University of Washington, and co-author of the influential 'Stochastic Parrots' paper

The key insight is that the model is not retrieving answers the way a search engine retrieves documents. It is generating them, one statistical step at a time. This is why the same prompt, asked twice, can produce slightly different responses — and why outputs can sound authoritative even when they're factually off-base.

What Training Actually Means

The fluency and apparent knowledge of an LLM comes from its training phase, during which the model was repeatedly exposed to enormous volumes of text — web pages, books, code, articles — and adjusted its internal parameters to become better at predicting text patterns. Modern large models are trained on hundreds of billions to trillions of tokens.

Training does not store facts like a filing cabinet. Instead, it shapes the statistical weights inside the network so that certain patterns of language reliably trigger certain patterns of response. The model learned that questions about photosynthesis tend to be followed by explanations involving chlorophyll and sunlight — not because it remembers a textbook, but because that pattern appeared consistently across its training data.

This also explains a key limitation: if something wasn't well-represented in training data, or if the real world has changed since training ended, the model has no reliable way to know. It will still produce a confident-sounding answer, which is why critical evaluation of LLM output matters. For a deeper look at how this dynamic plays out in productivity tools, see our analysis of AI writing assistants.

Why This Knowledge Changes How You Use AI Tools

Understanding the mechanics of an LLM reframes how to interact with one productively. Because the model is predicting plausible text rather than verifying truth, the quality and specificity of your prompt directly shapes what it has to work with. Vague inputs produce vague outputs — not because the model is being lazy, but because your prompt sets the probabilistic context it navigates.

It also explains why LLMs excel at tasks like summarizing, drafting, reformatting, and brainstorming — all activities where fluency and pattern-matching are the main requirements — and why they're less reliable for tasks requiring current data, precise citations, or logical deduction across many steps. The same mechanism that powers modern predictive text is at work here, scaled up dramatically.

When you type a prompt into an AI chatbot, you're not consulting a database or a reasoning agent. You're providing a statistical starting point for a very large, very well-trained prediction machine. That's both a powerful capability and a genuine constraint worth keeping in mind.