Tokens: The Invisible Currency of Every AI Conversation

You just asked an AI a question. Before it even started thinking — before it formed a single word of its reply — it did something you never saw: it counted. Every letter, every space, every punctuation mark in your message was quietly broken down into tiny units called tokens. And only then did the thinking begin.

Tokens are the invisible currency of every AI conversation. You don't see them, you don't spend them directly, but they determine what the AI can remember, how much it costs to run, and even how smart it seems on a good day versus a bad one. Understanding tokens is like understanding what's actually happening under the hood of the AI tools millions of people use every day.

So What Exactly Is a Token?

A token is not a word. It's not a character either. It's something in between — a chunk of text that an AI model has learned to recognise as a meaningful unit.

In English, the rough rule of thumb is that one token equals about three-quarters of a word. So a sentence with 100 words contains roughly 130 tokens. But it gets more interesting when you look at specific examples. The word "ChatGPT" is two tokens — "Chat" and "GPT." The word "unbelievable" might be three tokens. A single full stop "." is one token. The word "I" is one token. "Anthropic" is three tokens: "Ant," "hrop," and "ic."

Why such strange splits? Because AI models learn tokenisation from massive amounts of text data, and they break words down based on how frequently certain letter combinations appear together. Common short words stay whole. Rare or long words get fragmented. Numbers, punctuation, and emojis follow their own quirky rules.

Here's a fun way to feel it: the sentence "The quick brown fox" is 7 tokens. "Supercalifragilisticexpialidocious" is probably 10 or more. And "lol" — just three letters — is one single token, because the internet has used it so often that every model knows it by heart.

How Tokens Flow Through a Conversation

Here's where it gets genuinely fascinating — and a little counterintuitive.

When you send a message to an AI, you might imagine the AI just reads your latest message and replies. That's not what happens. What actually gets processed is the entire conversation from the beginning — your first message, the AI's first reply, your second message, its second reply, and so on, all the way to your latest question. On top of that, there's usually a hidden "system prompt" — instructions set by the app developer that tell the AI how to behave, what persona to adopt, what topics to avoid. All of this gets fed in together, every single turn.

So if you've been chatting with an AI for an hour, and the conversation has grown to 10,000 words, then every single reply the AI generates requires processing all 10,000 words plus your new message plus the system prompt. The token pile keeps growing with every exchange.

This is why AI feels "expensive" to run. It's not just answering your latest question — it's re-reading the entire conversation history each time.

The Context Window: Why AI Has a Memory Limit

Every AI model has what's called a context window — the maximum number of tokens it can hold in its "working memory" at once. Think of it like a whiteboard. The AI can only see what's written on the whiteboard. Once it's full, something has to be erased to make room for new information.

Context windows have grown dramatically in recent years. Early models had windows of 4,000 tokens. Today, frontier models like Claude can handle up to 200,000 tokens — roughly the length of a short novel. GPT-4 variants handle 128,000. These are genuinely impressive numbers.

But they do have a limit. And when you hit it, the AI starts quietly forgetting the beginning of your conversation. It might forget the task you defined in your first message. It might lose track of a character's name from the story you were co-writing. It might seem to contradict something it told you an hour ago — because that part of the conversation has already scrolled off its whiteboard.

This is why very long chats sometimes feel like the AI is "going dumb." It's not dumb — it just literally cannot see the earlier parts of the conversation anymore. The context window is a hard architectural limit, not a bug.

Does It Cost Real Money?

For companies and developers using AI through an API, yes — absolutely. Cloud AI services charge per token, both for input (what goes in) and output (what comes back). Pricing varies by model, but to give you a real sense of scale: a short casual chat of a few exchanges might cost a fraction of a cent — around $0.001. A long research session with document uploads and detailed back-and-forth might run $0.05 to $0.20. Running an AI assistant for thousands of users a day adds up to real infrastructure costs fast.

For individual users on consumer apps — ChatGPT, Claude, Gemini — the token costs are bundled into your subscription or absorbed by the company on the free tier. You don't see a per-token bill. But the economics are very real behind the scenes, which is why free tiers have usage limits and why "Pro" plans exist.

The token economy is also why model providers are constantly optimising. A model that can give the same quality answer using fewer output tokens saves money at massive scale. Efficiency matters just as much as intelligence.

Why This Actually Matters for You

Understanding tokens changes how you use AI — and makes you better at it.

First, shorter, clearer questions get better answers. Not because the AI rewards brevity, but because a crisp, focused question gives the model a clean signal. Vague, rambling prompts add noise. They use more tokens and often produce more generic responses.

Second, pasting giant documents into a chat is costly in more ways than one. A 50-page PDF might eat 20,000–30,000 tokens of context space, leaving much less room for the actual back-and-forth conversation. If you're doing serious document analysis, tools with dedicated document processing pipelines handle this better than just pasting text into a chat box.

Third, starting a new chat is sometimes the smartest move. If your conversation has gone on long enough that the AI seems to be losing the thread, a fresh session with a clean summary of what you need is often more effective than continuing in a bloated context.

And finally, system prompts matter. If you're building something with AI — a chatbot, an assistant, an automated workflow — the tokens in your system prompt get charged on every single API call. A bloated system prompt that could be half the length is literally costing double with every interaction.

Token Limits Across Popular Models

Different AI models come with different context windows — the maximum number of tokens they can handle in a single conversation. This number has grown dramatically over the past few years, and it's one of the clearest ways to compare models.

GPT-3.5, one of the earlier widely-used models, had a context window of around 4,000 tokens. That's roughly 3,000 words — enough for a decent conversation, but it would fill up quickly with long documents. GPT-4 extended this to 8,000 tokens in its standard version, with a 32,000-token variant available for power users. OpenAI's later models pushed this further still.

Anthropic's Claude models have made context window size a defining feature. Claude 3 and later versions support up to 200,000 tokens in a single context — that's roughly 150,000 words, or an entire novel. Google's Gemini 1.5 Pro similarly offers a million-token context window, which is long enough to load dozens of books at once.

But a large context window doesn't mean unlimited memory or perfect recall. Research has consistently shown that AI models tend to pay more attention to content at the beginning and end of a long context, with the "middle" sometimes receiving less focus. So even with a 200,000-token window, how you structure information within it still matters.

Tokens and the Future of AI

The token is a foundational concept in modern AI, but it's not fixed in stone. Researchers are actively working on alternatives — newer architectures that might process language more like how humans actually read and think, rather than through discrete token chunks. Some experiments use pixel-based or byte-level inputs that bypass tokenisation entirely.

For now, though, tokens remain the common currency. Every time a model generates a response, it does so one token at a time — predicting what the most likely next token is, then the one after that, and so on. The fluency, coherence, and apparent intelligence of the output emerge from billions of these tiny token-level predictions happening in sequence.

Understanding this doesn't make AI less impressive. If anything, it makes it more so. The fact that stringing together token predictions — essentially a very sophisticated form of autocomplete — can produce legal analysis, creative writing, working code, and empathetic conversation is one of the more remarkable things happening in technology right now.

A Fun Closer to Leave You With

This blog post is approximately 900–1,000 tokens long. If you were to copy it and paste it into a conversation with an AI — asking the AI to summarise it, critique it, or continue it — the model would process every one of those tokens before generating its first word of reply.

The AI reading this post to answer your next question will add another 300–500 tokens of its own response on top. And if you then reply, and it replies back, that entire exchange gets carried forward into the next turn.

It's turtles all the way down — or rather, tokens all the way down.

The next time you chat with an AI, spare a thought for the quiet arithmetic happening before the first word appears. Every conversation is a river of tokens, flowing through silicon, being counted, processed, and transformed into something that feels — remarkably, improbably — like thought.