Tokens

How many pieces is the word “strawberry”? Keep your guess in mind.

A chatbot like ChatGPT does not take in text letter by letter. First, the text is cut into small pieces called tokens. A piece can be a whole word, part of a word, or a single letter.

Chatbots like ChatGPT run on language models. A language model is a program trained on a huge amount of text. It works out a likely next piece of text.

OpenAI’s tiktoken library puts it this way: “Language models don’t see text like you and I, instead they see a sequence of numbers (known as tokens).” Google’s course on language models says a token “could be a word, a subword (a subset of a word), or even a single character.” It adds that “Most modern language models tokenize by subwords”.

Each piece has its own number, called its token ID. The language model works with the list of numbers, not the letters.

A tool called a tokeniser does the cutting. This lesson uses one called o200k_base. OpenAI, the company behind ChatGPT, publishes a free tool called tiktoken. It lists o200k_base for several OpenAI models, such as GPT-4o.

The o200k_base tokeniser cuts “strawberry” into 3 pieces: “st”, “raw” and “berry”. Their numbers are 302, 1618 and 19772.

A tokeniser has a fixed list of pieces, called its vocabulary. The o200k_base vocabulary has 199,998 pieces for text, about 200,000. Beside them, OpenAI defines 2 special tokens that mark where text ends or where a prompt ends.

A token ID is only the piece’s place in that list. It says nothing about meaning or size. In o200k_base, ID 0 is the exclamation mark “!”, ID 1 is a double quote mark, and ID 2 is “#”.

A space or a capital letter makes a different piece. In o200k_base, “cat” is 8837, “ cat” with a space before it is 9059, and “Cat” is 23546.

Every split in this lesson was measured on 26 and 27 September 2026: the numbers with gpt-tokenizer 4.0.0, a library that runs o200k_base, and each piece from OpenAI’s own vocabulary files.

Different tokenisers can cut the same word in different ways, even two from one company.

An older tokeniser, cl100k_base, is listed for GPT-4. It cuts “strawberry” into “str”, “aw” and “berry”, with other numbers. So every split in this lesson names its tokeniser.

OpenAI’s tiktoken library lists cl100k_base for GPT-4 and GPT-3.5-turbo. Its vocabulary has 100,256 pieces for text, about 100,000. The same piece gets a different number in each tokeniser: “berry” is 19772 in o200k_base and 15717 in cl100k_base.

People made these tokenisers with a method called byte pair encoding. It picks its pieces from data, not from grammar. It gives common runs of characters their own piece, so common parts of words often get one. The tiktoken README says the method “works on arbitrary text, even text that is not in the tokeniser’s training data”. It is also “reversible and lossless”: the pieces always join back into the exact text.

So the pieces do not always match the parts of a word a person would see. The o200k_base tokeniser cuts “unbelievable” into “un”, “bel” and “ievable”. The cl100k_base tokeniser cuts it into “un”, “belie” and “vable”. Google’s course says a tokeniser “might” cut “cats” into “cat” and “s”. The o200k_base tokeniser keeps “cats” as one piece.

A piece can even be part of one character. A tokeniser works on the bytes a computer stores for the text, and one character can take more than one byte. The emoji of a student is 6 pieces in o200k_base. Five of them are only part of a character. The sixth is a whole character, an invisible joining mark.

For English only, Google’s course gives a rough rule: one token is about 4 characters, or about three quarters of a word. It says the number of characters per token differs across languages.

The pieces matter in three ways: letters, languages, and how much text fits.

The o200k_base tokeniser puts the three r’s in “strawberry” into two pieces. One is in “raw” and two are in “berry”. The model gets the pieces’ numbers, not the letters.

In a study published in December 2024, eight models counted the letters in 10,000 words. Most got more than half of the words wrong. The best, GPT-4o, got about 1 in 6 (17%) wrong. The researchers found the counting itself was the harder part, more than the pieces.

The OECD (Organisation for Economic Co-operation and Development) and the European Union wrote an AI framework. One sentence from it is 16 pieces in English with o200k_base.

The same sentence in its Irish-language edition is 30 pieces, 14 more. So the same text can take more pieces in another language. That is this sentence with this tokeniser, not a rule.

More pieces take longer to process and can cost more. They also fill up the room a model has for text. Lesson 1.8 shows how much text fits.

The study (Fu and others, published December 2024) tested GPT-4o, GPT-4o-mini, Llama 3.1 8B and 70B, Llama 3.2 11B, Mistral-7B, Mixtral-8x7B and Gemma-2-9B. It reported the same split of “strawberry” for GPT-4o’s tokeniser. It found that “models are capable of recognizing the letters but not counting them”. Errors rose sharply when a letter came more than once in a word. The authors concluded that “tokenization does not play a fundamental role” in counting.

A larger benchmark, CharBench (first posted in August 2025, revised in April 2026), tested seven models in English only: GPT-4o, GPT-4o-mini, GPT-3.5-turbo, DeepSeek-V3, Llama-3.3-70B, Llama-3.1-405B and Mistral-7B. For counting, it found that tokenization properties are “weakly correlated with correctness”. For saying where a letter sits in a word, models did worse when the letter was inside a longer piece.

Each study tested only the models it names. The first was published in December 2024. The second was first posted in August 2025 and revised in April 2026. Newer models may do better at counting letters.

The two sentences come from printed page 7 of the framework and of its official Irish edition. The English is “While AI can generate outputs that resemble human interactions, AI itself is not human.” With the older cl100k_base tokeniser, they are 16 and 44 pieces.

The framework’s short sentence “AI is not human.” and its Irish translation, “Ní duine é IS.”, are 5 pieces each in o200k_base. In cl100k_base they are 5 and 7. So the difference depends on the sentence and on the tokeniser.

A study of 2023’s tokenisers (Petrov and others, NeurIPS 2023) found that the same text, translated, can take very different numbers of tokens in different languages, “with differences up to 15 times in some cases”. The authors say this can make “commercial language services” cost more for some language communities, make them slower, and let less text fit.

Why Irish: the Higher Education Authority (HEA) asks colleges to check how AI tools work in Irish. Lesson 3.2 will look at that.

A language model does not take in text the way a person reads it. A tokeniser cuts the text into pieces, and the model gets a list of numbers.

That is one difference, and only one. It does not show that a model understands nothing. Lesson 1.1 covers two mistakes: treating a chatbot as a mind with feelings, and saying it understands nothing.

Lesson 1.4 shows how a chatbot picks a likely next piece.

The English sentence in this lesson is the framework’s own words: “While AI can generate outputs that resemble human interactions, AI itself is not human.”

Some people think a chatbot may have a mind. In July 2023 (published April 2024), 2 in 3 (67%) of 300 US adults did not rule out that ChatGPT has experiences such as feelings; 1 in 3 (33%) said it definitely does not. In a survey of 500 US adults who use large language models, asked 21 to 23 January 2025 (Elon University and SSRS), 4 in 10 (40%) said the one they use most acts like it understands them, at least some of the time. That question asks how it seems, not what people believe.

The opposite mistake is to say that it only takes in numbers, so it understands nothing. A 2026 paper calls that “the mirror-image error”.

Both surveys used US participants. They do not report results for people in Ireland or for neurodivergent learners.

A language model works out a likely next token, again and again. Lesson 1.4 shows how.