Data, Text, and Naive Tokenization

So our first goal is to create a piece of software that can learn to generate language through imitation, much like infants and toddlers learn to generate words and sentences by imitating adults. To do so, it will need something to learn from, meaning it will have to process lots of data. As we are effectively using this data to train our software, we will refer to it as our Training Data in the following.

Choosing a data source

What form should this training data take, exactly? Infants and toddlers mostly use auditory data to learn language, with some help from visual cues — they hear older people speak and watch them interact. Using audio and video files to train our chatbot would have some advantages, since variations in pitch, rhythm, volume, eye movement, facial expressions, etc. are likely to contain information that could be useful for language learning.

But it would also be overkill — since we will type out our prompts in text form, we don’t actually need our chatbot to understand spoken language, so forcing it to learn how to translate raw audio into sequences of words would add a whole additional layer to the learning problem and likely make the task more complicated than it needs to be. It would also require a lot more memory, since audio and video files containing spoken language are orders of magnitude larger than equivalent text files.

So let's go with the much more intuitive choice — if our goal is to generate text, let's use a bunch of text data to train our model. How to best gather and process large amounts of such data off the internet is a major question in its own right and the importance of dataset quality for virtually any machine-learning endeavor is hard to overstate, but let's stay focussed on the learning-part of our journey for now and simply assume we have access to any dataset of high-quality English text we can conceive of, ranging from individual books or websites to all high-quality text on the internet.

Building a vocabulary

Since our goal is to create a program that can learn how to generate coherent English text, it seems intuitive to start by creating a Vocabulary that contains all the text elements our chatbot should be able to use. In machine learning, such vocabulary elements are referred to as Tokens.

In principle, we could define any snippet of text to be a token, from single characters to entire sentences or paragraphs. However, forcing our chatbot to think in terms of pre-formulated sentences feels oddly restrictive, and using individual characters would simply make the learning problem harder — after all, the software would then first have to learn how to chain these characters together to get anything meaningful to begin with, much like human infants have to learn what sequences of sounds correspond to words. So it seems like there must be some optimal solution somewhere in between, where our token definitions line up nicely with the "base units" of meaning in text.

It doesn't seem obvious what these base units would be. Words certainly seem like an intuitive start, but what about prefixes, suffixes, compounds, and so on? Figuring this out might be quite imporant, but we also don't want to get bogged down with excessive detail right off the bat, so let's just add this to our mental list of things to revisit down the road for now:

A list of problems we are refusing to confront. :)

A list of problems we are refusing to confront. :)

For the time being, let's go with the choice that seems the most intuitive to begin with — we’ll use words as tokens wherever we can, and only use individual symbols or characters as needed (spaces, punctuation marks, numbers, etc.).

The next thing we have to figure out is where to get these words from. In principle, we could use a pre-existing English dictionary to define the word-part of our vocabulary. However, such a vocabulary would likely be missing at least some of the words contained in the data, such as names, proper nouns,  slang terms, exotic word creations etc., while pointlessly including other words not contained in it, which our chatbot could never learn to use properly. This seems inconvenient.

So instead of using a pre-existing dictionary that might have limited overlap with the training data vocabulary, let’s begin embracing the spirit of imitation learning and learn our vocabulary from the data.

A straightforward way of doing this is to simply take all the words contained in the text, and to then add any other single-character symbols that appear in it, such as spaces, punctuation marks, digits, and so on. Perhaps the simplest robust definition of a word we could think of is "a sequence of nothing but letters", so let's just keep it simple and use that for now.

Let's make this more concrete using a specific training dataset. We are still far from having a model whose performance we are worried about, so let's simply choose something illustrative and go with a famously large book, say Leo Tolstoy’s War and Peace . If we apply our simple tokenization method to that, the result is a vocabulary of about 20,000 tokens.

With this vocabulary in place, we can build up text by chaining together arbitrary tokens from the vocabulary, and since there is no ambiguity to our simple tokenization rules, we can also break down any given text into the corresponding token sequence, so long as the text does not contain any tokens that aren't part of the vocabulary.

For example, the sentence

can be thought of as a chain of 14 tokens: