Machine learning models operate on numbers, not words. The fundamental challenge of NLP is converting text into numerical representations that preserve meaning. The simplest approach is one-hot encoding: represent each word as a vector with a 1 in its position and 0s everywhere else. With a vocabulary of 50,000 words, each word becomes a 50,000-dimensional vector with a single 1. This is wildly inefficient — 99.998% of every vector is zeros. Worse, one-hot vectors treat every word as equally different from every other word: "cat" is as far from "dog" as it is from "democracy." There's no notion of similarity. The history of text representation is the story of finding better ways to encode meaning into numbers.

Bag of Words represents a document as a vector of word counts. Each dimension corresponds to a word in the vocabulary, and the value is how many times that word appears. "The cat sat on the mat" becomes a count vector where "the" = 2, "cat" = 1, "sat" = 1, "on" = 1, "mat" = 1. The name "bag" reflects that word order is completely discarded — "dog bites man" and "man bites dog" have identical BoW representations despite opposite meanings. Despite this limitation, BoW is a strong baseline for document-level tasks like topic classification and spam detection, where the presence of certain words matters more than their arrangement. BoW vectors are sparse (mostly zeros) and high-dimensional, but they're fast to compute and easy to interpret.

TF-IDF improves on raw counts by weighting words based on how discriminative they are. Term Frequency (TF) measures how often a word appears in a document. Inverse Document Frequency (IDF) measures how rare a word is across all documents — words that appear in every document (like "the") get low IDF, while words that appear in few documents (like "quantum") get high IDF. The product TF × IDF gives high scores to words that are frequent in a specific document but rare overall — exactly the words that characterize that document. TF-IDF remains a remarkably strong baseline. A large-scale study across 73 datasets found that TF-IDF outperformed neural embeddings on 61 of them for text classification, often by 3–5% in F1 score.

Word2Vec (Mikolov et al., 2013) was the breakthrough that launched the dense embedding era. It's built on the distributional hypothesis: "You shall know a word by the company it keeps" (J.R. Firth, 1957). Words that appear in similar contexts have similar meanings, so they should have similar vectors. Word2Vec trains a shallow neural network on a simple task: given a word, predict its surrounding words (Skip-gram), or given surrounding words, predict the center word (CBOW). The hidden layer weights become the word vectors — typically 100–300 dimensions. The result: words with similar meanings cluster together, and vector arithmetic captures semantic relationships. The famous example: vec("king") − vec("man") + vec("woman") ≈ vec("queen").

GloVe (Global Vectors, Stanford 2014) combines the best of two worlds: the global co-occurrence statistics of matrix factorization methods with the local context window approach of Word2Vec. It builds a word-word co-occurrence matrix from the entire corpus, then factorizes it to produce dense vectors. GloVe often produces more stable embeddings than Word2Vec, especially for less frequent words. fastText (Facebook, 2016) adds a crucial innovation: it represents each word as a bag of character n-grams. The word "where" is represented by its character n-grams: "<wh", "whe", "her", "ere", "re>" plus the word itself. This means fastText can generate vectors for words it has never seen by composing their subword vectors — solving the OOV problem for morphologically rich languages.

Word2Vec, GloVe, and fastText all share a fundamental limitation: each word gets one vector regardless of context. The word "bank" has the same vector whether it means a financial institution or a river bank. ELMo (Embeddings from Language Models, 2018) solved this with contextual embeddings: it runs a bidirectional LSTM over the entire sentence and uses the hidden states as word representations. Now "bank" in "I deposited money at the bank" gets a different vector than "bank" in "We sat on the river bank." BERT (2018) took this further with transformer-based contextual embeddings, producing even richer representations. Contextual embeddings are the foundation of modern NLP — they're why BERT, GPT, and their successors understand language so much better than earlier models.

The choice between sparse and dense representations depends on your task, data, and constraints. Sparse methods (BoW, TF-IDF) excel when you have limited labeled data, need interpretability, or are doing keyword-based tasks like search and topic modeling. A large-scale study across 73 classification datasets found TF-IDF outperformed neural embeddings on 61 of 73 datasets by 3–5% average F1. Static dense embeddings (Word2Vec, GloVe) are best when you need semantic similarity, have moderate data, and want pre-trained representations without GPU costs. Contextual embeddings (BERT, GPT) dominate when you have sufficient compute, need to handle polysemy, or are tackling complex tasks like question answering and natural language inference. The best practitioners don't default to the newest method — they match the representation to the problem.

The evolution of text representation follows a clear arc: from no semantics (one-hot) to statistical semantics (TF-IDF) to learned semantics (Word2Vec) to contextual semantics (BERT). Each step added a new dimension of understanding. One-hot encoding treats words as arbitrary symbols. BoW and TF-IDF capture word importance but not meaning. Word2Vec and GloVe capture meaning but not context. Contextual embeddings capture both meaning and context. The next chapter on text classification will show how these representations are used as input to models that make predictions. The representation you choose determines which patterns your model can learn — and which it can't.