Real-world text is messy. It contains HTML tags, special characters, inconsistent casing, extra whitespace, emojis, URLs, and encoding artifacts. The same word appears in dozens of surface forms: "U.S.A.", "USA", "us", "United States." Preprocessing transforms this chaos into a consistent, clean representation that models can learn from. Without it, your model treats "Running", "running", and "RUNNING" as three completely different words. Preprocessing typically accounts for 60–70% of the effort in a classical NLP project. Even with modern transformers that handle raw text, understanding preprocessing helps you debug when models fail on edge cases.

Text cleaning removes elements that add noise without contributing meaning. HTML/XML stripping removes markup tags. URL removal replaces links with a placeholder or removes them entirely. Special character handling removes or normalizes characters like &, @, #. Whitespace normalization collapses multiple spaces, tabs, and newlines into single spaces. Encoding fixes handle UTF-8 artifacts like "don’t" appearing as "don’t." Number handling can normalize numbers to a token like <NUM> or spell them out. The key principle is to remove what doesn't carry meaning for your task while preserving what does.

Normalization reduces the vocabulary by mapping different surface forms to the same canonical representation. Lowercasing is the most common: "Apple", "apple", and "APPLE" become one token. Unicode normalization (NFC or NFKC) ensures that characters composed in different ways are treated identically — the accented "é" can be stored as one codepoint or two, and normalization picks one. Accent removal maps "café" to "cafe." Abbreviation expansion maps "Dr." to "Doctor." Each normalization step reduces vocabulary size but potentially loses information. Lowercasing "Apple" (the company) and "apple" (the fruit) merges two distinct concepts. The right level of normalization depends on your task and vocabulary size constraints.

Tokenization splits text into discrete units (tokens) that become the input to your model. The simplest approach is word-level tokenization: split on whitespace and punctuation. But even this is tricky. Should "New York" be one token or two? Is "don't" one token ("don't"), two ("do" + "n't"), or three ("do" + "not")? What about "state-of-the-art" or "3.14"? Different tokenizers make different choices. spaCy uses linguistic rules to handle contractions and special cases. NLTK provides multiple tokenizers with different trade-offs. Word-level tokenization creates a fixed vocabulary from the training data, and any word not seen during training becomes an unknown token (UNK) — a major limitation for production systems.

Subword tokenization solves the OOV problem by splitting words into smaller, reusable pieces. Common words stay whole ("the", "and"), while rare words are broken into subwords ("unhappiness" → "un" + "happiness"). Byte Pair Encoding (BPE), used by GPT models, iteratively merges the most frequent character pairs until reaching a target vocabulary size. WordPiece, used by BERT, is similar but optimizes for likelihood rather than frequency. SentencePiece treats the input as a raw byte stream (no pre-tokenization needed), supporting both BPE and Unigram algorithms. Typical vocabulary sizes range from 30,000 to 50,000 tokens. The key advantage: any text can be tokenized without UNK tokens, because the algorithm can always fall back to individual characters.

Stemming chops off word endings using simple rules. The Porter Stemmer (1980) applies cascading suffix-stripping rules: "running" → "run", "studies" → "studi", "operational" → "oper". It's fast but crude — "studi" isn't a real word. Lemmatization uses vocabulary lookup and morphological analysis to find the true dictionary form (lemma): "running" → "run", "studies" → "study", "better" → "good." It's slower but linguistically correct. Lemmatization requires knowing the word's part of speech — "saw" as a verb lemmatizes to "see", but "saw" as a noun stays "saw." In practice, stemming is used when speed matters and exact forms don't (search engines), while lemmatization is preferred when linguistic accuracy matters (text analysis, chatbots).

Stop words are high-frequency words like "the", "is", "at", "which" that appear in almost every document. Removing them reduces vocabulary size and computational cost while (sometimes) improving signal-to-noise ratio. Standard stop word lists contain 100–300 words. But stop word removal is dangerous in many contexts. "To be or not to be" becomes meaningless without stop words. Negation words like "not", "no", "never" are often on stop word lists but carry critical semantic meaning — "not good" and "good" have opposite meanings. For bag-of-words models and TF-IDF, stop word removal often helps. For neural models and transformers, it usually hurts because these models learn to use function words as structural signals. The modern best practice is to let the model decide what to ignore through attention weights.

A preprocessing pipeline chains cleaning, normalization, tokenization, and filtering steps in a specific order. The order matters: you should clean HTML before tokenizing, and tokenize before removing stop words. For classical ML pipelines (Naive Bayes, SVM, logistic regression), a typical pipeline is: clean → lowercase → tokenize (word-level) → remove stop words → stem/lemmatize → vectorize (TF-IDF). For transformer-based models, preprocessing is minimal: clean → use the model's pre-trained tokenizer (BPE/WordPiece). The pre-trained tokenizer was trained on specific preprocessing, so you must match it exactly. Adding lowercasing to a cased BERT model will degrade performance. The golden rule: your preprocessing must match what the model expects.