Text classification assigns one or more labels to a piece of text. Given a document, predict its category. It's the most widely deployed NLP task in industry. Sentiment analysis: "This movie was terrible" → negative. Spam detection: "You've won $1,000,000!" → spam. Topic classification: an article about GDP growth → economics. Intent detection: "Book a flight to London" → book_flight. Classification can be binary (spam/not spam), multi-class (one of N categories), or multi-label (multiple tags per document). The task seems simple, but doing it well at scale — handling edge cases, class imbalance, domain shift, and evolving categories — is where the real engineering challenge lies.

Naive Bayes applies Bayes' theorem with a "naive" assumption: all features (words) are conditionally independent given the class. This assumption is obviously wrong — "New" and "York" are highly correlated — but the model works remarkably well in practice. For text, Multinomial Naive Bayes models word counts: P(class | document) is proportional to P(class) × ∏ P(word | class). Training is just counting: how often does each word appear in documents of each class? Prediction is fast: multiply the probabilities. Naive Bayes excels with small training sets (as few as dozens of examples), is extremely fast to train and predict, and provides a strong baseline that more complex models must beat. It's still the go-to first model for text classification prototypes.

Logistic regression learns a weighted sum of features, passed through a sigmoid function to produce a probability. For text, the features are typically TF-IDF values, and the model learns which words are most predictive of each class. It consistently achieves 83–88% accuracy on sentiment analysis benchmarks, often matching or beating more complex models. Logistic regression works so well for text because TF-IDF features are high-dimensional and sparse — exactly the regime where linear models shine. With L1 regularization (Lasso), it performs automatic feature selection, zeroing out irrelevant words. With L2 regularization (Ridge), it handles correlated features gracefully. The learned weights are directly interpretable: the highest-weighted words for "positive" might be "excellent", "amazing", "loved."

Support Vector Machines find the hyperplane that maximizes the margin between classes. In the high-dimensional space of TF-IDF vectors, SVMs are particularly effective because they handle sparse, high-dimensional data well and are robust to overfitting through the margin constraint. Linear SVMs (LinearSVC) are the standard choice for text — kernel SVMs are usually unnecessary because the feature space is already high-dimensional enough for linear separability. SVMs were the dominant text classification method from the late 1990s through the early 2010s, winning most shared tasks and competitions. They remain competitive today, especially when combined with careful feature engineering. The key advantage over logistic regression: SVMs focus on the hardest examples (support vectors near the boundary) rather than all examples, making them more robust to noisy data.

Deep learning brought three waves of neural text classifiers. TextCNN (Kim, 2014) applies convolutional filters over word embeddings to capture local n-gram patterns — fast and effective for short texts. LSTMs and BiLSTMs process text sequentially, capturing long-range dependencies that bag-of-words models miss. Transformer-based models (BERT, RoBERTa) achieve state-of-the-art by fine-tuning pre-trained language models on classification tasks. BERT adds a classification head on top of the [CLS] token representation and fine-tunes the entire model. The accuracy gains over classical models are real but task-dependent: for simple binary sentiment, BERT might improve 2–3% over logistic regression; for nuanced multi-class tasks with subtle distinctions, the gap can be 10%+. The trade-off is always compute cost and complexity.

For classical models, feature engineering is where the real performance gains come from. Beyond unigram TF-IDF, you can add n-gram features: bigrams ("not good", "very bad") capture negation and intensification that unigrams miss. Character n-grams capture morphological patterns and are robust to typos. Metadata features like document length, punctuation count, capitalization ratio, and emoji presence add signal. Domain-specific features like sentiment lexicon scores (AFINN, VADER) provide expert knowledge. Feature selection using chi-squared tests or mutual information removes noise features. The best classical systems combine multiple feature types: TF-IDF unigrams + bigrams + character n-grams + metadata, often matching neural models without GPU costs.

Class imbalance is the most common pitfall. If 95% of emails are not spam, a model that always predicts "not spam" gets 95% accuracy but catches zero spam. Use F1 score (not accuracy) as your primary metric, and consider oversampling, undersampling, or class weights. Data leakage happens when test data information leaks into training — fitting TF-IDF on the entire dataset before splitting means the model has seen test vocabulary statistics. Always fit preprocessing on training data only. Domain shift: a model trained on movie reviews may fail on product reviews because the vocabulary and sentiment expressions differ. Label noise: human annotators disagree 10–20% of the time on subjective tasks like sentiment, setting a ceiling on model performance. Overfitting on spurious correlations: the model learns that reviews mentioning "Oscar" are positive, not because of sentiment but because Oscar-nominated films get more positive reviews.

A production text classification pipeline follows a structured flow. Data collection: gather labeled examples (manually annotated, crowd-sourced, or from existing systems). Preprocessing: clean and normalize text using the pipeline from Chapter 2. Feature extraction: convert text to numerical features (TF-IDF, embeddings). Model selection: start with logistic regression as a baseline, then try more complex models only if needed. Evaluation: use stratified cross-validation with F1 as the primary metric. Error analysis: examine misclassified examples to find systematic patterns. Iteration: improve features, add data for weak categories, adjust thresholds. Deployment: serve predictions via API with monitoring for drift. The most important step is error analysis — understanding why the model fails teaches you more than any hyperparameter search.