A disease affects 1 in 1,000 people. A test is 99% accurate (99% true positive rate, 1% false positive rate). You test positive. What’s the probability you actually have the disease?

Most people guess ~99%. The real answer: ~9%. Why? Because the disease is so rare that the 1% false positives from 999 healthy people (≈10 false alarms) vastly outnumber the 1 true positive.

Bayes’ theorem formalizes this:

P(disease | positive) = P(positive | disease) × P(disease) / P(positive)

Prior: P(disease) = 0.001 — your belief before seeing evidence.
Likelihood: P(positive | disease) = 0.99 — how likely the evidence is given the hypothesis.
Evidence: P(positive) = 0.001×0.99 + 0.999×0.01 = 0.01089
Posterior: P(disease | positive) = 0.99 × 0.001 / 0.01089 = 0.091

To classify an email with 10,000 words, we need P(word₁, word₂, …, word₁₀₀₀₀ | spam). Estimating this joint probability requires more data than exists in the universe — the number of possible word combinations is astronomical.

The naive assumption: features are conditionally independent given the class. This means:

P(x₁, x₂, …, xₙ | y) = ∏ P(xᵢ | y)

Instead of one impossibly complex joint distribution, we estimate d simple 1D distributions. For 10,000 words, that’s 10,000 simple probabilities instead of one 10,000-dimensional one.

Is it true? Almost never. “Free” and “money” in an email are clearly correlated. But Naive Bayes doesn’t need the assumption to be true — it just needs the resulting decision boundary to be approximately correct. The ranking of classes is often right even when the probabilities are miscalibrated.

Assumes each feature follows a normal distribution within each class:

P(xᵢ | y) = (1/√(2πσ²)) · exp(−(xᵢ − μ)² / 2σ²)

For each class y and feature i, estimate μ (mean) and σ (standard deviation) from training data. Best for continuous features like height, weight, temperature.

Models count data using the multinomial distribution. P(xᵢ | y) is proportional to how often feature i appears in class y. Best for text classification with word counts or TF-IDF vectors. The go-to for spam filtering, sentiment analysis, and document categorization.

Models binary features (present/absent). Each feature is a coin flip: P(xᵢ = 1 | y). Best for binary feature vectors like “does the email contain the word ‘free’?” (yes/no). Unlike Multinomial, it explicitly penalizes the absence of features.

Bag-of-Words (BoW) represents a document as a vector of word counts, ignoring word order. “The cat sat on the mat” becomes {the: 2, cat: 1, sat: 1, on: 1, mat: 1}.

With a vocabulary of V words, each document becomes a V-dimensional vector. Most entries are zero (sparse). A corpus of 10,000 emails with a 50,000-word vocabulary produces a [10,000 × 50,000] matrix.

TF-IDF (Term Frequency × Inverse Document Frequency) improves on raw counts by downweighting common words:

TF-IDF(t, d) = TF(t, d) × log(N / DF(t))

Words like “the” appear everywhere (high DF), so their IDF is low. Words like “viagra” appear in few documents (low DF), so their IDF is high — making them more discriminative for spam detection.

Suppose the word “cryptocurrency” never appeared in any spam email during training. Then P(“cryptocurrency” | spam) = 0.

Since Naive Bayes multiplies all feature probabilities, one zero kills the entire product: P(spam | email) = 0, regardless of how many other spam-like words appear. A single unseen word vetoes the entire classification.

Laplace smoothing (additive smoothing) fixes this by adding a small count α to every feature:

P(word | class) = (count(word, class) + α) / (total_words_in_class + α × V)

where V is the vocabulary size. With α = 1, every word gets at least 1 “phantom” observation. This ensures no probability is ever exactly zero.

scikit-learn’s alpha parameter controls this. Default is 1.0. Smaller values (0.01–0.1) often work better in practice — tune via cross-validation.

Generative models (Naive Bayes) learn the full joint distribution P(x, y) = P(x|y) × P(y). They model how the data was generated for each class, then use Bayes’ theorem to classify.

Discriminative models (Logistic Regression, SVM, Trees) learn P(y|x) directly. They only care about the decision boundary between classes, not how the data was generated.

Analogy: To tell cats from dogs, a generative model learns “what does a typical cat look like?” and “what does a typical dog look like?” then compares. A discriminative model learns “what features distinguish cats from dogs?” directly.

Discriminative models are generally more accurate because they focus on what matters (the boundary). But generative models have advantages: they handle missing data naturally, work well with very small training sets, and can generate synthetic data.

1. Very small training sets: With 50–100 labeled examples, NB often outperforms logistic regression, SVMs, and trees. Its simple model has few parameters to estimate, so it doesn’t overfit.

2. Very high-dimensional data: Text with 50,000+ word features. NB handles this naturally because it estimates each feature independently. No matrix inversions, no curse of dimensionality.

3. Real-time prediction: NB prediction is a simple sum of log-probabilities — O(d) per sample. No matrix multiplications, no tree traversals. It can classify millions of documents per second.

4. Incremental learning: NB can update its model with new data without retraining from scratch (partial_fit() in scikit-learn). Useful for streaming data.

5. Multi-class is natural: NB handles K classes with no extra machinery — just compute P(y=k | x) for each class. No one-vs-rest needed.

1. Probability calibration: NB probabilities are often poorly calibrated — it tends to push probabilities toward 0 and 1. The rankings are usually correct, but the actual probability values are unreliable. Use CalibratedClassifierCV if you need calibrated probabilities.

2. Feature correlations: Highly correlated features get double-counted. If “free” and “money” always appear together, NB treats them as independent evidence, overestimating their combined effect.

3. Continuous features: GaussianNB assumes normal distributions. If features are multimodal or heavily skewed, the Gaussian assumption fails. Consider discretizing features or using kernel density estimation.