21 Aug 2022 · 7 min read

LDA vs QDA, with examples in R

Linear and Quadratic Discriminant Analysis are attractive because there is nothing to tune. Both come from a classical probabilistic model via Bayes' rule, and both assume the classes are drawn from Gaussian distributions. The entire difference between them is one assumption.

The shared setup

Start from Bayes' rule: the posterior probability that an observation belongs to class k is proportional to the class prior times the class-conditional density. Assume that density is Gaussian, and you can write it out in closed form for each class. Since the denominator doesn't depend on the class, it drops out of the comparison.

Bayes' rule for the posterior probability of class k given x — Bayes' rule - the denominator is the same for every class, so only the numerator decides.

Multivariate Gaussian class-conditional density for class k — The class-conditional density, assumed multivariate Gaussian with mean mu_k and covariance Sigma_k. Whether Sigma_k varies by class is the entire LDA/QDA distinction.

LDA

Take the log of the posterior and assume every class shares the same covariance matrix. The quadratic terms cancel, and what remains is linear in x. The predicted class is whichever maximises that expression - and the decision boundaries between classes are straight lines (hyperplanes in higher dimensions).

Log posterior for LDA with a shared covariance matrix — LDA: one shared Σ, so the log-determinant term is constant and the discriminant is linear in x.

QDA

Drop the shared-covariance assumption and let each class have its own. The quadratic terms no longer cancel, so the discriminant is quadratic in x and the boundaries become curves. More flexible, and more parameters to estimate - one covariance matrix per class instead of one overall.

Log posterior for QDA with a per-class covariance matrix — QDA: Σ carries a k subscript. The log-determinant survives and the boundary bends.

Choosing between them

Few observations per class: prefer LDA. Estimating a separate covariance matrix per class needs data you may not have.
Clearly different class spreads: prefer QDA. Forcing a shared covariance will underfit visibly.
Neither is a substitute for checking the Gaussian assumption. If the classes aren't remotely normal, both are the wrong tool.

library(MASS)

lda_fit <- lda(class ~ ., data = train)
qda_fit <- qda(class ~ ., data = train)

mean(predict(lda_fit, test)$class == test$class)
mean(predict(qda_fit, test)$class == test$class)

Dataset 1 - two classes, unequal spread

Scatter plot of two simulated classes with different spreads — Two classes, visibly different spreads - the case where QDA earns its extra parameters.

LDA and QDA decision boundaries side by side on dataset 1 — LDA draws a straight line; QDA curves toward the tighter class. Small difference, right direction.

Dataset 2 - four classes, similar covariance

Scatter plot of four simulated classes — Four classes, roughly comparable spreads.

LDA and QDA decision boundaries side by side on dataset 2 — Almost indistinguishable. When the shared-covariance assumption holds, QDA only adds variance.

Dataset 3 - four classes, heavier overlap

Scatter plot of four overlapping simulated classes — More overlap, and class 4 spreads much wider than the rest.

LDA and QDA decision boundaries side by side on dataset 3 — Here the curvature matters: QDA wraps the wide class instead of slicing through it.

Dataset 4 - four dimensions, pairwise view

Pairwise scatter plot matrix of four variables for two classes — Four predictors, two classes. Separation is strong in some pairs and near-absent in others.

LDA decision regions and per-pair error rates across the scatter matrix — LDA per pair, with the error rate printed on each panel.

QDA decision regions and per-pair error rates across the scatter matrix — QDA on the same pairs. Slightly lower error almost everywhere - 0.047 → 0.035 on the best pair - but never dramatically so.

That's the honest summary: QDA is rarely much worse and often marginally better, and the gap only becomes interesting when class covariances genuinely differ. If you have plenty of data, start with QDA. If you don't, LDA's lower variance usually wins.