Lecture 3 · Week 3 · Gujarati ch. 3 · about 50 min

Estimation: OLS

The least-squares principle derived from scratch with ordinary calculus, the resulting formulas for the intercept and slope, a full hand-worked numerical example, and R-squared as a measure of fit.

By the end you can

State the least-squares criterion and derive the normal equations by minimising SSR
Compute OLS estimates b0 and b1 by hand from a small dataset
List the key algebraic properties of the OLS fitted line
Compute and interpret R-squared as a measure of goodness of fit

The least-squares principle

Infinitely many lines could be drawn through a scatter of points. Ordinary least squares (OLS) picks the one specific line that makes the residuals, as a whole, as small as possible — precisely by minimising the sum of squared residuals.

OLS objectiveKey formulaFormula sheet →

\min_{b_0,\,b_1} \; SSR = \sum_{i=1}^n \big(Y_i - b_0 - b_1X_i\big)^2

$SSR$: sum of squared residuals
$b_0, b_1$: the candidate intercept and slope being chosen

Use it when you need to explain what OLS is actually optimizing, before deriving the formulas for b0 and b1.

Deriving the normal equations

This is exactly the unconstrained optimization technique from the Mathematics I course, applied to a function of two variables $b_0$ and $b_1$ : set each partial derivative to zero.

OLS estimatorsKey formulasFormula sheet →

b_1 = \frac{\sum(X_i-\bar X)(Y_i-\bar Y)}{\sum(X_i-\bar X)^2} = \frac{S_{xy}}{S_{xx}} \qquad b_0 = \bar Y - b_1\bar X

$\bar X, \bar Y$: the sample means of X and Y
$S_{xy}$: sum of products of deviations from the means
$S_{xx}$: sum of squared deviations of X from its mean

Use it when you need to compute the OLS intercept and slope by hand from raw (X,Y) data — the single most-used formula pair in this course.

A full hand-worked example

Algebraic properties of the OLS line

Properties of OLSKey propertiesFormula sheet →

1.\ \sum e_i = 0 \qquad 2.\ \sum X_ie_i = 0 \qquad 3.\ \text{the fitted line passes through } (\bar X, \bar Y)

Use it when you need to check whether a claimed fitted line could actually be the OLS line, or explain a stated algebraic fact about residuals.

These are not assumptions — they are automatic mathematical consequences of the two normal equations derived above (the first normal equation directly implies property 1; combined with property 1, the second implies property 2).

Try it: drag the data, watch OLS refit

R-squared: goodness of fit

Decomposition of variationKey formulaFormula sheet →

\underbrace{\sum(Y_i-\bar Y)^2}_{TSS} = \underbrace{\sum(\hat Y_i-\bar Y)^2}_{ESS} + \underbrace{\sum(Y_i-\hat Y_i)^2}_{SSR}

$TSS$: total sum of squares
$ESS$: explained sum of squares
$SSR$: residual (unexplained) sum of squares

Use it when you need to break down the total variation in Y into the part the regression line explains and the part it leaves unexplained.

R-squaredKey formulaFormula sheet →

R^2 = \frac{ESS}{TSS} = 1 - \frac{SSR}{TSS}

$R^2$: the proportion of the total variation in Y explained by the regression, 0 \leq R^2 \leq 1

Use it when a question asks how well the fitted line explains the data — the standard, single-number goodness-of-fit measure for a two-variable regression.

Exam practice

Summary and review

OLS minimises $SSR = \sum(Y_i-b_0-b_1X_i)^2$ .
Setting $\partial SSR/\partial b_0=0$ and $\partial SSR/\partial b_1=0$ gives the two normal equations, solved to get $b_1=S_{xy}/S_{xx}$ and $b_0=\bar Y-b_1\bar X$ .
OLS residuals always sum to zero, and the fitted line always passes through $(\bar X,\bar Y)$ — automatic consequences of the normal equations.
$TSS = ESS+SSR$ ; $R^2 = ESS/TSS = 1-SSR/TSS$ , the proportion of variation in Y explained by X.
A high R² does not by itself validate a model’s specification or causal story.