Lecture 3 · Week 3 · Gujarati ch. 3 · about 50 min

Estimation: OLS

The least-squares principle derived from scratch with ordinary calculus, the resulting formulas for the intercept and slope, a full hand-worked numerical example, and R-squared as a measure of fit.

By the end you can

State the least-squares criterion and derive the normal equations by minimising SSR
Compute OLS estimates b1 and b2 by hand from a small dataset
List the key algebraic properties of the OLS fitted line
Compute and interpret R-squared as a measure of goodness of fit

Key idea · Exam recipe (memorise this)

Table with columns $X_i$ , $Y_i$ , $x_i = X_i - \bar X$ , $y_i = Y_i - \bar Y$ , $x_iy_i$ , $x_i^2$ (and $y_i^2$ if $r^2$ is asked). Means first.
Slope first: $b_2 = \dfrac{\sum x_iy_i}{\sum x_i^2}$ (or $\dfrac{\sum X_iY_i - n\bar X\bar Y}{\sum X_i^2 - n\bar X^2}$ ).
Intercept: $b_1 = \bar Y - b_2\bar X$ . Write $\hat Y_i = b_1 + b_2X_i$ .
Interpret with units: “one more unit of X goes with $b_2$ more units of Y, on average”.
Fit: $R^2 = ESS/TSS = 1 - RSS/TSS$ . Check: residuals sum to 0, line passes through $(\bar X, \bar Y)$ .
Graph: scatter, line through $(\bar X, \bar Y)$ , vertical residuals $e_i$ , “OLS minimises $\sum e_i^2$ ”.

How the exam asks it: four core exam templates

Midterm exams on OLS estimation test four standard formats: hand-calculating $b_1$ and $b_2$ from small datasets, computing $R^2$ via ANOVA decompositions, estimating parameters from given deviation sums, and interpreting regression output tables.

Exam template 1: Full hand calculation of the OLS line from raw data

Question: Five students report study hours $X$ and exam scores $Y$ : $X = (1, 2, 3, 4, 5)$ and $Y = (3, 5, 4, 6, 7)$ . a) Calculate the sample means $\bar{X}$ and $\bar{Y}$ .
b) Calculate $S_{xy} = \sum x_i y_i$ and $S_{xx} = \sum x_i^2$ , and solve for the slope $b_2$ .
c) Calculate the intercept $b_1$ , write the estimated regression line $\hat{Y}_i$ , and interpret the slope.

Exam template 2: Computing R-squared from residuals and total variation

Question: For the five students in Template 1, the residuals are $e = (-0.2, 0.9, -1.0, 0.1, 0.2)$ and $\bar{Y} = 5$ . a) Compute the residual sum of squares $SSR = \sum e_i^2$ .
b) Compute the total sum of squares $TSS = \sum (Y_i - \bar{Y})^2$ .
c) Compute $R^2$ and interpret its value.

Exam template 3: Estimating from summary sums

Question: An exam problem provides summary statistics for an empirical dataset: $\sum x_i^2 = 92.50$ , $\sum x_i y_i = 590.20$ , $\bar{X} = 10.35$ , and $\bar{Y} = 169.4$ . a) Calculate the OLS slope coefficient $b_2$ .
b) Calculate the OLS intercept $b_1$ .
c) State the estimated regression equation and predict $Y$ when $X = 15$ .

Exam template 4: Reading and interpreting an EViews output table

Question: An EViews regression table reports: Constant $C = 24.45455$ ( $SE = 6.413817$ ), regressor $X = 0.509091$ ( $SE = 0.035743$ ), $SSR = 337.2727$ , $R^2 = 0.962062$ , $n = 10$ . a) Compute the t-statistics for the constant and slope.
b) Calculate the estimated error variance $\hat{\sigma}^2$ and standard error of regression $\hat{\sigma}$ .
c) Interpret the coefficient of determination $R^2$ .

Cheat-sheet & the 4-step method

· Cheat-sheet: OLS estimation formulas and properties

Quantity	Formula	When to use it	How to write the exam answer
OLS slope	$b_2 = \dfrac{\sum x_i y_i}{\sum x_i^2} = \dfrac{S_{xy}}{S_{xx}}$	Estimate slope first	“Each 1-unit rise in $X$ is associated with a $b_2$ -unit rise in $Y$ on average.”
OLS intercept	$b_1 = \bar{Y} - b_2 \bar{X}$	Estimate intercept second	“When $X=0$ , baseline average $Y$ is $b_1$ [units].”
Fitted line	$\hat{Y}_i = b_1 + b_2 X_i$	Predict $Y$ given $X$	“Predicted $Y$ is $\hat{Y} = b_1 + b_2(c)$ [units].”
Residual	$e_i = Y_i - \hat{Y}_i$	Compute sample error	“Residual $e_i = Y_i - \hat{Y}_i$ ; algebraic check: $\sum e_i = 0$ .”
Variation split	$TSS = ESS + SSR$	Decompose total variance	” $TSS = \sum(Y-\bar{Y})^2$ , $ESS = \sum(\hat{Y}-\bar{Y})^2$ , $SSR = \sum e^2$ .”
R-squared	$R^2 = \dfrac{ESS}{TSS} = 1 - \dfrac{SSR}{TSS}$	Assess goodness of fit	” $R^2 \times 100\%$ of the variation in $Y$ is explained by $X$ .”
Error variance	$\hat{\sigma}^2 = \dfrac{SSR}{n-2}$	Measure residual spread	” $\hat{\sigma} = \sqrt{\hat{\sigma}^2}$ is the standard error of regression.”

Key idea · The 4-step method for OLS estimation problems

Step 1: Compute sample means and deviation sums first.
- Calculate $\bar{X} = \sum X / n$ and $\bar{Y} = \sum Y / n$ .
- Build columns $x_i = X_i - \bar{X}$ and $y_i = Y_i - \bar{Y}$ , then calculate $S_{xy} = \sum x_i y_i$ and $S_{xx} = \sum x_i^2$ .
Step 2: Solve for slope $b_2$ before intercept $b_1$ .
- Compute $b_2 = S_{xy} / S_{xx}$ . Note the sign: positive for supply/consumption, negative for demand.
- Compute $b_1 = \bar{Y} - b_2 \bar{X}$ .
Step 3: Write the fitted line and evaluate residuals.
- Write $\hat{Y}_i = b_1 + b_2 X_i$ .
- Calculate $e_i = Y_i - \hat{Y}_i$ and verify that $\sum e_i = 0$ .
Step 4: Compute $R^2$ and state the complete interpretation.
- Compute $SSR = \sum e_i^2$ and $TSS = \sum y_i^2$ ; evaluate $R^2 = 1 - SSR/TSS$ .
- Write a bold one-sentence answer stating the slope interpretation, prediction, and explained variance with exact units.

The least-squares principle

On the slides: OLS is a regression estimation technique that calculates the $b$ ‘s so as to minimise the sum of the squared residuals, where $e_i = Y_i - \hat Y_i$ is the gap between the actual $Y$ and the $Y$ estimated by the regression.

Infinitely many lines could be drawn through a scatter of points. Ordinary least squares (OLS) picks the one specific line that makes the residuals, as a whole, as small as possible — precisely by minimising the sum of squared residuals.

OLS objectiveKey formulaFormula sheet →

\min_{b_1,\,b_2} \; SSR = \sum_{i=1}^n \big(Y_i - b_1 - b_2X_i\big)^2

$SSR$: sum of squared residuals
$b_1, b_2$: the candidate intercept and slope being chosen

Use it when you need to explain what OLS is actually optimizing, before deriving the formulas for b1 and b2.

Deriving the normal equations

This is exactly the unconstrained optimization technique from the Mathematics I course, applied to a function of two variables $b_1$ and $b_2$ : set each partial derivative to zero.

OLS estimatorsKey formulasFormula sheet →

b_2 = \frac{\sum(X_i-\bar X)(Y_i-\bar Y)}{\sum(X_i-\bar X)^2} = \frac{S_{xy}}{S_{xx}} \qquad b_1 = \bar Y - b_2\bar X

$\bar X, \bar Y$: the sample means of X and Y
$S_{xy}$: sum of products of deviations from the means
$S_{xx}$: sum of squared deviations of X from its mean

Use it when you need to compute the OLS intercept and slope by hand from raw (X,Y) data — the single most-used formula pair in this course.

The slides write the same slope in two equivalent ways, with small letters for deviations from the mean ( $x_i = X_i - \bar X$ , $y_i = Y_i - \bar Y$ ):

OLS slope, two forms (slides)Key formulaFormula sheet →

b_2 = \frac{\sum x_i y_i}{\sum x_i^2} = \frac{\sum X_iY_i - n\bar X\bar Y}{\sum X_i^2 - n\bar X^2}

$x_i, y_i$: deviations from the means
$\sum X_iY_i, \sum X_i^2$: sums of the raw values
$n$: sample size

Use it when you are given raw sums (the second form) or a deviation table (the first form); both give the same number.

A full hand-worked example

The lecture’s numerical example

Algebraic properties of the OLS line

These are not assumptions — they are automatic mathematical consequences of the two normal equations derived above (the first normal equation directly implies property 1; combined with property 1, the second implies property 2).

Try it: drag the data, watch OLS refit

R-squared: goodness of fit

Decomposition of variationKey formulaFormula sheet →

\underbrace{\sum(Y_i-\bar Y)^2}_{TSS} = \underbrace{\sum(\hat Y_i-\bar Y)^2}_{ESS} + \underbrace{\sum(Y_i-\hat Y_i)^2}_{SSR}

$TSS$: total sum of squares
$ESS$: explained sum of squares
$SSR$: residual (unexplained) sum of squares

Use it when you need to break down the total variation in Y into the part the regression line explains and the part it leaves unexplained.

R-squaredKey formulaFormula sheet →

R^2 = \frac{ESS}{TSS} = 1 - \frac{SSR}{TSS}

$R^2$: the proportion of the total variation in Y explained by the regression, 0 \leq R^2 \leq 1

Use it when a question asks how well the fitted line explains the data — the standard, single-number goodness-of-fit measure for a two-variable regression.

Reading EViews output

The lecture tests Keynes’s consumption function on Gujarati’s Table 2.4 data: weekly income $X = 80, 100, \ldots, 260$ and weekly consumption $Y = 70, 65, 90, 95, 110, 115, 120, 140, 155, 150$ , with $n = 10$ . Keynes predicted that consumption rises with income but by less — a marginal propensity to consume between 0 and 1. EViews estimates the line by OLS and prints:

Variable	Coefficient	Std. Error	t-Statistic	Prob.
C	24.45455	6.413817	3.812791	0.0051
X	0.509091	0.035743	14.24317	0.0000

Statistic	Value	Statistic	Value
R-squared	0.962062	Mean dependent var	111.0000
Adjusted R-squared	0.957319	Sum squared resid	337.2727
S.E. of regression	6.493003	F-statistic	202.8679

Move the last family's consumption (X = 260)150Predict consumption at income X200

C (intercept b₁)	24.45455
X (slope b₂)	0.509091
Std. error of b₂	0.035743
t-statistic of b₂	14.24317
R-squared	0.962062
Adjusted R-squared	0.957319
S.E. of regression	6.4930
Sum squared resid	337.2727
Mean dependent var	111.00
F-statistic	202.868

b₂ = Σxy / Σx² = 16,800 / 33,000 = 0.5091: each extra 1 dollar of weekly income raises mean weekly consumption by about 51 cents (within incomes 80–260). It is the MPC, between 0 and 1 as Keynes said.

Interpreting the slope. $b_2 = 0.5091$ : within the sample range of X between 80 and 260 dollars per week, a one-dollar increase in weekly income is associated with an estimated increase of about 51 cents in average weekly consumption expenditure.

Interpreting the intercept. $b_1 = 24.4545$ : mechanically, the average weekly consumption when weekly income is zero. Taken literally this is not always meaningful, but it can be argued that a family with no income (unemployment, layoff) maintains some minimum consumption financed by borrowing or dissaving.

Interpreting R². $R^2 = 0.9621$ means about 96 percent of the variation in weekly consumption expenditure is explained by income. Since R² can be at most 1, the sample regression line fits the data very well.

Checking the printed numbers.

Each t-statistic is the coefficient divided by its standard error: $0.509091 / 0.035743 = 14.243$ (EViews prints 14.24317, the small difference is rounding of the displayed coefficient).
The S.E. of regression is the square root of the estimated error variance: $\sqrt{337.2727 / 8} = \sqrt{42.1591} = 6.493$ .
In a two-variable regression the F-statistic is just the square of the slope’s t: $14.24317^2 = 202.87$ .

Akaike info criterion, Schwarz criterion and Durbin-Watson appear in fuller output; what they mean comes later in the course.

Exam practice

Summary and review

OLS minimises $\sum e_i^2 = \sum(Y_i-b_1-b_2X_i)^2$ .
Setting $\partial SSR/\partial b_1=0$ and $\partial SSR/\partial b_2=0$ gives the two normal equations, solved to get $b_2=\sum x_iy_i/\sum x_i^2$ and $b_1=\bar Y-b_2\bar X$ .
OLS residuals always sum to zero, and the fitted line always passes through $(\bar X,\bar Y)$ — automatic consequences of the normal equations.
$TSS = ESS+SSR$ ; $R^2 = ESS/TSS = 1-SSR/TSS$ , the proportion of variation in Y explained by X.
A high R² does not by itself validate a model’s specification or causal story.
The statistical properties of $b_1, b_2$ (standard errors, BLUE) rest on the CLRM assumptions: Lesson 4.
EViews output pairs each coefficient with its standard error, t-statistic and p-value; in Gujarati’s consumption example $b_2 = 0.5091$ (about 51 cents per extra dollar) and $R^2 = 0.9621$ .

Step-by-step examples

Work through the ladder one level at a time: at every step, try to say what comes next before you reveal it. Easy builds the basic moves, medium combines the formulas, and the exam level has the shape of a real exam question.

Easy

Medium

X	Y
1	4
2	6
3	7
4	8
5	10

Each dot is one observation: a pair (X, Y). There are 5 of them. Before any formula, look: do the dots rise from left to right? Then the slope will be positive.

Exam level

X	Y
2	25
3	19
4	16
5	11
6	9

Each dot is one observation: a pair (X, Y). There are 5 of them. Before any formula, look: do the dots rise from left to right? Then the slope will be positive.