How to use this sheet

Full worked answers are in the solutions sheet.

There are three exercises based on material from Lectures 9 and 10 ( and at times go a little beyond it). Exercises 1 and 3 are pen-and-paper: work them out by hand first, as you would in a quiz or the midterm, and only then check your numbers in R. Exercise 2 is done in R.

Exercise 1. Least squares with matrices, by hand

Five observations:

\(x\) 1 2 4 5 8
\(Y\) 2 4 7 7 13

Questions.

  1. Write down the response vector \(\boldsymbol{Y}\) and the design matrix \(\boldsymbol{X}\) of the simple linear regression model. Compute \(\boldsymbol{X}'\boldsymbol{X}\) and \(\boldsymbol{X}'\boldsymbol{Y}\).
  2. Compute \(\det(\boldsymbol{X}'\boldsymbol{X})\) and \((\boldsymbol{X}'\boldsymbol{X})^{-1}\). Check that your answer agrees with \[(\boldsymbol{X}'\boldsymbol{X})^{-1} = \frac{1}{S_{xx}} \begin{bmatrix} \sum_i x_i^2 / n & -\bar x \\ -\bar x & 1 \end{bmatrix}.\]
  3. Compute \(\hat{\boldsymbol{\beta}} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{Y}\), and check it against \(\hat\beta_1 = S_{xY}/S_{xx}\) and \(\hat\beta_0 = \bar Y - \hat\beta_1 \bar x\).
  4. Compute the fitted values \(\hat{\boldsymbol{Y}} = \boldsymbol{X}\hat{\boldsymbol{\beta}}\) and the residuals \(\boldsymbol{e} = \boldsymbol{Y} - \hat{\boldsymbol{Y}}\). Then compute \(SSE = \boldsymbol{e}'\boldsymbol{e}\) and \(MSE\).
  5. Show that, for any data, \(\det(\boldsymbol{X}'\boldsymbol{X}) = n\,S_{xx}\). What happens when all the \(x_i\) are equal? Explain, in terms of the data rather than the algebra, why \(\beta_1\) cannot be estimated then.

To check your numbers once you are done:

X <- cbind(1, c(1, 2, 4, 5, 8)); Y <- c(2, 4, 7, 7, 13)
t(X) %*% X; t(X) %*% Y
beta_hat <- solve(t(X) %*% X) %*% t(X) %*% Y; beta_hat
e <- Y - X %*% beta_hat; t(e) %*% e

Exercise 2. Scottish hill races

The dataset hills, in the package MASS, gives the record time (time, in minutes) of 35 Scottish hill races in 1984, with the distance of each race (dist, in miles) and the total height climbed along its route (climb, in feet). The name of each race is the name of its row.

library(MASS)
head(hills)

Questions.

  1. Regress time on dist and climb. Write down the fitted equation, and interpret each of the two slopes in context. Does the intercept have an interpretation here?
  2. Now regress time on climb alone. Compare the slope of climb with the one of (a), and explain the difference. The correlation between dist and climb will help.
  3. Recover the coefficient of climb of (a) from simple regressions only, as in Lecture 10: remove from time and from climb what dist explains, then regress one set of residuals on the other.
  4. Build the design matrix \(\boldsymbol{X}\) of the model of (a). What are \(n\) and \(p\)? Using matrix operations only (t(), %*%, solve()), compute \(\hat{\boldsymbol{\beta}}\), then \(MSE = SSE/(n - p)\) and \(\boldsymbol{s^2}\{\hat{\boldsymbol{\beta}}\} = MSE\,(\boldsymbol{X}'\boldsymbol{X})^{-1}\). Check your results against coef(fit), sigma(fit) and vcov(fit). What is the standard error of the coefficient of climb?
  5. Plot the residuals of the model of (a) against its fitted values. What do you notice?

Exercise 3. Can the coefficients be estimated?

The least squares estimator \(\hat{\boldsymbol{\beta}} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{Y}\) exists only when \(\boldsymbol{X}'\boldsymbol{X}\) can be inverted, that is, when no column of \(\boldsymbol{X}\) is a linear combination of the others. Here are five regression models, each with an intercept.

  1. The predictors are the mean temperature of the day in degrees Celsius, and the same temperature in degrees Fahrenheit. (Recall that \(F = 32 + 1.8\,C\).)
  2. Soil samples are described by three predictors: their percentages of sand, of silt and of clay, which add up to 100.
  3. The predictors are \(x\) and \(x^2\), and \(x\) takes at least three different values in the data.
  4. There are two predictors, and \(n = 2\) observations.
  5. There are two predictors, whose sample correlation is 0.999.

Questions.

  1. For each model, say whether \(\boldsymbol{X}'\boldsymbol{X}\) can be inverted. When it cannot, write down the relation between the columns of \(\boldsymbol{X}\) that prevents it.
  2. In model 1, R can fit the regression once one of the two temperature columns is removed. Does it matter which one is removed, for the fitted values? For the coefficients?