← return to study.practicaldsc.org

The problems in this worksheet are taken from past exams in similar classes. Work on them on paper, since the exams you take in this course will also be on paper.

We encourage you to complete this worksheet in a live discussion section. Solutions will be made available after all discussion sections have concluded. You don’t need to submit your answers anywhere.

Note: We do not plan to cover all problems here in the live discussion section; the problems we don’t cover can be used for extra practice.

Problem 1

Every week, Lauren goes to her local grocery store and buys a varying amount of vegetable but always buys exactly one pound of meat (either beef, fish, or chicken). We use a linear regression model to predict her total grocery bill. We’ve collected a dataset containing the pounds of vegetables bought, the type of meat bought, and the total bill. Below we display the first few rows of the dataset and two plots generated using the entire training set.

Problem 1.1

Suppose we fit the following linear regression models to predict 'total' using the squared loss function. Based on the data and visualizations shown above, for each of the following models H(x), determine whether each fitted model coefficient w^* is positive (+), negative (-), or exactly 0. The notation \text{meat=beef} refers to the one-hot encoded 'meat' column with value 1 if the original value in the 'meat' column was 'beef' and 0 otherwise. Likewise, \text{meat=chicken} and \text{meat=fish} are the one-hot encoded 'meat' columns for 'chicken' and 'fish', respectively.

For example, in part (iv), you’ll need to provide three answers: one for w_0^* (either positive, negative, or 0), one for w_1^* (either positive, negative, or 0), and one for w_2^* (either positive, negative, or 0).

1. H(x) = w_0
2. H(x) = w_0 + w_1 \cdot \text{veg}
3. H(x) = w_0 + w_1 \cdot \text{(meat=chicken)}
4. H(x) = w_0 + w_1 \cdot \text{(meat=beef)} + w_2 \cdot \text{(meat=chicken)}
5. H(x) = w_0 + w_1 \cdot \text{(meat=beef)} + w_2 \cdot \text{(meat=chicken)} + w_3 \cdot \text{(meat=fish)}

Problem 1.2

Suppose we fit the model H(x) = w_0 + w_1 \cdot \text{veg} + w_2 \cdot \text{(meat=beef)} + w_3 \cdot \text{(meat=fish)}. After fitting, we find that \vec{w^*}=[-3, 5, 8, 12]

What is the prediction of this model on the first point in our dataset?

• -3

• 2

• 5

• 10

• 13

• 22

• 25

Problem 1.3

Following the same model H(x) and weights from the previous problem, what is the loss of this model on the second point in our dataset, using squared error loss?

• 0

• 1

• 5

• 6

• 8

• 24

• 25

• 169

Problem 2

Billy’s aunt owns a jewellery store, and gives him data on 5000 of the diamonds in her store. For each diamond, we have:

• carat: the weight of the diamond, in carats
• length: the length of the diamond, in centimeters
• width: the width of the diamond, in centimeters
• price: the value of the diamond, in dollars

The first 5 rows of the 5000-row dataset are shown below:

Billy has enlisted our help in predicting the price of a diamond given various other features.

Problem 2.1

Suppose we want to fit a linear prediction rule that uses two features, carat and length, to predict price. Specifically, our prediction rule will be of the form

\text{predicted price} = w_0 + w_1 \cdot \text{carat} + w_2 \cdot \text{length}

We will use least squares to find \vec{w}^* = \begin{bmatrix} w_0^* \\ w_1^* \\ w_2^* \end{bmatrix}.

Write out the first 5 rows of the design matrix, X. Your matrix should not have any variables in it.

Problem 2.2

Suppose the optimal parameter vector \vec{w}^* is given by

\vec{w}^* = \begin{bmatrix} 2000 \\ 10000 \\ -1000 \end{bmatrix}

What is the predicted price of a diamond with 0.65 carats and a length of 4 centimeters? Show your work.

Problem 2.3

Suppose \vec{e} = \begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix} is the error/residual vector, defined as

\vec{e} = \vec{y} - X \vec{w}^*

where \vec{y} is the observation vector containing the prices for each diamond.

For each of the following quantities, state whether they are guaranteed to be equal to 0 the scalar, \vec{0} the vector of all 0s, or neither. No justification is necessary.

• \sum_{i = 1}^n e_i
• || \vec{y} - X \vec{w}^* ||^2
• X^TX \vec{w}^*
• 2X^TX \vec{w}^* - 2X^T\vec{y}

Problem 2.4

Suppose we introduce two more features:

• width alone, and
• area, which is defined as length times width

Suppose we also decide to remove the intercept term of our prediction rule. With all of these changes, our prediction rule is now

\text{predicted price} = w_1 \cdot \text{carat} + w_2 \cdot \text{length} + w_3 \cdot \text{width} + w_4 \cdot (\text{length} \cdot \text{width})

• Write out just the first 2 rows of the design matrix X for this new prediction rule. You do not need to simplify the numbers in your matrix, it is fine if they involve the multiplication symbol.
• Is the optimal coefficient for carat, w_1^*, for this new prediction rule guaranteed to be equal to 10000, the optimal coefficient for carat in our original prediction rule? No justification is necessary.

Problem 3

The DataFrame new_releases contains the following information for songs that were recently released:

• "genre": the genre of the song (one of the following 5 possibilities: "Hip-Hop/Rap", "Pop", "Country", "Alternative", or "International")

• "rec_label": the record label of the artist who released the song (one of the following 4 possibilities: "EMI", "SME", "UMG", or "WMG")

• "danceability": how easy the song is to dance to, according to the Spotify API (between 0 and 1)

• "speechiness": what proportion of the song is made up of spoken words, according to the Spotify API (between 0 and 1)

• "first_month": the number of total streams the song had on Spotify in the first month it was released

The first few rows of new_releases are shown below (though new_releases has many more rows than are shown below).

We decide to build a linear regression model that predicts "first_month" given all other information. To start, we conduct a train-test split, splitting new_releases into X_train, X_test, y_train, and y_test.

We then fit two linear models (with intercept terms) to the training data:

• Model 1 (lr_one): Uses "danceability" only.

• Model 2 (lr_two): Uses "danceability" and "speechiness" only.

Problem 3.1

Consider the following outputs.

>>> X_train.shape[0]
50

>>> np.sum((y_train - lr_two.predict(X_train)) ** 2)
500000 # five hundred thousand

What is Model 2 (lr_two)’s training RMSE (square root of mean squared error)? Give your answer as an integer.

Now, suppose we fit two more linear models (with intercept terms) to the training data:

• Model 3 (\texttt{lr\_drop}): Uses "danceability" and "speechiness" as-is, and one-hot encodes "genre" and "rec_label", using OneHotEncoder(drop="first").

• Model 4 (\texttt{lr\_no\_drop}): Uses "danceability" and "speechiness" as-is, and one-hot encodes "genre" and "rec_label", using OneHotEncoder().

Note that the only difference between Model 3 and Model 4 is the fact that Model 3 uses drop="first".

Problem 3.2

How many one-hot encoded columns are used in each model? In other words, how many binary columns are used in each model? Give both answers as integers.

Hint: Make sure to look closely at the description of new_releases at the top of the previous page, and don’t include the already-quantitative features.

number of one-hot encoded columns in Model 3 (lr_drop) =

number of one-hot encoded columns in Model 4 (lr_no_drop) =

Problem 3.3

Recall, in Model 4 (lr_no_drop) we one-hot encoded "genre" and "rec_label", and did not use drop="first" when instantiating our OneHotEncoder.

Suppose we are given the following coefficients in Model 4:

• The coefficient on "genre_Pop" is 2000.

• The coefficient on "genre_Country" is 1000.

• The coefficient on "danceability" is 10^6 = 1{,}000{,}000.

Daisy and Billy are two artists signed to the same "rec_label" who each just released a new song with the same "speechiness". Daisy is a "Pop" artist while Billy is a "Country" artist.

Model 4 predicted that Daisy’s song and Billy’s song will have the same "first_month" streams. What is the absolute difference between Daisy’s song’s "danceability" and Billy’s song’s "danceability"? Give your answer as a simplified fraction.

Problem 4

Consider the dataset shown below.

Problem 4.1

We want to use multiple regression to fit a prediction rule of the form H(x^{(1)}, x^{(2)}, x^{(3)}) = w_0 + w_1 x^{(1)}x^{(3)} + w_2 (x^{(2)}-x^{(3)})^2. Write down the design matrix X and observation vector \vec{y} for this scenario. No justification needed.

Problem 4.2

For the X and \vec{y} that you have written down, let \vec{w} be the optimal parameter vector, which comes from solving the normal equations X^TX\vec{w}=X^T\vec{y}. Let \vec{e} = \vec{y} - X \vec{w} be the error vector, and let e_i be the ith component of this error vector. Show that 4e_1+e_2+4e_3+e_4=0.

Problem 5

Jasmine and Aritra are trying to build models that predict the number of open rooms a hotel room has. To do so, they use price, the average listing price of rooms at the hotel, along with a one hot encoded version of the hotel’s chain. For the purposes of this question, assume the only possible hotel chains are Marriott, Hilton, and Other.

Problem 5.1

First, Jasmine fits a linear model without an intercept term. Her prediction rule, H_1, looks like:

H_{1}(x) = w_1 \cdot \texttt{price} + w_2 \cdot \texttt{is\_Marriott} + w_3 \cdot \texttt{is\_Hilton} + w_4 \cdot \texttt{is\_Other}

After fitting her model, \vec{w}^* = \begin{bmatrix}−0.5 \\ 200 \\ 300 \\ 50 \end{bmatrix}.

1. The Marriott Marquis San Diego Marina, a hotel in the Marriott chain, has an average listing price of $250. How many open rooms does H_1 predict it has? Give your answer as a number with no variables.
2. The Marriott Marquis San Diego Marina actually has 45 open rooms. What’s the squared loss of your prediction above? Give your answer as a number with no variables.
3. True or False: Because your answer to (ii) above is not 0, it means there were no hotels in the Marriott chain with an average listing price of $250 in the training set. d ( ) True ( ) False

Problem 5.2

As a reminder,

H_{1}(x) = w_1 \cdot \texttt{price} + w_2 \cdot \texttt{is\_Marriott} + w_3 \cdot \texttt{is\_Hilton} + w_4 \cdot \texttt{is\_Other}

\vec{w}^* = \begin{bmatrix}−0.5 \\ 200 \\ 300 \\ 50 \end{bmatrix}

H_2(x) = \beta_0 + \beta_1 \cdot \texttt{price} + \beta_2 \cdot \texttt{is\_Marriott} + \beta_3 \cdot \texttt{is\_Hilton}

After fitting his model, Aritra finds \beta_{0}^{*} = 50. Given that, what are \beta_{1}^{*}, \beta_{2}^{*}, and \beta_{3}^{*}? Give your answers as numbers with no variables.

Problem 6

Suppose we want to predict how long it takes to run a Jupyter notebook on Datahub. For 100 different Jupyter notebooks, we collect the following 5 pieces of information:

• cells: number of cells in the notebook

• lines: number of lines of code

• max iterations: largest number of iterations in any loop in the notebook, or 1 if there are no loops

• variables: number of variables defined in the notebook

• runtime: number of seconds for the notebook to run on Datahub

Then we use multiple regression to fit a prediction rule of the form H(\text{cells, lines, max iterations, variables}) = w_0 + w_1 \cdot \text{cells} \cdot \text{lines} + w_2 \cdot (\text{max iterations})^{\text{variables} - 10}

Problem 6.1

What are the dimensions of the design matrix X?

\begin{bmatrix} & & & \\ & & & \\ & & & \\ \end{bmatrix}_{r \times c}

So, what should r and c be for: r rows \times c columns.

Problem 6.2

In one sentence, what does the entry in row 3, column 2 of the design matrix X represent? (Count rows and columns starting at 1, not 0).

Problem 7

and consider the two prediction rules

Let \text{MSE} represent the mean squared error of a prediction rule, and let \text{MAE} represent the mean absolute error of a prediction rule. Select the symbol that should go in each blank.

Problem 7.1

\text{MSE}(H^*) ___ \text{MSE}(H^{\circ})

• \leq

• \geq

• =

Problem 8

Let X be a design matrix with 4 columns, such that the first column is a column of all 1s. Let \vec{y} be an observation vector. Let \vec{w}^* = (X^TX)^{-1}X^T\vec{y}. We’ll name the components of \vec{w}^* as follows:

\vec{w}^* = \begin{bmatrix} w_0^* \\ w_1^* \\ w_2^* \\ w_3^* \end{bmatrix}

In this problem, we’ll consider various modifications to the design matrix and see how they affect the solution to the normal equations.

Problem 8.1

Let X_a be the design matrix that comes from interchanging the first two columns of X. Let \vec{w_a}^* = (X_a^TX_a)^{-1}X_a^T\vec{y}. Express the components \vec{w_a}^* in terms of w_0^*, w_1^*, w_2^*, and w_3^* (which were the components of \vec{w}^*).

Problem 8.2

Let X_b be the design matrix that comes from adding one to each entry of the first column of X. Let \vec{w_b}^* = (X_b^TX_b)^{-1}X_b^T\vec{y}. Express the components \vec{w_b}^* in terms of w_0^*, w_1^*, w_2^*, and w_3^* (which were the components of \vec{w}^*).

Problem 8.3

Let X_c be the design matrix that comes from adding one to each entry of the third column of X. Let \vec{w_c}^* = (X_c^TX_c)^{-1}X_c^T\vec{y}. Express the components \vec{w_c}^* in terms of w_0^*, w_1^*, w_2^*, and w_3^*, which were the components of \vec{w}^*.

Problem 9

The two plots below show the total number of boots (top) and sandals (bottom) purchased per month in the df table. Assume that there is one data point per month.

For each of the following regression models, use the visualizations shown above to select the value that is closest to the fitted model weights. If it is not possible to determine the model weight, select “Not enough info”. For the models below:

• The notation boot refers to the number of boots sold.
• The notation sandal refers to the number of sandals sold.
• summer=1 is a column with value 1 if the month is between March (03) and August (08), inclusive.
• winter=1 is a column with value 1 if the month is between September (09) and February (02), inclusive.

Problem 9.1

boot = w_0

w_0:

• 0

• 50

• 100

• Not enough info

Problem 9.2

boot = w_0 + w_1 \cdot \text{sandals}

w_0:

• -100

• -1

• 0

• 1

• 100

• Not enough info

w_1:

• -100

• -1

• 1

• 100

• Not enough info

Problem 9.3

boot = w_0 + w_1 \cdot (\text{summer=1})

w_0:

• -100

• -1

• 0

• 1

• 100

• Not enough info

w_1:

• -80

• -1

• 0

• 1

• 80

• Not enough info

Problem 9.4

sandal = w_0 + w_1 \cdot (\text{summer=1})

w_0:

• -20

• -1

• 0

• 1

• 20

• Not enough info

w_1:

• -80

• -1

• 0

• 1

• 80

• Not enough info

Problem 9.5

sandal = w_0 + w_1 \cdot (\text{summer=1}) + w_2 \cdot (\text{winter=1})

w_0:

• -20

• -1

• 0

• 1

• 20

• Not enough info

w_1:

• -80

• -1

• 0

• 1

• 80

• Not enough info

w_2:

• -80

• -1

• 0

• 1

• 80

• Not enough info

Problem 10

Reggie and Essie are given a dataset of real features x_i \in \mathbb{R} and observations y_i. Essie proposes the following linear prediction rule: H_1(\alpha_0,\alpha_1) = \alpha_0 + \alpha_1 x_i. and Reggie proposes to use v_i=(x_i)^2 and the prediction rule H_2(\gamma_0,\gamma_1) = \gamma_0 + \gamma_1 v_i.

Problem 10.1

Give an example of a dataset \{(x_i,y_i)\}_{i=1}^n for which minimum MSE(H_2) < minimum MSE(H_1). Explain.

Problem 10.2

Give an example of a dataset \{(x_i,y_i)\}_{i=1}^n for which minimum MSE(H_2) = minimum MSE(H_1). Explain.

Problem 10.3

A new feature z has been added to the dataset.

Essie proposes a linear regression model using two predictor variables x,z as H_3(w_0,w_1,w_2) = w_0 + w_1 x_i +w_2 z_i.

Explain if the following statement is True or False (prove or provide counter-example).

Reggie claims that having more features will lead to a smaller error, therefore the following prediction rule will give a smaller MSE: H_4(\alpha_0,\alpha_1,\alpha_2,\alpha_3) = \alpha_0 + \alpha_1 x_i +\alpha_2 z_i + \alpha_3 (2x_i-z_i)

👋 Feedback: Find an error? Still confused? Have a suggestion? Let us know here.