← return to study.practicaldsc.org

The problems in this worksheet are taken from past exams in similar classes. Work on them on paper, since the exams you take in this course will also be on paper.

We encourage you to complete this worksheet in a live discussion section. Solutions will be made available after all discussion sections have concluded. You don’t need to submit your answers anywhere.

Note: We do not plan to cover all problems here in the live discussion section; the problems we don’t cover can be used for extra practice.

Problem 1

Neerad wants to build a model that predicts the number of open rooms a hotel has, given various other features. He has a training set with 1200 rows available to him for the purposes of training his model.

Problem 1.1

Neerad fits a regression model using the GPTRegression class. GPTRegression models have several hyperparameters that can be tuned, including context_length and sentience.

To choose between 5 possible values of the hyperparameter context_length, Neerad performs k-fold cross-validation.

1. How many total times is a GPTRegression model fit?

• 4k

• 5k

• 240k

• 6000k

• 4(k − 1)

• 5(k − 1)

• 240(k − 1)

• 6000(k − 1)

2. Suppose that every time a GPTRegression model is fit, it appends the number of points in its training set to the list sizes. Note that after performing cross- validation, len(sizes) is equal to your answer to the previous subpart.

What is sum(sizes)?

• 4k

• 5k

• 240k

• 6000k

• 4(k − 1)

• 5(k − 1)

• 240(k − 1)

• 6000(k − 1)

Problem 1.2

The average training error and validation error for all 5 candidate values of context_length are given below.

Fill in the blanks: As context_length increases, model complexity __(i)__. The optimal choice of context_length is __(ii)__; if we choose a context_length any higher than that, our model will __(iii)__.

1. What goes in blank (i)?

• increases

• decreases

2. What goes in blank (ii)?

• 0.1

• 1

• 10

• 100

• 1000

3. What goes in blank (iii)?

• overfit the training data and have high bias

• underfit the training data and have high bias

• overfit the training data and have low bias

• underfit the training data and have low bias

Problem 2

Every week, Pranavi goes to her local grocery store and buys a varying amount of vegetable but always buys exactly one pound of meat (either beef, fish, or chicken). We use a linear regression model to predict her total grocery bill. We’ve collected a dataset containing the pounds of vegetables bought, the type of meat bought, and the total bill. Below we display the first few rows of the dataset and two plots generated using the entire training set.

Problem 2.1

Determine how each change below affects model bias and variance compared to the model H(x) described at the top of this page. For each change (i., ii., iii., iv.), choose all of the following that apply: increase bias, decrease bias, increase variance, decrease variance.

1. Add degree 3 polynomial features.
2. Add a feature of numbers chosen at random between 0 and 1.
3. Collect 100 more points for the training set.
4. Don’t use the 'veg' feature.

Problem 2.2

Suppose we predict 'total' from 'veg' using 8 models with different degree polynomial features (degrees 0 through 7). Which of the following plots display the training and validation errors of these models? Assume that we plot the degree of the polynomial features on the x-axis, mean squared error loss on the y-axis, and the plots share y-axes limits.

Training Error:

• A

• B

• C

• D

Validation Error:

• A

• B

• C

• D

Problem 3

Consider the least squares regression model, \vec{y} = X \vec{w}. Assume that X and \vec{y} refer to the design matrix and true response vector for our training data.

Let \vec{w}_\text{OLS}^* be the parameter vector that minimizes mean squared error without regularization. Specifically:

\vec{w}_\text{OLS}^* = \arg\underset{\vec{w}}{\min} \frac{1}{n} \| \vec{y} - X \vec{w} \|^2_2

Let \vec{w}_\text{ridge}^* be the parameter vector that minimizes mean squared error with L_2 regularization, using a non-negative regularization hyperparameter \lambda (i.e. ridge regression). Specifically:

\vec{w}_\text{ridge}^* = \arg\underset{\vec{w}}{\min} \frac{1}{n} \| y - X \vec{w} \|^2_2 + \lambda \sum_{j=1}^{p} w_j^2

For each of the following problems, fill in the blank.

Problem 3.1

If we set \lambda = 0, then \Vert \vec{w}_\text{OLS}^* \Vert^2_2 is ________ \Vert \vec{w}_\text{ridge}^* \Vert^2_2

• less than

• equal to

• greater than

• impossible to tell

Problem 3.2

For each of the remaining parts, you can assume that \lambda is set such that the predicted response vectors for our two models (\vec{y}^* = X \vec{w}_\text{OLS}^* and \vec{y}^* = X \vec{w}_\text{ridge}^*) is different.

The training MSE of the model \vec{y}^* = X \vec{w}_\text{OLS}^* is ________ than the model \vec{y}^* = X \vec{w}_\text{ridge}^*.

• less than

• equal to

• greater than

• impossible to tell

Problem 3.3

Now, assume we’ve fit both models using our training data, and evaluate both models on some unseen testing data.

The test MSE of the model \vec{y}^* = X \vec{w}_\text{OLS}^* is ________ than the model \vec{y}^* = X \vec{w}_\text{ridge}^*.

• less than

• equal to

• greater than

• impossible to tell

Problem 3.4

Assume that our design matrix X contains a column of all ones. The sum of the residuals of our model \vec{y}^* = X \vec{w}_\text{ridge}^* ________.

• equal to 0

• not necessarily equal to 0

Problem 3.5

As we increase \lambda, the bias of the model \vec{y}^* = X \vec{w}_\text{ridge}^* tends to ________.

• increase

• stay the same

• decrease

Problem 3.6

As we increase \lambda, the model variance of the model \vec{y}^* = X \vec{w}_\text{ridge}^* tends to ________.

• increase

• stay the same

• decrease

Problem 3.7

As we increase \lambda, the observation variance of the model \vec{y}^* = X \vec{w}_\text{ridge}^* tends to ________.

• increase

• stay the same

• decrease

Problem 4

One piece of information that may be useful as a feature is the proportion of SAT test takers in a state in a given year that qualify for free lunches in school. The Series lunch_props contains 8 values, each of which are either "low", "medium", or "high". Since we can’t use strings as features in a model, we decide to encode these strings using the following Pipeline:

# Note: The FunctionTransformer is only needed to change the result
# of the OneHotEncoder from a "sparse" matrix to a regular matrix
# so that it can be used with StandardScaler;
# it doesn't change anything mathematically.
pl = Pipeline([
    ("ohe", OneHotEncoder(drop="first")),
    ("ft", FunctionTransformer(lambda X: X.toarray())),
    ("ss", StandardScaler())
])

After calling pl.fit(lunch_props), pl.transform(lunch_props) evaluates to the following array:

array([[ 1.29099445, -0.37796447],
       [-0.77459667, -0.37796447],
       [-0.77459667, -0.37796447],
       [-0.77459667,  2.64575131],
       [ 1.29099445, -0.37796447],
       [ 1.29099445, -0.37796447],
       [-0.77459667, -0.37796447],
       [-0.77459667, -0.37796447]])

and pl.named_steps["ohe"].get_feature_names() evaluates to the following array:

array(["x0_low", "x0_med"], dtype=object)

Fill in the blanks: Given the above information, we can conclude that lunch_props has (a) value(s) equal to "low", (b) value(s) equal to "medium", and (c) value(s) equal to "high". (Note: You should write one positive integer in each box such that the numbers add up to 8.)

What goes in the blanks?

Problem 5

Suppose we have one qualitative variable that that we convert to numerical values using one- hot encoding. We’ve shown the first four rows of the resulting design matrix below:

Problem 5.1

Say we train a linear model m_1 on these data. Then, we replace all of the 1 values in column a with 3’s and all of the 1 values in column b with 2’s and train a new linear model m_2. Neither m_1 nor m_2 have an intercept term. On the training data, the average squared loss for m_1 will be ________ that of m_2.

• greater than

• less than

• equal to

• impossible to tell

Problem 5.2

To account for the intercept term, we add a column of all ones to our design matrix from part a. That is, the resulting design matrix has four columns: a with 3’s instead of 1’s, b with 2’s instead of 1’s, c, and a column of all ones. What is the rank of the new design matrix with these four columns?

• 1

• 2

• 3

• 4

Problem 5.3

Suppose we divide our sampling frame into three clusters of people, numbered 1, 2, and 3. After we survey people, along with our survey results, we save their cluster number as a new feature in our design matrix. Before training a model, what should we do with the cluster column? (Note: This part is independent of parts a and b.)

• Leave as is

• One-hot encode it

• Normalize it

• Use bag of words

Problem 6

We will aim to build a classifier that takes in demographic information about a state from a particular year and predicts whether or not the state’s mean math score is higher than its mean verbal score that year.

In honor of the rotisserie chicken event on UCSD’s campus in March of 2023, sklearn released a new classifier class called ChickenClassifier.

Problem 6.1

ChickenClassifiers have many hyperparameters, one of which is height. As we increase the value of height, the model variance of the resulting ChickenClassifier also increases.

First, we consider the training and testing accuracy of a ChickenClassifier trained using various values of height. Consider the plot below.

• Note that accuracy is a metric that measures how well a classifier performs by comparing the number of correct predictions to the total number of predictions.

Which of the following depicts training accuracy vs. height?

• Option 1

• Option 2

• Option 3

Which of the following depicts testing accuracy vs. height?

• Option 1

• Option 2

• Option 3

ChickenClassifiers have another hyperparameter, color, for which there are four possible values: "yellow", "brown", "red", and "orange". To find the optimal value of color, we perform k-fold cross-validation with k=4. The results are given in the table below.

Problem 6.2

Which value of color has the best average validation accuracy?

• "yellow"

• "brown"

• "red"

• "orange"

Problem 6.3

True or False: It is possible for a hyperparameter value to have the best average validation accuracy across all folds, but not have the best validation accuracy in any one particular fold.

• True

• False

Problem 6.4

Now, instead of finding the best height and best color individually, we decide to perform a grid search that uses k-fold cross-validation to find the combination of height and color with the best average validation accuracy.

For the purposes of this question, assume that:

• We are performing k-fold cross validation.
• Our training set contains n rows, where n is greater than 5 and is a multiple of k.
• There are h_1 possible values of height and h_2 possible values of color.

Consider the following three subparts:

• A. What is the size of each fold?
• B. How many times is row 5 in the training set used for training?
• C. How many times is row 5 in the training set used for validation?

Choose from the following options.

• k

• \frac{k}{n}

• \frac{n}{k}

• \frac{n}{k} \cdot (k - 1)

• h_1h_2k

• h_1h_2(k-1)

• \frac{nh_1h_2}{k}

• None of the above

Problem 7

Suppose we build a binary classifier that uses a song’s "track_name" and "artist_names" to predict whether its genre is "Hip-Hop/Rap" (1) or not (0).

For our classifier, we decide to use a brand-new model built into sklearn called the
BillyClassifier. A BillyClassifier instance has three hyperparameters that we’d like to tune. Below, we show a dictionary containing the values for each hyperparameter that we’d like to try:

hyp_grid = {
  "radius": [0.1, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 100], # 12 total
  "inflection": [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4], # 10 total
  "color": ["red", "yellow", "green", "blue", "purple"] # 5 total
}

To find the best combination of hyperparameters for our BillyClassifier, we first conduct a train-test split, which we use to create a training set with 800 rows. We then use GridSearchCV to conduct k-fold cross-validation for each combination of hyperparameters in hyp_grid, with k=4.

Problem 7.1

When we call GridSearchCV, how many times is a BillyClassifier instance trained in total? Give your answer as an integer.

Problem 7.2

In each of the 4 folds of the data, how large is the training set, and how large is the validation set? Give your answers as integers.

size of training set =

size of validation set =

Problem 7.3

Suppose that after fitting a GridSearchCV instance, its best_params_ attribute is

{"radius": 8, "inflection": 4, "color": "blue"}

Select all true statements below.

• The specific combination of hyperparameters in best_params_ had the highest average training accuracy among all combinations of hyperparameters in hyp_grid.

• The specific combination of hyperparameters in best_params_ had the highest average validation accuracy among all combinations of hyperparameters in hyp_grid.

• The specific combination of hyperparameters in best_params_ had the highest training accuracy among all combinations of hyperparameters in hyp_grid, in each of the 4 folds of the training data.

• The specific combination of hyperparameters in best_params_ had the highest validation accuracy among all combinations of hyperparameters in hyp_grid, in each of the 4 folds of the training data.

• A BillyClassifier that is fit using the specific combination of hyperparameters in best_params_ is guaranteed to have the best accuracy on unseen testing data among all combinations of hyperparameters in hyp_grid.

👋 Feedback: Find an error? Still confused? Have a suggestion? Let us know here.