This is absolute loss, and hence we’re looking for the minimizer of mean absolute error, which is the median, 15.

This is squared loss, and hence we’re looking for the minimizer of mean squared error, which is the mean, 16.

This is squared loss, multiplied by a constant. Note that when we go to minimize empirical risk for this loss function, we will take the derivative of empirical risk and set it equal to 0; at that point the constant factor of 4 can be divided from both sides, so this problem boils down to minimizing ordinary mean squared error. The only difference is that the graph of mean squared error will be stretched vertically by a factor of 4; the minimizing value will be in the same place.

For more justification, here we consider any general re-scaling $\alpha (y_i-h)^2$:

$$\begin{aligned} R_{sq}(h) &= \frac{1}{n} \sum_{i = 1}^n \alpha (y_i - h)^2 \\ &= \alpha \cdot \frac{1}{n} \sum_{i = 1}^n (y_i - h)^2 \\ \frac{d}{dh} R_{sq}(h) &= \alpha \cdot \frac{1}{n} \sum_{i = 1}^n 2(y_i - h)(-1) = 0\\ &\implies -\frac{2\alpha}{n}\sum_{i = 1}^n (y_i - h) = 0 \\ &\implies \sum_{i = 1}^n (y_i - h) = 0 \\ &\implies h^* = \frac{1}{n} \sum_{i = 1}^n y_i \end{aligned}$$

This is a scaled version of 0-1 loss. We know that empirical risk for 0-1 loss is minimized at the mode, so that also applies here. The mode, i.e. the most common value, is 12.

Note that we can write $(3y - 4h)^2$ as $\left( 3 \left( y - \frac{4}{3}h \right) \right)^2 = 9 \left( y - \frac{4}{3}h \right)^2$. As we’ve seen, the constant factor out front has no impact on the minimizing value. Using the same principle as in the last part, we can say that $$\frac{4}{3} h^* = \bar{x} \implies h^* = \frac{3}{4} \bar{x} = \frac{3}{4} \cdot 16 = 12$$

No minimizer.

Note that unlike $|y_i - h|$, $(y_i - h)^2$, and all of the other loss functions we’ve seen, $(y_i - h)^3$ tends towards $-\infty$, rather than having a minimum output of 0. This means that there is no $h$ that minimizes $\frac{1}{n} \sum_{i = 1}^n (y_i - h)^3$; the larger we make $h$, the more negative (and hence “smaller") this empirical risk becomes.

$$\begin{align*} \frac{d}{dh} L_B(y_i, h) &= \frac{d}{dh} \left[ \log \left( \frac{y_i}{h} \right) \right]^2 \\ &= 2 \cdot \log \left( \frac{y_i}{h} \right) \cdot \frac{d}{dh} \log \left( \frac{y_i}{h} \right) \\ &= 2 \cdot \log \left( \frac{y_i}{h} \right) \cdot \frac{d}{dh} \left( \log(y) - \log(h) \right) \\ &= 2 \cdot \log \left( \frac{y_i}{h} \right) \cdot \left( - \frac{1}{h} \right) \\ &= -\frac{2}{h} \log \left( \frac{y_i}{h} \right) \end{align*}$$

$$\begin{align*} R_B(h) &= \frac{1}{n} \sum_{i = 1}^n \left[ \log \left( \frac{y_i}{h} \right) \right]^2 \\ \frac{d}{dh} R_B(h) &= \frac{1}{n} \sum_{i = 1}^n \frac{d}{dh} \left[ \log \left( \frac{y_i}{h} \right) \right]^2 \\ &= \frac{1}{n} \sum_{i = 1}^n -\frac{2}{h} \log \left( \frac{y_i}{h} \right) \\ &= -\frac{2}{nh} \sum_{i = 1}^n \log \left( \frac{y_i}{h} \right) = 0 \\ 0 &= \sum_{i = 1}^n \log \left( \frac{y_i}{h} \right) = \sum_{i = 1}^n \left( \log(y_i) - \log(h)\right) \\ 0 &= \sum_{i = 1}^n \log(y_i) - \log(h) \sum_{i = 1}^n 1 \\ 0 &= \left( \log(y_1) + \log(y_2) + ... + \log(y_n) \right) - n \log(h) \\ \log(h^n) &= \log(y_1 \cdot y_2 \cdot ... \cdot y_n) \\ h^n &= y_1 \cdot y_2 \cdot ... \cdot y_n \\ h^* &= (y_1 \cdot y_2 \cdot ... \cdot y_n)^{\frac{1}{n}} \end{align*}$$

0.75.

It seems like there is a pretty strong, but not perfect, linear association between total bills and tips.

$M$ is less than 2.1. The variance is equal to the MSE of the constant prediction rule.

Note that the MSE of the best linear prediction rule will always be less than or equal to the MSE of the best constant prediction rule $h$. The only case in which these two MSEs are the same is when the best linear prediction rule is a flat line with slope 0, which is the same as a constant prediction. In all other cases, the linear prediction rule will make better predictions and hence have a lower MSE than the constant prediction rule.

In this case, the best linear prediction rule is clearly not flat, so $M < 2.1$.

We should add $w_1^*$ to each tip $y$.

First, we present the rigorous solution.

Let $\bar{x}_\text{old}$ represent the previous mean of the $x$’s and $\bar{x}_\text{new}$ represent the new mean of the $x$’s. Then, we know that $\bar{x}_\text{new} = \bar{x}_\text{old} + 1$.

Also, let $\bar{y}_\text{old}$ and $\bar{y}_\text{new}$ represent the old and new mean of the $y$’s. We will try and find a relationship between these two quantities.

We want the two intercepts to be the same. The intercept for the old line is $\bar{y}_\text{old} - w_1^* \bar{x}_\text{old}$ and the intercept for the new line is $\bar{y}_\text{new} - w_1^* \bar{x}_\text{new}$. Setting these equal yields

$$\begin{aligned} \bar{y}_\text{new} - w_1^* \bar{x}_\text{new} &= \bar{y}_\text{old} - w_1^* \bar{x}_\text{old} \\ \bar{y}_\text{new} - w_1^* (\bar{x}_\text{old} + 1) &= \bar{y}_\text{old} - w_1^* \bar{x}_\text{old} \\ \bar{y}_\text{new} &= \bar{y}_\text{old} - w_1^* \bar{x}_\text{old} + w_1^* (\bar{x}_\text{old} + 1) \\ \bar{y}_{\text{new}} &= \bar{y}_\text{old} + w_1^* \end{aligned}$$

Thus, in order for the intercepts to be equal, we need the mean of the new $y$’s to be $w_1^*$ greater than the mean of the old $y$’s. Since we’re told we’re adding the same constant to each $y$ that constant is $w_1^*$.

Another way to approach the question is as follows: consider any point that lies directly on a line with slope $w_1^*$ and intercept $w_0^*$. Consider how the slope between two points on a line is calculated: $\text{slope} = \frac{y_2 - y_1}{x_2 - x_1}$. If $x_2 - x_1 = 1$, in order for the slope to remain fixed we must have that $y_2 - y_1 = \text{slope}$. For a concrete example, think of the line $y = 5x + 2$. The point $(1, 7)$ is on the line, as is the point $(1 + 1, 7 + 5) = (2, 12)$.

In our case, none of our points are guaranteed to be on the line defined by slope $w_1^*$ and intercept $w_0^*$. Instead, we just want to be guaranteed that the points have the same regression line after being shifted. If we follow the same principle, though, and add 1 to every $x$ and $w_1^*$ to every $y$, the points’ relative positions to the line will not change (i.e. the vertical distance from each point to the line will not change), and so that will remain the line with the lowest MSE, and hence $w_0^*$ and $w_1^*$ won’t change.

$$\beta_1^* = \frac{r^2}{w_1^*}$$

Throughout this solution, let $x$ represent square footage and $y$ represent price.

We know that $w_1^* = r \frac{\sigma_y}{\sigma_x}$. But what about $\beta_1^*$?

When we take a rule that predicts price from square footage and transform it into a rule that predicts square footage from price, the roles of $x$ and $y$ have swapped; suddenly, square footage is no longer our independent variable, but our dependent variable, and vice versa for price. This means that the altered dataset we work with when using our new prediction rule has $\sigma_x$ standard deviation for its dependent variable (square footage), and $\sigma_y$ for its independent variable (price). So, we can write the formula for $\beta_1^*$ as follows: $$\beta_1^* = r \frac{\sigma_x}{\sigma_y}$$

In essence, swapping the independent and dependent variables of a dataset changes the slope of the regression line from $r \frac{\sigma_y}{\sigma_x}$ to $r \frac{\sigma_x}{\sigma_y}$.

From here, we can use a little algebra to get our $\beta_1^*$ in terms of one or more $n$, $r$, $w_0^*$, and $w_1^*$:

$$\begin{align*} \beta_1^* &= r \frac{\sigma_x}{\sigma_y} \\ w_1^* \cdot \beta_1^* &= w_1^* \cdot r \frac{\sigma_x}{\sigma_y} \\ w_1^* \cdot \beta_1^* &= ( r \frac{\sigma_y}{\sigma_x}) \cdot r \frac{\sigma_x}{\sigma_y} \end{align*}$$

The fractions $\frac{\sigma_y}{\sigma_x}$ and $\frac{\sigma_x}{\sigma_y}$ cancel out and we get:

$$\begin{align*} w_1^* \cdot \beta_1^* &= r^2 \\ \beta_1^* &= \frac{r^2}{w_1^*} \end{align*}$$

$$\beta_0^* = 1278.56$$

We start with the formula for the intercept of the regression line. Note that $x$ and $y$ are opposite what they’d normally be since we’re using price to predict square footage.

$$\beta_0^* = \bar{x} - \beta_1^* \bar{y}$$

We’re told that the average square footage of homes in the dataset is 2000, so $\bar{x} = 2000$. We also know from part (a) that $\beta_1^* = \frac{r^2}{w_1^*}$, and from the information given in this part this is $\beta_1^* = \frac{r^2}{w_1^*} = \frac{0.6^2}{250}$.

Finally, we need the average price of all homes in the dataset, $\bar{y}$. We aren’t given this information directly, but we can use the fact that $(\bar{x}, \bar{y})$ are on the regression line that uses square footage to predict price to find $\bar{y}$. Specifically, we have that $\bar{y} = w_0^* + w_1^* \bar{x}$; we know that $w_0^* = 1000$, $\bar{x} = 2000$, and $w_1^* = 250$, so $\bar{y} = 1000 + 2000 \cdot 250 = 501000$.

Putting these pieces together, we have

$$\begin{align*} \beta_0^* &= \bar{x} - \beta_1^* \bar{y} \\ &= 2000 - \frac{0.6^2}{250} \cdot 501000 \\ &= 2000 - 0.6^2 \cdot 2004 \\ &= 1278.56 \end{align*}$$

$54$.

To maximize the mean of the remaining 10 numbers, we want to minimize the numbers that are removed. The smallest possible non-negative number is 0, so to maximize the mean of the remaining 10, we should remove two 0s from the set of numbers. Recall that the sum of the 12 number set is $12 \cdot 45$; then, the maximum possible mean of the remaining 10 is

$$\frac{12 \cdot 45 - 2 \cdot 0}{10} = \frac{6}{5} \cdot 45 = 54$$

The dataset is {$3, 11$}.

We’ve already learned that $R_{sq}(h)$ is minimized at the mean of the data, and the minimum value of $R_sq(h)$ is the variance of the data. So we need to provide a dataset of two points with a mean of $7$ and a variance of $16$. Recall that the variance is the average squared distance of each data point to the mean. Since we want a variance of $16$, we can make each point $4$ units away from the mean. Therefore, our data set can be $y_1 = 3, y_2 = 11.$ In fact, this is the only solution.

A more calculative approach uses the formulas for mean and variance and solves a system of two equations:

$$\begin{aligned} \frac{y_1+y_2}{2} &= 7 \\ \frac12 \left((y_1 - 7)^2 + (y_2 - 7)^2 \right) &= 16 \end{aligned}$$

$$\text{Mean($D$)} = \frac{a}{5} + 3$$

$$\begin{cases} \text{Median($D$)} = 2 & \text{if a is in the range of } 0<a\leq2 \\ \text{Median($D$)} = a & \text{if a is in the range of } 2<a\leq5 \\ \text{Median($D$)} = 5 & \text{if a is in the range of } 5<a\leq\infty \\ \end{cases}$$

$$\dfrac{15}{4}<a<10$$

Since there are $3$ possible median values, we will have to discuss each situation separately.

In case $1$, when $0<a\leq2$, $\text{Median}(D) = 2$. So, we have:

$$\begin{align*} \text{Mean}(D) &< \text{Median}(D)\\ 3 + \frac{a}{5} &< 2\\ a&<-5 \end{align*}$$

But $a<-5$ is in conflict with the condition $0<a\leq2$, therefore there is no solution in this situation, and Median$(D) = 2$ is impossible.

In case $2$, when $2<a<5$, $\text{Median}(D) = a$. So, we have:

$$\begin{align*} \text{Mean}(D) &< \text{Median}(D)\\ 3 + \frac{a}{5} &< a\\ 3 &< \frac{4}{5} a\\ a &> \frac{15}{4}\\ \end{align*}$$

So $a$ has to be larger than $\frac{15}{4}$. But remember from the prerequisite condition that $2<a<5$.

To satisfy both conditions, we must have $\frac{15}{4}<a<5$.

In case 3, when $a\geq5$, $\text{Median}(D) = 5$. So, we have:

$$\begin{align*} \text{Mean}(D) &< \text{Median}(D)\\ 3 + \frac{a}{5} &< 5\\ a&<10 \end{align*}$$

combining with the prerequisite condition, we have $5\leq a<10$

Combining the range of all three cases, we have $\dfrac{15}{4}<a<10$ as our final answer.

$30.$

The minimizer of empirical risk for the constant model when using zero-one loss is the mode.

$7.$

The minimizer of empirical risk for the constant model when using absolute loss is the median. If the bar at 30 wasn’t there, the median would be 6, but the existence of that bar drags the “halfway” point up slightly, to 7.

$11.$

The minimizer of empirical risk for the constant model when using squared loss is the mean. The mean is heavily influenced by the presence of outliers, of which there are many at 30, dragging the mean up to 11. While you can’t calculate the mean here, given the large right tail, this question can be answered by understanding that the mean must be larger than the median, which is 7, and 11 is the next biggest option.

$15.$

The minimizer of empirical risk for the constant model when using infinity loss is the midrange, i.e. halfway between the min and max.

$6$

We know there is an even number of integers in this dataset because $10000 \% 2 = 0$. We can find the middle of the dataset as follows: $\frac{10000}{2} = 5000$. This means the element in the 5000th position and 5001st position can give us our median. The element at the 5000th position is a $5$ because $2500 + 2500 = 5000$. The element at the 5001st position is a $7$ because the next number after $5$ is $7$. We can then plug $5$ and $7$ into the equation: $$\frac{x_{5000} + x_{5001}}{2} = \frac{5 + 7}{2} = 6$$

The mean is smaller than the median.

We can calculate the mean as follows: $$\frac{2500 \cdot 3 + 2500 \cdot 5 + 4500 \cdot 7 + 500 \cdot 9}{10000} = 5.6$$ Using part (a) we know that $5.6 < 6$, which means the mean is smaller than the median.

This linear transformation reverses the order of the data because if $a<b$, then $-3a>-3b$ and so adding four to both sides gives $f(a)>f(b)$. Since $x_1\leq x_2 \leq \dots \leq x_n$, this means that the smallest of $f(x_1), f(x_2), \dots, f(x_n)$ is $f(x_n)$ and the largest is $f(x_1)$. Therefore,

$$\begin{aligned} EM(f(x_1), f(x_2), \dots, f(x_n)) &= \dfrac{f(x_n) + f(x_1)}{2} \\ &= \dfrac{-3x_n+4-3x_1+4}{2} \\ &= \dfrac{-3x_n-3x_1}{2} + 4\\ &= -3\left(\dfrac{x_1+x_n}{2}\right) + 4 \\ &= -3EM(x_1, x_2, \dots, x_n)+ 4\\ &= f(EM(x_1, x_2, \dots, x_n)). \end{aligned}$$

$\frac{\partial L}{\partial h} = -4w_ih(y_i^2 -h^2)$

To solve this problem we simply take the derivative of $L_\text{wolverine}(y_i, h) = w_i( y_i^2 - h^2 )^2$.

We can use the chain rule to find the derivative. The chain rule is: $\frac{\partial}{\partial h}[f(g(h))]=f'(g(h))g'(h)$.

Note that $(y_i^2 -h^2)^2$ is the area we care about inside of $L_\text{wolverine}(y_i, h) = w_i( y_i^2 - h^2 )^2$ because that is where $h$ is!. In this case $f(h) = h^2$ and $g(h) = y_i^2 - h^2$. We can then take the derivative of both to get: $f'(h) = 2h$ and $g'(x) = -2h$.

This tells us the derivative is: $\frac{\partial L}{\partial h} = (w_i) * 2(y_i^2 -h^2) * (-2h)$, which can be simplified to $\frac{\partial L}{\partial h} = -4w_ih(y_i^2 -h^2)$.

The recipe for average loss is to find the derivative of the risk function, set it equal to zero, and solve for $h^*$.

We know that average loss follows the equation $R(L(y_i, h)) = \frac{1}{n} \sum_{i=1}^n L(y_i, h)$. This means that $R_\text{wolverine}(h) = \frac{1}{n} \sum_{i = 1}^n w_i (y_i^2 - h^2)^2$.

Recall we have already found the derivative of $L_\text{wolverine}(y_i, h) = w_i ( y_i^2 - h^2)^2$. Which means that $\frac{\partial R}{\partial h}(h) = \frac{1}{n} \sum_{i = 1}^n \frac{\partial L}{\partial h}(h)$. So we can set $\frac{\partial}{\partial h}(h) R_\text{wolverine}(h) = \frac{1}{n} \sum_{i = 1}^n -4hw_i(y_i^2 -h^2)$.

We can now do the last two steps: $$\begin{align*} 0 &= \frac{1}{n} \sum_{i = 1}^n -4hw_i(y_i^2 -h^2)\\ 0&= \frac{-4h}{n} \sum_{i = 1}^n w_ih(y_i^2 -h^2)\\ 0&= \sum_{i = 1}^n w_i(y_i^2 -h^2)\\ 0&= \sum_{i = 1}^n w_iy_i^2 -w_ih^2\\ 0&= \sum_{i = 1}^n w_iy_i^2 - \sum_{i = 1}^n w_ih^2\\ \sum_{i = 1}^n w_ih^2 &= \sum_{i = 1}^n w_iy_i^2\\ h^2\sum_{i = 1}^n w_i &= \sum_{i = 1}^n w_iy_i^2\\ h^2 &= \frac{\sum_{i = 1}^n w_iy_i^2}{\sum_{i = 1}^n w_i}\\ h^* &= \sqrt{\frac{\sum_{i = 1}^n w_iy_i^2}{\sum_{i = 1}^n w_i}} \end{align*}$$

$y_1$

Recall from part b $h^* = \sqrt{\frac{\sum_{i = 1}^n w_i y_i^2}{\sum_{i = 1}^n w_i}}$.

The problem is asking us $\lim_{w_1 \rightarrow \infty} \sqrt{\frac{\sum_{i = 1}^n w_i y_i^2}{\sum_{i = 1}^n w_i}}$.

We can further rewrite the problem to get something like this: $\lim_{w_1 \rightarrow \infty} \sqrt{\frac{w_1 y_1^2 + \sum_{i=1}^{n-1}y_i^2}{w_1 + (n-1)}}$. Note that $\frac{\sum_{i=1}^{n-1}y_i^2}{n-1}$ is insignificant because it is a constant. Constants compared to infinity can be ignored. We now have something like $\sqrt{\frac{w_1y_1^2}{w_1}}$. We can cancel out the $w_1$ to get $\sqrt{y_1^2}$, which becomes $y_1$.

$8.$

Relative to the dataset with $x'$, the dataset with $x$ has an $x$-variable that’s “compressed” by a factor of 4, so the slope increases by a factor of 4 to $2 \cdot 4 = 8$.

Concretely, this can be shown by looking at the formula $2 = r\frac{SD(y')}{SD(x')}$, recognizing that $SD(y') = SD(y)$ since the $y$ values have the same spread in both datasets, and that $SD(x') = 4 SD(x)$.

$5$.

The key idea is that the regression line always passes through $(\text{mean } x, \text{mean } y)$ in the dataset we used to fit it. So, we know that: $2 \bar{x'} + 7 = \bar{y'}$. This first equation can be rewritten as: $2 \cdot (4\bar{x} - 3) + 7 = \bar{y} + 24$.

We’re also told this line passes through $(\bar{x}, \bar{y})$, which means that it’s also true that: $2 \bar{x} + 7 = \bar{y}$.

Now we have a system of two equations:

$\begin{cases} 2 \cdot (4\bar{x} - 3) + 7 = \bar{y} + 24 \\ 2 \bar{x} + 7 = \bar{y} \end{cases}$

$\dots$ and solving our system of two equations gives: $$\bar{x} = 5$$.