<style>
  body {
  text-align: justify}
</style>

```{css, echo=FALSE}
  .solution {
  background-color: #CCDDFF; /* Replace with your desired color */
  }
  
  blockquote {
  background-color: #CCDDFF; /* Replace with your desired color */
  font-family: "Times New Roman", serif; /* Change to your desired font family */
  /* font-size: 16px; /* Change to your desired font size */
  font-weight: bold; /* Makes the text bold */
  }
```

## Instruction

<span style='color:red'>__Please remove this section when submitting your homework.__</span>

Students are encouraged to work together on homework and/or utilize advanced AI tools. However, there are two basic rules:

  * __sharing, copying, or providing any part of a homework solution or code to others__ is an infraction of the [University's rules on Academic Integrity](https://studentcode.illinois.edu/article1/part4/1-401/). Any violation will be punished as severely as possible. __This simply means: write your own solution.__
  * __You must indicate clearly the level of involvement of AI tools (ChatGPT, Copilot, etc.) at the beginning of your homework.__ This includes whether you used it to generate code, debug code, or write explanations. If you did not use any AI tools, please state that explicitly. You are encouraged to briefly document your prompt(s) especially when AI is heavily involved. Please be aware that __you are responsible any consequences of using AI tools__, including incorrect or misleading information.

Final submissions must be uploaded to [Gradescope](https://www.gradescope.com/courses/570816). No email or hard copy will be accepted. Please refer to the course website for [**late submission policy and grading rubrics**](https://teazrq.github.io/stat432/syllabus.html).

  - You are required to submit the rendered file `HWx_yourNetID.pdf`. For example, `HW01_rqzhu.pdf`. Please note that this must be a `.pdf` file. __`.html` format will not be accepted__ because they are often not readable on gradescope. __Make all of your `R` code chunks visible for grading__. 
  - Include your Name and NetID at the beginning of your homework.
  - Make sure that you __set seed__ properly so that the results can be replicated if needed. Make sure the version of your `R` is $\geq 4.0.0$. This will ensure your random seed generation is the same as everyone else.
  - For some questions, there will be restrictions on what packages/functions you can use. Please read the requirements carefully. As long as the question does not specify such restrictions, you can use anything.
  - If you use this file or the example homework `.Rmd` file as a template, be sure to __remove this instruction__ section.


## Question 1 [40 pts]: Sparsity and Correlation

During our lecture, we considered a simulation model to analyze the variable selection property of Lasso. Now let's further investigate the prediction error of both Lasso and Ridge, and understand the bias-variance trade-off. Consider the linear model defined as:

\[
Y = X^\text{T} \boldsymbol \beta + \epsilon
\]

Where $\boldsymbol \beta = (\beta_1, \beta_2, \ldots, \beta_{100})^T$ with $\beta_1 = \beta_{11} = \beta_{21} = \beta_{31} = 0.4$ and all other $\beta$ parameters set to zero. The noise term $\epsilon \sim {\cal N}(0,1)$ is independent of $X$. The $p$-dimensional covariate $X$ follows a multivariate Gaussian distribution:

\[
\mathbf{X} \sim {\cal N}(\mathbf{0}, \Sigma_{p\times p}).
\]

In $\Sigma$, all diagonal elements are 1, and all off-diagonal elements are $\rho$.

a.  [10 points] A single Simulation Run

    -   Generate 200 training and 500 testing samples independently based on the above model.
    -   Use $\rho = 0.1$.
    -   Fit Lasso using `cv.glmnet()` on the training data with 10-fold cross-validation. Use `lambda.1se` to select the optimal $\lambda$.
    -   Report:
        -   Prediction error (MSE) on the test data.
        -   Report whether the true model was selected (you may refer to HW3 for this property).

b.  [15 points] Higher Correlation and Multiple Simulation Runs

    -   Write a code to compare the previous simulation with $\rho = 0.1, 0.3, 0.5, 0.7, 0.9$. (these numbers show keep the covariance matrix positive definite)
    -   Perform 100 simulation runs as in part a) and record the prediction error and the status of the variable selection for Lasso.
    -   Report the average prediction error and the proportion of runs where the correct model was selected for each value of $\rho$.
    -   Discuss the reasons behind any observed changes.

c.  [15 points] Ridge Regression

    -   Repeat task b) with the ridge regression. You do not need to record the variable selection status since ridge always select all variables.
    -   Report the average prediction error, do you see any difference between ridge and Lasso? Any performance differences within ridge regression as $\rho$ changes?
    -   Discuss the reasons behind any observed changes.


## Question 2 [25 pts]: Lasso Simulation

During our lecture, we considered a simulation model to analyze the variable selection property of Lasso. Now let's further investigate the prediction error cased by the $L1$ penalty under this model, and understand the bias-variance trade-off. For this question, your underlying true data generate should be 

\begin{align}
Y =& X^\text{T} \boldsymbol \beta + \epsilon \\
=& \sum_{j = 1}^p X_j 0.4^\sqrt{j} + \epsilon,
\end{align}

where `p`$= 30$, each $X_j$ is generated independently from $\cal{N}(0, 1)$, and $\epsilon$ also follows a standard normal, independent from $X$. The goal is to predict two target points and investigate how the prediction error changes under different penalties. The training data and two target testing points are defined by the following code. 

```{r}
    # target testing points
    p = 30
    xa = xb = rep(0, p)
    xa[2] = 1
    xb[10] = 1
```

Perform the following questions:

  * Use the grid of $\lambda$ values `exp(seq(-5, 5, 0.05))`
  * [10 pts] Repeat the following for `nsim`$= 200$ independent runs, with `n`$= 100$ observations in each run.
    * Generate the design matrix and $y$
    * Use the `glmnet()` function to fit Lasso on the $\lambda$ values
    * Predict two data points $\mathbf{x}_a$ and $\mathbf{x}_b$ on all the $\lambda$ values. To evaluate this prediction, we will use the following error metric:
    \[(\widehat\beta_\text{lasso}^T \mathbf{x} - \beta^T \mathbf{x})^2,\]
    where $\widehat\beta_\text{lasso}$ is the Lasso estimator corresponding to a particular $\lambda$, and $\beta$ is the true parameter
  * [5 pts] Based on your 200 simulation runs, average the results for each $\lambda$. You should then obtain two error curves (the prediction error curve across different $\lambda$ values), one for each prediction point, respectively. Plot your two curves using $\lambda$ (as x-axis) vs Error (as y-axis) in the same figure, and label these curves properly. The two should have displayed quite distinct patterns. 
  * [5 pts] The optimal $\lambda$ for `xb` should be much larger than `xa`. What are the corresponding best $\lambda$ and Error for each target point? 
  * [5 pts] Provide a discussion on why the the best $\lambda$ value for `xb` is larger than `xa`. Hint: pay attention to their covariate values and the associated $\widehat \beta$ parameters. Discuss how their predictions would trade bias and variance differently. 


## Question 3 [25 pts]: Shrinkage Methods and Testing Error

In this question, we will predict the number of applications received using the variables in the College dataset that can be found in `ISLR2` package. The output variable will be the number of applications (Apps) and the other variables are predictors. If you use Python, consider migrating the data to an excel file and read it in Python.

a.  [5 pts] Use the code below to divide the data set into a training set (600 observations) and a test set (177 observations). Fit a linear model (with all the input variables) using least squares on the training set using `lm()`, and report the test error (i.e., testing MSE).

```{r}
  library(ISLR2)
  data(College)
  
  # generate the indices for the testing data
  set.seed(7)
  test_idx = sample(nrow(College), 177)
  train = College[-test_idx,]
  test = College[test_idx,]
```


b.  [5 pts] Compare Lasso and Ridge regression on this problem. Train the model using cross-validation on the training set. Report the test error for both Lasso and Ridge regression. Use `lambda.min` and `lambda.1se` to select the optimal $\lambda$ for both methods.


c.  [15 pts] The `glmnet` package implemented a new feature called `relaxed` fits and the associated tuning parameter `gamma`. You can find some brief explanation of this feature at the documentation of this package. See

    * [CRAN Documentation](https://cran.r-project.org/web/packages/glmnet/glmnet.pdf)
    * [glmnet Vignette](https://glmnet.stanford.edu/articles/glmnet.html)

    Read these documentations regarding the `gamma` parameter, and summarize the idea of this feature in terms of the loss function being used. You need to write it specifically in terms of the data vectors $\mathbf y$ and matrix $\mathbf X$ and define any notations you need. Only consider the Lasso penalty for this question.

    After this, implement this feature and utilize the cross-validation to find the optimal $\lambda$ and $\gamma$ for the College dataset. Report the test error for the optimal model.



## Question 3 [10 pts]: Penalized Logistic Regression

In HW3, we used `golub` dataset from the `multtest` package. This dataset contains 3051 genes from 38 tumor mRNA samples from the leukemia microarray study Golub et al. (1999). The outcome `golub.cl` is an indicator for two leukemia types: Acute Lymphoblastic Leukemia (ALL) or Acute Myeloid Leukemia (AML). In genetic analysis, many gene expressions are highly correlated. Hence we could consider the Elastic net model for both sparsity and correlation.

```{r eval = FALSE}
  if (!require("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
  BiocManager::install("multtest")
```

Fit logistic regression to this dataset. Use a grid of $\alpha$ values in $[0, 1]$ and report the best $\alpha$ and $\lambda$ values using 19-fold cross-validation.





