Week 4 Discussion Questions

Selected STAT 432 Week 4 questions on lasso selection, prediction stability, penalty geometry, elastic net, standardization, validation, and coordinate descent.

These questions were selected and adapted from the Week 4 student submissions. Work through the reasoning with your group and be prepared to explain your conclusions.

Ten questions are listed below. Contributor NetIDs appear at the end of each question.

Question 01Identical predictors and variable importance

Does a Zero Coefficient Mean the Predictor Is Unimportant?

Two standardized predictors are identical. A lasso solution assigns coefficient $c>0$ to the first and zero to the second. Would splitting $c$ equally between them change the predictions or the lasso penalty? What does this imply about interpreting the zero coefficient? Would adding a ridge penalty distinguish the two allocations?

This question is contributed by kum2 and hx27.

Question 02Selection and prediction stability

How Different Are Predictions When Lasso Selects Different Variables?

Two centered predictors have unit variance and correlation $\rho$ . Two fitted models use $cX_1$ and $cX_2$ , respectively for prediction, for the same fixed coefficient $c$ . What happens to the prediction as $\rho$ approaches one? Does agreement between the predictions establish that either model predicts $Y$ accurately?

This question is contributed by zm25, bugatha2, shushim2, st58, and tingyun3.

Question 03Selection frequency and variable importance

Can an Irrelevant Predictor Be Selected More Often Than a Relevant One?

A predictor with true coefficient zero is highly correlated with a strong signal predictor. Another predictor has a small nonzero coefficient. Could lasso select the zero-coefficient predictor more frequently across repeated samples? What would this tell us about using selection frequency as a ranking of variable importance?

This question is contributed by wenhao7.

Question 04Penalty geometry and sparsity

Does Sparsity Disappear as Soon as the Penalty Becomes Smooth?

Consider the one-variable objective

$\frac12(\beta-a)^2+\lambda|\beta|^q, \qquad a\ne0,\quad \lambda>0,$

where $a$ is the unpenalized least-squares coefficient. Lasso uses $q=1$ , while ridge uses $q=2$ . If $1<q<2$ , can the minimizer be exactly zero? Explain by examining the objective near zero rather than relying only on a picture of the constraint region.

This question is contributed by heta2 and willm4.

Question 05Lasso and elastic net

Is the Same Numerical Penalty a Fair Comparison Between Lasso and Elastic Net?

At the same numerical $\lambda$ , elastic net retains both correlated signal predictors more often than lasso, but also selects more noise predictors. Does this establish an unavoidable cost of grouping correlated variables? Consider that the elastic-net absolute-value penalty has weight $\alpha\lambda$ , where $0<\alpha<1$ is the mixing parameter. How could we make a more informative comparison, and what should cross-validation choose if prediction is the goal?

This question is contributed by owenp3 and bjass2.

Question 06Cross-validation

Are We Averaging Validation Errors or Measuring Their Uncertainty?

Five-fold cross-validation produces five validation MSEs for each penalty. One student divides their sum by five; another divides their standard deviation by $\sqrt{5}$ . Are these competing estimates of the same quantity? Explain how each enters the minimum-error or one-standard-error rule, and why using more folds does not automatically guarantee a better penalty choice.

This question is adapted from related questions contributed by lb16, sgs11, and cudzich3.

Question 07Standardization and variable selection

Can Changing Measurement Units Change Which Variables Lasso Selects?

A predictor is converted from meters to centimeters. We refit lasso using the same numerical penalty without standardizing the predictors. Could the selected variables or predictions change, even though the underlying information is unchanged? Explain what standardization would change.

This question is adapted from related questions contributed by jingyi64 and melikah2.

Question 08Model selection and test-set leakage

Does Cross-Validation Protect Us If We Later Choose a Model Using the Test Set?

An analyst considers several elastic-net mixing parameters. For each one, cross-validation on the training data selects the penalty. The analyst then evaluates every resulting model on the final test set and reports the smallest test MSE. Is this an honest evaluation because every penalty was chosen by cross-validation? Explain how the comparison should be performed.

This question is adapted from a related question contributed by nkalele2.

Question 09Validation for future prediction

Is Random Cross-Validation Appropriate for Predicting the Future?

We use weekly sales data to build a model for predicting future sales. An analyst randomly divides the observations into five folds, allowing later weeks to help predict earlier weeks. Does this validation scheme represent the intended prediction task? Describe a more appropriate split and explain what it tests.

This question is adapted from a related question contributed by ericc13.

Question 10Coordinate descent and soft thresholding

When Should a Zero Coefficient Become Nonzero?

During coordinate descent for lasso, a coefficient is set to zero. After other coefficients are updated, could this coefficient need to become nonzero again? Explain how the partial residual changes and how the soft-thresholding update determines whether the coefficient stays zero.

This question is adapted from a related question contributed by prerith2.