Penalty + Ridge

[LucyMcGowan]

I am seeing unexpected behavior when trying to compare Ridge, Lasso, and Elastic Net (this is for a class lab and I want to make sure I'm pointing my students in the right direction). Below we create a grid of potential penalties and examine ridge (mixture = 0) lasso (mixture = 1) and elastic net (mixture = 0.5) -- we noticed that the penalty doesn't seem to be passing for ridge. I believe this is related to #431 and #486, and when we specify the path_values it does work but I'm just trying to understand whether this is expected behavior, if I am teaching students to use tidymodels + ridge do they always need to pass path_values for their penalties? it seems the penalty is ignored without notice (but maybe I missed something!)

suppressPackageStartupMessages(library(tidyverse))
library(tidymodels)
library(ISLR)

set.seed(7)
cv <- vfold_cv(Carseats, v = 5)

rec <- recipe(Sales ~ ., data = Carseats) |>
  step_dummy(all_nominal_predictors()) |>
  step_normalize(all_predictors())

elas_spec <- linear_reg(
  penalty = tune(),
  mixture = tune()
) |>
  set_engine("glmnet")

grid <- expand_grid(
  penalty = seq(0.01, 0.1, by = 0.01),
  mixture = c(0, 0.5, 1)
)

elas_wf <- workflow() |>
  add_recipe(rec) |>
  add_model(elas_spec)

elas_tune <- tune_grid(elas_wf, grid = grid, resamples = cv)

collect_metrics(elas_tune) |> 
  filter(.metric == "rmse") |>
  ggplot(aes(x = penalty, y = mean, group = mixture, color = mixture)) +
  geom_point() +
  geom_line()

## with path values

elas_spec_pv <- linear_reg(
  penalty = tune(),
  mixture = tune()
) |>
  set_engine(
    "glmnet",
    path_values = seq(0, 0.1, by = 0.01)
    )

elas_wf_pv <- workflow() |>
  add_recipe(rec) |>
  add_model(elas_spec_pv)

elas_tune_pv <- tune_grid(elas_wf_pv, grid = grid, resamples = cv)

collect_metrics(elas_tune_pv) |> 
  filter(.metric == "rmse") |>
  ggplot(aes(x = penalty, y = mean, group = mixture, color = mixture)) +
  geom_point() +
  geom_line()

^{Created on 2025-11-21 with reprex v2.1.1}

[topepo]

(For posterity, this is somewhat long)

Outside of using path_values, we don't pass specific penalties at fit time:

library(parsnip)

linear_reg(penalty = .1, mixture = 0) |> 
  set_engine("glmnet") |> 
  translate()
#> Linear Regression Model Specification (regression)
#> 
#> Main Arguments:
#>   penalty = 0.1
#>   mixture = 0
#> 
#> Computational engine: glmnet 
#> 
#> Model fit template:
#> glmnet::glmnet(x = missing_arg(), y = missing_arg(), weights = missing_arg(), 
#>     alpha = 0, family = "gaussian")

^{Created on 2025-11-21 with reprex v2.1.1}

When predicting, we do vary the penalty values when mixture = 0. Let's recreate things:

suppressPackageStartupMessages(library(tidymodels))
tidymodels_prefer()

rec <- recipe(Sales ~ ., data = ISLR::Carseats) |>
  step_dummy(all_nominal_predictors()) |>
  step_normalize(all_predictors())

Let's pull out the processed data set:

norm_data <- rec |> prep() |> bake(new_data = NULL)
# We'll use this later
design_matrix <- 
  rec |> 
  prep() |> 
  bake(new_data = NULL, all_predictors(), composition = "matrix")
design_matrix <- cbind(1, design_matrix)

Fit a single model. penalty is a placeholder for when we predict.

glmn_spec <- 
  linear_reg(penalty = 0.001, mixture = 0) |> 
  set_engine("glmnet")

fit_1 <- glmn_spec |> fit(Sales ~ ., data = norm_data)

Looking at the default coefficient path, the minimum penalty is not close to zero:

glmn_fit_1 <- fit_1 |> extract_fit_engine()

glmn_fit_1 |> 
  pluck("lambda") |> 
  min()
#> [1] 0.141173

I'm not sure what it does when approximating penalties outside of the path. Perhaps doing a linear interpolation to (0, 0)?

With tidymodels, does the penalty parameter have any effect at prediction time?

# uses placeholder of penalty = .1: 
predict(fit_1, new_data = norm_data[1,])$.pred
#> [1] 7.312714

predict(fit_1, new_data = norm_data[1,], penalty = 0.001)$.pred
#> [1] 7.312714
predict(fit_1, new_data = norm_data[1,], penalty = 0.1)$.pred
#> [1] 7.312714
predict(fit_1, new_data = norm_data[1,], penalty = 1)$.pred
#> [1] 7.528117

# doesn't change much given the range of the data
range(ISLR::Carseats$Sales)
#> [1]  0.00 16.27

This is the actual API that we use during tuning:

withr::with_options(
  list(pillar.sigfig = 15), 
  fit_1 |> 
    multi_predict(new_data = norm_data[1,], penalty = 10^(-4:0)) |> 
    pluck(".pred") |> 
    print()
  )
#> [[1]]
#> # A tibble: 5 × 2
#>   penalty              .pred
#>     <dbl>              <dbl>
#> 1  0.0001 7.31271376115355e0
#> 2  0.001  7.31271376115355e0
#> 3  0.01   7.31271376115355e0
#> 4  0.1    7.31271376115355e0
#> 5  1      7.52811678929796e0

^{Created on 2025-11-21 with reprex v2.1.1}

Not much is happening until > 0.1

Now that I'm fairly certain we didn't make any mistakes, here is what I think the actual issue is...

From Statistical learning with sparsity, the elastic net penalty is:

with footnote:

(the same equation is found in ?glmnet::glmnet)

In a way, this has the effect of putting their contributions on different scales. You'll often need higher pure ridge penalties to get the same effect as a penalty associated with a pure lasso model.

If I set the penalty range higher, I start to see the ridge penalty have an effect:

suppressPackageStartupMessages(library(tidymodels))
library(ISLR)

set.seed(7)
cv <- vfold_cv(Carseats, v = 5)

rec <- recipe(Sales ~ ., data = Carseats) |>
  step_dummy(all_nominal_predictors()) |>
  step_normalize(all_predictors())

elas_spec <- linear_reg(
  penalty = tune(),
  mixture = tune()
) |>
  set_engine("glmnet")

grid <- expand_grid(
  penalty = 10^seq(-2, 0, length.out = 20),
  mixture = c(0, 0.5, 1)
)

elas_wf <- workflow() |>
  add_recipe(rec) |>
  add_model(elas_spec)

elas_tune <- tune_grid(elas_wf, grid = grid, resamples = cv)

autoplot(elas_tune, metric = "rmse")

The y-axis scale gets blown up here but it's the same pattern at the micro level:

^{Created on 2025-11-21 with reprex v2.1.1}

Finally, tidymodels attempts to avoid the apples:oranges difference in the pure penalties by omitting a pure ridge penalty in the default grid. If you don't specify the range of the mixture parameter, the minimum value for that is set to 0.05 instead of zero. This can avoid issues like the one you are seeing by not displaying both pure penalties on the page at once. It's a little sneaky, but it's for a good cause 😎

If you know where to look, you could see this via:

elas_wf |> tunable() |> filter(name == "mixture") |> pluck("call_info")
#> [[1]]
#> [[1]]$pkg
#> [1] "dials"
#> 
#> [[1]]$fun
#> [1] "mixture"
#> 
#> [[1]]$range
#> [1] 0.05 1.00

These hidden differences in the ranges have been noted before (#725), and we want to document them more thoroughly. @hfrick and I recently discussed tidymodels/dials#384 to determine what kind of API we can create to illustrate this.