Skip to content

Add n_eff and r_eff to kfold diagnostics#1875

Open
florence-bockting wants to merge 5 commits intomasterfrom
pointwise-ess-kfold
Open

Add n_eff and r_eff to kfold diagnostics#1875
florence-bockting wants to merge 5 commits intomasterfrom
pointwise-ess-kfold

Conversation

@florence-bockting
Copy link
Copy Markdown
Collaborator

@florence-bockting florence-bockting commented Mar 30, 2026

Description

This PR adds n_eff and r_eff to the kfold() diagnostics slot (which currently only incorporates pareto_k). The computation logic is as follows:

  1. unnormalized log weights: $\log(w_{si}) = - \log[p(y_i \mid \theta_s)]$
  2. normalized weights: $\log(\tilde w_{si}) = \log(w_{si}) - \log[\sum^S_{s'=1} w_{s'i}] \rightarrow \tilde w_{si} = \exp[\log(\tilde w_{si})]$
  3. effective sample size: $n_{eff,i}= 1/\sum_{s=1}^{S} \tilde w_{si}^2$

whereby $r_{eff,i} = 1$.

TODO

  • add computation for n_eff and r_eff
  • add n_eff and r_eff to kfold diagnostics output
  • add unittests for updated output structure

@florence-bockting
Copy link
Copy Markdown
Collaborator Author

@avehtari Can you check whether the computation is correct? Additionally, I am uncertain about r_eff. It is currently hard coded at 1, not sure whether this is appropriate.

@avehtari
Copy link
Copy Markdown
Contributor

If the posterior inference has passed convergence checks, r_eff tends to not have much influence to MCSE, and for simplicity we decided to switch using default 1. loo() output includes now line telling r_eff range, so that it is possible to see whether 1 or comouted values are used. I'll check the rest later

florence-bockting and others added 4 commits March 30, 2026 13:13
* add dims to kfold output structure

* add test for new 'dims' attribute in kfold output structure

* fix: show warning in kfold only of high pareto-k values exist

* tests: add test for updated warning message

* style: trim whitespaces

---------

Co-authored-by: Florence Bockting <florence.bockting@aalto.fi>
@florence-bockting florence-bockting marked this pull request as ready for review April 23, 2026 06:41
@florence-bockting
Copy link
Copy Markdown
Collaborator Author

florence-bockting commented Apr 23, 2026

@avehtari Can you please check the final computation of r_eff, n_eff and k_threshold. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants