Debugging CISM Reprosums

[Katetc]

This is a local issue related to the reproducible sums PR. The purpose is to plan and take notes on what is likely to be a pretty complicated debugging task.

Here's my comment on the issue:

HI Bill, quick update. On Tuesday night I started an Intel "ERP" test - exact restart with changing processors in CESM. I checked today and it failed with a seg fault after the restart with the stack trace below. This test runs once for 3 years with 256 tasks and then restarts for 1 year with 128 tasks.

dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000001A82C30 cism_reprosum_mod 856 cism_reprosum_mod.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000000E8CC55 cism_parallel_mp_ 9600 parallel_mpi.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000000DD323D cism_parallel_mp_ 6222 parallel_mpi.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000000A4D5F6 glide_diagnostics 385 glide_diagnostics.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000000A47E73 glide_diagnostics 82 glide_diagnostics.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 0000000001A40E17 glad_initialise_m 277 glad_initialise.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 00000000009EAE50 glad_main_mp_glad 275 glad_main.F90
dec0385.hsn.de.hpc.ucar.edu 75: cesm.exe 00000000009C2178 glc_initmod_mp_gl 368 glc_InitMod.F90

Bill responded with:

@Katetc, thanks for the update. It looks like the failure is at l. 856 of cism_reprosum_mod.F90, which is this call:

    call mpi_allreduce (arr_lextremes, arr_gextremes, 2*(nflds+1), &
                        MPI_INTEGER, MPI_MIN, mpi_comm, ierr)

Backing up a bit, the parallel_global_sum call is here in glide_diagnostics:

  tot_area = parallel_global_sum(cell_area, parallel, ice_mask)

I don't see anything problematic about the cell_area field. But this is the first call to parallel_global_sum on restart, and I suspect that any call to parallel_global_sum will generate the same error.

Just to be sure, could you try commenting out the call to glide_write_diag at l. 82 and repeating the test? If you get another seg fault traceable to a different module, that would show that the issue is with the reprosums and not the cell_area field. From there, we can try to figure out what might go wrong on restart in CESM.

Meanwhile, could you point me to your config file? I'll try a similar test with the standalone model and see if it completes.

Further testing showed errors with singular matrices. I tried Claude who suggested a missing halo update but that didn't help. Here's my ideas for how to fix this:

• Notes on the exact error for each compiler/test case
• Notes on what Claude suggested
• Send Bill requested config file and case set up.
• Get DDT set up
• Run restart in debugger

[Katetc]

Here's some notes from my testing today:

1. Currently at hash 2edf195ea52b485a (Kate's output fixes), with the added halo update that Claude suggested (whatever).

2. With the Exact Restarts turned off (reproducible_sums = .false.) I had a 3 year Intel ERS T1850Gg case run successfully.

3. With the Exact Restarts turned on (reproducible_sums = .true.) I had a 3 year Intel ERS T1850Gg case run successfully.

4. With the Exact Restarts turned on (reproducible_sums = .true.) a 6 year Intel ERP T1850Gg test failed.
   It failed in the first run, with a floating overflow in glide_diagnostics, possibly line 449 calculating local energy? This happens during year 4, so wouldn't necessarily show up in a 3 year test like 2 & 3. I'll re-run those tests longer and see if I see the same issues.

5. With the Exact Restarts turned on (reproducible_sums = .true.) I had a 3 year gnu ERS T1850Gg case run fail.
   It failed almost immediately with the error * FATAL ERROR : Jacobian matrix is singular and no backtrace

6. With the Exact Restarts turned on (reproducible_sums = .true.) I had a 6 year gnu ERS T1850Gg case run fail.
   Same as above.

7. With the Exact Restarts turned on (reproducible_sums = .true.) I had a 6 year Intel ERS T1850Gg case run fail.
   Same failure as 4: Floating overflow in glide_diagnostics.

8. With the Exact Restarts turned off (reproducible_sums = .false.) I had a 6 year Intel ERS T1850Gg case run fail.
   Same failure as 4: Floating overflow in glide_diagnostics

9. With the Exact Restarts turned off (reproducible_sums = .false.) I had a 3 year gnu ERS T1850Gg case run fail
   Same failure as 5: Jacobian matrix is singular

So, the moral of the story is that, for me, the runs are crashing without restarting at all. The reproducible_sums flag doesn't matter at all. I get different errors for Intel and gnu compilers with the Intel compiler running for 4+ years and the gnu compiler crashing on the first velocity solve.

I'll hand this back to Bill L, with these final notes...

A good example Intel case can be found here:
/glade/derecho/scratch/katec/ERS_D_Ly6.f09_g17_gris4.T1850Gg.derecho_intel.20260416_141939_bdwc7k

With the config file here (and logs in the same directory):
/glade/derecho/scratch/katec/ERS_D_Ly6.f09_g17_gris4.T1850Gg.derecho_intel.20260416_141939_bdwc7k/run/cism.gris.config

A good example gnu case can be found here:
/glade/derecho/scratch/katec/ERS_D_Ly6.f09_g17_gris4.T1850Gg.derecho_gnu.20260416_141654_tntimg

With the config file here (and logs in the same directory):
/glade/derecho/scratch/katec/ERS_D_Ly6.f09_g17_gris4.T1850Gg.derecho_gnu.20260416_141654_tntimg/run/cism.gris.config

And more information about running CESM tests is now up on the wiki!

[Katetc]

quick note on the "fix" suggested by Claude. Here's the git diff:

diff --git a/libglissade/glissade.F90 b/libglissade/glissade.F90
index 2f3acb4f..6926f3cd 100644
--- a/libglissade/glissade.F90
+++ b/libglissade/glissade.F90
@@ -3211,6 +3211,11 @@ contains
 
     model%geometry%usrf(:,:) = max(0.d0, model%geometry%thck(:,:) + model%geometry%lsrf(:,:))
 
+    ! Halo update for upper surface elevation
+    ! This is needed before computing stagusrf, so that stagusrf is computed from 
+    ! correct values in halo cells when staggering values from neighboring processors.
+    call parallel_halo(model%geometry%usrf, parallel)
+
     ! ------------------------------------------------------------------------
     ! Compute some quantities on the staggered grid.
     ! These are not used by the Glissade velocity solver, but are used for diagnostics

It didn't make a difference and I'm reverting the change because Bill thinks his most recent commit fixed the problem.