PostProcess.py: add guardrail tests and fix OOM memory inefficiencies

[Copilot]

PostProcess.py caused OOM crashes due to redundant in-memory DataFrame copies accumulating across the per-sample processing loop in load_process_run3_samples.

Tests (tests/test_postprocess.py)

35 new unit and integration tests covering all key pure-Python functions:

• add_bdt_scores — binary, 3-class, 4-class, 5-class; with/without JEC shift suffix
• get_jets_for_txbb_sf — all process type branches
• get_nevents_data/signal/nosignal — sideband counting, mass window selection, cut masking
• fom_classic / fom_update — basic cases, zero-signal/background edges, ABCD correction
• Category assignment — VBF/Bin1/Bin2/Bin3/Fail logic, priority flag behavior

Memory fixes (load_process_run3_samples)

1. Progressive jshift merge — eliminates holding all JEC/JMR shift DataFrames simultaneously before concat:

# Before: holds all copies in memory until concat completes
bdt_events = pd.concat([bdt_events[jshift] for jshift in jshifts], axis=1)

# After: each DataFrame freed as soon as its new columns are transferred
bdt_events_merged = bdt_events.pop(jshifts[0])
for jshift in jshifts[1:]:
    _df = bdt_events.pop(jshift)
    for col in _df.columns:
        if col not in bdt_events_merged.columns:
            bdt_events_merged[col] = _df[col]
bdt_events = bdt_events_merged

2. Direct column assignment for more_vars and variation_vars — removes two intermediate pd.DataFrame() + pd.concat() round-trips:

# Before (two intermediate copies per block)
temp_df = pd.DataFrame(more_vars, index=bdt_events.index)
bdt_events = pd.concat([bdt_events, temp_df], axis=1)

# After
for col, values in more_vars.items():
    if col not in bdt_events.columns:
        bdt_events[col] = values

3. Eager events_dict deletion — frees the full raw sample DataFrame (the largest object per iteration) immediately after its last use (calculate_trigger_weights), rather than holding it alongside the growing bdt_events for the rest of the loop body.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[dprim7]

@copilot make PR into new branch called hh4b-dev