Binospec updates and first draft of IFU support#2080
Open
tepickering wants to merge 60 commits intopypeit:developfrom
Open
Binospec updates and first draft of IFU support#2080tepickering wants to merge 60 commits intopypeit:developfrom
tepickering wants to merge 60 commits intopypeit:developfrom
Conversation
Add MMTBINOSPECIFUSpectrograph as a SlicerIFU subclass of the existing Binospec spectrograph, implementing Phases 1-2 of IFU support: spectrograph definition with IFU metadata (atmospheric params for DAR correction), frame typing via MASK='IFU', grating- dependent sky subtraction parameters, IFU-tuned edge tracing and flat fielding, datacube WCS and binning, and fiber geometry methods including reference profile loading, sky fiber identification (40 dedicated fibers/side), fiber-to-sky position mapping, and cross- correlation-based fiber ID matching ported from the IDL pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add RST document comparing error propagation and sky subtraction between the IDL and PypeIt pipelines, with fiber-specific considerations for IFU support. Update mmt_binospec.rst with IFU mode documentation and link to the comparison. Add to spectrographs toctree. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use dedicated sky fibers for joint sky fit and flexure correction in SlicerIFU pipeline when spectrograph provides get_sky_fiber_mask() - Guard against empty arrays in DataContainer.__setitem__ - Reduce memory in fit2tiltimg and flatfield by evaluating tilts only at slit pixels instead of full-frame meshgrid arrays - Tune slit edge parameters for densely-packed IFU fibers: lower min_edge_side_sep and minimum_slit_length, override parent MOS settings in config_specific_par - Disable slit edge tweaking and object finding for fiber-fed IFU - Set trim_edge=(0,0) to avoid trimming narrow fiber slits Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nto binospec_updates
Handle masked slits (BADFLATCALIB/BADSKYSUB) gracefully throughout the datacube pipeline: skip empty slits in wavelength min/max calculation (coadd3d.py), skip masked slits in RA/DEC image generation instead of raising PypeItError (slittrace.py), and skip slits with no good pixels during subpixel resampling (datacube.py). Also add ra, dec, and exptime handlers to Binospec IFU compound_meta for coadd3d compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing pypeit_coadd_datacube script is designed for slicer-based IFUs and produces incorrect results for the fiber-fed Binospec IFU. This new script handles the fiber-based workflow: extracting 1D spectra per fiber, sky subtracting using dedicated sky fibers, mapping science fibers to sky positions via the layout file, combining both detectors, and interpolating onto a regular spatial grid using scipy.interpolate.griddata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The two Binospec detectors produce mirror-image spectra, so fibers on side B (DET02) run in the opposite spatial direction compared to side A. Reverse the sequential layout mapping for side B so that the first science fiber on the detector (leftmost) maps to the highest layout index (least negative X on sky) rather than the lowest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Default sky subtraction now uses PypeIt's per-fiber B-spline sky model from the spec2d file, which better handles bright sky features. The previous dedicated sky-fiber method is available via --use_fibers. Documentation updated with IFU reduction workflow, datacube script usage, command-line options, and output format. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Divide each fiber's extracted flux by its relative throughput from fiber_illumination.fits before sky subtraction. This corrects for up to ~30% sensitivity variation between fibers that was causing visible patterns in the datacube at bright wavelength channels. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each input spec2d file now produces its own output datacube. The processing is extracted into a _build_cube() helper called in a loop. The --output flag is restricted to single-file usage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Default fiber extraction is now Horne (1986) profile-weighted using a Gaussian spatial profile derived from the slit edges. This gives better S/N than boxcar by down-weighting noisy pixels at the fiber edges. Boxcar extraction remains available via --boxcar for extended sources that may not match the Gaussian profile assumption. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract fiber spatial profiles from the flat field calibration image (median across wavelength) instead of assuming a Gaussian shape. This better captures the true fiber profile and reduces noise in the output datacubes. Gaussian profiles remain available via --gaussian and are used automatically as a fallback if the flat field cannot be loaded. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add type annotations to all new functions and methods introduced in PR 2080: _load_flat, _build_empirical_profiles, _build_cube in the datacube script, and load_sky_layout, load_fiber_illumination, get_sky_fiber_mask, get_science_fiber_layout_indices, _ifu_calib_path, load_fiber_ref_profile in the spectrograph class. Also refactor _build_cube to use local imports instead of receiving modules as parameters, making the function signature cleaner and type-checkable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Always process both detectors since a single-detector cube would produce a half-field with off-center WCS and edge artifacts. If a detector is missing from the spec2d file it is skipped gracefully. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Set skip_second_find and skip_final_global to reduce the number of passes through the ~360 fibers per detector during object finding, which is unnecessary for fiber IFU data. A proper skip_objfind parameter would eliminate this overhead entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document that switching to an accelerated BLAS library (e.g. Apple newaccelerate on macOS 13.3+) can yield ~3x speedup for datacube construction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a new 'Fiber' pypeline path alongside MultiSlit, Echelle, and SlicerIFU. For fiber spectrographs, each fiber IS the object — no peak-detection object finding is needed. Instead, one SpecObj is created per fiber with the trace set to the fiber center from slit edges. New classes: - FiberFindObjects: creates fiber objects, uses joint sky subtraction with dedicated sky fibers, resets reduce_bpm after per-slit sky sub - FiberExtract: boxcar + Horne (1986) optimal extraction using empirical spatial profiles from the flat field Changes to Binospec IFU: - Switch from pypeline='SlicerIFU' with skip_extraction=True to pypeline='Fiber' with full extraction producing spec1d files - Fix compound_meta header cards (TEMPERAT->TEMP, HUMIDITY->HUMID, PARANGLE->PA) - Disable spectral flexure correction (active flexure control) Pypeline support added to: specobj, specobjs, slittrace, show_2dspec, spectrograph (IFU subheader), pypeit_steps (initial_slits). Robustness fixes: - Guard for empty slits in illum_profile_spectral_poly (flat.py) - Guard for empty data in 2D bspline flat fit (flatfield.py) - Guard for empty wavelength data in binospec_ifu_cube.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add get_fiber_metadata() to Spectrograph base class, allowing fiber-fed spectrographs to map detected trace positions to instrument-defined fiber IDs, names, and types. Override in BinospecIFUSpectrograph to cross-correlate against the reference fiber profile and assign physical fiber IDs (FIB_ID) and names (e.g. A42, SKY6-1) to each SpecObj via MASKDEF_ID and MASKDEF_OBJNAME. Refactor binospec_ifu_cube.py to derive sky fiber mask from get_fiber_metadata() fiber_type instead of get_sky_fiber_mask(). Disable spectral flexure correction for Binospec IFU (spec_method=skip) since the instrument has active flexure control. Update CLAUDE.md with Fiber pypeline and Binospec IFU documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Refactor binospec_ifu_cube.py to accept both spec1d and spec2d input files. For spec1d, reads pre-extracted OPT/BOX spectra (sky already subtracted by pipeline). Shared steps 2-7 extracted into _build_cube_common(). Update Binospec documentation to reflect Fiber pypeline, spec1d output, fiber naming, throughput correction, and spec1d/spec2d cube building workflows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use fiber IDs from metadata instead of sequential positional indexing for cube building. The previous approach assumed all fibers were present and in order, which garbled the spatial mapping when fibers were missing from spec1d files. - specobj.py: Use MASKDEF_OBJNAME (e.g., SCI_042) instead of SLITmmmm in spec1d extension names for Fiber pypeline - binospec_ifu_cube.py: Fix illumination correction to look up values by fiber ID via the reference profile, not by array position - mmt_binospec.py: Rewrite get_science_fiber_layout_indices to pair live reference fibers (sorted by detector position) with live layout entries, excluding dead fibers and reversing side B for the mirror- image detector geometry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create pypeit_binospec_ifu_illumcorr script that applies fiber-to-fiber throughput corrections directly to spec1d FITS files. This allows illumination correction before cube building, and sets an ILLUMCOR header flag so downstream tools can detect pre-corrected files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New pypeit_binospec_ifu_illumcorr command applies fiber-to-fiber illumination corrections directly to Binospec IFU spec1d FITS files, correcting for relative throughput differences between fibers. Sets ILLUMCOR header flag to prevent double-application by the cube builder. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… files Check ILLUMCOR header flag on spec1d files and skip Step 2 if the illumination correction was already applied by pypeit_binospec_ifu_illumcorr. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
Author
|
going to have to change tack on how the fiber illumination correction is being handled. need to fold it into the pixelflat or create an illumflat so that it gets folded properly into the sky model. more changes incoming... |
Add modify_pixelflat() hook to base Spectrograph class (no-op) and override in MMTBINOSPECIFUSpectrograph to scale each fiber's region in pixelflat_norm by its relative throughput from fiber_illumination.fits. Called from calibrations after building the flat and before saving to disk, so the correction flows through the normal PypeIt processing: the sky model fit, extraction, and all downstream steps see illumination-corrected data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When using dedicated sky fibers, the joint sky fit restricts inmask to sky fiber pixels only. But global_skysub computes the masked fraction relative to all pixels in thismask (all fibers), so with 40/360 sky fibers ~88% appears "masked", exceeding the 80% max_mask_frac threshold. The sky model was silently zeroed out. Override max_mask_frac=1.0 when dedicated sky fibers are in use, since the high masked fraction is expected (science fibers are intentionally excluded from the fit). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Measures bright sky emission line fluxes in each fiber, normalizes by the median across fibers, and interpolates to build a wavelength- dependent throughput correction applied before joint sky subtraction. This accounts for throughput differences between sky fibers (bare fibers) and science fibers (lenslet-fed) that cannot be captured by the dome-flat-based illumination correction. The correction is computed from 8 sky lines ([OI], O2, OH) and applied as an in-place division of the science image copy used for B-spline fitting. Variance propagation is handled via apply_relative_scale after the iterative loop converges. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fiber throughput correction is now handled entirely within the pipeline (dome-flat illumination baked into pixel flat + sky-line per-fiber correction during joint sky subtraction), so the post-processing illumcorr script and the cube builder's Step 2 are no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use the median ratio across the 3 best-measured sky lines per fiber instead of interpolating between lines across wavelength. The fiber throughput correction should be a single scalar per fiber; wavelength-dependent throughput belongs in spectral flux calibration from standard stars. Also fix spec2d doc to say detectors are written to separate extensions within a single file, not separate files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
Author
something about the IFU data is messing up overscan subtraction. sky fiber traces in some regions are getting significantly over-subtracted. the regions look normal in the raw data and in the IDL pipeline processed data. claude came up with a plan to port the way IDL handles overscan over. pypeit was doing it in a relatively rudimentary way that hadn't been touched in years. |
Replicates the IDL clean_overscan_vector function from bino_mosaic.pro: median-filter the 1D overscan vector, sigma-clip outliers, and interpolate over rejected pixels. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace simple median overscan with IDL bino_mosaic.pro approach: - Y-axis first, then X-axis (was X-first) - Sigma-clipped mean (resistant_mean) instead of simple median - Uses both prescan and postscan regions for X-axis (was prescan only) - Outlier cleaning via clean_overscan_vector on each 1D vector Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store degree-4 polynomial coefficients from IDL calibration file (scicam_bino_sep2017.fits) as a class attribute on MMTBINOSPECSpectrograph and apply correction in binospec_read_amp() after overscan subtraction, matching the IDL pipeline order (overscan -> nonlinearity -> gain). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…arity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generate pre-built bad pixel mask images for both Binospec detectors by porting all three masking layers from the IDL pipeline: individual bad pixels from badpix_binospec.fits, hard-coded bad columns, and detector trap regions including the diagonal defect on Side A. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the incomplete hard-coded bad pixel mask with comprehensive static BPM images ported from the IDL pipeline. The static files include ~12,500 individual bad pixels, 22 bad columns, and detector trap regions including a diagonal defect on Side A. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # pypeit/specobjs.py
Collaborator
Author
|
the binospec bad pixel mask was badly, badly out of date. the code comments even said so! fixed that here because it was messing up making IFU maps. |
…I change Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scattered light calibration needs a high-count frame to fit the model. Like KCWI, use the pixelflat as the scattlight frame so the calibration is built automatically without requiring separate scattlight-typed raw frames. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IDL pipeline builds a fresh scattered light model for every frame (flat and science). Set method='frame' for science frames so each gets its own fit to the inter-fiber gap pixels, matching the IDL approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…eded Remove scattered light subtraction from flat frames (not needed for geometry/pixel response). Allow method='frame' and method='archive' to work without a scattlight calibration frame by relaxing the None check in rawimage.process() and providing a pad fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move msscattlight.scattlight_param access into the method='model' block where a calibration is guaranteed. The method='frame' path gets its starting parameters from scattered_light_archive() and doesn't need the calibration object. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With a big assist from Claude Opus 4.6, this is the first cut at adding support for Binospec's fiber-based IFU. This is an ok starting point and provides something to build on and improve. The new
Fiberpypelineshould make it pretty straightforward to support other fiber-based spectrographs, both IFU and multi-object. The Binospec documentation has been updated to include information on the new functionality as well as a detailed comparison between the CfA-maintained IDL-based pipeline and PypeIt.The changes made elsewhere in the code were largely some missed validity checks that were tripped over during testing and development. A more significant change in
flatfield.pyis to fix an issue where 3 GB temporary arrays were being constructed for all 360 slits per detector. That obviously got pretty ugly, but the fix here did the trick to alleviate that.