Update from FFT validation

Corentin Ravoux - LPCA, Clermont-Ferrand

Lyman-alpha P1D Cancun 2024 session

12 december 2024

 

Y1 Validation results on Mocks

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Y1 Ohio mocks

  • 20 Y1 mocks made by Naim, with DLA and BAL variations.
  • A lot of FFT variations as tests (in Angstrom or km/s units, individual or combination, covariance computed)
  • Purpose: testing resolution, continuum fitting, pixel masking, etc...
  • Metric to evaluate discrepancy (from Naim):
\begin{align*} \chi^2 (\mathrm{true},\mathrm{corrected}) &= (P_{\mathrm{true}}(k,z) - P_{\mathrm{corrected}}(k,z)) \cdot C^{-1} \cdot (P_{\mathrm{true}}(k,z) - P_{\mathrm{corrected}}(k,z)) \\ &\sim \sum_{k,z} \left(\frac{P_{\mathrm{true}}(k,z) - P_{\mathrm{corrected}}(k,z)}{\sigma_{\mathrm{corrected}}(k,z)}\right)^2 \end{align*}

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Comparison with raw

Average over 20 mocks

Improvement of raw versus True Continuum:

  • Raw = no noise, no resolution (no picca)
  • True Continuum = picca with input continuum

Before: large differences at small and large scales.

Here: Most probably resolution modeling error. To correct with pixsim ?

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Continuum correction

  • Comparing picca true continuum with continuum fitting
  • We will probably not use the very high redshifts (z>4.3)
  • Smaller than eBOSS

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Continuum correction - χ² test

  • χ² histogram comparing average of 20 True continuum, with invidual picca continuum.
  • Corrected versus non-corrected version

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

DLA masking - χ² test

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Lines masking - chi2 test

Improves when removing very high z

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Y1 data status

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Y1 baseline

  • Cross-exposure for low-z
  • Full Y1 systematics included: Noise, resolution, metals, line & DLA masking correction, continuum correction, DLA completeness, EDR resolution correction

NERSC: /global/cfs/cdirs/desi/science/lya/y1-p1d/fft_measurement/v0/plots/baseline/notebook/measurement/p1d_fft_y1_measurement_kms_v3.fits

NERSC: /global/cfs/cdirs/desi/science/lya/y1-p1d/fft_measurement/v0/plots/baseline/notebook/measurement/p1d_fft_y1_measurement_v3.fits

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

FFT covariance estimator

  • A lot of iterations on the best shape of this estimator
  • Final derivation by Jean-Marc, unbiased estimator:
\begin{align*} \mathcal{\bar{C}}(A, B) = & \left(\frac{\sum_{i}^{N_{\mathrm{s}}} w_{A,i} \sum_{i}^{N_{\mathrm{s}}} w_{B,i}}{\sum_{i}^{N_{\mathrm{s}}} w_{A,i} w_{B,i}} -1\right)^{-1} \\&\left[ \frac{\sum_{i}^{N_{\mathrm{s}}} \sum_{j1}^{N_{A,i}} \sum_{j2}^{N_{B,i}} \frac{w_{A,i} w_{B,i}}{N_{A,i}N_{B,i}} \times p_{i,A,j1}p_{i,B,j2}}{\sum_{i}^{N_{\mathrm{s}}} \sum_{j1}^{N_{A,i}} \sum_{j2}^{N_{B,i}} \frac{w_{A,i} w_{B,i}}{N_{A,i}N_{B,i}}} - \left\langle P \right \rangle(A) \left\langle P \right \rangle(B)\right]\,, \end{align*}

= weight depending of sub-forest SNR and wavenumber bin A

w_{A,i}

= Number of modes in the wavenumber bin A

N_{A,i}

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

FFT variance and covariance improvements

  • Adding smoothing of the variance, useful at very high z for low statistics
  • Use of new JM covariance estimator
  • Fixing all numerical instabilities resulting in np.nan

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.

Validation/Mocks

Data

  • DLA mask correction
  • Continuum correction
  • DLA completeness correction
  • Skyline mask correction
  • Resolution modeling (pixsim)
  • Rerun covariance on mocks
  • Write the article
  • Optional: BAL, Correction multiplicativity,

TO DO

TO DO

TO DO

  • Cross exposure estimator
  • New covariance estimator
  • Metals power spectrum
  • P1D Corrections
  • Full systematics
  • Data splits (Marie-Lynn)
  • DLA variations
  • Write the article

Test covariance on mocks

TO DO

Corentin Ravoux                                                                                                                                                        P1D Session Cancun 2024 - 12 Dec.