Flaggers: RFI detection and dataset integration

This page documents the flagging subsystem in Pyralysis, including:

• the flagging contract (Flagger)

• threshold policies (ThresholdPolicy)

• SumThreshold implementation (SumThreshold)

Overview

Flagger is the single entry point for both:

1. Array-level detection: compute_mask() and apply().

2. Dataset-level updates: apply_dataset(), which walks Dataset.ms_list and writes merged masks into each SubMS FLAG column.

Concrete algorithms (e.g. SumThreshold) subclass Flagger and implement compute_mask() only; dataset orchestration is inherited.

Architecture

• Flagger - compute_mask(data, **kwargs), apply(data, flags=..., **kwargs), apply_dataset(dataset, vis_key=...).

• ThresholdPolicy - Interface for computing thresholds from window size.

• SumThreshold - Multi-axis SumThreshold driver over visibility cubes.

Flaggers UML (Flagger / SumThreshold / ThresholdPolicy).

SumThreshold algorithm mapped to code

At high level, SumThreshold does:

1. Reshape visibilities from (row, chan, corr) to (time, baseline, chan, corr) using contiguous time blocks.

2. For each configured direction (frequency/time), run 1D SumThreshold: - moving sums over windows - per-window average - threshold with chi(window_size) - expand window detections to sample-level flags

3. OR the new detections into a cumulative mask.

4. Flatten mask back to input shape.

Relevant methods:

• _reshape_to_4d()

• _run_iterations()

Threshold strategies

The threshold strategy controls chi as a function of window_size:

• ConstantThreshold

• InverseSqrtThreshold

• PowerLawThreshold

This uses the Strategy pattern so experiments can swap threshold behavior without changing the detector implementation.

Dataset-level usage

Call apply_dataset() on any concrete flagger:

from pyralysis.flaggers import SumThreshold
from pyralysis.flaggers.models import InverseSqrtThreshold

threshold_policy = InverseSqrtThreshold(base_threshold=0.5)
sum_threshold = SumThreshold(
    window_sizes=[1, 2, 4, 8, 16],
    threshold_policy=threshold_policy,
    directions=("freq", "time"),
    global_iterations=1,
)

dataset = sum_threshold.apply_dataset(dataset, vis_key="DATA")

vis_key options:

• "DATA", "MODEL_DATA", "CORRECTED_DATA"

• "ALL" for all *_DATA columns

• list/tuple of specific visibility-like columns

NumPy and Dask behavior

• The computational core accepts NumPy arrays and Dask arrays.

• Dask arrays keep lazy operations in the moving-sum and threshold pipeline.

• Some control-flow checks (for time-block validation) use concrete NumPy arrays because they validate metadata assumptions before reshape.

Future extension

A natural next step is a composite Flagger subclass that chains multiple detectors in sequence.