Flaggers: RFI detection and dataset integration
===============================================

This page documents the flagging subsystem in Pyralysis, including:

- the flagging contract (:class:`~pyralysis.flaggers.flagger.Flagger`)
- threshold policies (:class:`~pyralysis.flaggers.models.ThresholdPolicy`)
- SumThreshold implementation (:class:`~pyralysis.flaggers.operators.SumThreshold`)

Overview
--------

:class:`~pyralysis.flaggers.flagger.Flagger` is the single entry point for both:

1. **Array-level** detection: :meth:`~pyralysis.flaggers.flagger.Flagger.compute_mask`
   and :meth:`~pyralysis.flaggers.flagger.Flagger.apply`.
2. **Dataset-level** updates: :meth:`~pyralysis.flaggers.flagger.Flagger.apply_dataset`,
   which walks ``Dataset.ms_list`` and writes merged masks into each SubMS
   ``FLAG`` column.

Concrete algorithms (e.g. :class:`~pyralysis.flaggers.operators.SumThreshold`)
subclass :class:`~pyralysis.flaggers.flagger.Flagger` and implement
:meth:`~pyralysis.flaggers.flagger.Flagger.compute_mask` only; dataset
orchestration is inherited.

Architecture
------------

- :class:`~pyralysis.flaggers.flagger.Flagger`
  - ``compute_mask(data, **kwargs)``, ``apply(data, flags=..., **kwargs)``, ``apply_dataset(dataset, vis_key=...)``.
- :class:`~pyralysis.flaggers.models.ThresholdPolicy`
  - Interface for computing thresholds from window size.
- :class:`~pyralysis.flaggers.operators.SumThreshold`
  - Multi-axis SumThreshold driver over visibility cubes.

.. figure:: diagrams/png/fig14_flaggers-1.png
   :alt: Flaggers UML (Flagger / SumThreshold / ThresholdPolicy)
   :width: 100%

   **Flaggers UML** (``Flagger`` / ``SumThreshold`` / ``ThresholdPolicy``).

SumThreshold algorithm mapped to code
-------------------------------------

At high level, ``SumThreshold`` does:

1. Reshape visibilities from ``(row, chan, corr)`` to
   ``(time, baseline, chan, corr)`` using contiguous time blocks.
2. For each configured direction (frequency/time), run 1D SumThreshold:
   - moving sums over windows
   - per-window average
   - threshold with ``chi(window_size)``
   - expand window detections to sample-level flags
3. OR the new detections into a cumulative mask.
4. Flatten mask back to input shape.

Relevant methods:

- :meth:`~pyralysis.flaggers.operators.SumThreshold._reshape_to_4d`
- :meth:`~pyralysis.flaggers.operators.SumThreshold._run_iterations`

Threshold strategies
--------------------

The threshold strategy controls ``chi`` as a function of ``window_size``:

- :class:`~pyralysis.flaggers.models.ConstantThreshold`
- :class:`~pyralysis.flaggers.models.InverseSqrtThreshold`
- :class:`~pyralysis.flaggers.models.PowerLawThreshold`

This uses the Strategy pattern so experiments can swap threshold behavior
without changing the detector implementation.

Dataset-level usage
-------------------

Call :meth:`~pyralysis.flaggers.flagger.Flagger.apply_dataset` on any concrete
flagger:

.. code-block:: python

   from pyralysis.flaggers import SumThreshold
   from pyralysis.flaggers.models import InverseSqrtThreshold

   threshold_policy = InverseSqrtThreshold(base_threshold=0.5)
   sum_threshold = SumThreshold(
       window_sizes=[1, 2, 4, 8, 16],
       threshold_policy=threshold_policy,
       directions=("freq", "time"),
       global_iterations=1,
   )

   dataset = sum_threshold.apply_dataset(dataset, vis_key="DATA")

``vis_key`` options:

- ``"DATA"``, ``"MODEL_DATA"``, ``"CORRECTED_DATA"``
- ``"ALL"`` for all ``*_DATA`` columns
- list/tuple of specific visibility-like columns

NumPy and Dask behavior
-----------------------

- The computational core accepts NumPy arrays and Dask arrays.
- Dask arrays keep lazy operations in the moving-sum and threshold pipeline.
- Some control-flow checks (for time-block validation) use concrete NumPy arrays
  because they validate metadata assumptions before reshape.

Future extension
----------------

A natural next step is a composite :class:`~pyralysis.flaggers.flagger.Flagger`
subclass that chains multiple detectors in sequence.