# Raw-source alignment verification

This check establishes whether the legacy HDF5 sensor rows match the raw Parquet
series, independently of local time-of-day changes. It is separate from annotation
coverage, person identity, and model evaluation.

```sh title="Terminal"
uv sync --extra detection --extra source-audit
uv run --extra source-audit python -m sleepkit.recipes.detection.alignment \
  --data /path/to/cmidss --parquet /path/to/cmidss/train_series.parquet \
  --output /path/to/new-local-alignment.json
```

The reader uses optional PyArrow in bounded batches (default 65,536 rows). The
report contains source hashes and per-series details and stays local. Source files
are never written; an existing report is not overwritten. A failed alignment writes
the report and returns a failing exit code. Malformed/null source values fail
explicitly rather than being filled or resampled.

For every series the verifier checks:

- The first source step is zero and subsequent steps equal the HDF5 row indices.
- Absolute UTC timestamps advance exactly five seconds, including batch boundaries.
- Source and HDF5 row counts agree, and no raw series lacks its HDF5 counterpart.
- TS equals local seconds of day from the source timestamp; ENMO and ZANGLE match
  the source after float32 conversion. Matching NaNs count as preserved missing
  sensor values, not as evidence of signal quality.
- Local-clock jumps and timezone-offset changes are counted independently.
- Source hashes remain unchanged throughout verification.

## Verified local snapshot

On 2026-09-19, all **127,946,340 rows across 277 subjects passed**. There were no
step, UTC-cadence, row-count, or sensor-channel mismatches. All 44 local-clock changes
coincided with timezone-offset changes; UTC spacing remained five seconds. Thus the
44 apparent gaps in the annotation audit are local-clock shifts, not physical
sampling gaps in this snapshot.

Raw Parquet SHA-256:
`fd11f8c3af0507757cade38419f5ec34af28dd6e3057ab17ef05b0af717106af`.
The local report binds every HDF5 source individually. Do not generalize these
results to files with different hashes.

This resolves the step-origin and row-order assumption for the audited files. It
does not establish event-label correctness, that every series belongs to a different
person, or that historical model training excluded any particular subject.

## Reader change supported by this evidence

The legacy detector path validates cadence using local TS and rejects these 44 files.
The [independent sample-clock adapter](https://ambiqai.github.io/sleepkit/detection-sample-clock/) now preserves an explicit UTC clock alongside
local TS, validates continuity against it, and uses local TS only for the time-of-day
feature. This contract carries through caching, training, and unlabeled inference.
Inputs without independent clock evidence retain the strict local-TS check.

The [event-supported label policy](https://ambiqai.github.io/sleepkit/detection-label-policy/) records the adopted
derived annotated-period membership target. The [dataset adapter](https://ambiqai.github.io/sleepkit/detection-annotated-dataset/)
implements that target without overwriting source labels.

## Parquet dependency compatibility

The old fastparquet 2023 pin uses a NumPy 1 ABI and fails to import under the existing
NumPy 2 stack. The dependency is raised to `fastparquet>=2024.11.0,<2025`, whose
wheels support the project's Python 3.12/3.13 range. An explicit fastparquet-engine
round trip is tested so optional PyArrow cannot hide a broken legacy reader.

PyArrow is an optional `source-audit` extra, independent of TensorFlow and of the
legacy Parquet reader. CI installs that extra and requires the alignment tests.