Skip to content

Add lane-order invariance check to validation - #130

Merged
cnicholas merged 1 commit into
cnicholas:mainfrom
rabujamra:validation/mr-permutation-invariance
Sep 19, 2026
Merged

cnicholas merged 1 commit into
cnicholas:mainfrom
rabujamra:validation/mr-permutation-invariance

Conversation

@rabujamra

Copy link
Copy Markdown
Contributor

Closes the question raised in #114. Adds validation/mr_permutation_invariance.py,
a dependency-free script (csv + numpy) that holds every observation fixed and
varies only the lane order of the combined chart, then reports which chart
results move and which do not.

Why

With by=[] the moving range runs across the whole concatenated sequence, so the
step from one organisation to the next contributes a range like any other. On the
24-ACO file that is 23 of 95 ranges. The script asks what depends on which
organisation happens to be adjacent to which.

What it reports, on aco_per_capita_expenditure.csv

  within-ACO     72 ranges   mean    933.79     invariant to lane order
  boundaries     23 ranges   mean   3037.90     3.25x the within mean

  mRbar as loaded                    1443.21
  mRbar within-only                   933.79    +54.6%

ordering              mRbar    X-limit width   X signals   mR signals
  as loaded           1443.2         7677.9          4           3
  sorted by level     1272.5         6769.9          7           0
  interleaved         1497.6         7967.4          3           6
  20k random mean     1413.4         7519.4       3.92        4.48
             range  1269-1545     6753-8221        3-7         1-9

  random orderings with zero X signals    0 / 20000  (0.00%)
  shipped-order mRbar percentile          77.0

Two results, and they point in opposite directions:

  • The combined chart's conclusion is invariant. Across 20,000 random lane
    orderings the X chart never returned zero signals. "These 24 are not one
    process" survives every ordering tested.
  • The boundary-signal structure is not. The count ranges 1–9; under the
    shipped order it is 3, and ordering lanes by level gives none. The flagged
    pairs differ entirely between orderings.

The script also records, descriptively, that a boundary moving range changes
organisation and resets time in one number — organisation i's last
observation against organisation j's first. On this rising series the mean
boundary range is 1.66x the mean adjacent pure-level gap. That is a property of
the lane-major sequence on trending data, not of the two organisations, and the
script says so rather than attributing a share to the time step.

Confirmed against the library

Per #114, companion=True flags exactly the three transitions the script reports
for the shipped order, at the first-year points of ACO-019, ACO-021 and ACO-024,
plus the same four X points. The script uses the classical D4 = 3.267, which
matches the library's mR limit.

What it does not test

Runtime

About one second. No dependencies beyond numpy.

python3 validation/mr_permutation_invariance.py aco_per_capita_expenditure.csv

@cnicholas cnicholas left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed

@cnicholas
cnicholas merged commit 6db608f into cnicholas:main Sep 19, 2026
17 checks passed
cnicholas added a commit that referenced this pull request Sep 19, 2026
…et they run on

README.txt now lists short_series_sampling.py and short_series_bands.py (#119)
and mr_permutation_invariance.py (#130), all contributed by the #114 reporter,
and the synthetic 24-organisation x 4-year ACO file built with Tom Bishop for
that issue, so the permutation check runs out of the box from the repo root.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants