Conversation
The R0 models were not learning. Two separate degenerate modes were found and fixed, one objective was made learnable by modelling the right quantity, and the three-round campaign path now runs end to end. Model - Retire `default_current` to `legacy_matern_no_prior` and register `dim_scaled_prior` as the default. The old contract is kept byte-identical so archived Step 2B/2C runs stay reproducible; it must now be asked for by name. - BoTorch 0.15.1 already supplies a dimension-scaled LogNormal lengthscale prior; MOBO-Kit was discarding it by passing an explicit covar_module. Restored. Median ARD lengthscale 1121 -> 0.88, flat directions 6/10 -> 0/10. - The lengthscale prior alone opens a second degenerate mode: the outputscale collapses and the model calls the data pure noise (10/15 thickness folds, latent sd 1e-4 against fitted noise 0.93). Both priors are now required, and `_assert_signal_not_collapsed` checks it numerically on every fit, because a config name cannot prevent a degeneracy on refit. - Predictive interval coverage moves toward nominal on all three objectives. Objectives - Thickness trains on nanometres, not on its score. The score is a peaked Gaussian on 650 nm, so the map is 2-to-1 and destroys learnable signal. LOO R2 goes -0.503 (score) to +0.384 (nm with a structured mean). - Physics-informed mean functions, declared per objective in config. Thickness needs log(speed_1)+log(precur_conc); optoelectronic needs a single linear anneal_temp term. Opposite shapes; neither generalises. Optoelectronic's plain GP sat below the null at -0.342. - `ObjectiveSpec.model_link` records that a GP output is log-space. Link decode happens in exactly one place, so the sampling and quadrature paths cannot disagree; a test asserts they match to 4e-3. - Lognormal utility expectations use Gauss-Hermite quadrature. Moment-matching is ~500x less accurate and its bias changes sign across the range, which reorders candidates rather than shifting them. - Reference point declared in utility space. The old raw-scale point gave the optoelectronic axis 4.01x the uniformity axis. Campaign - `campaign.py`: run_r0_lhs / run_r1_ucb(5) / run_r2_qlognehvi(3), orchestrating the existing modules. 23 distinct conditions, three replicate films each. - The canonical config is runnable: resolved objectives, fixed scales, declared mean functions. - Debug/production approval tiers replaced by one validity check (count, uniqueness, on-grid, finite, spacing). Approval is a human decision recorded outside the code. - `assert_scaling_is_campaign_fixed` runs inside `build_objective_transform`, so no transform can exist without it. Data-derived scaling would make hypervolume incomparable between rounds. Workbook - `workbook_io.py` reads the source and writes candidates to a sibling file. openpyxl discards cached formula values on save, and Uniformity score is a formula column, so adding sheets to the source would blank a training column for every non-Excel reader. Verified directly. - Entry columns come from `model_source_columns(config)`, so thickness-in-nm is collected. Round detection fails closed on a partly scored sheet. - UTF-8 forced at every boundary; utf-8-sig for Excel-facing CSVs. Docs: GP_MODEL_DECISION.md records the numbers, both resolution floors, and the decision to keep sample 1. CAMPAIGN_STATUS.md orients collaborators and lists open issues. Not validated for fabrication. See CAMPAIGN_STATUS.md: an unexplained 0.089 discrepancy on optoelectronic, column AA is a stale pasted literal rather than a formula, and no proposed batch has been reviewed by a human. Tests: 517 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs a synthetic 3-objective, 10-input problem with a known Pareto front through run_r0_lhs -> run_r1_ucb(5) -> run_r2_qlognehvi(3). Nothing touches the experimental data, so it answers "does the algorithm work" separately from "are the measurements right". Two sign conventions fail silently and are now covered: - DTLZ2 minimises by default; negate=True is mandatory or the test would measure the opposite of optimisation. - BoTorch's Hypervolume assumes maximisation and silently DROPS points that do not dominate the reference -- no warning, no exception, just a smaller number or 0.0. The helper asserts at least one point dominates first. On the optimisation claim: cumulative hypervolume rises monotonically by construction, so that alone would pass for random sampling. The test therefore compares against a random baseline at equal budget, and asserts the MEAN gain rather than a per-seed win: measured 5 of 8 seeds, mean gain +0.075 vs +0.056, a ratio of 1.35x. With 8 added points in 10 dimensions that is the honest expectation; asserting a per-seed win would be flaky and false. Structural invariants covered: exact batch sizes (5 and 3, 23 distinct conditions), uniqueness, on-grid, in-bounds, finiteness, batch spacing well above the configured floor (0.735 and 0.859 against 0.15), proposals distinct from observed points, and determinism at a fixed seed. Pool sizes are shrunk for runtime; R2 is the bottleneck and mc_samples is the cheapest lever. Fast tests ~12 s, the multi-seed comparison ~33 s behind a new `slow` marker. Also records a finding from the recon: metrics.compute_ref_pareto_hv builds its automatic reference as mins - 1e-8, giving a degenerate 6e-8 hypervolume against 1.448 from botorch's infer_reference_point on the same data, and recomputes it per call so values are not comparable across iterations. The production path passes an explicit reference and is unaffected. Tests: 528 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
That machinery existed to audit a GP model this branch has since replaced. Its findings are recorded in docs/GP_MODEL_DECISION.md, so the code itself no longer earns its place: it was 16,442 lines that nothing on the campaign path imports. Removed 50 files -- 14 source modules, their tests, 5 examples, 4 configs, 9 docs and the Step 2B notebook -- plus stale demo/experiment output. Source drops from ~24k to ~9.6k lines; the campaign path itself is 7,073 lines across 16 modules. Everything is recoverable: tag pre-cleanup-2026-07-29 holds the full tree. git show pre-cleanup-2026-07-29:src/mobo_kit/<file>.py Kept despite not being on the campaign path yet, because they are genuinely useful rather than historical: discrete_refinement (exact-grid local search), sobol_pool (nested pools), candidate_diagnostics and plotting (both wanted for the visualization work). production_gate.py went with the ceremony. Its one valuable check -- that objective scales are never re-derived from observed data, which would make hypervolume incomparable between rounds -- was already extracted into campaign.assert_scaling_is_campaign_fixed, where it runs inside build_objective_transform and cannot be bypassed. main.py's propose_candidates branch was permanently blocked by that gate, i.e. dead. It now raises a message pointing at mobo_kit.campaign, which is the real proposal path and carries the objective contract and batch validity checks that the legacy path never had. Naming made consistent: configs are campaign_d2d_perovskite.yaml, example_demo.yaml, example_from_csv.yaml. No file is named after a step number any more. README rewritten around the campaign loop, the DTLZ2 acceptance evidence, the current parameter values, and where the removed history lives. Tests: 280 passed (was 528; the difference is tests for removed modules). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading order, the open issues in the order I would work them, the two resolution floors (-0.148 null and +/-0.236 resolution sd) that this project argued inside twice, the settled questions not to reopen (sample 1 stays, speed_2 does not explain the low-speed contradiction, uniformity has no learnable signal), and the two tooling traps that fail silently (openpyxl dropping cached formula values, BoTorch Hypervolume dropping non-dominating points). Also records the working advice: develop against DTLZ2, not the campaign workbook, since anything data-specific lives in config. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the workbook's derived score cells are pasted literals rather than formulas, so they do not update when the measurements behind them are edited. That is the failure that produced the original uniformity discrepancy, and an audit of all 15 R0 rows found the same divergence already present between AB and Y: Y evaluates the thickness Gaussian on the rounded X, AB was pasted from the same Gaussian on the unrounded T mean, and the two disagree by up to 1.7e-3. So the polarity is inverted. scores.py computes each objective from the raw measurement columns via a recipe declared in config, and the stored cells become cross-checks that warn on disagreement with a per-column tolerance -- a live formula and a deliberately rounded literal do not deserve the same one. On the R0 rows the recomputation reproduces Z to 1.1e-16, AA/R to 1.8e-15, and X to the 0.5 nm its rounding allows, so no campaign number changed except that thickness now reaches the GP unrounded. The formulas come from the removed d2d_scores.py with the polarity inverted; recover the original with git show pre-cleanup-2026-07-29:src/mobo_kit/d2d_scores.py Also here, because they share the read path: - read_candidate_results aggregates a filled-in candidate sheet to design points, so R1 -> R2 can advance at all. Thickness averages across replicate films in LOG space, matching the space the GP trains in and the train_Yvar pooling policy; within-film T1..T4 stays arithmetic, being a spatial average over one film. replicate_spread keeps the scatter that aggregation discards. - The candidate sheet now asks for raw measurement columns instead of derived scores, and round detection knows two thickness readings is complete while a missing Coverage is not. - fit_campaign_models and normalise_inputs are public, so callers stop reaching for _fit_models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compute_ref_pareto_hv defaulted to Y.min(dim=0) - 1e-8 when handed no reference, essentially the nadir of whatever data it was given, and re-derived it on every call. Two consequences: every hypervolume slab could be 1e-8 thick, and two rounds were measured against two different reference points, which is exactly what tracking hypervolume across rounds is supposed to rule out. Passing no reference now raises and names reference_point_utility. The precise condition, since the old issue text overstated it: mins - 1e-8 is harmless while some dominated point sets the per-objective minima, and collapses once the Pareto set itself sets them -- each point best in one objective and worst in another, which is what a genuine trade-off front is. So the default was worst exactly where the function mattered most. Pinned in a test. A reference that nothing dominates now also raises rather than reporting 0.0. BoTorch's Hypervolume assumes maximisation and silently drops non-dominating points, so an unreachable or wrongly signed reference is indistinguishable from a campaign that has not beaten its baseline yet. There were no tests on this function at all, which is how it survived: it returned a plausible number, and plausible numbers do not get questioned. main.py already passed an explicit reference. The demo notebook had one bare call, in a cell that then declared a different reference on the next line; its cached output came from the degenerate path and has been cleared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fifteen films is a real cost, and nothing here showed a human what a batch meant -- only that it passed its validity checks. batch_review.py writes a Review sheet into the candidate workbook and echoes the same text into the launcher pane, so the artifact can be forwarded to the experimental group on its own. Per candidate: inputs in physical units; predicted utility and sd per objective through posterior_utility_moments, the function the acquisition itself called; the prediction decoded into the measurement's own units; normalised distance to the nearest observed point; and which coordinates sit at a range edge, not only how many. For a log-link objective the decoded value is the posterior MEDIAN with a multiplicative interval, because exp of a mean of logs is not a mean. Labelling it as such matters: the two get quoted back later. Probes are declared in config under review:, because which counterfactual is worth asking is campaign knowledge. A probe holds every other coordinate and forces one input to a value worth interrogating. The informative comparison is the sd, not the mean: UCB pays for uncertainty, so a region skipped while the model still calls it uncertain is losing a trade-off, whereas a region skipped while the model calls it certain has been resolved -- possibly into an average. SD_MATERIALITY_RATIO makes that threshold explicit and the ratio is printed either way, because on the campaign's own R0 fit the probed sd came out 1-8% above the batch's on all three objectives, which a bare > reads as more uncertain while predicted thickness utility falls from 0.79 to 0.22. On the R0-trained batch the low-speed probe reports thickness utility 0.223 against 0.786 at an sd ratio of 1.02: skipped as known and bad, not unexplored. And because speed_1 is a feature of the thickness mean function, that confidence is a fitted global trend extrapolating to its range edge rather than a local average of the two contradictory observations there -- which the verdict says, derived from config rather than hardcoded. The launcher is a tkinter shell over inspect_campaign, gather_observations and generate_next_round, all testable without a display. It refuses an existing sheet before the ten seconds of model fitting rather than after, shows plain sentences for expected failures and a traceback only for unexpected ones, and approves nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he Step 1 spec Stale claims, all verified against the code before changing: - GP_MODEL_DECISION said the structured mean was not yet wired into campaign.py. It is; fit_campaign_models builds the mean modules. The same claim appeared as an open item and is now under Closed. - CAMPAIGN_STATUS listed removal of the legacy debug ceremony as not started. production_gate.py and 22 other modules went in 33f101f, and test_validity_report_carries_no_approval_flags holds the tiers out. - The config comment said to pool thickness replicate variance in NANOMETRES while claiming to match what the GP trains on. With response: log the GP trains on log T, so an nm^2 variance is wrong by a factor of T^2 -- 1.3e5 to 1.7e6 across the observed thicknesses, not even a constant rescaling. docs/D2D_CAMPAIGN_SPEC.md is deleted. Its grid rules are enforced by design.py and campaign.validate_batch, its thirteen unresolved decisions blocking real R1 are resolved in config, and its workbook audit describes a revision of the summary workbook that no longer exists -- it reports duplicate Uniformity score headers at Q/T, which the current workbook does not have. The one fact that lived nowhere else, the 177,816,994,740-recipe design space and why it must never be materialised, moved to the README. Recover with git show 19591cc:docs/D2D_CAMPAIGN_SPEC.md Also recorded: the workbook column audit and its numbers; sample 12's thickness being ROUND(mean(1600, 709)); the within-film log-T spread of 0.244 over 24 dof as a floor rather than an estimate of between-film variance; and a newly found issue -- the signal-collapse guard cannot distinguish a collapsed GP from a mean function that works, because it measures posterior variance, which a mean module does not enter, then reports that the posterior mean is effectively constant. Doc snippets now use the public fit_campaign_models and normalise_inputs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_assert_signal_not_collapsed compared the GP's latent sd against the fitted noise and stopped there. A mean module is not part of the covariance and so never enters posterior().variance, which means a structured mean that explains most of the data drove the residual GP's latent sd to ~0 and the fit was refused -- with a message asserting that the posterior mean is effectively constant, which is verifiably false in exactly that case. Two situations share the signature and now get different answers. True collapse -- zero-mean GP, outputscale to zero, posterior mean genuinely flat, nothing rankable -- still raises. The mean function having done its job now warns and lets the round proceed: refusing there dead-ends the campaign at the moment the physics model starts working, with no remedy available, because better data cannot be collected without first proposing conditions. The review artifact is the designed gate for a suspect batch. The warning is not a formality and says why: UCB's exploration term reads the latent posterior that just collapsed, and the mean module's coefficients are frozen buffers with no uncertainty of their own, so the narrow intervals such a model reports are understated rather than earned. It surfaces above the numbers in the launcher pane and the Review sheet, and in RoundResult.diagnostics[model_fit_warnings]. Two calibration decisions worth keeping: - Near-constant is measured against the objective's observed spread, floor 5%, not against the fitted noise sd. Noise-relative was the first attempt and is wrong: the noise is inflated precisely in the degenerate case, so the test co-varies with what it is trying to detect. Measured instance -- a linear mean on anneal_temp against a forced noise of 0.9 scored 0.38 on the noise yardstick and would have been called constant while it was tracking the data. - Only the guard's own warnings reach a human. record.warnings also collects every Python warning raised while fitting, about 18 numpy-2.0 deprecation notices per fit on this stack, and putting those in front of someone reviewing a batch is how people learn to ignore warnings. Whether a dataset trips the collapse is knife-edge: measured across residual magnitudes from 0 to 0.3 it fires at 0, 1e-4, 0.01 and 0.03 but not at 0.001 or 0.1, depending on where the MLL optimiser lands. So the guard's decision is tested directly on both branches, and the propagation tests force the condition rather than hoping data produces it. No fit on the current R0 data warns; the live campaign is unchanged. fit_campaign_models now returns (model, warnings). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_the_window_shows_a_readable_error_rather_than_a_traceback failed in a full-suite run and passed on its own. Two real problems, both mine: The launcher schedules a check() 200 ms after construction when it remembers a workbook. An earlier test in the same file checks a real workbook, which writes that path to ~/.mobo_kit/launcher.json. If pytest's tmp_path from that run still exists, the next window auto-checks it, and that result races the explicit check() the test performs -- overwriting the pane with a successful status where the test expected an error. Order-dependent, so it only appeared once the suite grew. Second, and true regardless of the race: a test suite has no business writing to the user's home directory. Both fixed by an isolated_settings fixture that stubs remembered_workbook to None and remember_workbook to a no-op. Verified with three consecutive runs of the file and two consecutive full-suite runs. Noted for the record: the previous commit was made in the same shell invocation as its verification run and did not stop when that run reported this failure. Verify first, commit second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rnings Two hardenings. The launcher runs its work off the main thread and reports back through a queue, so a reply could paint the pane with an answer to a question the user had moved on from. Two ways in: the auto-check scheduled 200 ms after startup against the remembered workbook, and a second press while the first was still running. Human reaction times made it unlikely rather than impossible, and the previous commit only papered over it in the test fixture. Now each dispatch takes a request id and a reply carrying a stale one is dropped; a status reply also names the workbook it examined and is dropped if the selection has changed since. A dropped reply still clears the busy state -- without that the fix would trade a rare wrong answer for a window that disables its own buttons forever. Pending auto-checks are cancelled as soon as the user browses or presses anything. The drop rules are tested by putting messages on the queue, not by racing two real threads. The first version of those tests did race them, passed alone, and failed intermittently in a full-suite run. A flaky test of a race-condition fix is worse than no test, because it teaches people to re-run until green. Second: the unfiltered fit-warning list is retained under diagnostics[fit_warnings_raw], unsurfaced. The human-facing channel stays guard-only -- 18 numpy deprecation notices per fit is how people learn to ignore the one warning that matters -- but a BoTorch or scipy convergence warning the filter dropped is exactly what someone will want when a fit looks strange weeks from now. Entries carry objective, stage and category. Also noted next to the guard's spread-based threshold: it is a diagnostic and never touches utility space, so it does not fall under the campaign-fixed-scaling rule, which governs the objective scales feeding hypervolume. Written down so it is not later "corrected" to a fixed constant. Verified: three consecutive runs of tests/test_launcher.py and three consecutive full-suite runs, 399 passed each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each proposed condition is run in triplicate, and the scatter across those films is the only direct measurement this campaign has of its own reproducibility. replicate_variance.py pools it and hands it to the GP as train_Yvar, so the marginal likelihood no longer has to guess the noise from 15 points in 10 dimensions. Wired now, enabled later: model.observation_noise stays fit_from_marginal_likelihood until the R1 triplicates land, and turning it on is then one config key. That is the point of building it before the data exists -- arrival should be a data event, not a code event -- so it is tested against synthetic replicates end to end. Four things it is careful about, three of which fail silently otherwise: - The variance handed over is of the MEAN, pooled / n_films, because the observation is an average of n films. Passing the single-film variance understates it threefold on a triplicate. - BoTorch accepts BOTH train_Yvar and an explicit likelihood and then silently ignores the variance: the likelihood wins, stays single-element, and the replicate information is dropped with no error. Verified on 0.15.1. _build_single_task_gp now passes one or the other, never both. - Standardize rescales train_Yvar along with the targets, so it must arrive in the target's own units -- and in the model's space, which for thickness is log T, not nanometres. Aggregation and pooling therefore share one space by construction. - Zero pooled variance is refused. Replicate films agreeing to the last digit are a transcription, not a measurement, and a zero train_Yvar tells the model the observation is exact. Between-film and within-film stay distinct, in the config comment and in the module docstring. Between-film is the quantity train_Yvar needs. The within-film 0.0593 on log T (24 dof) contains no run-to-run variation, so it is a FLOOR: sanity_floor_findings reports a pooled estimate below it, because films cannot be more reproducible than points on a single film. The collapse guard now averages the noise vector rather than reading its first element, since a FixedNoiseGaussianLikelihood carries one value per observation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two operator-facing scripts, plus the results of running the first one. scripts/dtlz2_parameter_sweep.py sweeps UCB beta against the local-penalization radius on DTLZ2, whose Pareto front is known, so the answer does not depend on whether the campaign's measurements are right. The decision rule is written in the script above the code and was committed before any number existed: keep 4.0 / 0.25 unless a cell beats its mean hypervolume gain by more than the per-seed sd of gains without reducing spacing. Result, 8 seeds per cell: NO CHANGE. The default scores +0.0780 with a per-seed sd of 0.0428, so a challenger needed +0.1208; the whole grid spans +0.0776 to +0.0961 against sds of 0.043 to 0.074. Seven cells have a higher mean and none is close. Flat within noise, which is the outcome that says the default was not a lucky pick. BO beats the random baseline on the mean in 9 of 9 cells, and the edge-coordinate count is flat at 16-17 of 80 everywhere -- so neither knob is what drove the live campaign's batches onto range edges. That was the monotone anneal_temp mean function, as the review artifact said. Stated as a limit rather than buried: radius is NOT properly tested by this sweep. DTLZ2 batches land 0.72-0.98 apart, far outside every radius tried, so local penalization rarely has two candidates close enough to penalise -- visible in beta=8 giving bit-identical results at radius 0.15 and 0.25. The sweep validates beta and says little about radius. scripts/intake_new_data.py is the one command to run when the group returns re-measured or corrected data. It audits the read, checks the anchors still span the data and that the campaign-fixed scaling guard passes, recomputes BOTH floors at the new N rather than reusing the N=15 values, and gives a per-objective verdict: keep the mean function only if it beats plain by more than the resolution floor, and name the exact config block to delete otherwise. It also reports the fit guard's status per objective, which exercises the warning path added in f528f84 -- cleaner re-measured thickness is exactly the data that would trip it. On the current 15 rows it reports uniformity failing to beat the null (-0.6809), optoelectronic keeping its mean function (-0.3420 -> +0.2670) and thickness keeping its (+0.1160 -> +0.3806), with a clean guard for both. Also repaired: scripts/plot_dtlz2_report.py called the private _fit_models, whose return type changed when fit warnings started travelling with the model. It now uses the public fit_campaign_models. The CAMPAIGN_STATUS snippet had the same staleness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 0.089 optoelectronic discrepancy is closed as a numerical artifact. The decisive experiment -- disable Standardize in both pipelines -- was run: the gap does NOT collapse, moving only 0.0881 to 0.0715. So the standardization scale is about a fifth of it, not the cause. The rest is the MLL optimiser. Without the outcome transform the two pipelines are mathematically the same model: a zero-mean GP on (y - trend) and a fixed-mean GP on y with mean trend have identical marginal likelihoods, because a fixed mean only shifts the data. Yet the fits land in different places -- across folds the outputscale differs by up to 2.7%, the noise by 2.7%, the median lengthscale by 9.6% -- and at N=15 that is worth 0.07 of LOO R2. Deterministic, not stochastic: the earlier seed sweep was bit-identical across four seeds. There was never a modelling question to answer, and 0.0881 sits well inside the +-0.236 resolution floor, so it was never evidence of anything. Second: the intake command and the decision record disagreed on plain thickness, +0.116 against +0.183. Same 15 rows and no resampling, so per this project's own rule that had to be explained rather than absorbed. It is entirely the DATA, not the method: rounded X two-stage +0.1830 mean module +0.1830 unrounded two-stage +0.1160 mean module +0.1160 The pipeline contributes exactly nothing -- with no mean function the two routes are the same code -- and the whole 0.067 comes from 7 of 15 rows changing by at most 0.50 nm. Half a nanometre on seven rows moves LOO R2 by 0.067, which is the same fragility that produced the 0.089. The structured numbers move by 0.006 and the swing that justifies the mean function clears the floor either way, so no conclusion changes. intake_new_data.py is now declared canonical, and GP_MODEL_DECISION says at the top which instrument produced its tables and that it reads a column the model no longer trains on. Third: radius is now verified by construction. The DTLZ2 sweep could not test it -- batches land 0.72-0.98 apart, outside every radius tried -- and the live campaign's R1 spacing of 0.921 says the same. Inert is fine for a safety knob, but its function should not be inferred from a campaign that never exercised it, so there is now a test with three candidates crowded 0.02 apart scoring above an isolated fourth: greedy takes the two best, penalization pushes the second pick past the radius. A companion pins the inert case. Also: the single-film variance comment had its direction inverted -- passing it would OVERSTATE the mean's variance threefold, not understate it. The implemented choice was right, the sentence was not. And the plot script's unused import is gone with its blanket warnings filter scoped, so a fit-guard message can still get through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project had two floors to check before comparing LOO R2 values. It has three. Two measurements this week, from completely different directions, each moved LOO R2 by about 0.07 at N=15 without changing anything meaningful: the MLL optimiser landing elsewhere on an IDENTICAL likelihood surface (0.0715, from outputscale and noise differing by under 2.7% and the median lengthscale by 9.6%), and rounding the thickness input to whole nanometres (0.0670, from at most 0.5 nm on 7 of 15 rows). Neither is sampling noise -- both are deterministic and reproducible. So a second-decimal difference in LOO R2 at this N is below what the metric can reproduce even on identical data with identical models. Where the sampling floor says a difference may be luck, the numerical floor says it may not be a difference at all. Both were learned by walking into them. A corollary worth its own line: under the same rounding the plain GP moved 0.067 while the structured model moved 0.006. A model whose answer turns on half a nanometre is reporting arithmetic; one that ignores it is reporting a trend. That is independent evidence for the mean function, arrived at without looking at either model's score. Also in this pass: - The 0.089 closure now leads with the mechanism and cites the floor as a corollary, so the record shows this project's "explained, not absorbed" rule was satisfied rather than sidestepped. - Confirmed and strengthened the sentence stating that the thickness mean function's evidentiary weight is the rank permutation (p = 0.0350, CI [0.0270, 0.0446]), with the R2 swing merely consistent -- it survived the reconciliation edits, and now says explicitly that the reconciliation could not have touched it, being about ranks rather than a regression score. - CAMPAIGN_STATUS's model-state table now carries the canonical intake numbers (+0.116 -> +0.381 thickness, -0.342 -> +0.267 optoelectronic) with the older figures explained rather than silently contradicted. - HANDOFF rewritten as a conclusion: where the project stands, what is actually open, the three floors, the four tooling traps, and which instrument produced which numbers. Its old "open issues" list was mostly closed items. - Stale text removed: issue 3 still described the 0.089 as unexplained with untested suspects; issue 8 was headed "Not started" for work that is done; a cross-reference pointed at an issue number that had shifted. - README gained a runnable snippet showing where Y_model comes from -- the computed values, not the workbook's stored score cells -- and a note that the two tuned parameters were checked under a pre-committed rule. - Sampling floor and resolution floor are the same number under two names; the floors section now says so rather than leaving a reader to guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audited the notebook and the requirements, neither of which had been checked against this arc's changes. The requirements are clean: all 17 pins match what is installed, and every pyproject range contains its pin. Its header still pointed at docs/STEP1_HANDOFF.md, removed in the cleanup commit, and said nothing about why linear_operator is pinned when pyproject does not declare it. Both fixed -- the pin matters because it is a gpytorch/botorch transitive dependency whose version affects fitted numbers, and a second-decimal change in LOO R2 is inside the numerical floor. The notebook is structurally sound: every mobo_kit import resolves, no call passes a keyword the current signature does not accept, and it no longer reads stored score columns, reaches for private helpers, or calls compute_ref_pareto_hv without a reference. What it does do is build its GPs with models.fit_gp_models and models.loocv_select_models -- the prior-free ScaleKernel(MaternKernel(nu=2.5, ard)) construction that GP_MODEL_DECISION.md documents as degenerate on small data: ARD lengthscales from 0.13 to 38,000, six to nine of ten directions switched off, noise pinned at its floor so the model believed the data were noiseless. That is the model this project spent a session replacing, and a reader following the notebook as the recommended path would rebuild it. Rather than rewrite a demo that works and cannot be re-executed here, the notebook now opens with a scope note: it is the general toolkit API on an arbitrary CSV, not the campaign path; the campaign uses fit_model_variant with dim_scaled_prior via campaign.fit_campaign_models; objective values are computed rather than read; and compute_ref_pareto_hv now requires an explicit reference. HANDOFF says the same in one paragraph, and that models.py is legacy-but-live rather than dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs R0 -> R1 -> R2 end to end with a GP fitted to the 15 real films standing in for the measurements, so the loop can be exercised before the R1 triplicates exist. 13 OFAT cells (9 radii at beta 4, 5 betas at radius 0.25), 45 input pairs as views of one 10-D run per cell, and a manifest of batch hashes that answers whether a knob changes the proposed batch at all. Derived from Annie Xu's examples/round_simulations.py on ax_plots_simulation. Her directory convention, slug rule, round legend and overlaid-boxplot figure are preserved. More importantly, her branch already carried the fix for the R1 baseline mis-encoding that colin still has: run_r1_ucb hands the objective transform measurement-space nanometres, which it exponentiates a second time, pinning every observation's thickness utility to exactly 0.0. Her _physical_to_model_output is that fix, written before we knew the bug existed. This script uses the corrected encoding and records both baseline hypervolumes (0.436442 against 0.004659) on every manifest row. campaign.py is deliberately untouched here; the fix lands as its own change. Two corrections recorded in docs/ROUND_SIM_DELTA.md are corrections to the brief, not to her code: her exp(mu + v/2) was the correct lognormal mean and her physical-mean label was accurate for it, and her slice fixing already used the median with a grid snap. The oracle here still reports the median, for a different reason -- a variance-dependent oracle bulges wherever R0 is sparse, so the simulated ground truth would encode sampling history rather than belief. Main finding: radius is not inert on this landscape. Achieved R1 spacing staircases 0.455 -> 0.921 as radius goes 0.05 -> 0.45, saturating around 0.30, and buys that spacing at a cost of 11 -> 15 range-edge coordinates. DTLZ2's not-exercised conclusion does not transfer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_r1_ucb handed its observed HVI baseline to ObjectiveTransform.transform in MEASUREMENT space. That transform is a model-output decoder: it applies exp() to a log-link objective before computing utility, so thickness in nanometres was exponentiated a second time. exp(360..1303) saturates the 650 nm Gaussian to exactly 0.0 -- finite, so neither the transform's own finiteness check nor the caller's fired -- and every observation's thickness utility was zero. Measured on the 15 R0 rows: baseline hypervolume 0.004659 against a correct 0.436442, a factor of 94, with the baseline Pareto set collapsing from 5 points to 2. Every candidate's improvement was scored against a front with no thickness axis. On the live campaign's own R1 step this moves one of the five proposed conditions (speed_1 4000 -> 2500, precur_conc 1.70 -> 1.45) and the batch's minimum spacing from 0.9209 to 0.6337 -- so the 0.921 on record was itself an artifact of the defect. The fix is ObjectiveTransform.encode_measurements, with transform_measurements as the one-call safe route, and run_r1_ucb encoding before it proposes. The acquisition modules are untouched: ucb_hvi.py stays byte-identical and the defect was in campaign.py orchestration. Annie Xu found this independently on ax_plots_simulation and fixed it there as _physical_to_model_output, before we knew it existed. This promotes her fix to the public contract. Audit of every ObjectiveTransform.transform call site: objectives.py 403/481/516 and ucb_hvi.py:342 all operate on posterior samples, which are already in model space; batch_review.py never routes measurements through the transform at all (it decodes exp(mu) itself and labels it a median). Exactly one call site was defective, ucb_hvi.py:698, reached only from run_r1_ucb. R2 was never affected: qLogNEHVI takes train_X_norm and derives its baseline through the model. Two things about why it survived 446 tests, both now closed. The baseline was invisible. It was a plausible finite number that nothing reproduced independently -- the same shape as the hypervolume auto-reference and the swallowed train_Yvar. run_r1_ucb now reports observed_baseline_hypervolume and observed_baseline_pareto_size, and the tests recompute them by a separate route. Reverting the encoding while keeping the diagnostic fails on the value, not merely on a missing key; that was checked rather than assumed. The synthetic acceptance test could not have caught it. Every DTLZ2 objective is affine, and for an affine objective measurement space and model space are the same numbers, so a link-encoding mistake is invisible by construction. DTLZ2 now also runs with its third objective reached through a log link -- reported as exp(f2), modelled with response: log, so the GP trains on the same latent quantity by a different route -- and asserts that no measurement strictly inside its anchors scores exactly zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The R1 batch was built on the mis-encoded UCB-HVI baseline fixed in 4b76670, so it is withdrawn. Regenerated through launcher.generate_next_round -- the same path the double-click launcher uses -- into local_inputs/Summary Table_R1_Candidates.xlsx. Four of the five conditions are unchanged. One is replaced: speed_1 4000 / precur_conc 1.70 out, speed_1 2500 / precur_conc 1.45 in. Baseline hypervolume 0.004659 -> 0.436442, baseline Pareto set 2 -> 5 points, minimum spacing 0.9209 -> 0.6337. NO FILMS WERE FABRICATED from the withdrawn batch; the human review gate caught it, which is what it is for. There was no prior candidate workbook on disk to archive: the withdrawn batch had been described in CAMPAIGN_STATUS and echoed to the launcher pane but never written. The withdrawal is declared in configs under review.notes, so it travels with the Review sheet if that is forwarded on its own. Delete that note once R1 is measured. Full diff in docs/R1_BATCH_WITHDRAWAL.md. Two things the defect had written into the docs, now corrected. The claim that radius is "probably inert" on the live campaign rested on the R1 batch's minimum spacing of 0.921 -- a figure the mis-encoding itself inflated. Corrected, the live batch spaces at 0.6337, and the round simulation measures a monotone staircase: achieved R1 spacing 0.455 / 0.455 / 0.455 / 0.543 / 0.720 / 0.921 / 0.921 / 0.921 / 0.921 across radius 0.05 to 0.45, with nine cells producing six distinct batches. radius binds below about 0.30 and saturates above it, and buys spacing at a cost of 11 -> 15 range-edge coordinates. That trade-off is a policy choice for the group. 0.25 stays the default as a declared choice; the hypervolumes are single-seed and cannot rank cells, so re-run the arm at ~5 seeds before moving it on performance grounds. Issue 4's probe numbers turn out NOT to be artifact-born, and are kept rather than voided. The 0.786 -> 0.223 fall with an sd ratio of 1.02 describes R1_C01, one of the four conditions that survived the reissue, and re-reads identically from the regenerated artifact. The batch-level figures are added alongside: 0.791 -> 0.299, mean sd ratio 1.10. The speed_1 = 1000 corner is still skipped (minimum proposed speed_1 is still 1500) and anneal_temp still pins at 100-105. The bug is recorded as CAMPAIGN_STATUS issue 9, and as the third plausible-finite-number failure after the hypervolume auto-reference and the swallowed train_Yvar: all three were finite, ordinary-looking wrong answers that no test reproduced independently. GP_MODEL_DECISION files it deliberately OUTSIDE the three floors -- a floor says when a difference is too small to argue about, and never licenses accepting one that is simply wrong. plot_round_simulation.py now calls the public run_r1_ucb instead of its own corrected copy, verified to reproduce it hash-for-hash on three cells, so the 13-cell manifest already on disk stays valid. Its manifest carries the baseline tripwire: reported (from the acquisition) must equal independent (recomputed by another route) or run_cell raises, with the unencoded value kept beside them as the size of the historical mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…aims README now separates the original design from the reconstruction: initially designed by Ethan Schwartz, Daniel Abdoue, Nicky Evans and Tonio Buonassisi, updated and reconstructed by Ziyang (Colin) Qi and Annie Xu. Three stale claims the first doc sweep missed, all traceable to the same defect or to work that has since landed. test_batch_selection.py still carried "the knob is probably inert there too" in a docstring, resting on the 0.921 spacing the mis-encoded baseline produced. The test itself is unaffected -- it verifies the mechanism by construction, which is exactly why it kept its value when the campaign evidence turned out to be wrong, and the docstring now says so. HANDOFF listed plotting as "the obvious next build"; it landed in f191730. Its "settled, do not reopen" entry for radius = 0.25 is split: beta = 4.0 stays settled, radius = 0.25 stays as a DECLARED POLICY CHOICE with the measured staircase and the diversity-versus-edge-pinning trade-off, because the evidence that made it settled was itself an artifact. CAMPAIGN_STATUS's plotting section now points at the implementation, and its round-comparison colours are corrected to the palette actually shipped (R0 #2a78d6 / R1 #eb6834 / R2 #1baf7a, shared with plot_dtlz2_report.py) rather than the grey/blue/orange it had described. Also strips a UTF-8 BOM accidentally written into plot_round_simulation.py by a PowerShell Set-Content during an earlier refactor. Python's loader tolerates it, so nothing failed, but it is the only such file in the repo and would confuse any tool reading the source as plain UTF-8. Not done deliberately: `black` would reformat 22 files here, including pre-existing core modules, so the repo is not black-clean as a baseline and reformatting now would bury this work in unrelated churn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request establishes the tested D2D MOBO workflow through Step 2C:
Why
Step 2B showed that candidate selection is sensitive to control inclusion, boundary behavior, and GP assumptions. Step 2C adds the validation and convergence evidence needed to assess those sensitivities reproducibly before any experimental batch is considered.
Safety and privacy
Validation
mainand zero commits behindReviewer focus
Please review the public/private configuration boundary, objective and reference-point contracts, GP validation gates, robust-region consensus criteria, and whether the current debug-only evidence is sufficient to proceed to a later experimental approval step.