Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ in primary and secondary extraction. A failed RT fit never reinterprets an index
as seconds; initial extraction searches unrestricted RT. Secondary queries still
center on detected apex RT and observed mobility.

Results format 5 includes `library_rt_axis` Parquet metadata. Unavailable library
Results format 6 removes the unused competition-group margin columns. Format 5
added `library_rt_axis` Parquet metadata. Unavailable library
RT, calibrated RT and RT residuals are NaN; observed apex RT remains seconds.
Calibration format v4 records the input library axis and effective tolerance
enums; an empty RT snapshot means no RT fit, while residuals can still contain
Expand Down Expand Up @@ -84,7 +85,7 @@ intensities, so those reader routes remain extraction-only.
Standalone `calib_dash` reads saved `calibration.json`, not a spectral library.

[Analyte module documentation](../rust/timsquery/src/chemistry/analyte.rs) documents chemistry storage, reader mappings,
library-wide sequence eligibility, and results format version 5.
library-wide sequence eligibility, and the format 5 changes.

Precursor isotope envelopes retain the three-bin C/S approximation. Scoring
finalization includes known modification C/S deltas or an explicitly based
Expand Down
4 changes: 2 additions & 2 deletions rust/timsseek/DATA_FLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ hits, then re-extract everything with the calibrated windows.
| 1. Prescore | `phase1_prescore` -- `timsseek_cli/src/processing.rs` | Broad, uncalibrated extraction of the whole library; keeps the best-scoring peptides as calibrants. |
| 2. Calibrate | `calibrate_from_phase1` | Fits the iRT→RT curve from the calibrants, measures m/z and mobility error, derives the narrow tolerances. |
| 3. Score | `phase3_score` | Re-extracts every peptide at its calibrated RT with the derived tolerances and computes the full feature set. |
| 4. Compete | `target_decoy_compete` | Dedups by sequence, competes within decoy groups, adds log-space group-difference features. |
| 4. Compete | `target_decoy_compete` | Dedups by sequence and keeps the highest-scoring member of each decoy group. |
| 5. Rescore | `ml::rescore_with` -- `timsseek/src/ml/mod.rs` | Fits a discriminant over feature projections and assigns q-values. LDA/MLP stream raw rows; GBM retains its input frame. |
| 6. Write | `execute_pipeline` -- `timsseek_cli/src/processing.rs` | Writes filtered results and optional feature-statistics sidecars, then completes the performance report. |

Expand Down Expand Up @@ -83,7 +83,7 @@ sides read that from `RowMajorDataset::get_fold`.

## The candidate chain

`ScoredCandidate` (Phase 3) → `CompetedCandidate` (`+ delta_group_ln1p_diff`, `+ delta_group_ln1p_ratio`) →
`ScoredCandidate` (Phase 3) → `CompetedCandidate` (competition winners) →
`FinalResult` (`+ discriminant_score`, `qvalue`) → parquet. All three live in
`scoring/results.rs`; `ScoringFields` is composed there from an ordered block
list, and that order is what parquet columns and the ML lanes both follow.
50 changes: 21 additions & 29 deletions rust/timsseek/src/ml/qvalues.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1115,8 +1115,6 @@ mod feature_tests {
s.identity.row = test_handles::row(0);
s
},
delta_group_ln1p_diff: 1.0,
delta_group_ln1p_ratio: 0.5,
discriminant_score: 0.0,
qvalue: 1.0,
}
Expand Down Expand Up @@ -1284,15 +1282,23 @@ mod feature_tests {
}
}

#[test]
fn competition_group_margins_are_not_features() {
let names = all_feature_name_set(library());
assert!(
!names
.iter()
.any(|name| name.starts_with("delta_group_ln1p_"))
);
}

/// Build a non-degenerate synthetic candidate set: `n` rows, alternating
/// target/decoy, distinct arena rows, with the LINEAR-lane count fields
/// varied by label + row so the cross-fit LDA has real within-class scatter
/// and a class-mean gap (i.e. it actually fits, exercising the score path).
///
/// Every field varied here lands in the linear lane, including the two
/// log-space group-delta features. The nonlinear lane is therefore constant.
///
/// The nonlinear lane is intentionally constant; use
/// Every model input varied here lands in the linear lane. The nonlinear
/// lane is intentionally constant; use
/// [`synthetic_competed_nonlinear_signal`] when a nonlinear split is needed.
fn synthetic_competed(n: u32) -> Vec<CompetedCandidate> {
(0..n)
Expand All @@ -1308,20 +1314,6 @@ mod feature_tests {
c.scoring.counts.falling_cycles = base.saturating_sub(jitter);
c.scoring.counts.npeaks = base + (i % 3) as u8;
c.scoring.finalize_counts.n_scored_fragments = base + (i % 4) as u8;
c.delta_group_ln1p_diff = if is_target { 2.0 } else { 0.5 } + (i % 7) as f32 * 0.1;
c.delta_group_ln1p_ratio = if is_target { 0.8 } else { 0.3 };
c
})
.collect()
}

fn synthetic_competed_linear_only(n: u32) -> Vec<CompetedCandidate> {
synthetic_competed(n)
.into_iter()
.map(|mut c| {
let sample = sample_competed_candidate();
c.delta_group_ln1p_diff = sample.delta_group_ln1p_diff;
c.delta_group_ln1p_ratio = sample.delta_group_ln1p_ratio;
c
})
.collect()
Expand Down Expand Up @@ -1956,9 +1948,9 @@ mod feature_tests {

#[test]
fn hybrid_lda_score_carries_the_linear_lane_into_the_gbm() {
assert_nonlinear_lane_is_flat(&synthetic_competed_linear_only(360), "lda_score");
assert_nonlinear_lane_is_flat(&synthetic_competed(360), "lda_score");

let (out, stats) = rescore_hybrid(synthetic_competed_linear_only(360), library()).unwrap();
let (out, stats) = rescore_hybrid(synthetic_competed(360), library()).unwrap();

let split_on_it = stats
.iter()
Expand Down Expand Up @@ -2058,16 +2050,16 @@ mod feature_tests {
/// Every consumer indexes on that contract -- `rescore_dash` sweeps the
/// matrix a row at a time with every column's accumulator live, so an
/// interleaved or transposed write would silently mix features together
/// rather than fail. Rows carry distinct `delta_group_ln1p_diff` values so the
/// rather than fail. Rows carry distinct `rising_cycles` values so the
/// assertion can tell them apart; a length check alone passes even when
/// the layout is wrong.
#[test]
fn feature_frame_rows_are_contiguous() {
let rows: Vec<_> = [1.0f32, 2.0, 3.0]
let rows: Vec<_> = [1u8, 2, 3]
.into_iter()
.map(|delta_group_ln1p_diff| {
.map(|rising_cycles| {
let mut c = sample_competed_candidate();
c.delta_group_ln1p_diff = delta_group_ln1p_diff;
c.scoring.counts.rising_cycles = rising_cycles;
c.into_final()
})
.collect();
Expand All @@ -2078,13 +2070,13 @@ mod feature_tests {

let j = names
.iter()
.position(|n| &**n == "delta_group_ln1p_diff")
.expect("delta_group_ln1p_diff is an ALL-lane feature");
.position(|n| &**n == "rising_cycles")
.expect("rising_cycles is an ALL-lane feature");
for (i, expected) in [1.0f64, 2.0, 3.0].into_iter().enumerate() {
assert_eq!(
got[i * nf + j],
expected,
"row {i}'s delta_group_ln1p_diff is not at matrix[{i} * {nf} + {j}]"
"row {i}'s rising_cycles is not at matrix[{i} * {nf} + {j}]"
);
}
}
Expand Down
12 changes: 1 addition & 11 deletions rust/timsseek/src/scoring/blocks/result_meta.rs
Original file line number Diff line number Diff line change
@@ -1,20 +1,10 @@
//! Result-meta family -- post-model, output-only fields (filled in place after
//! rescoring, not computed via Inputs). `columns` emits all four; only the two
//! delta-group fields carry `#[feat(raw)]`, so they are the only ones in a
//! feature lane -- the LINEAR one, since `linear = true` is the default
//! (`discriminant_score`/`qvalue` are Parquet-only).
//! Result metadata written to Parquet after rescoring.

use timsseek_macros::ScoreBlock;

/// Stage: post-model (output-only).
#[derive(Debug, Clone, Copy, ::serde::Serialize, ScoreBlock)]
pub struct ResultMeta {
/// `ln_1p(best_score) - ln_1p(runner_up_score)` within the competition group.
#[feat(raw)]
pub delta_group_ln1p_diff: f32,
/// `ln_1p(runner_up_score) / ln_1p(best_score)` within the competition group.
#[feat(raw)]
pub delta_group_ln1p_ratio: f32,
pub discriminant_score: f32,
pub qvalue: f32,
}
20 changes: 6 additions & 14 deletions rust/timsseek/src/scoring/parquet_writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,14 +39,15 @@ use super::results::FinalResult;
/// numeric id is written as its digits.
/// - 4: sequence is canonical and nullable, resolved from stored analyte facts;
/// entry_name, molecular_formula and formula_basis are separate nullable columns.
/// - 6: removed the unused competition-group margin columns.
///
/// `decoy_group_id` equals `library_id` on a row whose library declared no
/// competition group: the row competes only with its own mass-shift variants
/// and is its own group. mzSpecLib can declare a group (`related spectrum
/// keys`), and then a target and its shipped decoy share one `decoy_group_id`
/// across two `library_id`s. Other formats declare nothing, so for them the
/// duplication is expected, not a bug.
pub const RESULTS_FORMAT_VERSION: u32 = 5;
pub const RESULTS_FORMAT_VERSION: u32 = 6;

// ---------------------------------------------------------------------------
// Build a RecordBatch from a slice of FinalResult
Expand Down Expand Up @@ -304,8 +305,6 @@ impl<'a> ResultParquetWriter<'a> {
assert!(self.raw, "raw scores require the raw output schema");
self.add(FinalResult {
scoring: result.scoring,
delta_group_ln1p_diff: f32::NAN,
delta_group_ln1p_ratio: f32::NAN,
discriminant_score: f32::NAN,
qvalue: f32::NAN,
})
Expand Down Expand Up @@ -360,12 +359,7 @@ mod tests {
for rows in [&[][..], std::slice::from_ref(&row)] {
let batch = output_batch(rows, &geom, true).unwrap();
assert_eq!(batch.num_rows(), rows.len());
for name in [
"qvalue",
"discriminant_score",
"delta_group_ln1p_diff",
"delta_group_ln1p_ratio",
] {
for name in ["qvalue", "discriminant_score"] {
assert!(batch.schema().field_with_name(name).is_err());
}
assert!(batch.schema().field_with_name("main_score").is_ok());
Expand Down Expand Up @@ -563,14 +557,12 @@ mod tests {
}

#[test]
fn delta_columns_name_their_ln1p_formulas() {
fn unused_group_margin_columns_are_absent() {
let batch = build_record_batch(&[], &one_row_arena()).expect("schema");
let schema = batch.schema();

assert!(schema.index_of("delta_group_ln1p_diff").is_ok());
assert!(schema.index_of("delta_group_ln1p_ratio").is_ok());
assert!(schema.index_of("delta_group").is_err());
assert!(schema.index_of("delta_group_ratio").is_err());
assert!(schema.index_of("delta_group_ln1p_diff").is_err());
assert!(schema.index_of("delta_group_ln1p_ratio").is_err());
}

#[test]
Expand Down
27 changes: 3 additions & 24 deletions rust/timsseek/src/scoring/results.rs
Original file line number Diff line number Diff line change
Expand Up @@ -144,20 +144,16 @@ pub struct ScoredCandidate {
#[derive(Debug, Clone, Serialize)]
pub struct CompetedCandidate {
pub scoring: ScoringFields,
pub delta_group_ln1p_diff: f32,
pub delta_group_ln1p_ratio: f32,
/// Scratch field for CrossValidatedScorer (written during rescore)
pub(crate) discriminant_score: f32,
/// Scratch field for q-value assignment
pub(crate) qvalue: f32,
}

impl CompetedCandidate {
/// The post-model meta block (used for the ML delta-group features).
/// The post-model metadata block.
pub(crate) fn result_meta(&self) -> ResultMeta {
ResultMeta {
delta_group_ln1p_diff: self.delta_group_ln1p_diff,
delta_group_ln1p_ratio: self.delta_group_ln1p_ratio,
discriminant_score: self.discriminant_score,
qvalue: self.qvalue,
}
Expand All @@ -168,8 +164,6 @@ impl CompetedCandidate {
#[derive(Debug, Clone, Serialize)]
pub struct FinalResult {
pub scoring: ScoringFields,
pub delta_group_ln1p_diff: f32,
pub delta_group_ln1p_ratio: f32,
pub discriminant_score: f32,
pub qvalue: f32,
}
Expand All @@ -180,8 +174,6 @@ impl FinalResult {
pub fn sample() -> Self {
Self {
scoring: ScoringFields::sample_default(),
delta_group_ln1p_diff: 0.0,
delta_group_ln1p_ratio: 0.0,
discriminant_score: 0.0,
qvalue: 0.0,
}
Expand All @@ -190,8 +182,6 @@ impl FinalResult {
/// The post-model meta block (used for the Parquet meta columns).
pub(crate) fn result_meta(&self) -> ResultMeta {
ResultMeta {
delta_group_ln1p_diff: self.delta_group_ln1p_diff,
delta_group_ln1p_ratio: self.delta_group_ln1p_ratio,
discriminant_score: self.discriminant_score,
qvalue: self.qvalue,
}
Expand All @@ -215,19 +205,10 @@ impl FinalResult {
// ---------------------------------------------------------------------------

impl ScoredCandidate {
/// Convert into a `CompetedCandidate` with the given log-space delta values.
///
/// Items that are alone in their group (no competitor) should pass
/// `f32::NAN` for both deltas.
pub fn into_competed(
self,
delta_group_ln1p_diff: f32,
delta_group_ln1p_ratio: f32,
) -> CompetedCandidate {
/// Convert a competition winner into a `CompetedCandidate`.
pub fn into_competed(self) -> CompetedCandidate {
CompetedCandidate {
scoring: self.scoring,
delta_group_ln1p_diff,
delta_group_ln1p_ratio,
discriminant_score: f32::NAN,
qvalue: f32::NAN,
}
Expand All @@ -239,8 +220,6 @@ impl CompetedCandidate {
pub fn into_final(self) -> FinalResult {
FinalResult {
scoring: self.scoring,
delta_group_ln1p_diff: self.delta_group_ln1p_diff,
delta_group_ln1p_ratio: self.delta_group_ln1p_ratio,
discriminant_score: self.discriminant_score,
qvalue: self.qvalue,
}
Expand Down
Loading
Loading