Client support for patient/specimen/extraction, naming a VCF's extraction, and specimen measures (SACGF/variantgrid#1707, #1559)

[davmlaw]

🤖 Written by Claude

Background

TSO 500 phase 4 has landed server side (SACGF/variantgrid#1707 and #1559, PR SACGF/variantgrid#1716). VariantGrid can now be told which patient, specimen and extraction a lab's sequencing belongs to, which extraction each uploaded VCF came off, and what the specimen-level measures (TMB, MSI, GIS, tumour fraction, ploidy) were. Nothing consumes any of it yet — this client is what the lab codes against.

The join key is Patient → Specimen → Extraction → Sample. A TSO 500 block is one specimen with two extractions, the DNA and RNA arms (2600000001C / 2600000001B), so both arms post against the same specimen and the arm is named by nucleic_acid_source on the extraction.

How identifiers work. Every one of Patient / Specimen / Extraction is reachable two ways:

"2600000001C"                                                     // the local reference
                                                                  // (reference_id, or patient_code for a patient)
{"code": "H12345", "external_type": "HelixID"}                    // an ExternalPK - code AND external_type
{"code": "H12345", "external_type": "HelixID",
 "external_manager": "HELIX"}                                     // narrowed to one manager's namespace

code without external_type is a 400, since ExternalPK is unique on (code, external_type, external_manager) and a code alone names nothing.

Ordering doesn't matter. A VCF may name an extraction that doesn't exist yet: the server parks the claim on the sample and a reconcile task attaches it once the extraction is created. So the client can post files and accession records in either order, and never needs to re-send.

Wanted

1. Dataclasses — src/variantgrid_api/data_models.py

• Patient — patient_code, family_code, first_name, last_name, date_of_birth, date_of_death, sex, affected, optional external_pk
• Specimen — patient (a reference, see below), reference_id, description, collected_by, collection_date, received_date, tissue_status, optional external_pk
• Extraction — specimen (a reference), reference_id, nucleic_acid_source, extraction_date, optional external_pk
• SpecimenMeasure — measure_type, value, unit, call, threshold, threshold_source, method, source_payload, measured_date, optional extraction
• ExternalPK — code, external_type, external_manager, all three required
• ExternalReference — the thing a parent or an upload is named by: reference_id and/or code + external_type + optional external_manager, serialising to a bare string when it only carries reference_id

Enum values as the server stores them: nucleic_acid_source is D/R; tissue_status is R (reference/unaffected), A (affected/lesional) or U (unknown); measure_type is T TMB, M MSI, G GIS, F tumour fraction, P ploidy.

2. Client methods — src/variantgrid_api/api_client.py

Each of the create calls is an upsert keyed on the identifiers sent, so re-posting a run returns the same rows rather than duplicates.

create_patient(patient)                                  # POST patients/api/v1/patient/
create_specimen(specimen)                                # POST patients/api/v1/specimen/
create_extraction(extraction)                            # POST patients/api/v1/extraction/
create_specimen_measure(specimen_reference, measure)     # POST patients/api/v1/specimen_measure/
create_specimen_measures(specimen_reference, measures)   # POST patients/api/v1/specimen_measure/bulk_create
                                                         #   body: {"specimen": <reference>, "measures": [...]}
link_sequencing_sample_extraction(sequencing_sample_lookup, extraction_reference)
                                                         # POST seqauto/api/v1/sequencing_sample/link_extraction
                                                         #   body: {"sequencing_sample": <SequencingSampleLookup>,
                                                         #          "extraction": <reference>}

A specimen naming a patient the server doesn't have is a 400 (it has nowhere to live), as is a measure naming an unknown specimen. The link call is the one that answers differently: an unknown sequencing sample is a 400, but an unknown extraction is a 202 carrying match_status and match_error — the claim is parked and will attach itself. _handle_json_response already returns the body for a 2xx, so this is about surfacing match_status to the caller rather than treating 202 as failure.

One link call per sequencing sample is enough: the server carries the extraction down to every Sample created from that arm's VCFs, and across to the new rows if the sample sheet is re-sent.

3. upload_file gains upload metadata — src/variantgrid_api/api_client.py:234

The server takes upload metadata as query params on upload/api/v1/file_upload, and upload_file only ever sends path, so none of it is reachable from here. A metadata: Optional[dict] = None kwarg merged into params covers the lot:

• extraction — one reference for every sample in the file, which is every TSO 500 file
• sample_extractions — a JSON object keyed on VCF sample name, for a multi-sample VCF ({"TUMOUR": "2600000001C", "NORMAL": "2600000002C"}). Sending both keys is a 400
• genome_build — from TSO 500 phase 2: per-file loader refinements variantgrid#1711, also unreachable today. Send a build's own name (GRCh37), never an alias (hg19)
• source — the caller/software string, likewise (DRAGEN TSO500 SmallVariant, DRAGEN TSO500 CNV)

An unknown key is a 400 while the client is still connected. An unknown extraction is not — that is the ordering race the server absorbs.

This is the only route on a deployment not running seqauto, and the one a hand-uploaded file uses anywhere.

4. Example — examples/example_tso500.py

Alongside examples/example_haem_20_999.py, showing the whole run against the five-file TSO 500 set: accession patient → specimen → both extractions, post the sequencing run and sample sheet, one link call per arm, upload the DNA arm's VCFs and the RNA arm's, then post the run's TMB/MSI/GIS against the specimen.

5. CHANGELOG entry and version bump

Current version is 1.4.0.

Transcribe, never compute

The measures API takes what the client read out of vendor output — MetricsOutput.tsv, the MSI JSON, the GIS output — plus source_payload holding the raw block it came from, so "which file and which pipeline version produced this number" stays answerable at report time. Deriving a value client-side would move a clinical calculation into an unversioned client, and the server stores both the score and the lab's call precisely because the threshold between them is lab policy rather than vendor output.

Compatibility

Purely additive: every existing method and payload is untouched, and upload_file without metadata sends exactly what it sends today. Requires a server at or after SACGF/variantgrid#1716.