From 1b25f54aef42a36912e17e2eab45ad03b44ea005 Mon Sep 17 00:00:00 2001 From: "Stephanie S. Hong" Date: Mon, 26 Oct 2026 12:09:52 -0400 Subject: [PATCH] Concept map v1.8.1; switch off NRBC lab value (not collected; Labcorp reports do not include it) --- CHANGELOG.md | 39 +++++++++++++++++++ CITATION.cff | 4 +- codemeta.json | 6 +-- ...i_readi_redcap_value_set_mapping_data.yaml | 8 ++-- ...ed_cap_data_dictionary_with_extensions.csv | 4 +- .../scripts/count_concept_map.sql | 22 ++++++----- .../scripts/validate_concept_map_vocab.sql | 2 +- ...i_readi_redcap_value_set_mapping.sssom.tsv | 4 +- ..._readi_redcap_value_set_mapping.sssom.yaml | 2 +- ..._readi_redcap_value_set_mapping_common.csv | 4 +- ...dcap_value_set_mapping_common_map2omop.csv | 2 - ...ng_map2omop_open_source_public_version.csv | 2 - .../scripts/01_create_unit_lookup.sql | 5 ++- .../01a_load_unit_concept_lookup_csv.sql | 4 +- .../scripts/02_view_unit_lookup.sql | 4 +- .../scripts/README.md | 9 ++--- .../value-set-mapping-common/scripts/USAGE.md | 4 +- .../scripts/unit_concept_lookup.csv | 1 - 18 files changed, 83 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f90227..5633826 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,45 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +## [v1.8.1] - 2026-10-26 + +NRBC is switched off: it is not collected. The data team confirmed that the Labcorp *CBC, Platelet, No Differential* reports returned to AI-READI do not include NRBC, and that no REDCap field will be added ("We do not collect data for NRBC. This value is not provided in the reports returned to us from LabCorp."). A redacted Labcorp report (UAB) lists exactly 9 results: WBC, RBC, hemoglobin, hematocrit, MCV, MCH, MCHC, RDW and platelets. Two rows are edited in place, so every line number is the same as in v1.8. No concept changes; the ETL is not affected (the field matches no data). + +### Changed — Map_to_OMOP + +- `lbscat_nrbc` (line 2127, → 40761514) and its unit row `lbscat_nrbc_u` (line 2128, → 8554): `Map_to_OMOP` Yes → No and `transferred_to_fairhub_REDCap_dataset` Yes → No, so the transferred rows match the fields FAIRhub receives. The rows are kept (as for the superseded MoCA unit rows); the v1.7 note under *Added — NRBC* is superseded. + +### Reviewed and not changed — CBC LOINC codes + +The 9 CBC fields on the report map to the Labcorp test 028142 LOINC codes (v1.7), all standard Measurement concepts confirmed in `omop_vocab`, exactMatch 1; the report units match the unit rows (lines 2032–2040; the report writes "uL" for "µL"). + +| Line | Field | LOINC | Concept | +|---|---|---|---| +| 672 | `lbscat_wbc` WBC (x10E3/µL) | 6690-2 | 3000905 *Leukocytes [#/volume] in Blood by Automated count* | +| 673 | `lbscat_rbc` RBC (x10E6/µL) | 789-8 | 3020416 *Erythrocytes [#/volume] in Blood by Automated count* | +| 674 | `lbscat_a1c` Hemoglobin (g/dL) | 718-7 | 3000963 *Hemoglobin [Mass/volume] in Blood* | +| 675 | `lbscat_hct` Hematocrit (%) | 4544-3 | 3023314 *Hematocrit [Volume Fraction] of Blood by Automated count* | +| 676 | `lbscat_mcv` MCV (fL) | 787-2 | 3023599 *MCV [Entitic mean volume] in Red Blood Cells by Automated count* | +| 677 | `lbscat_mch` MCH (pg) | 785-6 | 3012030 *MCH [Entitic mass] by Automated count* | +| 678 | `lbscat_mchc` MCHC (g/dL) | 786-4 | 3009744 *MCHC [Entitic Mass/volume] in Red Blood Cells by Automated count* | +| 679 | `lbscat_rdw` RDW (%) | 788-0 | 3019897 *Erythrocyte [DistWidth] in Red Blood Cells by Automated count* | +| 680 | `lbscat_plt` Platelets (x10E3/µL) | 777-3 | 3024929 *Platelets [#/volume] in Blood by Automated count* | + +### Changed — scripts + +- `unit_concept_lookup.csv`: 114 rows (field 89, string 25); `lbscat_nrbc` removed. +- `01_create_unit_lookup.sql`: uses only `Map_to_OMOP` = Yes rows, so a switched-off unit row never enters the lookup (the result equals the CSV). Row counts updated in `01a_`, `02_`, the scripts' README and USAGE. +- `count_concept_map.sql`: the two NRBC rows are reported as designated-but-unmapped, "Not collected (NRBC)"; expected values updated to v1.8.1. `validate_concept_map_vocab.sql`: V0 comment (2,142 rows, unchanged). + +### Changed — master REDCap data dictionary + +- The two NRBC rows (master records 3138–3139) take the new values; `Origin` = Changed, `Date_of_Variable_Modification` 10/26/2026. Every concept map row still has a master row with the same values; 3,150 rows. + +### Regenerated + +- map2omop (2,112 rows) and open-source public (1,997) versions, SSSOM (2,067 rows; 75 incomplete) and LinkML (2,142 records) from the v1.8.1 common file. +- Counts: standard 1,515 (399 concepts); custom 489 (342); mapped 2,004 (69.0%, 741 concepts); designated but unmapped 138 (+2 "Not collected (NRBC)"); not designated 764. LOINC 1,190/276; UCUM 132/40. Standard predicates: exact 1,255; close 258 (170 at 1, 88 at 0.75); related 2. + ## [v1.8] - 2026-10-25 Corrections from the review of the concept map against the REDCap codebook (2026-10-25), checked against the delivered OMOP survey data, an ECG procedure row, four protected fields that had no row, and the master REDCap data dictionary brought in line with the concept map. 15 rows are appended at the end of the file (lines 2129–2143) and the other changes are in place, so every existing line number is the same as in v1.7. Concept IDs and names were confirmed in `omop_vocab`. diff --git a/CITATION.cff b/CITATION.cff index 1e55fd4..7252a9e 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -16,8 +16,8 @@ repository-code: "https://github.com/AI-READI/DataElementMaps" repository-artifact: "https://github.com/AI-READI/DataElementMaps/blob/main/manuscript-tvst2026/ai_readi_redcap_value_set_mapping_common.csv" url: "https://aireadi.org" license: MIT -version: "v1.8" -date-released: "2026-10-25" +version: "v1.8.1" +date-released: "2026-10-26" identifiers: - type: doi value: "10.5281/zenodo.23375723" diff --git a/codemeta.json b/codemeta.json index b007f41..11f8fc6 100644 --- a/codemeta.json +++ b/codemeta.json @@ -7,11 +7,11 @@ "codeRepository": "https://github.com/AI-READI/DataElementMaps", "downloadUrl": "https://github.com/AI-READI/DataElementMaps", "issueTracker": "https://github.com/AI-READI/DataElementMaps/issues", - "version": "v1.8", - "datePublished": "2026-10-25", + "version": "v1.8.1", + "datePublished": "2026-10-26", "releaseNotes": "Concept map v1.6: maps the A/G ratio, BUN/creatinine ratio (stored as creatinine/BUN), BMI and waist circumference to standard LOINC concepts, adds the unit ratio for the three ratio fields, and excludes the leftover answers of two removed questions. ETL re-run required. See CHANGELOG.md.", "dateCreated": "2023-06-28", - "dateModified": "2026-10-25", + "dateModified": "2026-10-26", "applicationCategory": "Scientific", "developmentStatus": "active", "isPartOf": "https://aireadi.org/", diff --git a/redCapDataDictionaryWithExtension/linkml/ai_readi_redcap_value_set_mapping_data.yaml b/redCapDataDictionaryWithExtension/linkml/ai_readi_redcap_value_set_mapping_data.yaml index 2606a63..8804dee 100644 --- a/redCapDataDictionaryWithExtension/linkml/ai_readi_redcap_value_set_mapping_data.yaml +++ b/redCapDataDictionaryWithExtension/linkml/ai_readi_redcap_value_set_mapping_data.yaml @@ -45843,8 +45843,8 @@ mappings: confidence: 1.0 modifier: fixed unit; no REDCap unit field - mapping_record_id: AIREADI_MAPPING_2126 - map_to_omop: 'Yes' - transferred_to_fairhub_redcap_dataset: 'Yes' + map_to_omop: 'No' + transferred_to_fairhub_redcap_dataset: 'No' protected_or_open_source: open-source extension_needed: 'No' form_name: lab_results @@ -45862,8 +45862,8 @@ mappings: predicate_id: skos:exactMatch confidence: 1.0 - mapping_record_id: AIREADI_MAPPING_2127 - map_to_omop: 'Yes' - transferred_to_fairhub_redcap_dataset: 'Yes' + map_to_omop: 'No' + transferred_to_fairhub_redcap_dataset: 'No' protected_or_open_source: open-source extension_needed: 'No' form_name: lab_results diff --git a/redCapDataDictionaryWithExtension/redcap-data-dictionary/_master_red_cap_data_dictionary_with_extensions.csv b/redCapDataDictionaryWithExtension/redcap-data-dictionary/_master_red_cap_data_dictionary_with_extensions.csv index 2c6fdd2..a9e3eb1 100644 --- a/redCapDataDictionaryWithExtension/redcap-data-dictionary/_master_red_cap_data_dictionary_with_extensions.csv +++ b/redCapDataDictionaryWithExtension/redcap-data-dictionary/_master_red_cap_data_dictionary_with_extensions.csv @@ -3228,8 +3228,8 @@ Added,10/24/2026,Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rttr_osm2,,Answ Added,10/24/2026,Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rtfl_os,,Answer,1,rtfl_os_1,Yes,4063911,,Ocular fundus photography,Procedure,SNOMED,Procedure,20067007,S,skos:closeMatch,0.75,left eye|45888270,,45888270,Left eye,, Added,10/24/2026,Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio,,Question,31045-8,Creatinine/BUN ratio,Creatinine/BUN ratio,3033837,,Creatinine/Urea nitrogen [Mass Ratio] in Serum or Plasma,Measurement,LOINC,Lab Test,31045-8,S,skos:exactMatch,1,,,,,, Added,10/24/2026,Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio_u,,Unit,ratio,import_bun_creatinine_ratio_u,ratio,8523,,ratio,Unit,UCUM,Unit,{ratio},S,skos:exactMatch,1,fixed unit; no REDCap unit field,,,,, -Added,10/24/2026,Yes,Yes,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,,,, -Added,10/24/2026,Yes,Yes,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,,,, +Changed,10/26/2026,No,No,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,,,, +Changed,10/26/2026,No,No,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,,,, Added,10/25/2026,Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,19,cmdosu_19,Milligram per Gram (mg/g),8723,,milligram per gram,Unit,UCUM,Unit,mg/g,S,skos:exactMatch,1,,,,,, Added,10/25/2026,Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,20,cmdosu_20,Milligram per Milliliter (mg/mL),8861,,milligram per milliliter,Unit,UCUM,Unit,mg/mL,S,skos:exactMatch,1,,,,,, Added,10/25/2026,Yes,Yes,open-source,No,device_return,yesno,dvamwenyn,,Answer,1,dvamwenyn_1,Yes,45877994,,Yes,Meas Value,LOINC,Answer,LA33-6,S,skos:exactMatch,1,,,,,, diff --git a/redCapDataDictionaryWithExtension/scripts/count_concept_map.sql b/redCapDataDictionaryWithExtension/scripts/count_concept_map.sql index 643b0cc..4082c32 100644 --- a/redCapDataDictionaryWithExtension/scripts/count_concept_map.sql +++ b/redCapDataDictionaryWithExtension/scripts/count_concept_map.sql @@ -35,6 +35,8 @@ -- * Unit rows switched off because a newer row replaces them (Map_to_OMOP = 'No', -- the old MoCA *_time_unit / *_score_unit rows since v1.7) are reported as -- designated-but-unmapped ("Superseded unit row"). +-- * The NRBC rows (lbscat_nrbc, lbscat_nrbc_u; Map_to_OMOP = 'No' since v1.8.1: NRBC is not +-- collected) are reported as designated-but-unmapped ("Not collected (NRBC)"). -- * Answer rows switched off because the answer is not a REDCap choice (Map_to_OMOP = 'No', -- answer 0 of susmkstoage, sualcage, suvptoyr since v1.8) are counted with -- "Answer to an excluded question". @@ -74,6 +76,8 @@ SELECT THEN 'REDCap survey timestamp' WHEN TRY_CAST(TARGET_CONCEPT_ID AS BIGINT) = 0 THEN 'No matching concept' WHEN Map_to_OMOP <> 'Yes' AND SRC_CODE LIKE 'moca_paper%' THEN 'MoCA paper-administration flag' + WHEN Map_to_OMOP <> 'Yes' AND (SRC_CODE = 'lbscat_nrbc' OR FIELD_ID = 'lbscat_nrbc_u') + THEN 'Not collected (NRBC)' WHEN Map_to_OMOP <> 'Yes' AND temp_Question_or_Answer = 'Answer' THEN 'Answer to an excluded question' WHEN Map_to_OMOP <> 'Yes' AND TARGET_DOMAIN_ID = 'Unit' THEN 'Superseded unit row' @@ -117,8 +121,8 @@ FROM rows ORDER BY CASE category WHEN 'Standard' THEN 1 WHEN 'Custom' THEN 2 WHEN 'Mapped total' THEN 3 WHEN 'Designated but unmapped' THEN 4 WHEN 'Not designated' THEN 5 ELSE 6 END; --- Expected (v1.8): Standard 1,517 (52.2%) / 400; Custom 489 (16.8%) / 342; --- Mapped 2,006 (69.0%) / 742; Designated-unmapped 136 (4.7%); +-- Expected (v1.8.1): Standard 1,515 (52.1%) / 399; Custom 489 (16.8%) / 342; +-- Mapped 2,004 (69.0%) / 741; Designated-unmapped 138 (4.7%); -- Not designated 764 (26.3%); Total 2,906. -- 2. Elements and distinct concepts by vocabulary ---------------------------------- @@ -130,7 +134,7 @@ FROM mapping_classified WHERE mapping_category IN ('Standard','Custom') GROUP BY mapping_category, TARGET_VOCABULARY_ID ORDER BY mapping_category DESC, elements DESC; --- Expected (v1.8): LOINC 1,191/277; SNOMED 171/74; UCUM 133/40; CPT4 12/1; PPI 5/5; +-- Expected (v1.8.1): LOINC 1,190/276; SNOMED 171/74; UCUM 132/40; CPT4 12/1; PPI 5/5; -- Ethnicity 4/2; OMOP Extension 1/1; AIREADI 390/265; AIREADI-Vision 69/47; AIREADI-MOCA 30/30. -- 3. Designated-but-unmapped breakdown ------------------------------------------------- @@ -139,9 +143,9 @@ FROM mapping_classified WHERE mapping_category = 'Designated but unmapped' GROUP BY unmapped_reason ORDER BY n DESC; --- Expected (v1.8): reference range 46; survey timestamp 34; superseded unit row 20; specimen handling 12; --- BCVA 10; answer to an excluded question 7; no matching concept 5; --- REDCap administrative 1; MoCA paper flag 1 (total 136). +-- Expected (v1.8.1): reference range 46; survey timestamp 34; superseded unit row 20; specimen handling 12; +-- BCVA 10; answer to an excluded question 7; no matching concept 5; not collected (NRBC) 2; +-- REDCap administrative 1; MoCA paper flag 1 (total 138). -- 4. Confidence and SKOS predicate distribution ----------------------------------------- SELECT mapping_category, @@ -152,9 +156,9 @@ FROM mapping_classified WHERE mapping_category IN ('Standard','Custom') GROUP BY mapping_category, PREDICATE_ID, CONFIDENCE ORDER BY mapping_category DESC, PREDICATE_ID, CONFIDENCE DESC; --- Expected (v1.8, standard): exact 1,257; close 258 (170 at 1, 88 at 0.75); related 2; --- confidence 1 = 1,427; 0.75 = 89; 0.5 = 1. --- Expected (v1.8, custom): all 489 exact match at confidence 1. +-- Expected (v1.8.1, standard): exact 1,255; close 258 (170 at 1, 88 at 0.75); related 2; +-- confidence 1 = 1,425; 0.75 = 89; 0.5 = 1. +-- Expected (v1.8.1, custom): all 489 exact match at confidence 1. -- 5. Laterality qualifiers --------------------------------------------------------------- SELECT qualifier_concept_id, qualifier_source_value, COUNT(*) AS n diff --git a/redCapDataDictionaryWithExtension/scripts/validate_concept_map_vocab.sql b/redCapDataDictionaryWithExtension/scripts/validate_concept_map_vocab.sql index ff4b857..eacd1ee 100644 --- a/redCapDataDictionaryWithExtension/scripts/validate_concept_map_vocab.sql +++ b/redCapDataDictionaryWithExtension/scripts/validate_concept_map_vocab.sql @@ -19,7 +19,7 @@ SELECT *, TRY_CAST(TRIM(CAST(qualifier_concept_id AS STRING)) AS BIGINT) AS qid FROM redcapsurveydb.ai_readi_redcap_value_set_mapping_common; --- V0. Row count (v1.4 and v1.5: 2,076; v1.6: 2,079; v1.7: 2,127; v1.8: 2,142) +-- V0. Row count (v1.4 and v1.5: 2,076; v1.6: 2,079; v1.7: 2,127; v1.8 and v1.8.1: 2,142) SELECT COUNT(*) AS n_rows FROM cmap; -- V1. Standard (non-custom) targets missing from the vocabulary, not standard, or invalid. Expect 0 rows. diff --git a/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.tsv b/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.tsv index fb0d996..9e782f5 100644 --- a/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.tsv +++ b/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.tsv @@ -2049,8 +2049,8 @@ AIREADI_REDCAP:retinal_imaging_v2.1.rttr_osm2_1 Yes skos:closeMatch OMOP:745759 AIREADI_REDCAP:retinal_imaging_v2.1.rtfl_os_1 Yes skos:closeMatch OMOP:4063911 Ocular fundus photography semapv:ManualMappingCuration 0.75 AI-READI REDCap SNOMED AI-READI manual_curation left eye|45888270 retinal_imaging_v2 yesno rtfl_os 1 rtfl_os_1 Answer 4063911 Procedure SNOMED Procedure 20067007 S No open-source Yes 45888270 Left eye AIREADI_REDCAP:imported.31045-8.Creatinine_BUN_ratio Creatinine/BUN ratio skos:exactMatch OMOP:3033837 Creatinine/Urea nitrogen [Mass Ratio] in Serum or Plasma semapv:ManualMappingCuration 1 AI-READI REDCap LOINC AI-READI manual_curation imported text import_bun_creatinine_ratio 31045-8 Creatinine/BUN ratio Question 3033837 Measurement LOINC Lab Test 31045-8 S No open-source Yes AIREADI_REDCAP:imported.ratio.import_bun_creatinine_ratio_u ratio skos:exactMatch OMOP:8523 ratio semapv:ManualMappingCuration 1 AI-READI REDCap UCUM AI-READI manual_curation fixed unit; no REDCap unit field imported text import_bun_creatinine_ratio_u ratio import_bun_creatinine_ratio_u Unit 8523 Unit UCUM Unit {ratio} S No open-source Yes -AIREADI_REDCAP:lab_results.lbscat_nrbc NRBC - % skos:exactMatch OMOP:40761514 Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count semapv:ManualMappingCuration 1 AI-READI REDCap LOINC AI-READI manual_curation lab_results text lbscat_nrbc Question 40761514 Measurement LOINC Lab Test 58413-6 S No open-source Yes -AIREADI_REDCAP:lab_results.missing.lbscat_nrbc_u percent skos:exactMatch OMOP:8554 percent semapv:ManualMappingCuration 1 AI-READI REDCap UCUM AI-READI manual_curation fixed unit; no REDCap unit field lab_results text lbscat_nrbc_u % lbscat_nrbc_u Unit 8554 Unit UCUM Unit % S No open-source Yes +AIREADI_REDCAP:lab_results.lbscat_nrbc NRBC - % skos:exactMatch OMOP:40761514 Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count semapv:ManualMappingCuration 1 AI-READI REDCap LOINC AI-READI manual_curation lab_results text lbscat_nrbc Question 40761514 Measurement LOINC Lab Test 58413-6 S No open-source No +AIREADI_REDCAP:lab_results.missing.lbscat_nrbc_u percent skos:exactMatch OMOP:8554 percent semapv:ManualMappingCuration 1 AI-READI REDCap UCUM AI-READI manual_curation fixed unit; no REDCap unit field lab_results text lbscat_nrbc_u % lbscat_nrbc_u Unit 8554 Unit UCUM Unit % S No open-source No AIREADI_REDCAP:current_medications.19.cmdosu_19 Milligram per Gram (mg/g) skos:exactMatch OMOP:8723 milligram per gram semapv:ManualMappingCuration 1 AI-READI REDCap UCUM AI-READI manual_curation current_medications dropdown cmdosu 19 cmdosu_19 Answer 8723 Unit UCUM Unit mg/g S No protected Yes AIREADI_REDCAP:current_medications.20.cmdosu_20 Milligram per Milliliter (mg/mL) skos:exactMatch OMOP:8861 milligram per milliliter semapv:ManualMappingCuration 1 AI-READI REDCap UCUM AI-READI manual_curation current_medications dropdown cmdosu 20 cmdosu_20 Answer 8861 Unit UCUM Unit mg/mL S No protected Yes AIREADI_REDCAP:device_return.1.dvamwenyn_1 Yes skos:exactMatch OMOP:45877994 Yes semapv:ManualMappingCuration 1 AI-READI REDCap LOINC AI-READI manual_curation device_return yesno dvamwenyn 1 dvamwenyn_1 Answer 45877994 Meas Value LOINC Answer LA33-6 S No open-source Yes diff --git a/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.yaml b/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.yaml index 7558680..beb56bd 100644 --- a/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.yaml +++ b/redCapDataDictionaryWithExtension/sssom/ai_readi_redcap_value_set_mapping.sssom.yaml @@ -15,4 +15,4 @@ curie_map: semapv: https://w3id.org/semapv/vocab/ subject_source: AI-READI REDCap object_source: OMOP Standardized Vocabularies -mapping_date: '2026-10-25' +mapping_date: '2026-10-26' diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common.csv b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common.csv index 6a5c274..20e96e5 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common.csv +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common.csv @@ -2124,8 +2124,8 @@ Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rttr_osm2,,Answer,1,rttr_osm2_1, Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rtfl_os,,Answer,1,rtfl_os_1,Yes,4063911,,Ocular fundus photography,Procedure,SNOMED,Procedure,20067007,S,skos:closeMatch,0.75,left eye|45888270,,45888270,Left eye Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio,,Question,31045-8,Creatinine/BUN ratio,Creatinine/BUN ratio,3033837,,Creatinine/Urea nitrogen [Mass Ratio] in Serum or Plasma,Measurement,LOINC,Lab Test,31045-8,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio_u,,Unit,ratio,import_bun_creatinine_ratio_u,ratio,8523,,ratio,Unit,UCUM,Unit,{ratio},S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, -Yes,Yes,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,, -Yes,Yes,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, +No,No,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,, +No,No,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,19,cmdosu_19,Milligram per Gram (mg/g),8723,,milligram per gram,Unit,UCUM,Unit,mg/g,S,skos:exactMatch,1,,,, Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,20,cmdosu_20,Milligram per Milliliter (mg/mL),8861,,milligram per milliliter,Unit,UCUM,Unit,mg/mL,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,device_return,yesno,dvamwenyn,,Answer,1,dvamwenyn_1,Yes,45877994,,Yes,Meas Value,LOINC,Answer,LA33-6,S,skos:exactMatch,1,,,, diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common_map2omop.csv b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common_map2omop.csv index 57b8823..3950a45 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common_map2omop.csv +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common_map2omop.csv @@ -2096,8 +2096,6 @@ Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rttr_osm2,,Answer,1,rttr_osm2_1, Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rtfl_os,,Answer,1,rtfl_os_1,Yes,4063911,,Ocular fundus photography,Procedure,SNOMED,Procedure,20067007,S,skos:closeMatch,0.75,left eye|45888270,,45888270,Left eye Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio,,Question,31045-8,Creatinine/BUN ratio,Creatinine/BUN ratio,3033837,,Creatinine/Urea nitrogen [Mass Ratio] in Serum or Plasma,Measurement,LOINC,Lab Test,31045-8,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio_u,,Unit,ratio,import_bun_creatinine_ratio_u,ratio,8523,,ratio,Unit,UCUM,Unit,{ratio},S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, -Yes,Yes,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,, -Yes,Yes,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,19,cmdosu_19,Milligram per Gram (mg/g),8723,,milligram per gram,Unit,UCUM,Unit,mg/g,S,skos:exactMatch,1,,,, Yes,Yes,protected,No,current_medications,dropdown,cmdosu,,Answer,20,cmdosu_20,Milligram per Milliliter (mg/mL),8861,,milligram per milliliter,Unit,UCUM,Unit,mg/mL,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,device_return,yesno,dvamwenyn,,Answer,1,dvamwenyn_1,Yes,45877994,,Yes,Meas Value,LOINC,Answer,LA33-6,S,skos:exactMatch,1,,,, diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_map2omop_open_source_public_version.csv b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_map2omop_open_source_public_version.csv index 36bb866..5c4dd05 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_map2omop_open_source_public_version.csv +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_map2omop_open_source_public_version.csv @@ -1987,8 +1987,6 @@ Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rttr_osm2,,Answer,1,rttr_osm2_1, Yes,Yes,open-source,No,retinal_imaging_v2,yesno,rtfl_os,,Answer,1,rtfl_os_1,Yes,4063911,,Ocular fundus photography,Procedure,SNOMED,Procedure,20067007,S,skos:closeMatch,0.75,left eye|45888270,,45888270,Left eye Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio,,Question,31045-8,Creatinine/BUN ratio,Creatinine/BUN ratio,3033837,,Creatinine/Urea nitrogen [Mass Ratio] in Serum or Plasma,Measurement,LOINC,Lab Test,31045-8,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,imported,text,import_bun_creatinine_ratio_u,,Unit,ratio,import_bun_creatinine_ratio_u,ratio,8523,,ratio,Unit,UCUM,Unit,{ratio},S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, -Yes,Yes,open-source,No,lab_results,text,,,Question,lbscat_nrbc,,NRBC - %,40761514,,Nucleated erythrocytes/Leukocytes [Ratio] in Blood by Automated count,Measurement,LOINC,Lab Test,58413-6,S,skos:exactMatch,1,,,, -Yes,Yes,open-source,No,lab_results,text,lbscat_nrbc_u,,Unit,%,lbscat_nrbc_u,percent,8554,,percent,Unit,UCUM,Unit,%,S,skos:exactMatch,1,fixed unit; no REDCap unit field,,, Yes,Yes,open-source,No,device_return,yesno,dvamwenyn,,Answer,1,dvamwenyn_1,Yes,45877994,,Yes,Meas Value,LOINC,Answer,LA33-6,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,device_return,yesno,dvamwenyn,,Answer,0,dvamwenyn_0,No,45878245,,No,Meas Value,LOINC,Answer,LA32-8,S,skos:exactMatch,1,,,, Yes,Yes,open-source,No,device_return,yesno,dcvgmenyn,,Answer,1,dcvgmenyn_1,Yes,45877994,,Yes,Meas Value,LOINC,Answer,LA33-6,S,skos:exactMatch,1,,,, diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01_create_unit_lookup.sql b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01_create_unit_lookup.sql index 1db268f..be03dcb 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01_create_unit_lookup.sql +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01_create_unit_lookup.sql @@ -5,7 +5,7 @@ -- unit_lookup has one row per (key_type, lookup_key): -- key_type = 'field' : lookup_key = measurement field name (e.g. lbscat_rbc) -> 90 rows -- key_type = 'string' : lookup_key = unit text (e.g. 'mEq/L') -> 25 rows --- v1.7: 115 rows (v1.6: 113; v1.4 and v1.5: 110). Same content as unit_concept_lookup.csv (option A: 01a_load_unit_concept_lookup_csv.sql). +-- v1.8.1: 114 rows (v1.7 and v1.8: 115; v1.6: 113; v1.4 and v1.5: 110). Only Map_to_OMOP = 'Yes' rows are used (v1.8.1). Same content as unit_concept_lookup.csv (option A: 01a_load_unit_concept_lookup_csv.sql). -- -- Requires the concept map as a table or view named concept_map (see 00_load_concept_map.sql). -- Dialect: Databricks / Spark SQL. @@ -20,6 +20,7 @@ SELECT DISTINCT TRIM(SRC_CODE) AS unit_source_value FROM concept_map WHERE TARGET_DOMAIN_ID = 'Unit' + AND Map_to_OMOP = 'Yes' AND RIGHT(TRIM(FIELD_ID), 2) = '_u' UNION ALL -- LDL: the measurement field and its unit field have different names @@ -28,6 +29,7 @@ SELECT DISTINCT CAST(TARGET_CONCEPT_ID AS INT), TRIM(SRC_CODE) FROM concept_map WHERE TARGET_DOMAIN_ID = 'Unit' + AND Map_to_OMOP = 'Yes' AND TRIM(FIELD_ID) = 'import_ldl_cholesterol_u' UNION ALL -- by unit text: generic Unit rows (blank FIELD_ID) plus 's', 'second', 'score' @@ -36,4 +38,5 @@ SELECT DISTINCT CAST(TARGET_CONCEPT_ID AS INT), TRIM(SRC_CODE) FROM concept_map WHERE TARGET_DOMAIN_ID = 'Unit' + AND Map_to_OMOP = 'Yes' AND (COALESCE(TRIM(FIELD_ID), '') = '' OR TRIM(SRC_CODE) IN ('s', 'second', 'score')); diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01a_load_unit_concept_lookup_csv.sql b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01a_load_unit_concept_lookup_csv.sql index 3c8acc6..5774b0c 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01a_load_unit_concept_lookup_csv.sql +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/01a_load_unit_concept_lookup_csv.sql @@ -6,7 +6,7 @@ -- 01_create_unit_lookup.sql (option B), without loading the concept map. -- -- unit_concept_lookup.csv columns: key_type, lookup_key, unit_concept_id, unit_source_value --- v1.7: 115 rows (field 90, string 25), one row per (key_type, lookup_key). +-- v1.8.1: 114 rows (field 89, string 25), one row per (key_type, lookup_key). -- The file is UTF-8 (x10E3/µL and x10E6/µL use the micro sign U+00B5). -- -- Set the path to your copy of unit_concept_lookup.csv, then run. @@ -28,5 +28,5 @@ SELECT TRIM(key_type) AS key_type, TRIM(unit_source_value) AS unit_source_value FROM unit_concept_lookup_csv; --- Check: 115 rows; field 90, string 25. +-- Check: 114 rows; field 89, string 25. SELECT key_type, COUNT(*) AS n FROM unit_lookup GROUP BY key_type ORDER BY key_type; diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/02_view_unit_lookup.sql b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/02_view_unit_lookup.sql index 55fc774..87f98c6 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/02_view_unit_lookup.sql +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/02_view_unit_lookup.sql @@ -5,13 +5,13 @@ -- Requires unit_lookup (01_create_unit_lookup.sql). Dialect: Databricks / Spark SQL. -- ============================================================================= --- Row counts by key type. v1.7: field 90, string 25. +-- Row counts by key type. v1.8.1: field 89, string 25. SELECT key_type, COUNT(*) AS n FROM unit_lookup GROUP BY key_type ORDER BY key_type; --- Lookup by measurement field name (90 rows). +-- Lookup by measurement field name (89 rows). SELECT lookup_key AS field_name, unit_concept_id, unit_source_value FROM unit_lookup WHERE key_type = 'field' diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/README.md b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/README.md index 23505e7..68f6665 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/README.md +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/README.md @@ -6,9 +6,9 @@ Scripts to fill `MEASUREMENT.unit_concept_id` from the AI-READI concept map (`.. |---|---| | `USAGE.md` | **Step-by-step usage** | | `00_load_concept_map.sql` | Loads the mapping file as `concept_map` (only needed for option B; set the file path) | -| `unit_concept_lookup.csv` | **The ready-made lookup table** (115 rows): load it and skip building it | +| `unit_concept_lookup.csv` | **The ready-made lookup table** (114 rows): load it and skip building it | | `01a_load_unit_concept_lookup_csv.sql` | Option A: creates `unit_lookup` from `unit_concept_lookup.csv` | -| `01_create_unit_lookup.sql` | Option B: **one query** that builds `unit_lookup` from the concept map (same 115 rows) | +| `01_create_unit_lookup.sql` | Option B: **one query** that builds `unit_lookup` from the concept map (same 114 rows) | | `02_view_unit_lookup.sql` | Review `unit_lookup`: counts, field and unit-text lists, duplicate check | | `03_left_join_unit_lookup_template.sql` | **The LEFT JOIN** to add to the ETL (replace the placeholders) | | `04_example_omop_survey_measurement.sql` | The same join, ready to run on `omop_survey.measurement` | @@ -38,7 +38,7 @@ LEFT JOIN unit_lookup u | key_type | lookup_key | Rows | Source in the concept map | |---|---|---|---| -| `field` | measurement field name | 90 | Unit rows named `_u`: the imported-lab unit fields (e.g. `import_potassium_u`) and the v1.4 fixed-unit rows for fields with no unit field (CBC, physical assessment, autorefractor, globulin, bilirubin, urine albumin/creatinine, MoCA scores and times; v1.6: the BUN/creatinine, A/G and waist/hip ratios; v1.7: `import_bun_creatinine_ratio` and the proposed `lbscat_nrbc`). Plus `import_ldl_cholesterol_calculation`, whose unit field has a different name (`import_ldl_cholesterol_u`). | +| `field` | measurement field name | 89 | Unit rows named `_u`: the imported-lab unit fields (e.g. `import_potassium_u`) and the v1.4 fixed-unit rows for fields with no unit field (CBC, physical assessment, autorefractor, globulin, bilirubin, urine albumin/creatinine, MoCA scores and times; v1.6: the BUN/creatinine, A/G and waist/hip ratios; v1.7: `import_bun_creatinine_ratio`). The v1.7 NRBC unit row (`lbscat_nrbc_u`) has `Map_to_OMOP` = No since v1.8.1 (NRBC is not collected) and is not in the lookup. Plus `import_ldl_cholesterol_calculation`, whose unit field has a different name (`import_ldl_cholesterol_u`). | | `string` | unit text | 25 | Generic Unit rows (blank `FIELD_ID`) plus `s`, `second`, `score` | Notes: @@ -46,7 +46,7 @@ Notes: - MoCA: score fields → 44777566 (score), time fields → 8555 (second). - Do not join on all Unit rows of the concept map by `SRC_CODE`: the same text appears in several rows (e.g. `g/dL` in 6), which would multiply measurement rows. `unit_lookup` avoids this. -### key_type = 'field' (90 rows) +### key_type = 'field' (89 rows) | lookup_key | unit_concept_id | unit_source_value | |---|---|---| @@ -104,7 +104,6 @@ Notes: | `lbscat_mch` | 8564 | pg | | `lbscat_mchc` | 8713 | g/dL | | `lbscat_mcv` | 8583 | fL | -| `lbscat_nrbc` | 8554 | % | | `lbscat_plt` | 8848 | x10E3/µL | | `lbscat_rbc` | 8815 | x10E6/µL | | `lbscat_rdw` | 8554 | % | diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/USAGE.md b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/USAGE.md index 405a3e6..04674f6 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/USAGE.md +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/USAGE.md @@ -4,7 +4,7 @@ How to fill `MEASUREMENT.unit_concept_id` from the AI-READI concept map: create ## Step 1. Create the unit_lookup table (choose one) -**Option A: load the ready-made file (simplest).** `unit_concept_lookup.csv` (in this folder) is the complete lookup table, 115 rows: +**Option A: load the ready-made file (simplest).** `unit_concept_lookup.csv` (in this folder) is the complete lookup table, 114 rows: | Column | Meaning | |---|---| @@ -19,7 +19,7 @@ Upload it, set the path in `01a_load_unit_concept_lookup_csv.sql`, and run it. T | key_type | lookup_key | Rows | Example | |---|---|---|---| -| `field` | measurement field name | 90 | `lbscat_rbc` → 8815 (`x10E6/µL`) | +| `field` | measurement field name | 89 | `lbscat_rbc` → 8815 (`x10E6/µL`) | | `string` | unit text | 25 | `mEq/L` → 9557 | Each key appears once. diff --git a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/unit_concept_lookup.csv b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/unit_concept_lookup.csv index d6c4060..42b553c 100644 --- a/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/unit_concept_lookup.csv +++ b/redCapDataDictionaryWithExtension/value-set-mapping-common/scripts/unit_concept_lookup.csv @@ -53,7 +53,6 @@ field,lbscat_hct,8554,% field,lbscat_mch,8564,pg field,lbscat_mchc,8713,g/dL field,lbscat_mcv,8583,fL -field,lbscat_nrbc,8554,% field,lbscat_plt,8848,x10E3/µL field,lbscat_rbc,8815,x10E6/µL field,lbscat_rdw,8554,%