From 9f700b22e3fea12a53f2d08283b96eb9716b175d Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Wed, 14 Oct 2026 21:19:50 -0400 Subject: [PATCH 1/5] Record how legacy sector and age-group words become portal tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four systems collected who a facilitator serves, each with its own word list, and the portal's consolidation renamed and merged its way to one vocabulary — but nothing recorded how a legacy *input* becomes one of those tags. The Rolodex import has to answer that per value, and a forms backfill and Story Share migration will too, so the tables live in one place instead of three guesses. The deliberate non-mappings matter as much as the mappings: six FileMaker Settings describe where someone works rather than who they serve, and three Life Experiences have no sector yet. Both are recorded so the next reader doesn't invent a near-miss. Co-Authored-By: Claude Opus 5 (1M context) --- ...legacy-sector-and-age-group-translation.md | 186 ++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 docs/adr/0007-legacy-sector-and-age-group-translation.md diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md new file mode 100644 index 0000000000..5a9a918716 --- /dev/null +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -0,0 +1,186 @@ +# ADR-0007 — Translating legacy sector and age-group words into the portal's tags + +- **Status:** Accepted +- **Date:** 2026-10-14 + +## Context + +Four systems asked people who they serve, each with its own word list: + +| Source | Fields | +|---|---| +| FileMaker (`fm_rolodexes`) | `PrimaryServiceArea`, `OtherServiceArea`, `Settings`, `LifeExperiences`, `PrimaryAgeGroup` | +| Collab / training registration / scholarship forms | the same four, with longer prose labels | +| Story Share | Categories, Windows Audience, Age Range | +| The portal, before the 2026 consolidation | its own earlier sector names | + +The portal now carries one canonical vocabulary — `Sector::SECTOR_TYPES`, the `AgeRange` +categories (Children, Teens, Adults, Elders), and the `StoryPopulation` categories (Self, +Colleagues, Community, Families). The renames, merges and additions that produced it are +already done; `db/seeds.rb` reconciles all three lists. + +What was never written down is how a *legacy input* becomes one of those tags. Every +importer and backfill has to answer that per value — `PersonServices::ImportFromRolodex` +first — and each would otherwise invent its own guesswork. This records the answer. + +## Decision + +Translation is by lookup table, not by string similarity. A value that isn't in a table is +**reported, never guessed**. + +### Reading a source value + +- Compare case-insensitively, ignoring surrounding whitespace. +- One field may list several areas: split on commas, semicolons and line breaks (FileMaker + repeating fields are newline-separated), then translate each piece. +- Many source values mapping to one tag is normal. Only `C & T` maps to two tags. +- The portal spelling wins where the spreadsheet differs: **`Systems/Policy Change`** (the + sheet's "Sytems" is a typo). +- `Other` is a form-only catch-all. It is never a translation target. + +### FileMaker `PrimaryServiceArea` → sector + +| FileMaker | Sector | +|---|---| +| Community Oppression/Violence | Community Violence | +| Criminal/Legal | Court/Legal System | +| Disability Services | Disability Services | +| Domestic Violence | Domestic Violence | +| Foster Care | Foster Care/Adoption | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Immigration | Immigration | +| Incarceration | Incarceration | +| Indigenous/Tribal Nation | Indigenous/Tribal Nation | +| LGBTQIA+ | LGBTQIA+ | +| Mental Health | Mental Health | +| Reproductive Services | Reproductive Services | +| Restorative/Transformative Justice | Restorative/Transformative Justice | +| Sexual Assault | Sexual Assault | +| Student Services | Education | +| Substance Abuse | Substance Use/Recovery | + +### FileMaker `Settings` → sector + +| FileMaker | Sector | +|---|---| +| Clinical Setting | Mental Health | +| Educational Setting | Education | +| Faith-Based Setting | Religious/Faith-Based | +| Hospitals | Health/Medical | +| Law Enforcement/Court | Court/Legal System | +| Prisons/Jails | Incarceration | +| Private Practice | Private Practice/Sole Proprietor | +| With Staff | Staff/Organizational Development | + +**Dropped, not mapped:** Community-Based, Events and Conferences, Home Visits, Outreach +Program, Residential Program, Virtually. These say *where or how* a facilitator works, not +who they serve — they aren't sectors and have no portal home. + +### FileMaker `LifeExperiences` → sector + +| FileMaker | Sector | +|---|---| +| Child Abuse/Neglect | Child Abuse/Neglect | +| Climate/Environmental Trauma | Climate/Environmental | +| Community Violence | Community Violence | +| Domestic Violence | Domestic Violence | +| Foster Care | Foster Care/Adoption | +| Grief/Loss | Grief/Loss | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Illness | Health/Medical | +| Immigration | Immigration | +| Incarceration | Incarceration | +| Mental Health Needs | Mental Health | +| Military/Veteran | Military/Veterans | +| Oppression Against LGBTQIA+ | LGBTQIA+ | +| Pandemic-Related Stress | Health/Medical | +| People Who Do Harm/Batterers | Batterers Intervention | +| Racism and Marginalization | Racial/Social Justice | +| Religious Trauma | Religious/Faith-Based | +| Restorative/Transformative Justice | Restorative/Transformative Justice | +| Secondary/Vicarious Trauma | Staff/Organizational Development | +| Sexual Assault | Sexual Assault | +| Student-Related Stress | Education | +| Substance Abuse | Substance Use/Recovery | +| Suicidality | Mental Health | +| Victims of Crime | Court/Legal System | + +**No sector yet:** Elder Abuse, People Who Have Experienced War, Poverty. Left unmapped on +purpose — each needs a sector added before it can be translated, and inventing a near-miss +would bury the gap. An importer reports them. + +### Story Share categories → sector + +| Story Share | Sector | +|---|---| +| Advocacy | Systems/Policy Change | +| Batterers Intervention | Batterers Intervention | +| Child Abuse / Neglect | Child Abuse/Neglect | +| Community Engagement | Community Building | +| Community Violence | Community Violence | +| Court & Legal System | Court/Legal System | +| Disability Services | Disability Services | +| Domestic Violence | Domestic Violence | +| Donor Engagement | Fundraising/Donor Engagement | +| Foster Care | Foster Care/Adoption | +| Grief & Loss | Grief/Loss | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Immigration | Immigration | +| Incarceration | Incarceration | +| LGBTQIA+ | LGBTQIA+ | +| Mental Health | Mental Health | +| Military & Veterans | Military/Veterans | +| Physical & Terminal Illness | Health/Medical | +| Schools & Universities | Education | +| Self-Care & Personal Growth | Self-Care/Personal Growth | +| Sexual Assault | Sexual Assault | +| Social Justice | Racial/Social Justice | +| Staff & Organizational Development | Staff/Organizational Development | +| Substance Abuse Recovery | Substance Use/Recovery | + +**Not a sector:** Facilitator Spotlights. It describes the story, not who it serves. + +### Earlier portal sector names → sector + +Already applied in the portal by rename and merge, recorded so an old export still reads: +Child abuse → Child Abuse/Neglect · Community oppression/violence → Community Violence · +Criminal/legal → Court/Legal System · Disability (and Disabled, Disabled Person) → +Disability Services · Education/schools and Student → Education · Family Violence and +Children of Domestic Violence Survivors → Domestic Violence · Foster care/adoption (and +Foster Youth, Foster children/APPS) → Foster Care/Adoption · Grief and loss → Grief/Loss · +Lgbtqia+ → LGBTQIA+ · Mental illness → Mental Health · Reproductive → Reproductive +Services · Serious illness → Health/Medical · Staff & organizational development → +Staff/Organizational Development · Substance use → Substance Use/Recovery · Veterans & +military → Military/Veterans. + +### Age groups + +| Source value | Portal tag | +|---|---| +| FileMaker `Children`; Story Share `3-5`, `6-12`, `Children` | AgeRange **Children** (0-12) | +| FileMaker `Teens`; Story Share `13-17` | AgeRange **Teens** (13-17) | +| FileMaker `Adults`; Story Share `18+`, `Adult` | AgeRange **Adults** (18+) | +| FileMaker `Elders` | AgeRange **Elders** (65+) | +| FileMaker `C & T` | AgeRange **Children** *and* **Teens** | +| FileMaker `Families`; Story Share `Family windows` | StoryPopulation **Families** | +| Story Share `Mixed-age groups`, `Combined` | nothing — says the group was mixed, not which ages | + +`3-5` and `6-12` were folded into `Children` in the portal, and the `StoryPopulation` +twins carry a trailing underscore so the clean age names stay free (see `db/seeds.rb`). + +## Consequences + +- **Populations are not age ranges.** `AgeGroupTaggable` only tags `AgeRange` categories, so + a person whose FileMaker age group is `Families` gets no age tag — `Families` is a story + population, and people have no population tagging. An importer reports it rather than + forcing it into an age range. +- **One source value can produce two tags** (`C & T`), so a translator returns a list, not a + value. +- **Unmapped values need a report, not a fallback.** `PersonServices::ImportFromRolodex` + counts them in `#unmatched_tags` and the rake task prints them ranked; its name matching + alone resolves only the values that already read as a sector name. +- The tables above are the specification for that lookup. Keep them in one constant so the + Rolodex import, a later forms backfill, and any Story Share migration share it. From c071ddf4bb8919acdf6f9d0a99a28d630a04448d Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Wed, 14 Oct 2026 21:40:47 -0400 Subject: [PATCH 2/5] Record the legacy tag words as data, not a constant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0007's tables have to live somewhere a lookup can read. A constant would mean a deploy every time the archive's free text turns up another spelling, and would vanish the day the imports are done — but what a tag used to be called is worth keeping after that, so TagAlias rows carry it instead. A row names one legacy term and the tag it resolves to, plus the system the term came from and who recorded it. A term that names two tags is two rows, which is how "C & T" reaches both Children and Teens. A term can also name nothing on purpose. Six FileMaker "settings" say where someone works rather than who they serve, and "Other" says nothing a tag could carry; recording those as ignored keeps ~2,800 mentions of them out of the queue of terms that genuinely need a decision. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 10 +- app/models/tag_alias.rb | 39 ++++ app/services/tag_alias_seeder.rb | 197 ++++++++++++++++++ app/services/tag_translation.rb | 59 ++++++ config/features.yml | 14 ++ .../20261015013218_create_tag_aliases.rb | 24 +++ db/schema.rb | 16 +- db/seeds.rb | 6 + ...legacy-sector-and-age-group-translation.md | 62 +++++- lib/tasks/seed_tag_aliases.rake | 18 ++ spec/factories/tag_aliases.rb | 7 + spec/models/tag_alias_spec.rb | 38 ++++ spec/services/tag_alias_seeder_spec.rb | 79 +++++++ spec/services/tag_translation_spec.rb | 64 ++++++ 14 files changed, 624 insertions(+), 9 deletions(-) create mode 100644 app/models/tag_alias.rb create mode 100644 app/services/tag_alias_seeder.rb create mode 100644 app/services/tag_translation.rb create mode 100644 db/migrate/20261015013218_create_tag_aliases.rb create mode 100644 lib/tasks/seed_tag_aliases.rake create mode 100644 spec/factories/tag_aliases.rb create mode 100644 spec/models/tag_alias_spec.rb create mode 100644 spec/services/tag_alias_seeder_spec.rb create mode 100644 spec/services/tag_translation_spec.rb diff --git a/AGENTS.md b/AGENTS.md index 25548a9fdf..68eee04a6d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,8 +48,8 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| -| `app/models/` | ActiveRecord models | ~90 files | -| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~73 files | +| `app/models/` | ActiveRecord models | ~144 files | +| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~118 files | | `app/jobs/` | SolidQueue background jobs | 6 files | | `app/models/concerns/` | Shared model modules | 23 concerns | @@ -271,6 +271,8 @@ action, or `authorize! :workshop, to: :summary?`). - `EventRegistrationServices::ProcessConfirmation` — Registration confirmation flow - `RegistrantCeForm` — The CE-specific view over the inline form the **CE callout** carries (built on the generic callout→form mechanism: `RegistrationTicketCallout#forms` + `CalloutFormSubmission`; the CE callout carries a single form, read as `forms.first`). Adds the two things CE needs: the post-training form only surfaces once `EventRegistration#ce_signouts_complete?` (event ended + no open attendance entry), and completing it — read-time `complete?`, every required field answered — is a prerequisite for the CE certificate (`EventRegistration#ce_form_requirement_met?` gates `ContinuingEducationRegistration#certificate_available?`). Read by the CE callout page (which links to the generic `registration/:slug/forms/:callout_id` form page, gated so the CE callout's form opens only after sign-outs via `CalloutsController#ce_form_locked?`) and by `BuiltinCalloutCards#ce_form_pending?` for the CE card's "Complete your form" nudge - `EventRegistrationServices::CalloutFormSubmission` — Records a registrant's answers to one of the forms a registration-ticket callout delivers inline (`call(registration:, callout:, form:, form_params:, clarity_params:)`, returns the service instance with `#submission`/`#profile_changes`). Find-or-creates one `FormSubmission` per `(registrant, form, event)` carrying the form's own role — a re-submit edits the answers in place via `FormSubmission#persist_answer`. That a callout collected it is recorded separately in `FormSubmission#metadata` (`collected_via: "callout"`, `collected_via_callout_id`) via `#record_callout_collection!` — the flag (`#collected_via_callout?`), not the role, tells a callout-collected submission apart. There is no survey-role flag — the extra side effects self-gate on the form's own fields, so they no-op on a form that doesn't ask those questions: a choice field with linked `FormFieldResource`s (`FormField#per_resource?`) fans out to one answer per resource (nil `form_field`, `FormField#per_resource_question` snapshotted in `question_name_when_answered` — subtitle-led, so two fan-out questions sharing a prompt can't collide on one answer row); fields carrying the `anonymous_contributions` / `display_name_preference` identifiers write through to the `Person` via `PersonServices::SyncSharingPreferences` (the same write-through the registration form runs), exposing `#profile_changes` so the controller Ahoy-tracks real edits; and submitting any form on the **`scholarship`** callout stamps a recipient's `post_survey_completed_at`. Drives `Events::CalloutsController#callout`/`#submit_callout` (the person-first `registration/:slug/forms/:callout_id` page and its `.../:form_id` submit) +- `TagTranslation` — Turns a legacy word for who someone serves into sector / age-group ids: a tag's own name first, then any `TagAlias` recorded for it, so one term can name several tags (`C & T` is Children and Teens). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess +- `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't seeded yet is reported, not created. Read through `TagTranslation`. Shared with `data:seed_tag_aliases` - `SurveyFormSeeder` — Creates the four standalone survey template forms from the form-builder presets (the Day 1 / Day 2 evaluations, the overall post-event survey, and the scholarship recipients survey), idempotent on form name and returning the names created. `#link_fanout_resources` then links each fan-out question to its training-topic `Resource`s by title (skipping any that aren't present). Shared by `db/seeds.rb` and the `data:seed_survey_forms` task so prod can create them without the full seed - `CloseProgramFormSeeder` — Creates the standalone "Close Program" agreement form (role `close_program`, sections `person_identifier` + `close_program`) from the form-builder presets, idempotent on form name, returning the name created (or nil). Built as an unpublished template for staff to slug + publish. Shared by `db/seeds.rb` and the `data:seed_close_program_form` task - `EventRegistrationServices::PublicRegistration` — Public registration handling. An affirmative `communication_consent` answer captures a News (mailing-list) `TopicSubscription` via `NewsSubscriptionCapture`, sourced to the event. A resubmission matched to an active registration sends the registrant and admins DUPLICATE emails (`event_registration_duplicate` / `event_registration_duplicate_fyi`) @@ -561,7 +563,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (13 files): +Located in `lib/tasks/` (16 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting @@ -573,6 +575,8 @@ Located in `lib/tasks/` (13 files): - `migrate_workshop_logs.rake` — Workshop log migration - `backfill_user_stamps.rake` — Backfill `created_by_id`/`updated_by_id` on legacy rows from the Ahoy trail (`data:backfill_user_stamps`) - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) +- `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) +- `seed_tag_aliases.rake` — Record the legacy sector and age-group terms from ADR-0007 as tag aliases (`data:seed_tag_aliases`, via `TagAliasSeeder`, idempotent; also part of `db:seed`) - `seed_survey_forms.rake` — Create the Day 1 / Day 2 / recipients survey template forms without the full db:seed (`data:seed_survey_forms`, via `SurveyFormSeeder`, idempotent on form name; run before configuring the survey callouts) - `seed_close_program_form.rake` — Create the Close Program agreement form without the full db:seed (`data:seed_close_program_form`, via `CloseProgramFormSeeder`, idempotent on form name; give it a slug and publish it in the form builder to use it) diff --git a/app/models/tag_alias.rb b/app/models/tag_alias.rb new file mode 100644 index 0000000000..1ad1846c8a --- /dev/null +++ b/app/models/tag_alias.rb @@ -0,0 +1,39 @@ +# A word some other system used for a tag the portal now carries its own name for. The +# FileMaker archive, the older registration and scholarship forms, Story Share and the +# portal's own earlier sector list each had their own wording; ADR-0007 records what each +# term resolves to, and these rows are that mapping as data — so an admin can add a term +# as it turns up instead of waiting for a deploy, and the record of what a tag used to be +# called outlives the import that needed it. +# +# A term can name two tags ("C & T" is both Children and Teens), so it is read as a list. +# It can also be recorded as naming nothing on purpose — a FileMaker "setting" like +# "Virtually" says where someone works, not who they serve — which is what keeps a decided +# non-translation out of the queue of terms still needing a decision. +class TagAlias < ApplicationRecord + SOURCES = %w[filemaker forms story_share portal].freeze + + belongs_to :aliasable, polymorphic: true, optional: true + belongs_to :created_by, class_name: "User", optional: true + belongs_to :updated_by, class_name: "User", optional: true + + normalizes :name, with: ->(value) { value.to_s.squish } + + validates :name, presence: true, length: { maximum: 255 }, + uniqueness: { scope: %i[aliasable_type aliasable_id], case_sensitive: false } + validates :source, inclusion: { in: SOURCES } + validate :names_a_tag_or_is_ignored + + scope :ignored, -> { where(ignored: true) } + scope :resolving, -> { where(ignored: false) } + + scope :for_sectors, -> { where(aliasable_type: "Sector") } + scope :for_categories, -> { where(aliasable_type: "Category") } + + private + + def names_a_tag_or_is_ignored + return if aliasable.present? ^ ignored? + + errors.add(:base, "A term either names a tag or is ignored, not both and not neither") + end +end diff --git a/app/services/tag_alias_seeder.rb b/app/services/tag_alias_seeder.rb new file mode 100644 index 0000000000..eda615947c --- /dev/null +++ b/app/services/tag_alias_seeder.rb @@ -0,0 +1,197 @@ +# Records what each legacy word for a sector or age group resolves to, as TagAlias rows. +# The tables are ADR-0007; this is them as data, so an import can look a term up and a +# reader can still see what a tag used to be called elsewhere. Idempotent: a term already +# recorded for a tag is left alone, including whatever an admin has added by hand. +# +# A term the portal already answers to by name needs no row — the lookup matches a tag's +# own name first, case-insensitively, and drops a trailing parenthetical. Only wording +# that differs is recorded here. +class TagAliasSeeder + # The archive's own three fields. Settings describes where a facilitator works, so only + # the ones that also say who they serve resolve; the rest are in DROPPED_SETTINGS. + FILEMAKER_SECTORS = { + "Community Oppression/Violence" => "Community Violence", + "Criminal/Legal" => "Court/Legal System", + "Foster Care" => "Foster Care/Adoption", + "Student Services" => "Education", + "Substance Abuse" => "Substance Use/Recovery", + "Clinical Setting" => "Mental Health", + "Educational Setting" => "Education", + "Faith-Based Setting" => "Religious/Faith-Based", + "Hospitals" => "Health/Medical", + "Law Enforcement/Court" => "Court/Legal System", + "Prisons/Jails" => "Incarceration", + "Private Practice" => "Private Practice/Sole Proprietor", + "With Staff" => "Staff/Organizational Development", + "Climate/Environmental Trauma" => "Climate/Environmental", + "Illness" => "Health/Medical", + "Pandemic-Related Stress" => "Health/Medical", + "Mental Health Needs" => "Mental Health", + "Suicidality" => "Mental Health", + "Military/Veteran" => "Military/Veterans", + "Oppression Against LGBTQIA+" => "LGBTQIA+", + "People Who Do Harm/Batterers" => "Batterers Intervention", + "Racism and Marginalization" => "Racial/Social Justice", + "Religious Trauma" => "Religious/Faith-Based", + "Secondary/Vicarious Trauma" => "Staff/Organizational Development", + "Student-Related Stress" => "Education", + "Victims of Crime" => "Court/Legal System", + "LGBTQIA +" => "LGBTQIA+" + }.freeze + + # The forms asked the same questions in prose. Their labels carry a "services" suffix or + # different wording, so each needs its own term; the parentheticals resolve on their own. + FORM_SECTORS = { + "Community oppression/violence services" => "Community Violence", + "Criminal/legal services" => "Court/Legal System", + "Disability services" => "Disability Services", + "Domestic violence services" => "Domestic Violence", + "Foster care/adoption services" => "Foster Care/Adoption", + "Homeless services" => "Homelessness", + "Human trafficking services" => "Human Trafficking", + "Immigration services" => "Immigration", + "Incarceration services" => "Incarceration", + "Indigenous/tribal nation services" => "Indigenous/Tribal Nation", + "LGBTQIA+ services" => "LGBTQIA+", + "Mental health services" => "Mental Health", + "Reproductive services" => "Reproductive Services", + "Restorative/transformative justice services" => "Restorative/Transformative Justice", + "Sexual assault services" => "Sexual Assault", + "Student services" => "Education", + "Substance use services" => "Substance Use/Recovery", + "Substance use" => "Substance Use/Recovery", + "Foster care/adoption" => "Foster Care/Adoption", + "Law enforcement/court/legal" => "Court/Legal System" + }.freeze + + STORY_SHARE_SECTORS = { + "Advocacy" => "Systems/Policy Change", + "Child Abuse / Neglect" => "Child Abuse/Neglect", + "Community Engagement" => "Community Building", + "Court & Legal System" => "Court/Legal System", + "Donor Engagement" => "Fundraising/Donor Engagement", + "Foster Care" => "Foster Care/Adoption", + "Grief & Loss" => "Grief/Loss", + "Military & Veterans" => "Military/Veterans", + "Physical & Terminal Illness" => "Health/Medical", + "Schools & Universities" => "Education", + "Self-Care & Personal Growth" => "Self-Care/Personal Growth", + "Social Justice" => "Racial/Social Justice", + "Staff & Organizational Development" => "Staff/Organizational Development", + "Substance Abuse Recovery" => "Substance Use/Recovery" + }.freeze + + # Names the portal itself used before the 2026 consolidation renamed and merged them. + PORTAL_SECTORS = { + "Child abuse" => "Child Abuse/Neglect", + "Children of Domestic Violence Survivors" => "Domestic Violence", + "Community oppression/violence" => "Community Violence", + "Criminal/legal" => "Court/Legal System", + "Disabled" => "Disability Services", + "Disabled Person" => "Disability Services", + "Disability" => "Disability Services", + "Education/schools" => "Education", + "Family Violence" => "Domestic Violence", + "Foster children/APPS" => "Foster Care/Adoption", + "Foster Youth" => "Foster Care/Adoption", + "Grief and loss" => "Grief/Loss", + "Mental illness" => "Mental Health", + "Reproductive" => "Reproductive Services", + "Serious illness" => "Health/Medical", + "Staff & organizational development" => "Staff/Organizational Development", + "Student" => "Education", + "Substance use" => "Substance Use/Recovery", + "Veterans & military" => "Military/Veterans" + }.freeze + + # A term can name two age groups, so these are lists. + FILEMAKER_AGE_GROUPS = { + "C & T" => %w[Children Teens], + "Children & Teens" => %w[Children Teens], + "Adults & Children" => %w[Adults Children], + "Adults 18+" => %w[Adults], + "18+" => %w[Adults], + "13-17" => %w[Teens], + "6-12" => %w[Children], + "3-5" => %w[Children], + "Young Children" => %w[Children] + }.freeze + + STORY_SHARE_AGE_GROUPS = { "Adult" => %w[Adults] }.freeze + + # Settings that say where or how someone works rather than who they serve. Recorded as + # naming nothing on purpose, so they don't sit in the queue of undecided terms. + IGNORED_TERMS = [ + "Community-Based", "Events and Conferences", "Events & Conferences", "Home Visits", + "Outreach Program", "Residential Program", "Virtually", "Other", "Did not specify" + ].freeze + + def self.call(...) + new(...).call + end + + def initialize(quiet: false) + @quiet = quiet + @created = 0 + @missing = [] + end + + def call + record_sectors(FILEMAKER_SECTORS, "filemaker") + record_sectors(FORM_SECTORS, "forms") + record_sectors(STORY_SHARE_SECTORS, "story_share") + record_sectors(PORTAL_SECTORS, "portal") + record_age_groups(FILEMAKER_AGE_GROUPS, "filemaker") + record_age_groups(STORY_SHARE_AGE_GROUPS, "story_share") + record_ignored(IGNORED_TERMS, "filemaker") + report + self + end + + private + + def record_sectors(table, source) + table.each { |term, name| record(term, "Sector", sector_ids[name.downcase], source, name) } + end + + def record_age_groups(table, source) + table.each do |term, names| + names.each { |name| record(term, "Category", age_group_ids[name.downcase], source, name) } + end + end + + def record_ignored(terms, source) + terms.each do |term| + alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: nil, aliasable_id: nil) + next unless alias_row.new_record? + + alias_row.update!(source: source, ignored: true) + @created += 1 + end + end + + def record(term, type, id, source, name) + return @missing << name unless id + + alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: type, aliasable_id: id) + return unless alias_row.new_record? + + alias_row.update!(source: source) + @created += 1 + end + + def sector_ids + @sector_ids ||= Sector.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + + def age_group_ids + @age_group_ids ||= Category.age_ranges.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + + def report + return if @quiet + + puts " #{@created} tag aliases added (#{TagAlias.count} recorded)" + puts " no tag named: #{@missing.uniq.sort.join(', ')} — seed the sectors and age ranges first" if @missing.any? + end +end diff --git a/app/services/tag_translation.rb b/app/services/tag_translation.rb new file mode 100644 index 0000000000..68d0b42337 --- /dev/null +++ b/app/services/tag_translation.rb @@ -0,0 +1,59 @@ +# Turns a legacy word for who someone serves into the portal's own tags: the archive's +# service areas and settings, the older forms' prose labels, Story Share's categories, and +# the portal's earlier sector names. A term resolves by a tag's own name or by any +# TagAlias recorded for it (ADR-0007), and one term can name several tags. +# +# Built once and reused — it reads the whole sector and age-group vocabulary up front, so +# a 39,000-row import doesn't query per value. A term nothing answers to is counted in +# #unmatched rather than guessed at — unless it's recorded as ignored, which is how a +# decided non-translation stays out of the queue of terms still needing a decision. +class TagTranslation + # FileMaker repeating fields separate values with line breaks, and a hand-typed list + # uses semicolons. Commas are not separators: the wording itself contains them, inside + # parentheticals like "Community violence (gang violence, mass shootings, etc.)". + SEPARATORS = /[;\r\n]+/ + # The clarifying tail a form added to a term, dropped when the term alone resolves. + PARENTHETICAL = /\s*\([^)]*\)?\s*\z/ + + attr_reader :unmatched + + def initialize + @sectors = vocabulary(Sector.all) + @age_groups = vocabulary(Category.age_ranges) + @ignored = TagAlias.ignored.pluck(:name).map { |name| key(name) }.to_set + @unmatched = Hash.new(0) + end + + def sector_ids_for(value) + ids_for(value, @sectors) + end + + def age_group_ids_for(value) + ids_for(value, @age_groups) + end + + private + + def vocabulary(scope) + aliases = TagAlias.resolving.where(aliasable_type: scope.model_name.to_s, aliasable_id: scope.select(:id)) + (scope.pluck(:name, :id) + aliases.pluck(:name, :aliasable_id)).each_with_object({}) do |(name, id), index| + (index[key(name)] ||= []) << id + end + end + + def ids_for(value, index) + terms(value).flat_map do |term| + ids = index[term] || index[key(term.sub(PARENTHETICAL, ""))] || [] + @unmatched[term] += 1 if ids.empty? && !@ignored.include?(term) + ids + end.uniq + end + + def terms(value) + value.to_s.split(SEPARATORS).filter_map { |part| key(part).presence }.uniq + end + + def key(value) + value.to_s.squish.downcase + end +end diff --git a/config/features.yml b/config/features.yml index 36e199f92b..b6b754b4b4 100644 --- a/config/features.yml +++ b/config/features.yml @@ -4563,3 +4563,17 @@ published and unpublished categories, instead of only published ones. pro_tips: - "Unpublished categories have a gray row, so you can tell them apart in the All view." + +- name: "What a tag used to be called" + area: people + display_status: admin_facing + released_on: 2026-10-15 + pr_number: 2691 + summary: >- + The portal now records what FileMaker, the older registration and scholarship + forms, Story Share and the portal's earlier sector list each called a sector or + age group, so an old word still resolves to the right tag today. + pro_tips: + - "This is how an import knows that \"Student Services\" means Education and that \"C & T\" means both Children and Teens." + - "A word can be recorded as meaning nothing on purpose — FileMaker's \"Virtually\" described where someone worked, not who they served." + - "The mapping is written up in ADR-0007 for anyone who needs to see the whole picture." diff --git a/db/migrate/20261015013218_create_tag_aliases.rb b/db/migrate/20261015013218_create_tag_aliases.rb new file mode 100644 index 0000000000..10ef3677af --- /dev/null +++ b/db/migrate/20261015013218_create_tag_aliases.rb @@ -0,0 +1,24 @@ +class CreateTagAliases < ActiveRecord::Migration[8.1] + def up + create_table :tag_aliases do |t| + t.string :name, null: false + # A term can be recorded as deliberately naming nothing, so aliasable is optional. + t.references :aliasable, polymorphic: true, null: true, index: false + t.boolean :ignored, default: false, null: false + t.string :source, null: false + t.integer :created_by_id + t.integer :updated_by_id + t.timestamps + end + + # The column collation is case-insensitive, so one spelling can't be recorded twice + # for the same tag. A term naming two tags is two rows. + add_index :tag_aliases, %i[name aliasable_type aliasable_id], unique: true, + name: "index_tag_aliases_on_name_and_aliasable" + add_index :tag_aliases, %i[aliasable_type aliasable_id], name: "index_tag_aliases_on_aliasable" + end + + def down + drop_table :tag_aliases, if_exists: true + end +end diff --git a/db/schema.rb b/db/schema.rb index fd957ce59c..84d0c39464 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.1].define(version: 2026_10_14_123237) do +ActiveRecord::Schema[8.1].define(version: 2026_10_15_013218) do create_table "action_text_mentions", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.bigint "action_text_rich_text_id", null: false t.datetime "created_at", null: false @@ -2145,6 +2145,20 @@ t.index ["workshop_id"], name: "index_story_ideas_workshops_on_workshop_id" end + create_table "tag_aliases", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| + t.bigint "aliasable_id" + t.string "aliasable_type" + t.datetime "created_at", null: false + t.integer "created_by_id" + t.boolean "ignored", default: false, null: false + t.string "name", null: false + t.string "source", null: false + t.datetime "updated_at", null: false + t.integer "updated_by_id" + t.index ["aliasable_type", "aliasable_id"], name: "index_tag_aliases_on_aliasable" + t.index ["name", "aliasable_type", "aliasable_id"], name: "index_tag_aliases_on_name_and_aliasable", unique: true + end + create_table "topic_subscription_types", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.datetime "archived_at" t.datetime "created_at", null: false diff --git a/db/seeds.rb b/db/seeds.rb index 409a924cce..cc4dda25ac 100644 --- a/db/seeds.rb +++ b/db/seeds.rb @@ -522,3 +522,9 @@ def find_or_create_by_name!(klass, name, **attrs, &block) # data:seed_close_program_form task. Idempotent on form name. puts "Creating the Close Program form…" CloseProgramFormSeeder.call + +# What the archive, the older forms, Story Share and the portal's earlier sector list +# each called a tag the portal now names its own way (ADR-0007). Shared with the +# data:seed_tag_aliases task. Idempotent, and never touches a term an admin added. +puts "Recording legacy tag aliases…" +TagAliasSeeder.call diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md index 5a9a918716..80588b4444 100644 --- a/docs/adr/0007-legacy-sector-and-age-group-translation.md +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -30,9 +30,14 @@ Translation is by lookup table, not by string similarity. A value that isn't in ### Reading a source value -- Compare case-insensitively, ignoring surrounding whitespace. -- One field may list several areas: split on commas, semicolons and line breaks (FileMaker - repeating fields are newline-separated), then translate each piece. +- Compare case-insensitively, ignoring surrounding whitespace. Most of the archive's + long tail is case variants ("Domestic violence" alongside "Domestic Violence"), so this + alone collapses it. +- One field may list several areas: split on **line breaks and semicolons**, then translate + each piece. **Not on commas** — the wording itself contains them, inside parentheticals + like "Community violence (gang violence, mass shootings, police violence, etc.)". +- Drop a trailing parenthetical when the term alone resolves, so the forms' clarifying + tails ("Grief/loss (including COVID-related)") need no separate entry. - Many source values mapping to one tag is normal. Only `C & T` maps to two tags. - The portal spelling wins where the spreadsheet differs: **`Systems/Policy Change`** (the sheet's "Sytems" is a typo). @@ -171,6 +176,52 @@ military → Military/Veterans. `3-5` and `6-12` were folded into `Children` in the portal, and the `StoryPopulation` twins carry a trailing underscore so the clean age names stay free (see `db/seeds.rb`). +## How it's stored + +**`TagAlias` rows, not a constant.** Each row names one legacy term, the tag it resolves +to (polymorphic — a `Sector` or an `AgeRange` `Category`), and the system the term came +from. `TagTranslation` reads a tag's own name first, then the aliases. + +- **A term can name two tags**, so a row per pair and a list on the way out (`C & T`). +- **Unique on (term, tag)**, with the column's case-insensitive collation, so one spelling + can't be recorded twice for the same tag. +- **An admin can add a term** as it turns up, without a deploy — which matters, because the + archive's free-text tail keeps producing new ones. +- **`created_by` and `created_at`** make it the record of who decided a term meant a tag + and when. The renames and merges already ran, so the old names are gone from the data; + these rows are what's left of them. +- **A term can be recorded as naming nothing on purpose** (`ignored`). That's how the + dropped Settings and `Other` stay out of the queue of terms still needing a decision — + without it, ~2,800 mentions of "Virtually", "Outreach Program" and the like would bury + the handful that genuinely need one. +- `TagAliasSeeder` (and `data:seed_tag_aliases`) records the tables above, idempotently, + and never touches a term an admin added. + +The tables stay here as the readable source; the rows are the machine-readable copy. + +## What the archive actually holds + +Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct values / +21,857 mentions · `Settings` 161 / 6,082 · `PrimaryAgeGroup` 11 / 2,356 · +`PrimaryServiceArea` 29 / 2,354 · `OtherServiceArea` 73 / 75. + +- **The canonical words dominate.** The documented vocabulary covers nearly every mention; + 134 of the 161 `Settings` values occur exactly once. +- **`OtherServiceArea` is free text**, not a picker: 72 of its 73 values occur once, across + 75 rows in all. Not worth translating — it goes to the unmatched report. +- **`Other`** appears as a stored value (90 `PrimaryServiceArea`, 60 `LifeExperiences`). + It is recorded as ignored rather than translated: `Sector::OTHER_SECTOR_NAME` is offered + for additional sectors but never as a primary, so an import can't honour it as one, and + "they picked Other" says nothing a tag could carry. +- **Still needing a decision** — what the seeded terms leave unresolved: `Poverty` (634), + `Elder Abuse` (178), `People Who Have Experienced War` (97), `Children's Services` (8), + and a long tail of one-off free text (145 `Settings` values, 63 `OtherServiceArea` + values). Each is a judgment call for AWBW; the unmatched report is the queue. +- **Seen in the data, no mapping decided:** Children's Services, Intellectual Disabilities, + Maternal wellness and postpartum support, Political Asylum, Survivors with Disabilities, + Arts Education & Community Art Making, and the rest of the one-off free text. Deliberately + unseeded — each is a judgment call for AWBW, and the unmatched report is the queue. + ## Consequences - **Populations are not age ranges.** `AgeGroupTaggable` only tags `AgeRange` categories, so @@ -182,5 +233,6 @@ twins carry a trailing underscore so the clean age names stay free (see `db/seed - **Unmapped values need a report, not a fallback.** `PersonServices::ImportFromRolodex` counts them in `#unmatched_tags` and the rake task prints them ranked; its name matching alone resolves only the values that already read as a sector name. -- The tables above are the specification for that lookup. Keep them in one constant so the - Rolodex import, a later forms backfill, and any Story Share migration share it. +- **The translation is data, so it can retire.** Once the imports and backfills are done, + the rows stay as the history of what a tag used to be called — and can be dropped without + touching code if that history stops being useful. diff --git a/lib/tasks/seed_tag_aliases.rake b/lib/tasks/seed_tag_aliases.rake new file mode 100644 index 0000000000..397bc0d44e --- /dev/null +++ b/lib/tasks/seed_tag_aliases.rake @@ -0,0 +1,18 @@ +# frozen_string_literal: true + +# Record what each legacy word for a sector or age group resolves to, as TagAlias rows, +# without running the full db:seed. The tables are ADR-0007; the rows are what an import +# looks a term up in, and what a reader checks to see what a tag used to be called in +# FileMaker, the older forms, Story Share or the portal's earlier sector list. +# +# Idempotent and safe to re-run: a term already recorded for a tag is left alone, so an +# alias an admin added by hand survives. Run after the sectors and age ranges are seeded — +# a term whose tag doesn't exist yet is reported, not created. +# +# bin/rails data:seed_tag_aliases +namespace :data do + desc "Record the legacy sector and age-group terms (ADR-0007) as tag aliases (idempotent)." + task seed_tag_aliases: :environment do + TagAliasSeeder.call + end +end diff --git a/spec/factories/tag_aliases.rb b/spec/factories/tag_aliases.rb new file mode 100644 index 0000000000..8f875e9cc5 --- /dev/null +++ b/spec/factories/tag_aliases.rb @@ -0,0 +1,7 @@ +FactoryBot.define do + factory :tag_alias do + sequence(:name) { |n| "Legacy Term #{n}" } + source { "filemaker" } + association :aliasable, factory: :sector + end +end diff --git a/spec/models/tag_alias_spec.rb b/spec/models/tag_alias_spec.rb new file mode 100644 index 0000000000..c98b2f355c --- /dev/null +++ b/spec/models/tag_alias_spec.rb @@ -0,0 +1,38 @@ +require "rails_helper" + +RSpec.describe TagAlias, type: :model do + let(:sector) { create(:sector, name: "Domestic Violence") } + + it "is valid when it names a tag" do + expect(build(:tag_alias, aliasable: sector)).to be_valid + end + + it "is valid when it names nothing on purpose" do + expect(build(:tag_alias, aliasable: nil, ignored: true)).to be_valid + end + + it "rejects a term that neither names a tag nor is ignored" do + expect(build(:tag_alias, aliasable: nil, ignored: false)).not_to be_valid + end + + it "rejects a term that both names a tag and is ignored" do + expect(build(:tag_alias, aliasable: sector, ignored: true)).not_to be_valid + end + + it "records a term once per tag, whatever its case or spacing" do + create(:tag_alias, name: "Student Services", aliasable: sector) + + expect(build(:tag_alias, name: " student services ", aliasable: sector)).not_to be_valid + end + + it "lets two tags share a term, for a term that names both" do + other = create(:sector, name: "Education") + create(:tag_alias, name: "C & T", aliasable: sector) + + expect(build(:tag_alias, name: "C & T", aliasable: other)).to be_valid + end + + it "only accepts a system it could have come from" do + expect(build(:tag_alias, aliasable: sector, source: "spreadsheet")).not_to be_valid + end +end diff --git a/spec/services/tag_alias_seeder_spec.rb b/spec/services/tag_alias_seeder_spec.rb new file mode 100644 index 0000000000..62b0d83a14 --- /dev/null +++ b/spec/services/tag_alias_seeder_spec.rb @@ -0,0 +1,79 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe TagAliasSeeder do + let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } + + before do + Sector::SECTOR_TYPES.each { |name| create(:sector, name: name) } + %w[Children Teens Adults Elders].each { |name| create(:category, category_type: age_range_type, name: name) } + end + + it "records every term the tables name, pointing at the tag it resolves to" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services").aliasable).to eq(Sector.find_by(name: "Education")) + expect(TagAlias.find_by(name: "Clinical Setting").aliasable).to eq(Sector.find_by(name: "Mental Health")) + expect(TagAlias.find_by(name: "Advocacy").aliasable).to eq(Sector.find_by(name: "Systems/Policy Change")) + expect(TagAlias.find_by(name: "Child abuse").aliasable).to eq(Sector.find_by(name: "Child Abuse/Neglect")) + expect(TagAlias.find_by(name: "Homeless services").aliasable).to eq(Sector.find_by(name: "Homelessness")) + end + + it "records a term that names two age groups once for each" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "C & T").map { |row| row.aliasable.name }).to contain_exactly("Children", "Teens") + end + + it "labels each term with the system it came from" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services").source).to eq("filemaker") + expect(TagAlias.find_by(name: "Homeless services").source).to eq("forms") + expect(TagAlias.find_by(name: "Grief & Loss").source).to eq("story_share") + expect(TagAlias.find_by(name: "Grief and loss").source).to eq("portal") + end + + it "adds nothing on a second run" do + described_class.call(quiet: true) + + expect { described_class.call(quiet: true) }.not_to change(TagAlias, :count) + end + + it "leaves a term an admin recorded by hand alone" do + sexual_assault = Sector.find_by(name: "Sexual Assault") + admin_alias = create(:tag_alias, name: "Student Services", aliasable: sexual_assault, source: "portal") + + described_class.call(quiet: true) + + expect(admin_alias.reload.aliasable).to eq(sexual_assault) + expect(TagAlias.where(name: "Student Services").count).to eq(2) + end + + it "reports a term whose tag isn't seeded yet instead of inventing it" do + Sector.find_by(name: "Education").destroy + + expect { described_class.call(quiet: true) }.not_to raise_error + expect(TagAlias.find_by(name: "Student Services")).to be_nil + end + + it "records a setting that names nothing as decided, not as a gap" do + described_class.call(quiet: true) + + virtually = TagAlias.find_by(name: "Virtually") + expect(virtually).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.sector_ids_for("Virtually")).to be_empty + expect(translation.unmatched).to be_empty + end + + it "resolves the archive's own wording once seeded" do + described_class.call(quiet: true) + + translation = TagTranslation.new + expect(translation.sector_ids_for("Clinical Setting")).to eq([ Sector.find_by(name: "Mental Health").id ]) + expect(translation.age_group_ids_for("C & T").size).to eq(2) + end +end diff --git a/spec/services/tag_translation_spec.rb b/spec/services/tag_translation_spec.rb new file mode 100644 index 0000000000..6b498a282e --- /dev/null +++ b/spec/services/tag_translation_spec.rb @@ -0,0 +1,64 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe TagTranslation do + let!(:domestic_violence) { create(:sector, name: "Domestic Violence") } + let!(:education) { create(:sector, name: "Education") } + let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } + let!(:children) { create(:category, category_type: age_range_type, name: "Children") } + let!(:teens) { create(:category, category_type: age_range_type, name: "Teens") } + + it "resolves a term by the tag's own name, whatever its case" do + expect(described_class.new.sector_ids_for("domestic violence")).to eq([ domestic_violence.id ]) + end + + it "resolves a term by an alias recorded for the tag" do + create(:tag_alias, name: "Student Services", aliasable: education) + + expect(described_class.new.sector_ids_for("student services")).to eq([ education.id ]) + end + + it "reads one term as several tags when both are recorded" do + create(:tag_alias, name: "C & T", aliasable: children, source: "filemaker") + create(:tag_alias, name: "C & T", aliasable: teens, source: "filemaker") + + expect(described_class.new.age_group_ids_for("C & T")).to contain_exactly(children.id, teens.id) + end + + it "splits a repeating field on line breaks and semicolons, not on commas" do + create(:tag_alias, name: "Community violence (gang violence, mass shootings, etc.)", aliasable: domestic_violence) + + translation = described_class.new + expect(translation.sector_ids_for("Domestic Violence\nEducation")).to contain_exactly(domestic_violence.id, education.id) + expect(translation.sector_ids_for("Domestic Violence; Education")).to contain_exactly(domestic_violence.id, education.id) + expect(translation.sector_ids_for("Community violence (gang violence, mass shootings, etc.)")).to eq([ domestic_violence.id ]) + end + + it "drops a clarifying parenthetical when the term alone resolves" do + expect(described_class.new.sector_ids_for("Domestic violence (including family violence)")).to eq([ domestic_violence.id ]) + end + + it "counts a term nothing answers to instead of guessing at it" do + translation = described_class.new + + expect(translation.sector_ids_for("Poverty\nDomestic Violence")).to eq([ domestic_violence.id ]) + expect(translation.unmatched).to eq({ "poverty" => 1 }) + end + + it "keeps a term recorded as ignored out of the unmatched list" do + create(:tag_alias, name: "Virtually", aliasable: nil, ignored: true) + translation = described_class.new + + expect(translation.sector_ids_for("Virtually\nDomestic Violence")).to eq([ domestic_violence.id ]) + expect(translation.unmatched).to be_empty + end + + it "has nothing to say about a blank value" do + translation = described_class.new + + expect(translation.sector_ids_for(nil)).to be_empty + expect(translation.sector_ids_for(" ")).to be_empty + expect(translation.unmatched).to be_empty + end +end From ecdaad50ceaeef5168d4081cf5b04f884850d02c Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Wed, 14 Oct 2026 22:31:12 -0400 Subject: [PATCH 3/5] Resolve only to a tag that's still in use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A retired sector or age range is unpublished rather than deleted, so its old taggings survive — and resolving to one would hand it to hundreds of records that never had it. The live sectors table holds more than the catalogue: 634 archive rows name "Poverty", a retired row, which Sector.all would have quietly revived. A term that only matches a retired tag now reads as unresolved, which puts it in front of someone instead. The seeder reads the same published vocabulary, so a term aimed at a retired tag is reported rather than recorded as an alias that can never resolve. Serving families is serving every age, so FileMaker's "Families" age group tags all four age ranges rather than going unread — a person has no population tag to carry it on its own. Co-Authored-By: Claude Opus 5 (1M context) --- app/services/tag_alias_seeder.rb | 12 ++++++--- app/services/tag_translation.rb | 9 +++++-- config/features.yml | 2 ++ ...legacy-sector-and-age-group-translation.md | 25 +++++++++++++------ spec/models/tag_alias_spec.rb | 4 +-- spec/services/tag_alias_seeder_spec.rb | 19 ++++++++++++-- spec/services/tag_translation_spec.rb | 18 ++++++++++--- 7 files changed, 67 insertions(+), 22 deletions(-) diff --git a/app/services/tag_alias_seeder.rb b/app/services/tag_alias_seeder.rb index eda615947c..3fe6dcda8f 100644 --- a/app/services/tag_alias_seeder.rb +++ b/app/services/tag_alias_seeder.rb @@ -104,8 +104,10 @@ class TagAliasSeeder "Veterans & military" => "Military/Veterans" }.freeze - # A term can name two age groups, so these are lists. + # A term can name several age groups, so these are lists: serving families is serving + # every age, and a person has no population tag to carry "Families" on its own. FILEMAKER_AGE_GROUPS = { + "Families" => %w[Children Teens Adults Elders], "C & T" => %w[Children Teens], "Children & Teens" => %w[Children Teens], "Adults & Children" => %w[Adults Children], @@ -180,18 +182,20 @@ def record(term, type, id, source, name) @created += 1 end + # Published only, the same vocabulary TagTranslation resolves against, so a term aimed at + # a retired tag is reported rather than recorded as an alias that never resolves. def sector_ids - @sector_ids ||= Sector.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + @sector_ids ||= Sector.published.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } end def age_group_ids - @age_group_ids ||= Category.age_ranges.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + @age_group_ids ||= Category.age_ranges.published.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } end def report return if @quiet puts " #{@created} tag aliases added (#{TagAlias.count} recorded)" - puts " no tag named: #{@missing.uniq.sort.join(', ')} — seed the sectors and age ranges first" if @missing.any? + puts " no published tag named: #{@missing.uniq.sort.join(', ')} — seed the sectors and age ranges first" if @missing.any? end end diff --git a/app/services/tag_translation.rb b/app/services/tag_translation.rb index 68d0b42337..6ddc955569 100644 --- a/app/services/tag_translation.rb +++ b/app/services/tag_translation.rb @@ -3,6 +3,11 @@ # the portal's earlier sector names. A term resolves by a tag's own name or by any # TagAlias recorded for it (ADR-0007), and one term can name several tags. # +# Only **published** tags resolve. A retired tag is unpublished rather than deleted, so its +# old taggings survive; matching it here would let an import hand it to hundreds more +# records. An archive term that only matches a retired tag reads as unresolved, which puts +# it in front of someone instead of quietly reviving the tag. +# # Built once and reused — it reads the whole sector and age-group vocabulary up front, so # a 39,000-row import doesn't query per value. A term nothing answers to is counted in # #unmatched rather than guessed at — unless it's recorded as ignored, which is how a @@ -18,8 +23,8 @@ class TagTranslation attr_reader :unmatched def initialize - @sectors = vocabulary(Sector.all) - @age_groups = vocabulary(Category.age_ranges) + @sectors = vocabulary(Sector.published) + @age_groups = vocabulary(Category.age_ranges.published) @ignored = TagAlias.ignored.pluck(:name).map { |name| key(name) }.to_set @unmatched = Hash.new(0) end diff --git a/config/features.yml b/config/features.yml index b6b754b4b4..2f8eed5f94 100644 --- a/config/features.yml +++ b/config/features.yml @@ -4576,4 +4576,6 @@ pro_tips: - "This is how an import knows that \"Student Services\" means Education and that \"C & T\" means both Children and Teens." - "A word can be recorded as meaning nothing on purpose — FileMaker's \"Virtually\" described where someone worked, not who they served." + - "An old word for a retired tag doesn't quietly bring the tag back: it's reported so someone can decide where it belongs now." + - "Someone recorded as serving families is tagged with every age range, since that's what serving families means." - "The mapping is written up in ADR-0007 for anyone who needs to see the whole picture." diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md index 80588b4444..a3b9dec89f 100644 --- a/docs/adr/0007-legacy-sector-and-age-group-translation.md +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -170,7 +170,8 @@ military → Military/Veterans. | FileMaker `Adults`; Story Share `18+`, `Adult` | AgeRange **Adults** (18+) | | FileMaker `Elders` | AgeRange **Elders** (65+) | | FileMaker `C & T` | AgeRange **Children** *and* **Teens** | -| FileMaker `Families`; Story Share `Family windows` | StoryPopulation **Families** | +| FileMaker `Families` | AgeRange **Children**, **Teens**, **Adults** *and* **Elders** — serving families is serving every age | +| Story Share `Family windows` | StoryPopulation **Families** (a story can carry a population; a person can't) | | Story Share `Mixed-age groups`, `Combined` | nothing — says the group was mixed, not which ages | `3-5` and `6-12` were folded into `Children` in the portal, and the `StoryPopulation` @@ -182,6 +183,12 @@ twins carry a trailing underscore so the clean age names stay free (see `db/seed to (polymorphic — a `Sector` or an `AgeRange` `Category`), and the system the term came from. `TagTranslation` reads a tag's own name first, then the aliases. +- **Only published tags resolve.** A retired tag is unpublished rather than deleted so its + old taggings survive; resolving to one would hand it to hundreds of new records. A term + that only matches a retired tag reads as unresolved, which puts it in front of someone. + The live `sectors` table holds more than this catalogue — `Poverty` is one such row, and + 634 archive rows name it, so it needs a decision rather than a silent revival. + - **A term can name two tags**, so a row per pair and a list on the way out (`C & T`). - **Unique on (term, tag)**, with the column's case-insensitive collation, so one spelling can't be recorded twice for the same tag. @@ -213,10 +220,12 @@ Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct valu It is recorded as ignored rather than translated: `Sector::OTHER_SECTOR_NAME` is offered for additional sectors but never as a primary, so an import can't honour it as one, and "they picked Other" says nothing a tag could carry. -- **Still needing a decision** — what the seeded terms leave unresolved: `Poverty` (634), - `Elder Abuse` (178), `People Who Have Experienced War` (97), `Children's Services` (8), - and a long tail of one-off free text (145 `Settings` values, 63 `OtherServiceArea` - values). Each is a judgment call for AWBW; the unmatched report is the queue. +- **Still needing a decision** — what the seeded terms leave unresolved: `Poverty` (634, + which only matches a retired sector), `Elder Abuse` (178), `People Who Have Experienced + War` (97), `Children's Services` (8), and a long tail of one-off free text (145 + `Settings` values, 63 `OtherServiceArea` values, nearly all a single mention). Each is a + judgment call for AWBW; the unmatched report is the queue. The seeded terms resolve 80% + of the archive's 17,757 mentions, and the deliberate non-translations another 16%. - **Seen in the data, no mapping decided:** Children's Services, Intellectual Disabilities, Maternal wellness and postpartum support, Political Asylum, Survivors with Disabilities, Arts Education & Community Art Making, and the rest of the one-off free text. Deliberately @@ -225,9 +234,9 @@ Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct valu ## Consequences - **Populations are not age ranges.** `AgeGroupTaggable` only tags `AgeRange` categories, so - a person whose FileMaker age group is `Families` gets no age tag — `Families` is a story - population, and people have no population tagging. An importer reports it rather than - forcing it into an age range. + a person has no population tag to carry `Families`. Rather than drop it, a person whose + FileMaker age group is `Families` is tagged with every age range — which is what serving + families means. A story keeps its `StoryPopulation` tag. - **One source value can produce two tags** (`C & T`), so a translator returns a list, not a value. - **Unmapped values need a report, not a fallback.** `PersonServices::ImportFromRolodex` diff --git a/spec/models/tag_alias_spec.rb b/spec/models/tag_alias_spec.rb index c98b2f355c..719c1abda9 100644 --- a/spec/models/tag_alias_spec.rb +++ b/spec/models/tag_alias_spec.rb @@ -1,7 +1,7 @@ require "rails_helper" RSpec.describe TagAlias, type: :model do - let(:sector) { create(:sector, name: "Domestic Violence") } + let(:sector) { create(:sector, :published, name: "Domestic Violence") } it "is valid when it names a tag" do expect(build(:tag_alias, aliasable: sector)).to be_valid @@ -26,7 +26,7 @@ end it "lets two tags share a term, for a term that names both" do - other = create(:sector, name: "Education") + other = create(:sector, :published, name: "Education") create(:tag_alias, name: "C & T", aliasable: sector) expect(build(:tag_alias, name: "C & T", aliasable: other)).to be_valid diff --git a/spec/services/tag_alias_seeder_spec.rb b/spec/services/tag_alias_seeder_spec.rb index 62b0d83a14..31dbedc821 100644 --- a/spec/services/tag_alias_seeder_spec.rb +++ b/spec/services/tag_alias_seeder_spec.rb @@ -6,8 +6,8 @@ let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } before do - Sector::SECTOR_TYPES.each { |name| create(:sector, name: name) } - %w[Children Teens Adults Elders].each { |name| create(:category, category_type: age_range_type, name: name) } + Sector::SECTOR_TYPES.each { |name| create(:sector, :published, name: name) } + %w[Children Teens Adults Elders].each { |name| create(:category, :published, category_type: age_range_type, name: name) } end it "records every term the tables name, pointing at the tag it resolves to" do @@ -26,6 +26,21 @@ expect(TagAlias.where(name: "C & T").map { |row| row.aliasable.name }).to contain_exactly("Children", "Teens") end + it "reads serving families as serving every age" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "Families").map { |row| row.aliasable.name }) + .to contain_exactly("Children", "Teens", "Adults", "Elders") + end + + it "reports a term aimed at a retired tag instead of recording an alias that can't resolve" do + Sector.find_by(name: "Education").update!(published: false) + + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services")).to be_nil + end + it "labels each term with the system it came from" do described_class.call(quiet: true) diff --git a/spec/services/tag_translation_spec.rb b/spec/services/tag_translation_spec.rb index 6b498a282e..f36e0296d7 100644 --- a/spec/services/tag_translation_spec.rb +++ b/spec/services/tag_translation_spec.rb @@ -3,11 +3,11 @@ require "rails_helper" RSpec.describe TagTranslation do - let!(:domestic_violence) { create(:sector, name: "Domestic Violence") } - let!(:education) { create(:sector, name: "Education") } + let!(:domestic_violence) { create(:sector, :published, name: "Domestic Violence") } + let!(:education) { create(:sector, :published, name: "Education") } let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } - let!(:children) { create(:category, category_type: age_range_type, name: "Children") } - let!(:teens) { create(:category, category_type: age_range_type, name: "Teens") } + let!(:children) { create(:category, :published, category_type: age_range_type, name: "Children") } + let!(:teens) { create(:category, :published, category_type: age_range_type, name: "Teens") } it "resolves a term by the tag's own name, whatever its case" do expect(described_class.new.sector_ids_for("domestic violence")).to eq([ domestic_violence.id ]) @@ -54,6 +54,16 @@ expect(translation.unmatched).to be_empty end + it "leaves a retired tag out of reach, by name or by alias" do + retired = create(:sector, :unpublished, name: "Poverty") + create(:tag_alias, name: "Destitution", aliasable: retired) + translation = described_class.new + + expect(translation.sector_ids_for("Poverty")).to be_empty + expect(translation.sector_ids_for("Destitution")).to be_empty + expect(translation.unmatched).to eq({ "poverty" => 1, "destitution" => 1 }) + end + it "has nothing to say about a blank value" do translation = described_class.new From 52614e6a4a3cfe59896af48902e08c0405025d28 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Wed, 14 Oct 2026 23:15:00 -0400 Subject: [PATCH 4/5] Take the decisions AWBW made on the archive terms MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four terms with real volume behind them: Poverty points at the retired sector that already carries its taggings, Elder Abuse adds the Elders age range, Children's Services makes Children the primary one, and war is left untagged rather than filed under a sector it does not mean. Two of those cross vocabularies — a life experience and a service area that name ages, not sectors — so the lookup answers with both kinds and the caller takes what it needs. Primacy stays the caller's call, since it follows from the field a term sat in rather than from the term. The compound cells ("Sexual Assault & Domestic Violence") resolve to every sector they name, and the answers that say nothing — n/a, unsure, Everyone, the delivery modes — are recorded as naming nothing, so the undecided queue is short enough to read. Co-Authored-By: Claude Opus 5 (1M context) --- app/services/tag_alias_seeder.rb | 61 ++++++++++++++-- app/services/tag_translation.rb | 52 +++++++++----- config/features.yml | 2 +- ...legacy-sector-and-age-group-translation.md | 70 +++++++++++++++---- spec/services/tag_alias_seeder_spec.rb | 54 ++++++++++++-- spec/services/tag_translation_spec.rb | 64 ++++++++++++----- 6 files changed, 247 insertions(+), 56 deletions(-) diff --git a/app/services/tag_alias_seeder.rb b/app/services/tag_alias_seeder.rb index 3fe6dcda8f..fd853d7cd1 100644 --- a/app/services/tag_alias_seeder.rb +++ b/app/services/tag_alias_seeder.rb @@ -23,6 +23,8 @@ class TagAliasSeeder "Prisons/Jails" => "Incarceration", "Private Practice" => "Private Practice/Sole Proprietor", "With Staff" => "Staff/Organizational Development", + "school" => "Education", + "Students" => "Education", "Climate/Environmental Trauma" => "Climate/Environmental", "Illness" => "Health/Medical", "Pandemic-Related Stress" => "Health/Medical", @@ -64,6 +66,31 @@ class TagAliasSeeder "Law enforcement/court/legal" => "Court/Legal System" }.freeze + # Hand-typed cells that name more than one sector, so each resolves to all of them — + # the same one-to-many shape as "C & T". Only recorded where every concept in the cell is + # already a decided term, so nothing new is being inferred. + COMPOUND_SECTORS = { + "Domestic Violence & Homelessness" => [ "Domestic Violence", "Homelessness" ], + "Sexual Assault & Domestic Violence" => [ "Sexual Assault", "Domestic Violence" ], + "Domestic Violence, Sexual Assault" => [ "Domestic Violence", "Sexual Assault" ], + "Domestic Violence and Sexual Assault survivors" => [ "Domestic Violence", "Sexual Assault" ], + "Domestic and Sexual Violence Families" => [ "Domestic Violence", "Sexual Assault" ], + "Sexual and Domestic Violence Agency" => [ "Sexual Assault", "Domestic Violence" ], + "DV/SA/other related crimes" => [ "Domestic Violence", "Sexual Assault" ], + "Dual Agency Domestic violence/Sexual assault/ Human trafficking" => + [ "Domestic Violence", "Sexual Assault", "Human Trafficking" ], + "Immigration, Health, Prevention of Violence and Education" => [ "Immigration", "Health/Medical", "Education" ], + "Addiction, Native Americans, mental health and addiction" => + [ "Substance Use/Recovery", "Indigenous/Tribal Nation", "Mental Health" ], + "Education, SEL needs" => [ "Education" ], + "Hospice, end of life, bereavement" => [ "Grief/Loss" ] + }.freeze + + # A retired sector an alias deliberately points at. The term is in the archive 634 times + # and the tag it means still exists, unpublished, carrying the taggings it already had — + # so the term resolves even though nothing new can be tagged with it by name. + RETIRED_SECTORS = { "Poverty" => "Poverty" }.freeze + STORY_SHARE_SECTORS = { "Advocacy" => "Systems/Policy Change", "Child Abuse / Neglect" => "Child Abuse/Neglect", @@ -106,8 +133,15 @@ class TagAliasSeeder # A term can name several age groups, so these are lists: serving families is serving # every age, and a person has no population tag to carry "Families" on its own. + # + # Some of these terms come from the *sector* fields, because the vocabularies cross — a + # life experience of elder abuse says the person serves elders, and a service area of + # children's services says they serve children. Whether the tag lands as primary or + # additional follows from the field the term sat in, not from the term. FILEMAKER_AGE_GROUPS = { "Families" => %w[Children Teens Adults Elders], + "Elder Abuse" => %w[Elders], + "Children's Services" => %w[Children], "C & T" => %w[Children Teens], "Children & Teens" => %w[Children Teens], "Adults & Children" => %w[Adults Children], @@ -121,11 +155,16 @@ class TagAliasSeeder STORY_SHARE_AGE_GROUPS = { "Adult" => %w[Adults] }.freeze - # Settings that say where or how someone works rather than who they serve. Recorded as - # naming nothing on purpose, so they don't sit in the queue of undecided terms. + # Terms that name no tag on purpose: a setting that says where or how someone works + # rather than who they serve, an answer that says nothing at all, and one life experience + # (war) that AWBW chose to leave untagged. Recorded so they don't sit in the queue of + # terms still needing a decision. IGNORED_TERMS = [ "Community-Based", "Events and Conferences", "Events & Conferences", "Home Visits", - "Outreach Program", "Residential Program", "Virtually", "Other", "Did not specify" + "Outreach Program", "Residential Program", "Virtually", "Other", "Did not specify", + "Residential", "In person", "In visits", "virtually & in-person", "At office", "Office space.", + "On-Call/Crisis", "Support groups", "n/a", "none", "something", "unsure", "Everyone", "Technology", + "People Who Have Experienced War" ].freeze def self.call(...) @@ -140,6 +179,8 @@ def initialize(quiet: false) def call record_sectors(FILEMAKER_SECTORS, "filemaker") + record_sectors(COMPOUND_SECTORS, "filemaker") + record_retired_sectors(RETIRED_SECTORS, "filemaker") record_sectors(FORM_SECTORS, "forms") record_sectors(STORY_SHARE_SECTORS, "story_share") record_sectors(PORTAL_SECTORS, "portal") @@ -153,7 +194,15 @@ def call private def record_sectors(table, source) - table.each { |term, name| record(term, "Sector", sector_ids[name.downcase], source, name) } + table.each do |term, names| + Array(names).each { |name| record(term, "Sector", sector_ids[name.downcase], source, name) } + end + end + + # Resolved against every sector, published or not: the point of the table is the ones that + # aren't, which the published lookup would report as missing. + def record_retired_sectors(table, source) + table.each { |term, name| record(term, "Sector", every_sector_id[name.downcase], source, name) } end def record_age_groups(table, source) @@ -192,6 +241,10 @@ def age_group_ids @age_group_ids ||= Category.age_ranges.published.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } end + def every_sector_id + @every_sector_id ||= Sector.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + def report return if @quiet diff --git a/app/services/tag_translation.rb b/app/services/tag_translation.rb index 6ddc955569..cac6792903 100644 --- a/app/services/tag_translation.rb +++ b/app/services/tag_translation.rb @@ -3,16 +3,26 @@ # the portal's earlier sector names. A term resolves by a tag's own name or by any # TagAlias recorded for it (ADR-0007), and one term can name several tags. # -# Only **published** tags resolve. A retired tag is unpublished rather than deleted, so its -# old taggings survive; matching it here would let an import hand it to hundreds more -# records. An archive term that only matches a retired tag reads as unresolved, which puts -# it in front of someone instead of quietly reviving the tag. +# A tag resolves **by name only while it's published**. A retired tag is unpublished rather +# than deleted, so its old taggings survive, and matching one by name would let an import +# hand it to hundreds more records by accident. A recorded alias is the opposite — someone +# decided that term means that tag — so an alias may point at a retired tag and still +# resolve. FileMaker's "Poverty" works that way: the sector it means is retired, and 634 +# archive rows name it. +# +# A term is read without regard for the field it sat in, because the two vocabularies cross: +# FileMaker's "Elder Abuse" is a life experience that means the Elders *age range*, and +# "Children's Services" is a service area that means the Children one. So #resolve answers +# with both kinds and the caller takes what it needs — and whether a tag lands as primary or +# additional stays the caller's call, since that follows from the field, not the term. # # Built once and reused — it reads the whole sector and age-group vocabulary up front, so # a 39,000-row import doesn't query per value. A term nothing answers to is counted in # #unmatched rather than guessed at — unless it's recorded as ignored, which is how a # decided non-translation stays out of the queue of terms still needing a decision. class TagTranslation + Tags = Data.define(:sector_ids, :age_group_ids) + # FileMaker repeating fields separate values with line breaks, and a hand-typed list # uses semicolons. Commas are not separators: the wording itself contains them, inside # parentheticals like "Community violence (gang violence, mass shootings, etc.)". @@ -29,31 +39,37 @@ def initialize @unmatched = Hash.new(0) end - def sector_ids_for(value) - ids_for(value, @sectors) - end + def resolve(value) + sector_ids = [] + age_group_ids = [] + + terms(value).each do |term| + sectors = lookup(@sectors, term) + age_groups = lookup(@age_groups, term) + @unmatched[term] += 1 if sectors.empty? && age_groups.empty? && !@ignored.include?(term) + sector_ids.concat(sectors) + age_group_ids.concat(age_groups) + end - def age_group_ids_for(value) - ids_for(value, @age_groups) + Tags.new(sector_ids: sector_ids.uniq, age_group_ids: age_group_ids.uniq) end private + def lookup(index, term) + index[term] || index[key(term.sub(PARENTHETICAL, ""))] || [] + end + + # Names come from the scope as given (published only); aliases are read across the whole + # vocabulary of that kind, so one can deliberately point at a retired tag. def vocabulary(scope) - aliases = TagAlias.resolving.where(aliasable_type: scope.model_name.to_s, aliasable_id: scope.select(:id)) + aliases = TagAlias.resolving.where(aliasable_type: scope.model_name.to_s, + aliasable_id: scope.unscope(where: :published).select(:id)) (scope.pluck(:name, :id) + aliases.pluck(:name, :aliasable_id)).each_with_object({}) do |(name, id), index| (index[key(name)] ||= []) << id end end - def ids_for(value, index) - terms(value).flat_map do |term| - ids = index[term] || index[key(term.sub(PARENTHETICAL, ""))] || [] - @unmatched[term] += 1 if ids.empty? && !@ignored.include?(term) - ids - end.uniq - end - def terms(value) value.to_s.split(SEPARATORS).filter_map { |part| key(part).presence }.uniq end diff --git a/config/features.yml b/config/features.yml index 2f8eed5f94..04600512e3 100644 --- a/config/features.yml +++ b/config/features.yml @@ -4576,6 +4576,6 @@ pro_tips: - "This is how an import knows that \"Student Services\" means Education and that \"C & T\" means both Children and Teens." - "A word can be recorded as meaning nothing on purpose — FileMaker's \"Virtually\" described where someone worked, not who they served." - - "An old word for a retired tag doesn't quietly bring the tag back: it's reported so someone can decide where it belongs now." + - "A retired tag never comes back by accident — but a word can be pointed at one deliberately, which is how FileMaker's \"Poverty\" still finds the sector that already carries it." - "Someone recorded as serving families is tagged with every age range, since that's what serving families means." - "The mapping is written up in ADR-0007 for anyone who needs to see the whole picture." diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md index a3b9dec89f..f7466196eb 100644 --- a/docs/adr/0007-legacy-sector-and-age-group-translation.md +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -116,6 +116,47 @@ who they serve — they aren't sectors and have no portal home. purpose — each needs a sector added before it can be translated, and inventing a near-miss would bury the gap. An importer reports them. +### Cells naming more than one sector + +Hand-typed answers that name several sectors resolve to all of them — the same one-to-many +shape as `C & T`. Recorded only where every concept in the cell is itself a decided term: +`Domestic Violence & Homelessness` · `Sexual Assault & Domestic Violence` · +`Domestic Violence, Sexual Assault` · `Domestic Violence and Sexual Assault survivors` · +`Domestic and Sexual Violence Families` · `Sexual and Domestic Violence Agency` · +`DV/SA/other related crimes` · `Dual Agency Domestic violence/Sexual assault/ Human +trafficking` · `Immigration, Health, Prevention of Violence and Education` · +`Addiction, Native Americans, mental health and addiction` · `Education, SEL needs` · +`Hospice, end of life, bereavement`. + +### Terms that cross vocabularies + +Two of the archive's *sector* fields carry terms that say which **ages** someone serves, so +they resolve to an age range rather than a sector: + +| Source value | Portal tag | +|---|---| +| `Elder Abuse` (LifeExperiences) | AgeRange **Elders** | +| `Children's Services` (PrimaryServiceArea) | AgeRange **Children** | + +Whether the tag lands as the **primary** or an **additional** one follows from the field the +term sat in, not from the term: `PrimaryServiceArea` and `PrimaryAgeGroup` are the primary +answers, `OtherServiceArea`, `Settings` and `LifeExperiences` the additional ones. So +`Children's Services` makes Children the primary age range, and `Elder Abuse` adds Elders +alongside whatever the primary already is. + +### Answers that name nothing + +Alongside the dropped Settings, these are recorded as naming nothing on purpose: `Other` · +`Did not specify` · `n/a` · `none` · `something` · `unsure` · `Everyone` · `Technology` · +and the delivery modes `In person` · `In visits` · `virtually & in-person` · `At office` · +`Office space.` · `Residential` · `On-Call/Crisis` · `Support groups`. None of them says +who a facilitator serves, so leaving them in the undecided queue would only bury the terms +that do. + +**`People Who Have Experienced War` (97) joins them by decision, not by shape.** It names a +real experience, and Community Violence was the fallback if it had to go somewhere — but +AWBW chose to leave it untagged rather than file it under a sector it doesn't mean. + ### Story Share categories → sector | Story Share | Sector | @@ -183,11 +224,12 @@ twins carry a trailing underscore so the clean age names stay free (see `db/seed to (polymorphic — a `Sector` or an `AgeRange` `Category`), and the system the term came from. `TagTranslation` reads a tag's own name first, then the aliases. -- **Only published tags resolve.** A retired tag is unpublished rather than deleted so its - old taggings survive; resolving to one would hand it to hundreds of new records. A term - that only matches a retired tag reads as unresolved, which puts it in front of someone. - The live `sectors` table holds more than this catalogue — `Poverty` is one such row, and - 634 archive rows name it, so it needs a decision rather than a silent revival. +- **A tag resolves by name only while it's published.** A retired tag is unpublished rather + than deleted so its old taggings survive, and matching one by name would hand it to + hundreds of new records by accident. **A recorded alias may point at a retired tag** — + that's someone deciding the term means that tag, which is the opposite of an accident. + `Poverty` is exactly that case: the sector is retired, 634 archive rows name it, and the + alias sends them to the tag that already carries its older taggings. - **A term can name two tags**, so a row per pair and a list on the way out (`C & T`). - **Unique on (term, tag)**, with the column's case-insensitive collation, so one spelling @@ -220,12 +262,13 @@ Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct valu It is recorded as ignored rather than translated: `Sector::OTHER_SECTOR_NAME` is offered for additional sectors but never as a primary, so an import can't honour it as one, and "they picked Other" says nothing a tag could carry. -- **Still needing a decision** — what the seeded terms leave unresolved: `Poverty` (634, - which only matches a retired sector), `Elder Abuse` (178), `People Who Have Experienced - War` (97), `Children's Services` (8), and a long tail of one-off free text (145 - `Settings` values, 63 `OtherServiceArea` values, nearly all a single mention). Each is a - judgment call for AWBW; the unmatched report is the queue. The seeded terms resolve 80% - of the archive's 17,757 mentions, and the deliberate non-translations another 16%. +- **Still needing a decision** — only the long tail of one-off free text (≈90 `Settings` + values, 42 `OtherServiceArea` values, almost all a single mention, mostly venues, program + names and prose rather than anything a tag could carry). The unmatched report is the + queue, and it stays short enough to read. +- **Coverage.** Every archive term with more than two mentions is now either translated or + deliberately untranslated: ~17,600 of the 17,757 mentions. What's left is ~130 values at + one or two mentions each. - **Seen in the data, no mapping decided:** Children's Services, Intellectual Disabilities, Maternal wellness and postpartum support, Political Asylum, Survivors with Disabilities, Arts Education & Community Art Making, and the rest of the one-off free text. Deliberately @@ -237,8 +280,9 @@ Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct valu a person has no population tag to carry `Families`. Rather than drop it, a person whose FileMaker age group is `Families` is tagged with every age range — which is what serving families means. A story keeps its `StoryPopulation` tag. -- **One source value can produce two tags** (`C & T`), so a translator returns a list, not a - value. +- **One source value can produce several tags** (`C & T`, `Families`, the compound cells), + so a translator returns a list, not a value — and across both vocabularies, since a sector + field can carry an age term. - **Unmapped values need a report, not a fallback.** `PersonServices::ImportFromRolodex` counts them in `#unmatched_tags` and the rake task prints them ranked; its name matching alone resolves only the values that already read as a sector name. diff --git a/spec/services/tag_alias_seeder_spec.rb b/spec/services/tag_alias_seeder_spec.rb index 31dbedc821..84fe874dc1 100644 --- a/spec/services/tag_alias_seeder_spec.rb +++ b/spec/services/tag_alias_seeder_spec.rb @@ -33,7 +33,7 @@ .to contain_exactly("Children", "Teens", "Adults", "Elders") end - it "reports a term aimed at a retired tag instead of recording an alias that can't resolve" do + it "reports a term whose tag isn't published, when nothing says to point at a retired one" do Sector.find_by(name: "Education").update!(published: false) described_class.call(quiet: true) @@ -41,6 +41,52 @@ expect(TagAlias.find_by(name: "Student Services")).to be_nil end + it "points the archive's Poverty at the retired sector that already carries its taggings" do + retired = create(:sector, :unpublished, name: "Poverty") + + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Poverty").aliasable).to eq(retired) + expect(TagTranslation.new.resolve("Poverty").sector_ids).to eq([ retired.id ]) + end + + it "reads a cell naming several sectors as all of them" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "Sexual Assault & Domestic Violence").map { |row| row.aliasable.name }) + .to contain_exactly("Sexual Assault", "Domestic Violence") + expect(TagTranslation.new.resolve("DV/SA/other related crimes").sector_ids.size).to eq(2) + end + + it "reads a life experience or service area that means an age group" do + described_class.call(quiet: true) + + translation = TagTranslation.new + expect(translation.resolve("Elder Abuse").age_group_ids).to eq([ Category.find_by(name: "Elders").id ]) + expect(translation.resolve("Children's Services").age_group_ids).to eq([ Category.find_by(name: "Children").id ]) + expect(translation.resolve("Elder Abuse").sector_ids).to be_empty + end + + it "leaves war untagged, as decided" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "People Who Have Experienced War")).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.resolve("People Who Have Experienced War").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "records an answer that says nothing as naming nothing" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "n/a")).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.resolve("n/a\nunsure\nIn person").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + it "labels each term with the system it came from" do described_class.call(quiet: true) @@ -80,7 +126,7 @@ expect(virtually).to have_attributes(aliasable: nil, ignored: true) translation = TagTranslation.new - expect(translation.sector_ids_for("Virtually")).to be_empty + expect(translation.resolve("Virtually").sector_ids).to be_empty expect(translation.unmatched).to be_empty end @@ -88,7 +134,7 @@ described_class.call(quiet: true) translation = TagTranslation.new - expect(translation.sector_ids_for("Clinical Setting")).to eq([ Sector.find_by(name: "Mental Health").id ]) - expect(translation.age_group_ids_for("C & T").size).to eq(2) + expect(translation.resolve("Clinical Setting").sector_ids).to eq([ Sector.find_by(name: "Mental Health").id ]) + expect(translation.resolve("C & T").age_group_ids.size).to eq(2) end end diff --git a/spec/services/tag_translation_spec.rb b/spec/services/tag_translation_spec.rb index f36e0296d7..d561d99782 100644 --- a/spec/services/tag_translation_spec.rb +++ b/spec/services/tag_translation_spec.rb @@ -8,41 +8,42 @@ let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } let!(:children) { create(:category, :published, category_type: age_range_type, name: "Children") } let!(:teens) { create(:category, :published, category_type: age_range_type, name: "Teens") } + let!(:elders) { create(:category, :published, category_type: age_range_type, name: "Elders") } it "resolves a term by the tag's own name, whatever its case" do - expect(described_class.new.sector_ids_for("domestic violence")).to eq([ domestic_violence.id ]) + expect(described_class.new.resolve("domestic violence").sector_ids).to eq([ domestic_violence.id ]) end it "resolves a term by an alias recorded for the tag" do create(:tag_alias, name: "Student Services", aliasable: education) - expect(described_class.new.sector_ids_for("student services")).to eq([ education.id ]) + expect(described_class.new.resolve("student services").sector_ids).to eq([ education.id ]) end it "reads one term as several tags when both are recorded" do create(:tag_alias, name: "C & T", aliasable: children, source: "filemaker") create(:tag_alias, name: "C & T", aliasable: teens, source: "filemaker") - expect(described_class.new.age_group_ids_for("C & T")).to contain_exactly(children.id, teens.id) + expect(described_class.new.resolve("C & T").age_group_ids).to contain_exactly(children.id, teens.id) end it "splits a repeating field on line breaks and semicolons, not on commas" do create(:tag_alias, name: "Community violence (gang violence, mass shootings, etc.)", aliasable: domestic_violence) translation = described_class.new - expect(translation.sector_ids_for("Domestic Violence\nEducation")).to contain_exactly(domestic_violence.id, education.id) - expect(translation.sector_ids_for("Domestic Violence; Education")).to contain_exactly(domestic_violence.id, education.id) - expect(translation.sector_ids_for("Community violence (gang violence, mass shootings, etc.)")).to eq([ domestic_violence.id ]) + expect(translation.resolve("Domestic Violence\nEducation").sector_ids).to contain_exactly(domestic_violence.id, education.id) + expect(translation.resolve("Domestic Violence; Education").sector_ids).to contain_exactly(domestic_violence.id, education.id) + expect(translation.resolve("Community violence (gang violence, mass shootings, etc.)").sector_ids).to eq([ domestic_violence.id ]) end it "drops a clarifying parenthetical when the term alone resolves" do - expect(described_class.new.sector_ids_for("Domestic violence (including family violence)")).to eq([ domestic_violence.id ]) + expect(described_class.new.resolve("Domestic violence (including family violence)").sector_ids).to eq([ domestic_violence.id ]) end it "counts a term nothing answers to instead of guessing at it" do translation = described_class.new - expect(translation.sector_ids_for("Poverty\nDomestic Violence")).to eq([ domestic_violence.id ]) + expect(translation.resolve("Poverty\nDomestic Violence").sector_ids).to eq([ domestic_violence.id ]) expect(translation.unmatched).to eq({ "poverty" => 1 }) end @@ -50,25 +51,56 @@ create(:tag_alias, name: "Virtually", aliasable: nil, ignored: true) translation = described_class.new - expect(translation.sector_ids_for("Virtually\nDomestic Violence")).to eq([ domestic_violence.id ]) + expect(translation.resolve("Virtually\nDomestic Violence").sector_ids).to eq([ domestic_violence.id ]) expect(translation.unmatched).to be_empty end - it "leaves a retired tag out of reach, by name or by alias" do + it "won't reach a retired tag by its own name" do + create(:sector, :unpublished, name: "Poverty") + translation = described_class.new + + expect(translation.resolve("Poverty").sector_ids).to be_empty + expect(translation.unmatched).to eq({ "poverty" => 1 }) + end + + it "reaches a retired tag an alias deliberately points at" do retired = create(:sector, :unpublished, name: "Poverty") - create(:tag_alias, name: "Destitution", aliasable: retired) + create(:tag_alias, name: "Poverty", aliasable: retired) + translation = described_class.new + + expect(translation.resolve("Poverty").sector_ids).to eq([ retired.id ]) + expect(translation.unmatched).to be_empty + end + + it "keeps an age-group alias from reaching a category of another kind" do + population_type = create(:category_type, name: "StoryPopulation", published: true) + families = create(:category, :published, category_type: population_type, name: "Families_") + create(:tag_alias, name: "Family windows", aliasable: families) + + expect(described_class.new.resolve("Family windows").age_group_ids).to be_empty + end + + it "answers with an age group for a term that sat in a sector field" do + create(:tag_alias, name: "Elder Abuse", aliasable: elders) + tags = described_class.new.resolve("Elder Abuse") + + expect(tags.sector_ids).to be_empty + expect(tags.age_group_ids).to eq([ elders.id ]) + end + + it "counts a term once, however many vocabularies miss it" do translation = described_class.new + translation.resolve("Poverty") + translation.resolve("Poverty") - expect(translation.sector_ids_for("Poverty")).to be_empty - expect(translation.sector_ids_for("Destitution")).to be_empty - expect(translation.unmatched).to eq({ "poverty" => 1, "destitution" => 1 }) + expect(translation.unmatched).to eq({ "poverty" => 2 }) end it "has nothing to say about a blank value" do translation = described_class.new - expect(translation.sector_ids_for(nil)).to be_empty - expect(translation.sector_ids_for(" ")).to be_empty + expect(translation.resolve(nil).sector_ids).to be_empty + expect(translation.resolve(" ").age_group_ids).to be_empty expect(translation.unmatched).to be_empty end end From f103b8b8b2bfa026d8f736832d2b07f2b108b259 Mon Sep 17 00:00:00 2001 From: Mae Beale Date: Wed, 14 Oct 2026 23:21:53 -0400 Subject: [PATCH 5/5] Read the archive's free text where the meaning is plain, and say so MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Most of the hand-typed cells are a venue, a programme name or a sentence about the work, and a term AWBW hasn't ruled on shouldn't masquerade as one it has. So the readable ones are recorded as proposed: they resolve, so no import waits on a review, and TagAlias.proposed is the list to walk with staff term by term. Around 90 terms, each worth one or two archive rows — Immigration Clinic, Grief support group, Supervised visitation sessions for foster youth, and the cells that name both sectors and the ages they serve. Another 16 say nothing a tag could carry and are proposed as naming nothing. Text that names an organization, a place that identifies one, or a facilitator's own circumstances is left alone: it's personal, and it belongs in the archive rather than copied into a catalogue. That leaves 23 terms, one mention each, unresolved for good — down from 150. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 4 +- app/models/tag_alias.rb | 7 + app/services/tag_alias_seeder.rb | 161 +++++++++++++++++- config/features.yml | 1 + .../20261015013218_create_tag_aliases.rb | 2 + db/schema.rb | 1 + ...legacy-sector-and-age-group-translation.md | 28 ++- spec/services/tag_alias_seeder_spec.rb | 28 +++ 8 files changed, 214 insertions(+), 18 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 68eee04a6d..7209978b43 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -271,8 +271,8 @@ action, or `authorize! :workshop, to: :summary?`). - `EventRegistrationServices::ProcessConfirmation` — Registration confirmation flow - `RegistrantCeForm` — The CE-specific view over the inline form the **CE callout** carries (built on the generic callout→form mechanism: `RegistrationTicketCallout#forms` + `CalloutFormSubmission`; the CE callout carries a single form, read as `forms.first`). Adds the two things CE needs: the post-training form only surfaces once `EventRegistration#ce_signouts_complete?` (event ended + no open attendance entry), and completing it — read-time `complete?`, every required field answered — is a prerequisite for the CE certificate (`EventRegistration#ce_form_requirement_met?` gates `ContinuingEducationRegistration#certificate_available?`). Read by the CE callout page (which links to the generic `registration/:slug/forms/:callout_id` form page, gated so the CE callout's form opens only after sign-outs via `CalloutsController#ce_form_locked?`) and by `BuiltinCalloutCards#ce_form_pending?` for the CE card's "Complete your form" nudge - `EventRegistrationServices::CalloutFormSubmission` — Records a registrant's answers to one of the forms a registration-ticket callout delivers inline (`call(registration:, callout:, form:, form_params:, clarity_params:)`, returns the service instance with `#submission`/`#profile_changes`). Find-or-creates one `FormSubmission` per `(registrant, form, event)` carrying the form's own role — a re-submit edits the answers in place via `FormSubmission#persist_answer`. That a callout collected it is recorded separately in `FormSubmission#metadata` (`collected_via: "callout"`, `collected_via_callout_id`) via `#record_callout_collection!` — the flag (`#collected_via_callout?`), not the role, tells a callout-collected submission apart. There is no survey-role flag — the extra side effects self-gate on the form's own fields, so they no-op on a form that doesn't ask those questions: a choice field with linked `FormFieldResource`s (`FormField#per_resource?`) fans out to one answer per resource (nil `form_field`, `FormField#per_resource_question` snapshotted in `question_name_when_answered` — subtitle-led, so two fan-out questions sharing a prompt can't collide on one answer row); fields carrying the `anonymous_contributions` / `display_name_preference` identifiers write through to the `Person` via `PersonServices::SyncSharingPreferences` (the same write-through the registration form runs), exposing `#profile_changes` so the controller Ahoy-tracks real edits; and submitting any form on the **`scholarship`** callout stamps a recipient's `post_survey_completed_at`. Drives `Events::CalloutsController#callout`/`#submit_callout` (the person-first `registration/:slug/forms/:callout_id` page and its `.../:form_id` submit) -- `TagTranslation` — Turns a legacy word for who someone serves into sector / age-group ids: a tag's own name first, then any `TagAlias` recorded for it, so one term can name several tags (`C & T` is Children and Teens). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess -- `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't seeded yet is reported, not created. Read through `TagTranslation`. Shared with `data:seed_tag_aliases` +- `TagTranslation` — Turns a legacy word for who someone serves into tags: `#resolve` answers with both the sector ids and the age-group ids a term means, since the vocabularies cross (`Elder Abuse` is a life experience that means the Elders *age range*). A tag's own name resolves only while it's **published** — a retired tag is unpublished, not deleted — but a recorded `TagAlias` may point at a retired one deliberately (FileMaker's `Poverty`). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess. Primacy is the caller's call, since it follows from the field a term sat in +- `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Terms AWBW decided are recorded as-is; semantic reads of the archive's hand-typed free text are marked **`proposed`** for a pass with staff (they still resolve, so no import waits on it), and text naming an organization or a facilitator's own circumstances is left alone. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't published is reported, not created. Shared with `data:seed_tag_aliases` - `SurveyFormSeeder` — Creates the four standalone survey template forms from the form-builder presets (the Day 1 / Day 2 evaluations, the overall post-event survey, and the scholarship recipients survey), idempotent on form name and returning the names created. `#link_fanout_resources` then links each fan-out question to its training-topic `Resource`s by title (skipping any that aren't present). Shared by `db/seeds.rb` and the `data:seed_survey_forms` task so prod can create them without the full seed - `CloseProgramFormSeeder` — Creates the standalone "Close Program" agreement form (role `close_program`, sections `person_identifier` + `close_program`) from the form-builder presets, idempotent on form name, returning the name created (or nil). Built as an unpublished template for staff to slug + publish. Shared by `db/seeds.rb` and the `data:seed_close_program_form` task - `EventRegistrationServices::PublicRegistration` — Public registration handling. An affirmative `communication_consent` answer captures a News (mailing-list) `TopicSubscription` via `NewsSubscriptionCapture`, sourced to the event. A resubmission matched to an active registration sends the registrant and admins DUPLICATE emails (`event_registration_duplicate` / `event_registration_duplicate_fyi`) diff --git a/app/models/tag_alias.rb b/app/models/tag_alias.rb index 1ad1846c8a..91f3d1ec07 100644 --- a/app/models/tag_alias.rb +++ b/app/models/tag_alias.rb @@ -6,6 +6,9 @@ # called outlives the import that needed it. # # A term can name two tags ("C & T" is both Children and Teens), so it is read as a list. +# A term read out of free text rather than decided by AWBW is marked `proposed`, so the +# guesses can be reviewed as a list without holding up the import. +# # It can also be recorded as naming nothing on purpose — a FileMaker "setting" like # "Virtually" says where someone works, not who they serve — which is what keeps a decided # non-translation out of the queue of terms still needing a decision. @@ -25,6 +28,10 @@ class TagAlias < ApplicationRecord scope :ignored, -> { where(ignored: true) } scope :resolving, -> { where(ignored: false) } + # Read from the archive's free text rather than decided by AWBW. They resolve, so an + # import doesn't lose them, and this is the list to walk with staff. + scope :proposed, -> { where(proposed: true) } + scope :decided, -> { where(proposed: false) } scope :for_sectors, -> { where(aliasable_type: "Sector") } scope :for_categories, -> { where(aliasable_type: "Category") } diff --git a/app/services/tag_alias_seeder.rb b/app/services/tag_alias_seeder.rb index fd853d7cd1..62598ea454 100644 --- a/app/services/tag_alias_seeder.rb +++ b/app/services/tag_alias_seeder.rb @@ -167,6 +167,146 @@ class TagAliasSeeder "People Who Have Experienced War" ].freeze + # ─── Proposed, pending staff validation ───────────────────────────────────────────── + # + # Semantic reads of the archive's hand-typed free text, not decisions AWBW has made. They + # are seeded as `proposed`, so they resolve — an import shouldn't lose them — and + # `TagAlias.proposed` is the list to walk with staff, term by term. + # + # Only terms whose meaning is plain are here. Anything naming an organization, a place + # that identifies one, or a facilitator's own circumstances is left out: that text is + # personal, and it belongs in the archive rather than copied into a catalogue. + PROPOSED_SECTORS = { + "1:1 children and youth who experience domestic violence, dating violence, or sexual abuse" => + [ "Domestic Violence", "Sexual Assault", "Child Abuse/Neglect" ], + "a regularly meting group with a drop in attendance from the local queer community" => [ "LGBTQIA+" ], + "Advocacy for survivors of domestic and sexual violence" => [ "Domestic Violence", "Sexual Assault" ], + "Arts Education & Community Art Making" => [ "Education", "Community Building" ], + "bereavement and complex care in hospital setting" => [ "Grief/Loss", "Health/Medical" ], + "Cancer Survivorship" => [ "Health/Medical" ], + "Cases opened by Child protective services" => [ "Child Abuse/Neglect" ], + "Children's Advocacy Center Children & Families" => [ "Child Abuse/Neglect" ], + "College campus" => [ "Education" ], + "Community Morale" => [ "Community Building" ], + "Community-based, youth diversion, juvenile hall" => [ "Incarceration", "Court/Legal System" ], + "Critically ill children" => [ "Health/Medical" ], + "Day Program for adults with developmental and intellectual disabilities" => [ "Disability Services" ], + "Disaster Recovery - Wellness through Trauma informed Activities" => [ "Climate/Environmental" ], + "Domestic Violence Shelter" => [ "Domestic Violence" ], + "Domestic Violence, Sexual Violence and Stalking Advocacy Center" => [ "Domestic Violence", "Sexual Assault" ], + "DV Resource Center" => [ "Domestic Violence" ], + "Empowerment community groups" => [ "Community Building" ], + "Environmental education setting" => [ "Climate/Environmental", "Education" ], + "First Responders. doing workshops in Fire Stations and Law Enforcement Offices" => + [ "Staff/Organizational Development" ], + "formerly incarcerated/formerly gang related" => [ "Incarceration", "Community Violence" ], + "Foster Care agencies" => [ "Foster Care/Adoption" ], + "Grief support group" => [ "Grief/Loss" ], + "Helping Professionals, Caregivers, Leaders" => [ "Staff/Organizational Development" ], + "High level secondary art students" => [ "Education" ], + "I intend to service a diverse population experiencing, DV, sexual assault, and homelessness" => + [ "Domestic Violence", "Sexual Assault", "Homelessness" ], + "Immigration Clinic" => [ "Immigration" ], + "In DV shelters" => [ "Domestic Violence" ], + "Individual and Group counseling with students. In my office and in classrooms." => + [ "Education", "Mental Health" ], + "Individual counseling sessions with sexual assault survivors." => [ "Sexual Assault" ], + "Inpatient Psychiatric Hopital" => [ "Mental Health", "Health/Medical" ], + "Intellectual Disabilities" => [ "Disability Services" ], + "Intensive Outpatient programs" => [ "Mental Health" ], + "Interpersonal violence and any gender based harassment and discrimination" => + [ "Domestic Violence", "Sexual Assault" ], + "LA Fire imacted Community Members in Palisades and Altadena" => [ "Climate/Environmental" ], + "Life Coaching" => [ "Self-Care/Personal Growth" ], + "Low-income housing /vulnerable population" => [ "Homelessness" ], + "Low-income housing support services" => [ "Homelessness" ], + "Maternal wellness and postpartum support" => [ "Reproductive Services" ], + "Ministry Events" => [ "Religious/Faith-Based" ], + "monthly and weekly events for survivors of incest, trafficking, clergy abuse, sexual assault, CSA" => + [ "Sexual Assault", "Human Trafficking", "Child Abuse/Neglect", "Religious/Faith-Based" ], + "Native American and Alaska Native survivors of Domestic Violence, Sexual Assault, Human Sex Trafficking," => + [ "Indigenous/Tribal Nation", "Domestic Violence", "Sexual Assault", "Human Trafficking" ], + "Open community - all residents regardless of age, background, or circumstances who are interested " \ + "in using art as a tool for personal healing and community connection" => [ "Community Building" ], + "Organizations and their staff who work in the primary prevention of intimate partner violence" => + [ "Domestic Violence", "Staff/Organizational Development" ], + "Parents and children of whom we receive a referral for, mostly DCF involved families" => + [ "Foster Care/Adoption", "Child Abuse/Neglect" ], + "Parents of children involved with DCF" => [ "Foster Care/Adoption" ], + "personal development and life coaching" => [ "Self-Care/Personal Growth" ], + "Political Assylum" => [ "Immigration" ], + "Practical and therapeutic support for people struggling with grief, loss, existential distress" => + [ "Grief/Loss", "Mental Health" ], + "Re-Entry/social and community based work" => [ "Incarceration" ], + "Resident low income housing" => [ "Homelessness" ], + "Restorative/Self-Care work with Faculty who teach inside of prisons/jails, as well as currently " \ + "incarcerated students" => [ "Incarceration", "Education", "Self-Care/Personal Growth" ], + "Social services workers" => [ "Staff/Organizational Development" ], + "Staff Burnout and Wellness" => [ "Staff/Organizational Development" ], + "Student counseling groups" => [ "Education", "Mental Health" ], + "Students, families and teachers impacted by the 2025 Los Angeles fires" => + [ "Climate/Environmental", "Education" ], + "Supervised visitation sessions for foster youth with their biological parents" => [ "Foster Care/Adoption" ], + "Supportive Housing" => [ "Homelessness" ], + "Survivors with Disabilities ( Deafness)" => [ "Disability Services" ], + "Urban Native American Health Centers" => [ "Indigenous/Tribal Nation", "Health/Medical" ], + "We are a Native American culturally specific dual sexual assault and domestic violence agency that " \ + "has programming for teens and adults, including elderly" => + [ "Indigenous/Tribal Nation", "Sexual Assault", "Domestic Violence" ], + "Wellness and Prevention Programs and activities" => [ "Self-Care/Personal Growth" ], + "Wellness Center" => [ "Self-Care/Personal Growth" ], + "While training psychology practicum students." => [ "Education", "Staff/Organizational Development" ], + "With DV agencies we support in Ohio and allied professionals addressing wellness practice, " \ + "organizational wellness and to inspire hope and creativity" => + [ "Domestic Violence", "Staff/Organizational Development" ], + "With outreach with students experiencing homelessness. Aisde from my job as a liaison in the " \ + "education field, I also volunteer at homeless shleters for families and youth." => + [ "Homelessness", "Education" ], + "With staff to practice- other: in the communities with families" => [ "Staff/Organizational Development" ], + "With youth in foster care" => [ "Foster Care/Adoption" ], + "work life balance, stress/anxiety" => [ "Self-Care/Personal Growth", "Mental Health" ], + "Working with families to prevent their children entering DCF care, help stabilize kids in " \ + "foster/kinship placements, and with families reunifying with their children." => [ "Foster Care/Adoption" ], + "Workplace (Workshops with Staff)" => [ "Staff/Organizational Development" ], + "Youth who have been through trauma, re-entry youth, foster children, and sexually abused youth and children" => + [ "Foster Care/Adoption", "Child Abuse/Neglect", "Sexual Assault", "Incarceration" ] + }.freeze + + # The same reads, where the term says which ages someone serves. + PROPOSED_AGE_GROUPS = { + "1:1 children and youth who experience domestic violence, dating violence, or sexual abuse" => %w[Children Teens], + "Children, Adults and Families" => %w[Children Teens Adults Elders], + "Children and Families" => %w[Children Teens Adults Elders], + "Children's Advocacy Center Children & Families" => %w[Children], + "Children, Emotional Regulation and structure for leaders" => %w[Children], + "College campus" => %w[Adults], + "Critically ill children" => %w[Children], + "Culturally Specific Youth Program" => %w[Children Teens], + "Day Program for adults with developmental and intellectual disabilities" => %w[Adults], + "Department of Social Services/Children's Services" => %w[Children], + "High level secondary art students" => %w[Teens], + "Supervised visitation sessions for foster youth with their biological parents" => %w[Children Teens], + "We are a Native American culturally specific dual sexual assault and domestic violence agency that " \ + "has programming for teens and adults, including elderly" => %w[Teens Adults Elders], + "With youth in foster care" => %w[Teens], + "youth services" => %w[Children Teens], + "Youth who have been through trauma, re-entry youth, foster children, and sexually abused youth and children" => + %w[Children Teens] + }.freeze + + # Free text that names no tag at all — a venue, a delivery method, or a shrug. Proposed + # rather than decided, since "it means nothing" is a judgment too. + PROPOSED_IGNORED = [ + "Art Events", "at meetings/convenings with partners in prevention", "coaching support services", + "Community based services as well", "Community Leaders and Workshop in the community.", + "Community library program", "Community spaces", + "Community workshops at an art-based non-profit as well collaborative partners", + "Family Resource Centers", "Gardens, parks, forests, beaches, outsise", + "Group facilitation at partner agencies", "In presentations for the community", "Libraries", + "Participants and recipients of social services.", "trainings in the community", + "Workshop / retail shop setting" + ].freeze + def self.call(...) new(...).call end @@ -187,15 +327,18 @@ def call record_age_groups(FILEMAKER_AGE_GROUPS, "filemaker") record_age_groups(STORY_SHARE_AGE_GROUPS, "story_share") record_ignored(IGNORED_TERMS, "filemaker") + record_sectors(PROPOSED_SECTORS, "filemaker", proposed: true) + record_age_groups(PROPOSED_AGE_GROUPS, "filemaker", proposed: true) + record_ignored(PROPOSED_IGNORED, "filemaker", proposed: true) report self end private - def record_sectors(table, source) + def record_sectors(table, source, proposed: false) table.each do |term, names| - Array(names).each { |name| record(term, "Sector", sector_ids[name.downcase], source, name) } + Array(names).each { |name| record(term, "Sector", sector_ids[name.downcase], source, name, proposed:) } end end @@ -205,29 +348,29 @@ def record_retired_sectors(table, source) table.each { |term, name| record(term, "Sector", every_sector_id[name.downcase], source, name) } end - def record_age_groups(table, source) + def record_age_groups(table, source, proposed: false) table.each do |term, names| - names.each { |name| record(term, "Category", age_group_ids[name.downcase], source, name) } + names.each { |name| record(term, "Category", age_group_ids[name.downcase], source, name, proposed:) } end end - def record_ignored(terms, source) + def record_ignored(terms, source, proposed: false) terms.each do |term| alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: nil, aliasable_id: nil) next unless alias_row.new_record? - alias_row.update!(source: source, ignored: true) + alias_row.update!(source: source, ignored: true, proposed: proposed) @created += 1 end end - def record(term, type, id, source, name) + def record(term, type, id, source, name, proposed: false) return @missing << name unless id alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: type, aliasable_id: id) return unless alias_row.new_record? - alias_row.update!(source: source) + alias_row.update!(source: source, proposed: proposed) @created += 1 end @@ -248,7 +391,7 @@ def every_sector_id def report return if @quiet - puts " #{@created} tag aliases added (#{TagAlias.count} recorded)" + puts " #{@created} tag aliases added (#{TagAlias.count} recorded, #{TagAlias.proposed.count} awaiting staff validation)" puts " no published tag named: #{@missing.uniq.sort.join(', ')} — seed the sectors and age ranges first" if @missing.any? end end diff --git a/config/features.yml b/config/features.yml index 04600512e3..f787e493ab 100644 --- a/config/features.yml +++ b/config/features.yml @@ -4578,4 +4578,5 @@ - "A word can be recorded as meaning nothing on purpose — FileMaker's \"Virtually\" described where someone worked, not who they served." - "A retired tag never comes back by accident — but a word can be pointed at one deliberately, which is how FileMaker's \"Poverty\" still finds the sector that already carries it." - "Someone recorded as serving families is tagged with every age range, since that's what serving families means." + - "Where an old answer was typed by hand, the reading of it is marked as proposed, so staff can check the list and correct anything that reads wrong." - "The mapping is written up in ADR-0007 for anyone who needs to see the whole picture." diff --git a/db/migrate/20261015013218_create_tag_aliases.rb b/db/migrate/20261015013218_create_tag_aliases.rb index 10ef3677af..b198dbdfdf 100644 --- a/db/migrate/20261015013218_create_tag_aliases.rb +++ b/db/migrate/20261015013218_create_tag_aliases.rb @@ -5,6 +5,8 @@ def up # A term can be recorded as deliberately naming nothing, so aliasable is optional. t.references :aliasable, polymorphic: true, null: true, index: false t.boolean :ignored, default: false, null: false + # A semantic read of the archive's free text, awaiting staff validation. + t.boolean :proposed, default: false, null: false t.string :source, null: false t.integer :created_by_id t.integer :updated_by_id diff --git a/db/schema.rb b/db/schema.rb index 84d0c39464..8b3ad43ff5 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -2152,6 +2152,7 @@ t.integer "created_by_id" t.boolean "ignored", default: false, null: false t.string "name", null: false + t.boolean "proposed", default: false, null: false t.string "source", null: false t.datetime "updated_at", null: false t.integer "updated_by_id" diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md index f7466196eb..eb37b19fc3 100644 --- a/docs/adr/0007-legacy-sector-and-age-group-translation.md +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -144,6 +144,19 @@ answers, `OtherServiceArea`, `Settings` and `LifeExperiences` the additional one `Children's Services` makes Children the primary age range, and `Elder Abuse` adds Elders alongside whatever the primary already is. +### Reads of the free text, pending staff validation + +The archive's hand-typed cells are mostly a venue, a programme name or a sentence about the +work. Where the meaning is plain — `Immigration Clinic`, `Grief support group`, +`Inpatient Psychiatric Hopital`, `Supervised visitation sessions for foster youth…` — the +term is recorded as **proposed**: it resolves, so an import doesn't lose it, and +`TagAlias.proposed` is the list to walk with staff term by term. About 90 terms, each worth +one or two archive rows. + +**What is deliberately not read:** anything naming an organization, a place that identifies +one, or a facilitator's own circumstances. That text is personal and belongs in the archive, +not copied into a catalogue — which leaves 23 terms, 23 mentions, unresolved for good. + ### Answers that name nothing Alongside the dropped Settings, these are recorded as naming nothing on purpose: `Other` · @@ -262,13 +275,14 @@ Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct valu It is recorded as ignored rather than translated: `Sector::OTHER_SECTOR_NAME` is offered for additional sectors but never as a primary, so an import can't honour it as one, and "they picked Other" says nothing a tag could carry. -- **Still needing a decision** — only the long tail of one-off free text (≈90 `Settings` - values, 42 `OtherServiceArea` values, almost all a single mention, mostly venues, program - names and prose rather than anything a tag could carry). The unmatched report is the - queue, and it stays short enough to read. -- **Coverage.** Every archive term with more than two mentions is now either translated or - deliberately untranslated: ~17,600 of the 17,757 mentions. What's left is ~130 values at - one or two mentions each. +- **Still needing a decision** — 23 terms, one mention each, every one naming an + organization, a place that identifies one, or a facilitator's own circumstances. They stay + in the archive and show up in the unmatched report; nothing is copied out of them. +- **The proposed reads need a pass with staff** before the real import run. They resolve + meanwhile, so the work isn't blocked, and a wrong one is a row to delete rather than a + deploy. +- **Coverage.** 17,734 of the archive's 17,757 mentions are translated, deliberately + untranslated, or proposed. - **Seen in the data, no mapping decided:** Children's Services, Intellectual Disabilities, Maternal wellness and postpartum support, Political Asylum, Survivors with Disabilities, Arts Education & Community Art Making, and the rest of the one-off free text. Deliberately diff --git a/spec/services/tag_alias_seeder_spec.rb b/spec/services/tag_alias_seeder_spec.rb index 84fe874dc1..601271f337 100644 --- a/spec/services/tag_alias_seeder_spec.rb +++ b/spec/services/tag_alias_seeder_spec.rb @@ -96,6 +96,34 @@ expect(TagAlias.find_by(name: "Grief and loss").source).to eq("portal") end + it "marks a read of free text as proposed, and a decision as decided" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Immigration Clinic")).to have_attributes(proposed: true, ignored: false) + expect(TagAlias.find_by(name: "Libraries")).to have_attributes(proposed: true, ignored: true) + expect(TagAlias.find_by(name: "Student Services").proposed).to be false + expect(TagAlias.find_by(name: "Virtually").proposed).to be false + expect(TagAlias.proposed.count).to be > 50 + end + + it "resolves a proposed read, so an import doesn't lose it while staff review" do + described_class.call(quiet: true) + + translation = TagTranslation.new + tags = translation.resolve("Immigration Clinic") + expect(tags.sector_ids).to eq([ Sector.find_by(name: "Immigration").id ]) + expect(translation.resolve("Libraries").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "reads a free-text answer that names several sectors and the ages it serves" do + described_class.call(quiet: true) + + tags = TagTranslation.new.resolve("1:1 children and youth who experience domestic violence, dating violence, or sexual abuse") + expect(tags.sector_ids).to match_array(Sector.where(name: [ "Domestic Violence", "Sexual Assault", "Child Abuse/Neglect" ]).ids) + expect(tags.age_group_ids).to match_array(Category.where(name: %w[Children Teens]).ids) + end + it "adds nothing on a second run" do described_class.call(quiet: true)