diff --git a/AGENTS.md b/AGENTS.md index 25548a9fdf..7209978b43 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,8 +48,8 @@ This codebase (Rails 8.1) | Directory | Purpose | Count | |---|---|---| -| `app/models/` | ActiveRecord models | ~90 files | -| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~73 files | +| `app/models/` | ActiveRecord models | ~144 files | +| `app/services/` | Service objects and POROs (e.g. `MoneyFormatter` for currency display, `StoryImporter` for the stories spreadsheet import) | ~118 files | | `app/jobs/` | SolidQueue background jobs | 6 files | | `app/models/concerns/` | Shared model modules | 23 concerns | @@ -271,6 +271,8 @@ action, or `authorize! :workshop, to: :summary?`). - `EventRegistrationServices::ProcessConfirmation` — Registration confirmation flow - `RegistrantCeForm` — The CE-specific view over the inline form the **CE callout** carries (built on the generic callout→form mechanism: `RegistrationTicketCallout#forms` + `CalloutFormSubmission`; the CE callout carries a single form, read as `forms.first`). Adds the two things CE needs: the post-training form only surfaces once `EventRegistration#ce_signouts_complete?` (event ended + no open attendance entry), and completing it — read-time `complete?`, every required field answered — is a prerequisite for the CE certificate (`EventRegistration#ce_form_requirement_met?` gates `ContinuingEducationRegistration#certificate_available?`). Read by the CE callout page (which links to the generic `registration/:slug/forms/:callout_id` form page, gated so the CE callout's form opens only after sign-outs via `CalloutsController#ce_form_locked?`) and by `BuiltinCalloutCards#ce_form_pending?` for the CE card's "Complete your form" nudge - `EventRegistrationServices::CalloutFormSubmission` — Records a registrant's answers to one of the forms a registration-ticket callout delivers inline (`call(registration:, callout:, form:, form_params:, clarity_params:)`, returns the service instance with `#submission`/`#profile_changes`). Find-or-creates one `FormSubmission` per `(registrant, form, event)` carrying the form's own role — a re-submit edits the answers in place via `FormSubmission#persist_answer`. That a callout collected it is recorded separately in `FormSubmission#metadata` (`collected_via: "callout"`, `collected_via_callout_id`) via `#record_callout_collection!` — the flag (`#collected_via_callout?`), not the role, tells a callout-collected submission apart. There is no survey-role flag — the extra side effects self-gate on the form's own fields, so they no-op on a form that doesn't ask those questions: a choice field with linked `FormFieldResource`s (`FormField#per_resource?`) fans out to one answer per resource (nil `form_field`, `FormField#per_resource_question` snapshotted in `question_name_when_answered` — subtitle-led, so two fan-out questions sharing a prompt can't collide on one answer row); fields carrying the `anonymous_contributions` / `display_name_preference` identifiers write through to the `Person` via `PersonServices::SyncSharingPreferences` (the same write-through the registration form runs), exposing `#profile_changes` so the controller Ahoy-tracks real edits; and submitting any form on the **`scholarship`** callout stamps a recipient's `post_survey_completed_at`. Drives `Events::CalloutsController#callout`/`#submit_callout` (the person-first `registration/:slug/forms/:callout_id` page and its `.../:form_id` submit) +- `TagTranslation` — Turns a legacy word for who someone serves into tags: `#resolve` answers with both the sector ids and the age-group ids a term means, since the vocabularies cross (`Elder Abuse` is a life experience that means the Elders *age range*). A tag's own name resolves only while it's **published** — a retired tag is unpublished, not deleted — but a recorded `TagAlias` may point at a retired one deliberately (FileMaker's `Poverty`). Splits a repeating field on line breaks and semicolons — **not commas**, which the archive's own wording contains — and drops a trailing parenthetical. Built once and reused (it reads the whole vocabulary up front); a term nothing answers to lands in `#unmatched` rather than a guess. Primacy is the caller's call, since it follows from the field a term sat in +- `TagAliasSeeder` — Records what the FileMaker archive, the older registration/scholarship forms, Story Share and the portal's earlier sector list each called a tag the portal now names its own way (**ADR-0007**) as `TagAlias` rows, so the translation is data an admin can extend and a record of what a tag used to be called. Terms AWBW decided are recorded as-is; semantic reads of the archive's hand-typed free text are marked **`proposed`** for a pass with staff (they still resolve, so no import waits on it), and text naming an organization or a facilitator's own circumstances is left alone. Idempotent on (term, tag) and never touches a term added by hand; a term whose tag isn't published is reported, not created. Shared with `data:seed_tag_aliases` - `SurveyFormSeeder` — Creates the four standalone survey template forms from the form-builder presets (the Day 1 / Day 2 evaluations, the overall post-event survey, and the scholarship recipients survey), idempotent on form name and returning the names created. `#link_fanout_resources` then links each fan-out question to its training-topic `Resource`s by title (skipping any that aren't present). Shared by `db/seeds.rb` and the `data:seed_survey_forms` task so prod can create them without the full seed - `CloseProgramFormSeeder` — Creates the standalone "Close Program" agreement form (role `close_program`, sections `person_identifier` + `close_program`) from the form-builder presets, idempotent on form name, returning the name created (or nil). Built as an unpublished template for staff to slug + publish. Shared by `db/seeds.rb` and the `data:seed_close_program_form` task - `EventRegistrationServices::PublicRegistration` — Public registration handling. An affirmative `communication_consent` answer captures a News (mailing-list) `TopicSubscription` via `NewsSubscriptionCapture`, sourced to the event. A resubmission matched to an active registration sends the registrant and admins DUPLICATE emails (`event_registration_duplicate` / `event_registration_duplicate_fyi`) @@ -561,7 +563,7 @@ RuboCop linting on PRs and pushes to main. ## Rake Tasks -Located in `lib/tasks/` (13 files): +Located in `lib/tasks/` (16 files): - `dev.rake` — Development database seeding from XML/CSV - `rhino_migrator.rake` — Rich text editor migration - `attachment_report.rake` — Attachment reporting @@ -573,6 +575,8 @@ Located in `lib/tasks/` (13 files): - `migrate_workshop_logs.rake` — Workshop log migration - `backfill_user_stamps.rake` — Backfill `created_by_id`/`updated_by_id` on legacy rows from the Ahoy trail (`data:backfill_user_stamps`) - `backfill_notification_stamps.rake` — Backfill notifications' `created_by_id`/`updated_by_id` from `sender_id`, nil-only (`data:backfill_notification_stamps`) +- `fm_import.rake` — Load the FileMaker CSV exports into the `fm_*` archive tables (`fm:import`, reads each CSV from the `FM Archive` resources, batch `insert_all`, skips rows whose key is already stored) - `resolve_resource_organizations.rake` — Link resources to organizations from the legacy `agency` free-text name (`data:resolve_resource_organizations`, match-or-create, idempotent, DRY_RUN) +- `seed_tag_aliases.rake` — Record the legacy sector and age-group terms from ADR-0007 as tag aliases (`data:seed_tag_aliases`, via `TagAliasSeeder`, idempotent; also part of `db:seed`) - `seed_survey_forms.rake` — Create the Day 1 / Day 2 / recipients survey template forms without the full db:seed (`data:seed_survey_forms`, via `SurveyFormSeeder`, idempotent on form name; run before configuring the survey callouts) - `seed_close_program_form.rake` — Create the Close Program agreement form without the full db:seed (`data:seed_close_program_form`, via `CloseProgramFormSeeder`, idempotent on form name; give it a slug and publish it in the form builder to use it) diff --git a/app/models/tag_alias.rb b/app/models/tag_alias.rb new file mode 100644 index 0000000000..91f3d1ec07 --- /dev/null +++ b/app/models/tag_alias.rb @@ -0,0 +1,46 @@ +# A word some other system used for a tag the portal now carries its own name for. The +# FileMaker archive, the older registration and scholarship forms, Story Share and the +# portal's own earlier sector list each had their own wording; ADR-0007 records what each +# term resolves to, and these rows are that mapping as data — so an admin can add a term +# as it turns up instead of waiting for a deploy, and the record of what a tag used to be +# called outlives the import that needed it. +# +# A term can name two tags ("C & T" is both Children and Teens), so it is read as a list. +# A term read out of free text rather than decided by AWBW is marked `proposed`, so the +# guesses can be reviewed as a list without holding up the import. +# +# It can also be recorded as naming nothing on purpose — a FileMaker "setting" like +# "Virtually" says where someone works, not who they serve — which is what keeps a decided +# non-translation out of the queue of terms still needing a decision. +class TagAlias < ApplicationRecord + SOURCES = %w[filemaker forms story_share portal].freeze + + belongs_to :aliasable, polymorphic: true, optional: true + belongs_to :created_by, class_name: "User", optional: true + belongs_to :updated_by, class_name: "User", optional: true + + normalizes :name, with: ->(value) { value.to_s.squish } + + validates :name, presence: true, length: { maximum: 255 }, + uniqueness: { scope: %i[aliasable_type aliasable_id], case_sensitive: false } + validates :source, inclusion: { in: SOURCES } + validate :names_a_tag_or_is_ignored + + scope :ignored, -> { where(ignored: true) } + scope :resolving, -> { where(ignored: false) } + # Read from the archive's free text rather than decided by AWBW. They resolve, so an + # import doesn't lose them, and this is the list to walk with staff. + scope :proposed, -> { where(proposed: true) } + scope :decided, -> { where(proposed: false) } + + scope :for_sectors, -> { where(aliasable_type: "Sector") } + scope :for_categories, -> { where(aliasable_type: "Category") } + + private + + def names_a_tag_or_is_ignored + return if aliasable.present? ^ ignored? + + errors.add(:base, "A term either names a tag or is ignored, not both and not neither") + end +end diff --git a/app/services/tag_alias_seeder.rb b/app/services/tag_alias_seeder.rb new file mode 100644 index 0000000000..62598ea454 --- /dev/null +++ b/app/services/tag_alias_seeder.rb @@ -0,0 +1,397 @@ +# Records what each legacy word for a sector or age group resolves to, as TagAlias rows. +# The tables are ADR-0007; this is them as data, so an import can look a term up and a +# reader can still see what a tag used to be called elsewhere. Idempotent: a term already +# recorded for a tag is left alone, including whatever an admin has added by hand. +# +# A term the portal already answers to by name needs no row — the lookup matches a tag's +# own name first, case-insensitively, and drops a trailing parenthetical. Only wording +# that differs is recorded here. +class TagAliasSeeder + # The archive's own three fields. Settings describes where a facilitator works, so only + # the ones that also say who they serve resolve; the rest are in DROPPED_SETTINGS. + FILEMAKER_SECTORS = { + "Community Oppression/Violence" => "Community Violence", + "Criminal/Legal" => "Court/Legal System", + "Foster Care" => "Foster Care/Adoption", + "Student Services" => "Education", + "Substance Abuse" => "Substance Use/Recovery", + "Clinical Setting" => "Mental Health", + "Educational Setting" => "Education", + "Faith-Based Setting" => "Religious/Faith-Based", + "Hospitals" => "Health/Medical", + "Law Enforcement/Court" => "Court/Legal System", + "Prisons/Jails" => "Incarceration", + "Private Practice" => "Private Practice/Sole Proprietor", + "With Staff" => "Staff/Organizational Development", + "school" => "Education", + "Students" => "Education", + "Climate/Environmental Trauma" => "Climate/Environmental", + "Illness" => "Health/Medical", + "Pandemic-Related Stress" => "Health/Medical", + "Mental Health Needs" => "Mental Health", + "Suicidality" => "Mental Health", + "Military/Veteran" => "Military/Veterans", + "Oppression Against LGBTQIA+" => "LGBTQIA+", + "People Who Do Harm/Batterers" => "Batterers Intervention", + "Racism and Marginalization" => "Racial/Social Justice", + "Religious Trauma" => "Religious/Faith-Based", + "Secondary/Vicarious Trauma" => "Staff/Organizational Development", + "Student-Related Stress" => "Education", + "Victims of Crime" => "Court/Legal System", + "LGBTQIA +" => "LGBTQIA+" + }.freeze + + # The forms asked the same questions in prose. Their labels carry a "services" suffix or + # different wording, so each needs its own term; the parentheticals resolve on their own. + FORM_SECTORS = { + "Community oppression/violence services" => "Community Violence", + "Criminal/legal services" => "Court/Legal System", + "Disability services" => "Disability Services", + "Domestic violence services" => "Domestic Violence", + "Foster care/adoption services" => "Foster Care/Adoption", + "Homeless services" => "Homelessness", + "Human trafficking services" => "Human Trafficking", + "Immigration services" => "Immigration", + "Incarceration services" => "Incarceration", + "Indigenous/tribal nation services" => "Indigenous/Tribal Nation", + "LGBTQIA+ services" => "LGBTQIA+", + "Mental health services" => "Mental Health", + "Reproductive services" => "Reproductive Services", + "Restorative/transformative justice services" => "Restorative/Transformative Justice", + "Sexual assault services" => "Sexual Assault", + "Student services" => "Education", + "Substance use services" => "Substance Use/Recovery", + "Substance use" => "Substance Use/Recovery", + "Foster care/adoption" => "Foster Care/Adoption", + "Law enforcement/court/legal" => "Court/Legal System" + }.freeze + + # Hand-typed cells that name more than one sector, so each resolves to all of them — + # the same one-to-many shape as "C & T". Only recorded where every concept in the cell is + # already a decided term, so nothing new is being inferred. + COMPOUND_SECTORS = { + "Domestic Violence & Homelessness" => [ "Domestic Violence", "Homelessness" ], + "Sexual Assault & Domestic Violence" => [ "Sexual Assault", "Domestic Violence" ], + "Domestic Violence, Sexual Assault" => [ "Domestic Violence", "Sexual Assault" ], + "Domestic Violence and Sexual Assault survivors" => [ "Domestic Violence", "Sexual Assault" ], + "Domestic and Sexual Violence Families" => [ "Domestic Violence", "Sexual Assault" ], + "Sexual and Domestic Violence Agency" => [ "Sexual Assault", "Domestic Violence" ], + "DV/SA/other related crimes" => [ "Domestic Violence", "Sexual Assault" ], + "Dual Agency Domestic violence/Sexual assault/ Human trafficking" => + [ "Domestic Violence", "Sexual Assault", "Human Trafficking" ], + "Immigration, Health, Prevention of Violence and Education" => [ "Immigration", "Health/Medical", "Education" ], + "Addiction, Native Americans, mental health and addiction" => + [ "Substance Use/Recovery", "Indigenous/Tribal Nation", "Mental Health" ], + "Education, SEL needs" => [ "Education" ], + "Hospice, end of life, bereavement" => [ "Grief/Loss" ] + }.freeze + + # A retired sector an alias deliberately points at. The term is in the archive 634 times + # and the tag it means still exists, unpublished, carrying the taggings it already had — + # so the term resolves even though nothing new can be tagged with it by name. + RETIRED_SECTORS = { "Poverty" => "Poverty" }.freeze + + STORY_SHARE_SECTORS = { + "Advocacy" => "Systems/Policy Change", + "Child Abuse / Neglect" => "Child Abuse/Neglect", + "Community Engagement" => "Community Building", + "Court & Legal System" => "Court/Legal System", + "Donor Engagement" => "Fundraising/Donor Engagement", + "Foster Care" => "Foster Care/Adoption", + "Grief & Loss" => "Grief/Loss", + "Military & Veterans" => "Military/Veterans", + "Physical & Terminal Illness" => "Health/Medical", + "Schools & Universities" => "Education", + "Self-Care & Personal Growth" => "Self-Care/Personal Growth", + "Social Justice" => "Racial/Social Justice", + "Staff & Organizational Development" => "Staff/Organizational Development", + "Substance Abuse Recovery" => "Substance Use/Recovery" + }.freeze + + # Names the portal itself used before the 2026 consolidation renamed and merged them. + PORTAL_SECTORS = { + "Child abuse" => "Child Abuse/Neglect", + "Children of Domestic Violence Survivors" => "Domestic Violence", + "Community oppression/violence" => "Community Violence", + "Criminal/legal" => "Court/Legal System", + "Disabled" => "Disability Services", + "Disabled Person" => "Disability Services", + "Disability" => "Disability Services", + "Education/schools" => "Education", + "Family Violence" => "Domestic Violence", + "Foster children/APPS" => "Foster Care/Adoption", + "Foster Youth" => "Foster Care/Adoption", + "Grief and loss" => "Grief/Loss", + "Mental illness" => "Mental Health", + "Reproductive" => "Reproductive Services", + "Serious illness" => "Health/Medical", + "Staff & organizational development" => "Staff/Organizational Development", + "Student" => "Education", + "Substance use" => "Substance Use/Recovery", + "Veterans & military" => "Military/Veterans" + }.freeze + + # A term can name several age groups, so these are lists: serving families is serving + # every age, and a person has no population tag to carry "Families" on its own. + # + # Some of these terms come from the *sector* fields, because the vocabularies cross — a + # life experience of elder abuse says the person serves elders, and a service area of + # children's services says they serve children. Whether the tag lands as primary or + # additional follows from the field the term sat in, not from the term. + FILEMAKER_AGE_GROUPS = { + "Families" => %w[Children Teens Adults Elders], + "Elder Abuse" => %w[Elders], + "Children's Services" => %w[Children], + "C & T" => %w[Children Teens], + "Children & Teens" => %w[Children Teens], + "Adults & Children" => %w[Adults Children], + "Adults 18+" => %w[Adults], + "18+" => %w[Adults], + "13-17" => %w[Teens], + "6-12" => %w[Children], + "3-5" => %w[Children], + "Young Children" => %w[Children] + }.freeze + + STORY_SHARE_AGE_GROUPS = { "Adult" => %w[Adults] }.freeze + + # Terms that name no tag on purpose: a setting that says where or how someone works + # rather than who they serve, an answer that says nothing at all, and one life experience + # (war) that AWBW chose to leave untagged. Recorded so they don't sit in the queue of + # terms still needing a decision. + IGNORED_TERMS = [ + "Community-Based", "Events and Conferences", "Events & Conferences", "Home Visits", + "Outreach Program", "Residential Program", "Virtually", "Other", "Did not specify", + "Residential", "In person", "In visits", "virtually & in-person", "At office", "Office space.", + "On-Call/Crisis", "Support groups", "n/a", "none", "something", "unsure", "Everyone", "Technology", + "People Who Have Experienced War" + ].freeze + + # ─── Proposed, pending staff validation ───────────────────────────────────────────── + # + # Semantic reads of the archive's hand-typed free text, not decisions AWBW has made. They + # are seeded as `proposed`, so they resolve — an import shouldn't lose them — and + # `TagAlias.proposed` is the list to walk with staff, term by term. + # + # Only terms whose meaning is plain are here. Anything naming an organization, a place + # that identifies one, or a facilitator's own circumstances is left out: that text is + # personal, and it belongs in the archive rather than copied into a catalogue. + PROPOSED_SECTORS = { + "1:1 children and youth who experience domestic violence, dating violence, or sexual abuse" => + [ "Domestic Violence", "Sexual Assault", "Child Abuse/Neglect" ], + "a regularly meting group with a drop in attendance from the local queer community" => [ "LGBTQIA+" ], + "Advocacy for survivors of domestic and sexual violence" => [ "Domestic Violence", "Sexual Assault" ], + "Arts Education & Community Art Making" => [ "Education", "Community Building" ], + "bereavement and complex care in hospital setting" => [ "Grief/Loss", "Health/Medical" ], + "Cancer Survivorship" => [ "Health/Medical" ], + "Cases opened by Child protective services" => [ "Child Abuse/Neglect" ], + "Children's Advocacy Center Children & Families" => [ "Child Abuse/Neglect" ], + "College campus" => [ "Education" ], + "Community Morale" => [ "Community Building" ], + "Community-based, youth diversion, juvenile hall" => [ "Incarceration", "Court/Legal System" ], + "Critically ill children" => [ "Health/Medical" ], + "Day Program for adults with developmental and intellectual disabilities" => [ "Disability Services" ], + "Disaster Recovery - Wellness through Trauma informed Activities" => [ "Climate/Environmental" ], + "Domestic Violence Shelter" => [ "Domestic Violence" ], + "Domestic Violence, Sexual Violence and Stalking Advocacy Center" => [ "Domestic Violence", "Sexual Assault" ], + "DV Resource Center" => [ "Domestic Violence" ], + "Empowerment community groups" => [ "Community Building" ], + "Environmental education setting" => [ "Climate/Environmental", "Education" ], + "First Responders. doing workshops in Fire Stations and Law Enforcement Offices" => + [ "Staff/Organizational Development" ], + "formerly incarcerated/formerly gang related" => [ "Incarceration", "Community Violence" ], + "Foster Care agencies" => [ "Foster Care/Adoption" ], + "Grief support group" => [ "Grief/Loss" ], + "Helping Professionals, Caregivers, Leaders" => [ "Staff/Organizational Development" ], + "High level secondary art students" => [ "Education" ], + "I intend to service a diverse population experiencing, DV, sexual assault, and homelessness" => + [ "Domestic Violence", "Sexual Assault", "Homelessness" ], + "Immigration Clinic" => [ "Immigration" ], + "In DV shelters" => [ "Domestic Violence" ], + "Individual and Group counseling with students. In my office and in classrooms." => + [ "Education", "Mental Health" ], + "Individual counseling sessions with sexual assault survivors." => [ "Sexual Assault" ], + "Inpatient Psychiatric Hopital" => [ "Mental Health", "Health/Medical" ], + "Intellectual Disabilities" => [ "Disability Services" ], + "Intensive Outpatient programs" => [ "Mental Health" ], + "Interpersonal violence and any gender based harassment and discrimination" => + [ "Domestic Violence", "Sexual Assault" ], + "LA Fire imacted Community Members in Palisades and Altadena" => [ "Climate/Environmental" ], + "Life Coaching" => [ "Self-Care/Personal Growth" ], + "Low-income housing /vulnerable population" => [ "Homelessness" ], + "Low-income housing support services" => [ "Homelessness" ], + "Maternal wellness and postpartum support" => [ "Reproductive Services" ], + "Ministry Events" => [ "Religious/Faith-Based" ], + "monthly and weekly events for survivors of incest, trafficking, clergy abuse, sexual assault, CSA" => + [ "Sexual Assault", "Human Trafficking", "Child Abuse/Neglect", "Religious/Faith-Based" ], + "Native American and Alaska Native survivors of Domestic Violence, Sexual Assault, Human Sex Trafficking," => + [ "Indigenous/Tribal Nation", "Domestic Violence", "Sexual Assault", "Human Trafficking" ], + "Open community - all residents regardless of age, background, or circumstances who are interested " \ + "in using art as a tool for personal healing and community connection" => [ "Community Building" ], + "Organizations and their staff who work in the primary prevention of intimate partner violence" => + [ "Domestic Violence", "Staff/Organizational Development" ], + "Parents and children of whom we receive a referral for, mostly DCF involved families" => + [ "Foster Care/Adoption", "Child Abuse/Neglect" ], + "Parents of children involved with DCF" => [ "Foster Care/Adoption" ], + "personal development and life coaching" => [ "Self-Care/Personal Growth" ], + "Political Assylum" => [ "Immigration" ], + "Practical and therapeutic support for people struggling with grief, loss, existential distress" => + [ "Grief/Loss", "Mental Health" ], + "Re-Entry/social and community based work" => [ "Incarceration" ], + "Resident low income housing" => [ "Homelessness" ], + "Restorative/Self-Care work with Faculty who teach inside of prisons/jails, as well as currently " \ + "incarcerated students" => [ "Incarceration", "Education", "Self-Care/Personal Growth" ], + "Social services workers" => [ "Staff/Organizational Development" ], + "Staff Burnout and Wellness" => [ "Staff/Organizational Development" ], + "Student counseling groups" => [ "Education", "Mental Health" ], + "Students, families and teachers impacted by the 2025 Los Angeles fires" => + [ "Climate/Environmental", "Education" ], + "Supervised visitation sessions for foster youth with their biological parents" => [ "Foster Care/Adoption" ], + "Supportive Housing" => [ "Homelessness" ], + "Survivors with Disabilities ( Deafness)" => [ "Disability Services" ], + "Urban Native American Health Centers" => [ "Indigenous/Tribal Nation", "Health/Medical" ], + "We are a Native American culturally specific dual sexual assault and domestic violence agency that " \ + "has programming for teens and adults, including elderly" => + [ "Indigenous/Tribal Nation", "Sexual Assault", "Domestic Violence" ], + "Wellness and Prevention Programs and activities" => [ "Self-Care/Personal Growth" ], + "Wellness Center" => [ "Self-Care/Personal Growth" ], + "While training psychology practicum students." => [ "Education", "Staff/Organizational Development" ], + "With DV agencies we support in Ohio and allied professionals addressing wellness practice, " \ + "organizational wellness and to inspire hope and creativity" => + [ "Domestic Violence", "Staff/Organizational Development" ], + "With outreach with students experiencing homelessness. Aisde from my job as a liaison in the " \ + "education field, I also volunteer at homeless shleters for families and youth." => + [ "Homelessness", "Education" ], + "With staff to practice- other: in the communities with families" => [ "Staff/Organizational Development" ], + "With youth in foster care" => [ "Foster Care/Adoption" ], + "work life balance, stress/anxiety" => [ "Self-Care/Personal Growth", "Mental Health" ], + "Working with families to prevent their children entering DCF care, help stabilize kids in " \ + "foster/kinship placements, and with families reunifying with their children." => [ "Foster Care/Adoption" ], + "Workplace (Workshops with Staff)" => [ "Staff/Organizational Development" ], + "Youth who have been through trauma, re-entry youth, foster children, and sexually abused youth and children" => + [ "Foster Care/Adoption", "Child Abuse/Neglect", "Sexual Assault", "Incarceration" ] + }.freeze + + # The same reads, where the term says which ages someone serves. + PROPOSED_AGE_GROUPS = { + "1:1 children and youth who experience domestic violence, dating violence, or sexual abuse" => %w[Children Teens], + "Children, Adults and Families" => %w[Children Teens Adults Elders], + "Children and Families" => %w[Children Teens Adults Elders], + "Children's Advocacy Center Children & Families" => %w[Children], + "Children, Emotional Regulation and structure for leaders" => %w[Children], + "College campus" => %w[Adults], + "Critically ill children" => %w[Children], + "Culturally Specific Youth Program" => %w[Children Teens], + "Day Program for adults with developmental and intellectual disabilities" => %w[Adults], + "Department of Social Services/Children's Services" => %w[Children], + "High level secondary art students" => %w[Teens], + "Supervised visitation sessions for foster youth with their biological parents" => %w[Children Teens], + "We are a Native American culturally specific dual sexual assault and domestic violence agency that " \ + "has programming for teens and adults, including elderly" => %w[Teens Adults Elders], + "With youth in foster care" => %w[Teens], + "youth services" => %w[Children Teens], + "Youth who have been through trauma, re-entry youth, foster children, and sexually abused youth and children" => + %w[Children Teens] + }.freeze + + # Free text that names no tag at all — a venue, a delivery method, or a shrug. Proposed + # rather than decided, since "it means nothing" is a judgment too. + PROPOSED_IGNORED = [ + "Art Events", "at meetings/convenings with partners in prevention", "coaching support services", + "Community based services as well", "Community Leaders and Workshop in the community.", + "Community library program", "Community spaces", + "Community workshops at an art-based non-profit as well collaborative partners", + "Family Resource Centers", "Gardens, parks, forests, beaches, outsise", + "Group facilitation at partner agencies", "In presentations for the community", "Libraries", + "Participants and recipients of social services.", "trainings in the community", + "Workshop / retail shop setting" + ].freeze + + def self.call(...) + new(...).call + end + + def initialize(quiet: false) + @quiet = quiet + @created = 0 + @missing = [] + end + + def call + record_sectors(FILEMAKER_SECTORS, "filemaker") + record_sectors(COMPOUND_SECTORS, "filemaker") + record_retired_sectors(RETIRED_SECTORS, "filemaker") + record_sectors(FORM_SECTORS, "forms") + record_sectors(STORY_SHARE_SECTORS, "story_share") + record_sectors(PORTAL_SECTORS, "portal") + record_age_groups(FILEMAKER_AGE_GROUPS, "filemaker") + record_age_groups(STORY_SHARE_AGE_GROUPS, "story_share") + record_ignored(IGNORED_TERMS, "filemaker") + record_sectors(PROPOSED_SECTORS, "filemaker", proposed: true) + record_age_groups(PROPOSED_AGE_GROUPS, "filemaker", proposed: true) + record_ignored(PROPOSED_IGNORED, "filemaker", proposed: true) + report + self + end + + private + + def record_sectors(table, source, proposed: false) + table.each do |term, names| + Array(names).each { |name| record(term, "Sector", sector_ids[name.downcase], source, name, proposed:) } + end + end + + # Resolved against every sector, published or not: the point of the table is the ones that + # aren't, which the published lookup would report as missing. + def record_retired_sectors(table, source) + table.each { |term, name| record(term, "Sector", every_sector_id[name.downcase], source, name) } + end + + def record_age_groups(table, source, proposed: false) + table.each do |term, names| + names.each { |name| record(term, "Category", age_group_ids[name.downcase], source, name, proposed:) } + end + end + + def record_ignored(terms, source, proposed: false) + terms.each do |term| + alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: nil, aliasable_id: nil) + next unless alias_row.new_record? + + alias_row.update!(source: source, ignored: true, proposed: proposed) + @created += 1 + end + end + + def record(term, type, id, source, name, proposed: false) + return @missing << name unless id + + alias_row = TagAlias.find_or_initialize_by(name: term.squish, aliasable_type: type, aliasable_id: id) + return unless alias_row.new_record? + + alias_row.update!(source: source, proposed: proposed) + @created += 1 + end + + # Published only, the same vocabulary TagTranslation resolves against, so a term aimed at + # a retired tag is reported rather than recorded as an alias that never resolves. + def sector_ids + @sector_ids ||= Sector.published.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + + def age_group_ids + @age_group_ids ||= Category.age_ranges.published.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + + def every_sector_id + @every_sector_id ||= Sector.pluck(:name, :id).to_h { |name, id| [ name.downcase, id ] } + end + + def report + return if @quiet + + puts " #{@created} tag aliases added (#{TagAlias.count} recorded, #{TagAlias.proposed.count} awaiting staff validation)" + puts " no published tag named: #{@missing.uniq.sort.join(', ')} — seed the sectors and age ranges first" if @missing.any? + end +end diff --git a/app/services/tag_translation.rb b/app/services/tag_translation.rb new file mode 100644 index 0000000000..cac6792903 --- /dev/null +++ b/app/services/tag_translation.rb @@ -0,0 +1,80 @@ +# Turns a legacy word for who someone serves into the portal's own tags: the archive's +# service areas and settings, the older forms' prose labels, Story Share's categories, and +# the portal's earlier sector names. A term resolves by a tag's own name or by any +# TagAlias recorded for it (ADR-0007), and one term can name several tags. +# +# A tag resolves **by name only while it's published**. A retired tag is unpublished rather +# than deleted, so its old taggings survive, and matching one by name would let an import +# hand it to hundreds more records by accident. A recorded alias is the opposite — someone +# decided that term means that tag — so an alias may point at a retired tag and still +# resolve. FileMaker's "Poverty" works that way: the sector it means is retired, and 634 +# archive rows name it. +# +# A term is read without regard for the field it sat in, because the two vocabularies cross: +# FileMaker's "Elder Abuse" is a life experience that means the Elders *age range*, and +# "Children's Services" is a service area that means the Children one. So #resolve answers +# with both kinds and the caller takes what it needs — and whether a tag lands as primary or +# additional stays the caller's call, since that follows from the field, not the term. +# +# Built once and reused — it reads the whole sector and age-group vocabulary up front, so +# a 39,000-row import doesn't query per value. A term nothing answers to is counted in +# #unmatched rather than guessed at — unless it's recorded as ignored, which is how a +# decided non-translation stays out of the queue of terms still needing a decision. +class TagTranslation + Tags = Data.define(:sector_ids, :age_group_ids) + + # FileMaker repeating fields separate values with line breaks, and a hand-typed list + # uses semicolons. Commas are not separators: the wording itself contains them, inside + # parentheticals like "Community violence (gang violence, mass shootings, etc.)". + SEPARATORS = /[;\r\n]+/ + # The clarifying tail a form added to a term, dropped when the term alone resolves. + PARENTHETICAL = /\s*\([^)]*\)?\s*\z/ + + attr_reader :unmatched + + def initialize + @sectors = vocabulary(Sector.published) + @age_groups = vocabulary(Category.age_ranges.published) + @ignored = TagAlias.ignored.pluck(:name).map { |name| key(name) }.to_set + @unmatched = Hash.new(0) + end + + def resolve(value) + sector_ids = [] + age_group_ids = [] + + terms(value).each do |term| + sectors = lookup(@sectors, term) + age_groups = lookup(@age_groups, term) + @unmatched[term] += 1 if sectors.empty? && age_groups.empty? && !@ignored.include?(term) + sector_ids.concat(sectors) + age_group_ids.concat(age_groups) + end + + Tags.new(sector_ids: sector_ids.uniq, age_group_ids: age_group_ids.uniq) + end + + private + + def lookup(index, term) + index[term] || index[key(term.sub(PARENTHETICAL, ""))] || [] + end + + # Names come from the scope as given (published only); aliases are read across the whole + # vocabulary of that kind, so one can deliberately point at a retired tag. + def vocabulary(scope) + aliases = TagAlias.resolving.where(aliasable_type: scope.model_name.to_s, + aliasable_id: scope.unscope(where: :published).select(:id)) + (scope.pluck(:name, :id) + aliases.pluck(:name, :aliasable_id)).each_with_object({}) do |(name, id), index| + (index[key(name)] ||= []) << id + end + end + + def terms(value) + value.to_s.split(SEPARATORS).filter_map { |part| key(part).presence }.uniq + end + + def key(value) + value.to_s.squish.downcase + end +end diff --git a/config/features.yml b/config/features.yml index 36e199f92b..f787e493ab 100644 --- a/config/features.yml +++ b/config/features.yml @@ -4563,3 +4563,20 @@ published and unpublished categories, instead of only published ones. pro_tips: - "Unpublished categories have a gray row, so you can tell them apart in the All view." + +- name: "What a tag used to be called" + area: people + display_status: admin_facing + released_on: 2026-10-15 + pr_number: 2691 + summary: >- + The portal now records what FileMaker, the older registration and scholarship + forms, Story Share and the portal's earlier sector list each called a sector or + age group, so an old word still resolves to the right tag today. + pro_tips: + - "This is how an import knows that \"Student Services\" means Education and that \"C & T\" means both Children and Teens." + - "A word can be recorded as meaning nothing on purpose — FileMaker's \"Virtually\" described where someone worked, not who they served." + - "A retired tag never comes back by accident — but a word can be pointed at one deliberately, which is how FileMaker's \"Poverty\" still finds the sector that already carries it." + - "Someone recorded as serving families is tagged with every age range, since that's what serving families means." + - "Where an old answer was typed by hand, the reading of it is marked as proposed, so staff can check the list and correct anything that reads wrong." + - "The mapping is written up in ADR-0007 for anyone who needs to see the whole picture." diff --git a/db/migrate/20261015013218_create_tag_aliases.rb b/db/migrate/20261015013218_create_tag_aliases.rb new file mode 100644 index 0000000000..b198dbdfdf --- /dev/null +++ b/db/migrate/20261015013218_create_tag_aliases.rb @@ -0,0 +1,26 @@ +class CreateTagAliases < ActiveRecord::Migration[8.1] + def up + create_table :tag_aliases do |t| + t.string :name, null: false + # A term can be recorded as deliberately naming nothing, so aliasable is optional. + t.references :aliasable, polymorphic: true, null: true, index: false + t.boolean :ignored, default: false, null: false + # A semantic read of the archive's free text, awaiting staff validation. + t.boolean :proposed, default: false, null: false + t.string :source, null: false + t.integer :created_by_id + t.integer :updated_by_id + t.timestamps + end + + # The column collation is case-insensitive, so one spelling can't be recorded twice + # for the same tag. A term naming two tags is two rows. + add_index :tag_aliases, %i[name aliasable_type aliasable_id], unique: true, + name: "index_tag_aliases_on_name_and_aliasable" + add_index :tag_aliases, %i[aliasable_type aliasable_id], name: "index_tag_aliases_on_aliasable" + end + + def down + drop_table :tag_aliases, if_exists: true + end +end diff --git a/db/schema.rb b/db/schema.rb index fd957ce59c..8b3ad43ff5 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.1].define(version: 2026_10_14_123237) do +ActiveRecord::Schema[8.1].define(version: 2026_10_15_013218) do create_table "action_text_mentions", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.bigint "action_text_rich_text_id", null: false t.datetime "created_at", null: false @@ -2145,6 +2145,21 @@ t.index ["workshop_id"], name: "index_story_ideas_workshops_on_workshop_id" end + create_table "tag_aliases", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| + t.bigint "aliasable_id" + t.string "aliasable_type" + t.datetime "created_at", null: false + t.integer "created_by_id" + t.boolean "ignored", default: false, null: false + t.string "name", null: false + t.boolean "proposed", default: false, null: false + t.string "source", null: false + t.datetime "updated_at", null: false + t.integer "updated_by_id" + t.index ["aliasable_type", "aliasable_id"], name: "index_tag_aliases_on_aliasable" + t.index ["name", "aliasable_type", "aliasable_id"], name: "index_tag_aliases_on_name_and_aliasable", unique: true + end + create_table "topic_subscription_types", charset: "utf8mb4", collation: "utf8mb4_unicode_ci", force: :cascade do |t| t.datetime "archived_at" t.datetime "created_at", null: false diff --git a/db/seeds.rb b/db/seeds.rb index 409a924cce..cc4dda25ac 100644 --- a/db/seeds.rb +++ b/db/seeds.rb @@ -522,3 +522,9 @@ def find_or_create_by_name!(klass, name, **attrs, &block) # data:seed_close_program_form task. Idempotent on form name. puts "Creating the Close Program form…" CloseProgramFormSeeder.call + +# What the archive, the older forms, Story Share and the portal's earlier sector list +# each called a tag the portal now names its own way (ADR-0007). Shared with the +# data:seed_tag_aliases task. Idempotent, and never touches a term an admin added. +puts "Recording legacy tag aliases…" +TagAliasSeeder.call diff --git a/docs/adr/0007-legacy-sector-and-age-group-translation.md b/docs/adr/0007-legacy-sector-and-age-group-translation.md new file mode 100644 index 0000000000..eb37b19fc3 --- /dev/null +++ b/docs/adr/0007-legacy-sector-and-age-group-translation.md @@ -0,0 +1,305 @@ +# ADR-0007 — Translating legacy sector and age-group words into the portal's tags + +- **Status:** Accepted +- **Date:** 2026-10-14 + +## Context + +Four systems asked people who they serve, each with its own word list: + +| Source | Fields | +|---|---| +| FileMaker (`fm_rolodexes`) | `PrimaryServiceArea`, `OtherServiceArea`, `Settings`, `LifeExperiences`, `PrimaryAgeGroup` | +| Collab / training registration / scholarship forms | the same four, with longer prose labels | +| Story Share | Categories, Windows Audience, Age Range | +| The portal, before the 2026 consolidation | its own earlier sector names | + +The portal now carries one canonical vocabulary — `Sector::SECTOR_TYPES`, the `AgeRange` +categories (Children, Teens, Adults, Elders), and the `StoryPopulation` categories (Self, +Colleagues, Community, Families). The renames, merges and additions that produced it are +already done; `db/seeds.rb` reconciles all three lists. + +What was never written down is how a *legacy input* becomes one of those tags. Every +importer and backfill has to answer that per value — `PersonServices::ImportFromRolodex` +first — and each would otherwise invent its own guesswork. This records the answer. + +## Decision + +Translation is by lookup table, not by string similarity. A value that isn't in a table is +**reported, never guessed**. + +### Reading a source value + +- Compare case-insensitively, ignoring surrounding whitespace. Most of the archive's + long tail is case variants ("Domestic violence" alongside "Domestic Violence"), so this + alone collapses it. +- One field may list several areas: split on **line breaks and semicolons**, then translate + each piece. **Not on commas** — the wording itself contains them, inside parentheticals + like "Community violence (gang violence, mass shootings, police violence, etc.)". +- Drop a trailing parenthetical when the term alone resolves, so the forms' clarifying + tails ("Grief/loss (including COVID-related)") need no separate entry. +- Many source values mapping to one tag is normal. Only `C & T` maps to two tags. +- The portal spelling wins where the spreadsheet differs: **`Systems/Policy Change`** (the + sheet's "Sytems" is a typo). +- `Other` is a form-only catch-all. It is never a translation target. + +### FileMaker `PrimaryServiceArea` → sector + +| FileMaker | Sector | +|---|---| +| Community Oppression/Violence | Community Violence | +| Criminal/Legal | Court/Legal System | +| Disability Services | Disability Services | +| Domestic Violence | Domestic Violence | +| Foster Care | Foster Care/Adoption | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Immigration | Immigration | +| Incarceration | Incarceration | +| Indigenous/Tribal Nation | Indigenous/Tribal Nation | +| LGBTQIA+ | LGBTQIA+ | +| Mental Health | Mental Health | +| Reproductive Services | Reproductive Services | +| Restorative/Transformative Justice | Restorative/Transformative Justice | +| Sexual Assault | Sexual Assault | +| Student Services | Education | +| Substance Abuse | Substance Use/Recovery | + +### FileMaker `Settings` → sector + +| FileMaker | Sector | +|---|---| +| Clinical Setting | Mental Health | +| Educational Setting | Education | +| Faith-Based Setting | Religious/Faith-Based | +| Hospitals | Health/Medical | +| Law Enforcement/Court | Court/Legal System | +| Prisons/Jails | Incarceration | +| Private Practice | Private Practice/Sole Proprietor | +| With Staff | Staff/Organizational Development | + +**Dropped, not mapped:** Community-Based, Events and Conferences, Home Visits, Outreach +Program, Residential Program, Virtually. These say *where or how* a facilitator works, not +who they serve — they aren't sectors and have no portal home. + +### FileMaker `LifeExperiences` → sector + +| FileMaker | Sector | +|---|---| +| Child Abuse/Neglect | Child Abuse/Neglect | +| Climate/Environmental Trauma | Climate/Environmental | +| Community Violence | Community Violence | +| Domestic Violence | Domestic Violence | +| Foster Care | Foster Care/Adoption | +| Grief/Loss | Grief/Loss | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Illness | Health/Medical | +| Immigration | Immigration | +| Incarceration | Incarceration | +| Mental Health Needs | Mental Health | +| Military/Veteran | Military/Veterans | +| Oppression Against LGBTQIA+ | LGBTQIA+ | +| Pandemic-Related Stress | Health/Medical | +| People Who Do Harm/Batterers | Batterers Intervention | +| Racism and Marginalization | Racial/Social Justice | +| Religious Trauma | Religious/Faith-Based | +| Restorative/Transformative Justice | Restorative/Transformative Justice | +| Secondary/Vicarious Trauma | Staff/Organizational Development | +| Sexual Assault | Sexual Assault | +| Student-Related Stress | Education | +| Substance Abuse | Substance Use/Recovery | +| Suicidality | Mental Health | +| Victims of Crime | Court/Legal System | + +**No sector yet:** Elder Abuse, People Who Have Experienced War, Poverty. Left unmapped on +purpose — each needs a sector added before it can be translated, and inventing a near-miss +would bury the gap. An importer reports them. + +### Cells naming more than one sector + +Hand-typed answers that name several sectors resolve to all of them — the same one-to-many +shape as `C & T`. Recorded only where every concept in the cell is itself a decided term: +`Domestic Violence & Homelessness` · `Sexual Assault & Domestic Violence` · +`Domestic Violence, Sexual Assault` · `Domestic Violence and Sexual Assault survivors` · +`Domestic and Sexual Violence Families` · `Sexual and Domestic Violence Agency` · +`DV/SA/other related crimes` · `Dual Agency Domestic violence/Sexual assault/ Human +trafficking` · `Immigration, Health, Prevention of Violence and Education` · +`Addiction, Native Americans, mental health and addiction` · `Education, SEL needs` · +`Hospice, end of life, bereavement`. + +### Terms that cross vocabularies + +Two of the archive's *sector* fields carry terms that say which **ages** someone serves, so +they resolve to an age range rather than a sector: + +| Source value | Portal tag | +|---|---| +| `Elder Abuse` (LifeExperiences) | AgeRange **Elders** | +| `Children's Services` (PrimaryServiceArea) | AgeRange **Children** | + +Whether the tag lands as the **primary** or an **additional** one follows from the field the +term sat in, not from the term: `PrimaryServiceArea` and `PrimaryAgeGroup` are the primary +answers, `OtherServiceArea`, `Settings` and `LifeExperiences` the additional ones. So +`Children's Services` makes Children the primary age range, and `Elder Abuse` adds Elders +alongside whatever the primary already is. + +### Reads of the free text, pending staff validation + +The archive's hand-typed cells are mostly a venue, a programme name or a sentence about the +work. Where the meaning is plain — `Immigration Clinic`, `Grief support group`, +`Inpatient Psychiatric Hopital`, `Supervised visitation sessions for foster youth…` — the +term is recorded as **proposed**: it resolves, so an import doesn't lose it, and +`TagAlias.proposed` is the list to walk with staff term by term. About 90 terms, each worth +one or two archive rows. + +**What is deliberately not read:** anything naming an organization, a place that identifies +one, or a facilitator's own circumstances. That text is personal and belongs in the archive, +not copied into a catalogue — which leaves 23 terms, 23 mentions, unresolved for good. + +### Answers that name nothing + +Alongside the dropped Settings, these are recorded as naming nothing on purpose: `Other` · +`Did not specify` · `n/a` · `none` · `something` · `unsure` · `Everyone` · `Technology` · +and the delivery modes `In person` · `In visits` · `virtually & in-person` · `At office` · +`Office space.` · `Residential` · `On-Call/Crisis` · `Support groups`. None of them says +who a facilitator serves, so leaving them in the undecided queue would only bury the terms +that do. + +**`People Who Have Experienced War` (97) joins them by decision, not by shape.** It names a +real experience, and Community Violence was the fallback if it had to go somewhere — but +AWBW chose to leave it untagged rather than file it under a sector it doesn't mean. + +### Story Share categories → sector + +| Story Share | Sector | +|---|---| +| Advocacy | Systems/Policy Change | +| Batterers Intervention | Batterers Intervention | +| Child Abuse / Neglect | Child Abuse/Neglect | +| Community Engagement | Community Building | +| Community Violence | Community Violence | +| Court & Legal System | Court/Legal System | +| Disability Services | Disability Services | +| Domestic Violence | Domestic Violence | +| Donor Engagement | Fundraising/Donor Engagement | +| Foster Care | Foster Care/Adoption | +| Grief & Loss | Grief/Loss | +| Homelessness | Homelessness | +| Human Trafficking | Human Trafficking | +| Immigration | Immigration | +| Incarceration | Incarceration | +| LGBTQIA+ | LGBTQIA+ | +| Mental Health | Mental Health | +| Military & Veterans | Military/Veterans | +| Physical & Terminal Illness | Health/Medical | +| Schools & Universities | Education | +| Self-Care & Personal Growth | Self-Care/Personal Growth | +| Sexual Assault | Sexual Assault | +| Social Justice | Racial/Social Justice | +| Staff & Organizational Development | Staff/Organizational Development | +| Substance Abuse Recovery | Substance Use/Recovery | + +**Not a sector:** Facilitator Spotlights. It describes the story, not who it serves. + +### Earlier portal sector names → sector + +Already applied in the portal by rename and merge, recorded so an old export still reads: +Child abuse → Child Abuse/Neglect · Community oppression/violence → Community Violence · +Criminal/legal → Court/Legal System · Disability (and Disabled, Disabled Person) → +Disability Services · Education/schools and Student → Education · Family Violence and +Children of Domestic Violence Survivors → Domestic Violence · Foster care/adoption (and +Foster Youth, Foster children/APPS) → Foster Care/Adoption · Grief and loss → Grief/Loss · +Lgbtqia+ → LGBTQIA+ · Mental illness → Mental Health · Reproductive → Reproductive +Services · Serious illness → Health/Medical · Staff & organizational development → +Staff/Organizational Development · Substance use → Substance Use/Recovery · Veterans & +military → Military/Veterans. + +### Age groups + +| Source value | Portal tag | +|---|---| +| FileMaker `Children`; Story Share `3-5`, `6-12`, `Children` | AgeRange **Children** (0-12) | +| FileMaker `Teens`; Story Share `13-17` | AgeRange **Teens** (13-17) | +| FileMaker `Adults`; Story Share `18+`, `Adult` | AgeRange **Adults** (18+) | +| FileMaker `Elders` | AgeRange **Elders** (65+) | +| FileMaker `C & T` | AgeRange **Children** *and* **Teens** | +| FileMaker `Families` | AgeRange **Children**, **Teens**, **Adults** *and* **Elders** — serving families is serving every age | +| Story Share `Family windows` | StoryPopulation **Families** (a story can carry a population; a person can't) | +| Story Share `Mixed-age groups`, `Combined` | nothing — says the group was mixed, not which ages | + +`3-5` and `6-12` were folded into `Children` in the portal, and the `StoryPopulation` +twins carry a trailing underscore so the clean age names stay free (see `db/seeds.rb`). + +## How it's stored + +**`TagAlias` rows, not a constant.** Each row names one legacy term, the tag it resolves +to (polymorphic — a `Sector` or an `AgeRange` `Category`), and the system the term came +from. `TagTranslation` reads a tag's own name first, then the aliases. + +- **A tag resolves by name only while it's published.** A retired tag is unpublished rather + than deleted so its old taggings survive, and matching one by name would hand it to + hundreds of new records by accident. **A recorded alias may point at a retired tag** — + that's someone deciding the term means that tag, which is the opposite of an accident. + `Poverty` is exactly that case: the sector is retired, 634 archive rows name it, and the + alias sends them to the tag that already carries its older taggings. + +- **A term can name two tags**, so a row per pair and a list on the way out (`C & T`). +- **Unique on (term, tag)**, with the column's case-insensitive collation, so one spelling + can't be recorded twice for the same tag. +- **An admin can add a term** as it turns up, without a deploy — which matters, because the + archive's free-text tail keeps producing new ones. +- **`created_by` and `created_at`** make it the record of who decided a term meant a tag + and when. The renames and merges already ran, so the old names are gone from the data; + these rows are what's left of them. +- **A term can be recorded as naming nothing on purpose** (`ignored`). That's how the + dropped Settings and `Other` stay out of the queue of terms still needing a decision — + without it, ~2,800 mentions of "Virtually", "Outreach Program" and the like would bury + the handful that genuinely need one. +- `TagAliasSeeder` (and `data:seed_tag_aliases`) records the tables above, idempotently, + and never touches a term an admin added. + +The tables stay here as the readable source; the rows are the machine-readable copy. + +## What the archive actually holds + +Counted across `fm_rolodexes` (October 2026): `LifeExperiences` 55 distinct values / +21,857 mentions · `Settings` 161 / 6,082 · `PrimaryAgeGroup` 11 / 2,356 · +`PrimaryServiceArea` 29 / 2,354 · `OtherServiceArea` 73 / 75. + +- **The canonical words dominate.** The documented vocabulary covers nearly every mention; + 134 of the 161 `Settings` values occur exactly once. +- **`OtherServiceArea` is free text**, not a picker: 72 of its 73 values occur once, across + 75 rows in all. Not worth translating — it goes to the unmatched report. +- **`Other`** appears as a stored value (90 `PrimaryServiceArea`, 60 `LifeExperiences`). + It is recorded as ignored rather than translated: `Sector::OTHER_SECTOR_NAME` is offered + for additional sectors but never as a primary, so an import can't honour it as one, and + "they picked Other" says nothing a tag could carry. +- **Still needing a decision** — 23 terms, one mention each, every one naming an + organization, a place that identifies one, or a facilitator's own circumstances. They stay + in the archive and show up in the unmatched report; nothing is copied out of them. +- **The proposed reads need a pass with staff** before the real import run. They resolve + meanwhile, so the work isn't blocked, and a wrong one is a row to delete rather than a + deploy. +- **Coverage.** 17,734 of the archive's 17,757 mentions are translated, deliberately + untranslated, or proposed. +- **Seen in the data, no mapping decided:** Children's Services, Intellectual Disabilities, + Maternal wellness and postpartum support, Political Asylum, Survivors with Disabilities, + Arts Education & Community Art Making, and the rest of the one-off free text. Deliberately + unseeded — each is a judgment call for AWBW, and the unmatched report is the queue. + +## Consequences + +- **Populations are not age ranges.** `AgeGroupTaggable` only tags `AgeRange` categories, so + a person has no population tag to carry `Families`. Rather than drop it, a person whose + FileMaker age group is `Families` is tagged with every age range — which is what serving + families means. A story keeps its `StoryPopulation` tag. +- **One source value can produce several tags** (`C & T`, `Families`, the compound cells), + so a translator returns a list, not a value — and across both vocabularies, since a sector + field can carry an age term. +- **Unmapped values need a report, not a fallback.** `PersonServices::ImportFromRolodex` + counts them in `#unmatched_tags` and the rake task prints them ranked; its name matching + alone resolves only the values that already read as a sector name. +- **The translation is data, so it can retire.** Once the imports and backfills are done, + the rows stay as the history of what a tag used to be called — and can be dropped without + touching code if that history stops being useful. diff --git a/lib/tasks/seed_tag_aliases.rake b/lib/tasks/seed_tag_aliases.rake new file mode 100644 index 0000000000..397bc0d44e --- /dev/null +++ b/lib/tasks/seed_tag_aliases.rake @@ -0,0 +1,18 @@ +# frozen_string_literal: true + +# Record what each legacy word for a sector or age group resolves to, as TagAlias rows, +# without running the full db:seed. The tables are ADR-0007; the rows are what an import +# looks a term up in, and what a reader checks to see what a tag used to be called in +# FileMaker, the older forms, Story Share or the portal's earlier sector list. +# +# Idempotent and safe to re-run: a term already recorded for a tag is left alone, so an +# alias an admin added by hand survives. Run after the sectors and age ranges are seeded — +# a term whose tag doesn't exist yet is reported, not created. +# +# bin/rails data:seed_tag_aliases +namespace :data do + desc "Record the legacy sector and age-group terms (ADR-0007) as tag aliases (idempotent)." + task seed_tag_aliases: :environment do + TagAliasSeeder.call + end +end diff --git a/spec/factories/tag_aliases.rb b/spec/factories/tag_aliases.rb new file mode 100644 index 0000000000..8f875e9cc5 --- /dev/null +++ b/spec/factories/tag_aliases.rb @@ -0,0 +1,7 @@ +FactoryBot.define do + factory :tag_alias do + sequence(:name) { |n| "Legacy Term #{n}" } + source { "filemaker" } + association :aliasable, factory: :sector + end +end diff --git a/spec/models/tag_alias_spec.rb b/spec/models/tag_alias_spec.rb new file mode 100644 index 0000000000..719c1abda9 --- /dev/null +++ b/spec/models/tag_alias_spec.rb @@ -0,0 +1,38 @@ +require "rails_helper" + +RSpec.describe TagAlias, type: :model do + let(:sector) { create(:sector, :published, name: "Domestic Violence") } + + it "is valid when it names a tag" do + expect(build(:tag_alias, aliasable: sector)).to be_valid + end + + it "is valid when it names nothing on purpose" do + expect(build(:tag_alias, aliasable: nil, ignored: true)).to be_valid + end + + it "rejects a term that neither names a tag nor is ignored" do + expect(build(:tag_alias, aliasable: nil, ignored: false)).not_to be_valid + end + + it "rejects a term that both names a tag and is ignored" do + expect(build(:tag_alias, aliasable: sector, ignored: true)).not_to be_valid + end + + it "records a term once per tag, whatever its case or spacing" do + create(:tag_alias, name: "Student Services", aliasable: sector) + + expect(build(:tag_alias, name: " student services ", aliasable: sector)).not_to be_valid + end + + it "lets two tags share a term, for a term that names both" do + other = create(:sector, :published, name: "Education") + create(:tag_alias, name: "C & T", aliasable: sector) + + expect(build(:tag_alias, name: "C & T", aliasable: other)).to be_valid + end + + it "only accepts a system it could have come from" do + expect(build(:tag_alias, aliasable: sector, source: "spreadsheet")).not_to be_valid + end +end diff --git a/spec/services/tag_alias_seeder_spec.rb b/spec/services/tag_alias_seeder_spec.rb new file mode 100644 index 0000000000..601271f337 --- /dev/null +++ b/spec/services/tag_alias_seeder_spec.rb @@ -0,0 +1,168 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe TagAliasSeeder do + let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } + + before do + Sector::SECTOR_TYPES.each { |name| create(:sector, :published, name: name) } + %w[Children Teens Adults Elders].each { |name| create(:category, :published, category_type: age_range_type, name: name) } + end + + it "records every term the tables name, pointing at the tag it resolves to" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services").aliasable).to eq(Sector.find_by(name: "Education")) + expect(TagAlias.find_by(name: "Clinical Setting").aliasable).to eq(Sector.find_by(name: "Mental Health")) + expect(TagAlias.find_by(name: "Advocacy").aliasable).to eq(Sector.find_by(name: "Systems/Policy Change")) + expect(TagAlias.find_by(name: "Child abuse").aliasable).to eq(Sector.find_by(name: "Child Abuse/Neglect")) + expect(TagAlias.find_by(name: "Homeless services").aliasable).to eq(Sector.find_by(name: "Homelessness")) + end + + it "records a term that names two age groups once for each" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "C & T").map { |row| row.aliasable.name }).to contain_exactly("Children", "Teens") + end + + it "reads serving families as serving every age" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "Families").map { |row| row.aliasable.name }) + .to contain_exactly("Children", "Teens", "Adults", "Elders") + end + + it "reports a term whose tag isn't published, when nothing says to point at a retired one" do + Sector.find_by(name: "Education").update!(published: false) + + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services")).to be_nil + end + + it "points the archive's Poverty at the retired sector that already carries its taggings" do + retired = create(:sector, :unpublished, name: "Poverty") + + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Poverty").aliasable).to eq(retired) + expect(TagTranslation.new.resolve("Poverty").sector_ids).to eq([ retired.id ]) + end + + it "reads a cell naming several sectors as all of them" do + described_class.call(quiet: true) + + expect(TagAlias.where(name: "Sexual Assault & Domestic Violence").map { |row| row.aliasable.name }) + .to contain_exactly("Sexual Assault", "Domestic Violence") + expect(TagTranslation.new.resolve("DV/SA/other related crimes").sector_ids.size).to eq(2) + end + + it "reads a life experience or service area that means an age group" do + described_class.call(quiet: true) + + translation = TagTranslation.new + expect(translation.resolve("Elder Abuse").age_group_ids).to eq([ Category.find_by(name: "Elders").id ]) + expect(translation.resolve("Children's Services").age_group_ids).to eq([ Category.find_by(name: "Children").id ]) + expect(translation.resolve("Elder Abuse").sector_ids).to be_empty + end + + it "leaves war untagged, as decided" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "People Who Have Experienced War")).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.resolve("People Who Have Experienced War").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "records an answer that says nothing as naming nothing" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "n/a")).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.resolve("n/a\nunsure\nIn person").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "labels each term with the system it came from" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Student Services").source).to eq("filemaker") + expect(TagAlias.find_by(name: "Homeless services").source).to eq("forms") + expect(TagAlias.find_by(name: "Grief & Loss").source).to eq("story_share") + expect(TagAlias.find_by(name: "Grief and loss").source).to eq("portal") + end + + it "marks a read of free text as proposed, and a decision as decided" do + described_class.call(quiet: true) + + expect(TagAlias.find_by(name: "Immigration Clinic")).to have_attributes(proposed: true, ignored: false) + expect(TagAlias.find_by(name: "Libraries")).to have_attributes(proposed: true, ignored: true) + expect(TagAlias.find_by(name: "Student Services").proposed).to be false + expect(TagAlias.find_by(name: "Virtually").proposed).to be false + expect(TagAlias.proposed.count).to be > 50 + end + + it "resolves a proposed read, so an import doesn't lose it while staff review" do + described_class.call(quiet: true) + + translation = TagTranslation.new + tags = translation.resolve("Immigration Clinic") + expect(tags.sector_ids).to eq([ Sector.find_by(name: "Immigration").id ]) + expect(translation.resolve("Libraries").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "reads a free-text answer that names several sectors and the ages it serves" do + described_class.call(quiet: true) + + tags = TagTranslation.new.resolve("1:1 children and youth who experience domestic violence, dating violence, or sexual abuse") + expect(tags.sector_ids).to match_array(Sector.where(name: [ "Domestic Violence", "Sexual Assault", "Child Abuse/Neglect" ]).ids) + expect(tags.age_group_ids).to match_array(Category.where(name: %w[Children Teens]).ids) + end + + it "adds nothing on a second run" do + described_class.call(quiet: true) + + expect { described_class.call(quiet: true) }.not_to change(TagAlias, :count) + end + + it "leaves a term an admin recorded by hand alone" do + sexual_assault = Sector.find_by(name: "Sexual Assault") + admin_alias = create(:tag_alias, name: "Student Services", aliasable: sexual_assault, source: "portal") + + described_class.call(quiet: true) + + expect(admin_alias.reload.aliasable).to eq(sexual_assault) + expect(TagAlias.where(name: "Student Services").count).to eq(2) + end + + it "reports a term whose tag isn't seeded yet instead of inventing it" do + Sector.find_by(name: "Education").destroy + + expect { described_class.call(quiet: true) }.not_to raise_error + expect(TagAlias.find_by(name: "Student Services")).to be_nil + end + + it "records a setting that names nothing as decided, not as a gap" do + described_class.call(quiet: true) + + virtually = TagAlias.find_by(name: "Virtually") + expect(virtually).to have_attributes(aliasable: nil, ignored: true) + + translation = TagTranslation.new + expect(translation.resolve("Virtually").sector_ids).to be_empty + expect(translation.unmatched).to be_empty + end + + it "resolves the archive's own wording once seeded" do + described_class.call(quiet: true) + + translation = TagTranslation.new + expect(translation.resolve("Clinical Setting").sector_ids).to eq([ Sector.find_by(name: "Mental Health").id ]) + expect(translation.resolve("C & T").age_group_ids.size).to eq(2) + end +end diff --git a/spec/services/tag_translation_spec.rb b/spec/services/tag_translation_spec.rb new file mode 100644 index 0000000000..d561d99782 --- /dev/null +++ b/spec/services/tag_translation_spec.rb @@ -0,0 +1,106 @@ +# frozen_string_literal: true + +require "rails_helper" + +RSpec.describe TagTranslation do + let!(:domestic_violence) { create(:sector, :published, name: "Domestic Violence") } + let!(:education) { create(:sector, :published, name: "Education") } + let!(:age_range_type) { create(:category_type, name: "AgeRange", published: true) } + let!(:children) { create(:category, :published, category_type: age_range_type, name: "Children") } + let!(:teens) { create(:category, :published, category_type: age_range_type, name: "Teens") } + let!(:elders) { create(:category, :published, category_type: age_range_type, name: "Elders") } + + it "resolves a term by the tag's own name, whatever its case" do + expect(described_class.new.resolve("domestic violence").sector_ids).to eq([ domestic_violence.id ]) + end + + it "resolves a term by an alias recorded for the tag" do + create(:tag_alias, name: "Student Services", aliasable: education) + + expect(described_class.new.resolve("student services").sector_ids).to eq([ education.id ]) + end + + it "reads one term as several tags when both are recorded" do + create(:tag_alias, name: "C & T", aliasable: children, source: "filemaker") + create(:tag_alias, name: "C & T", aliasable: teens, source: "filemaker") + + expect(described_class.new.resolve("C & T").age_group_ids).to contain_exactly(children.id, teens.id) + end + + it "splits a repeating field on line breaks and semicolons, not on commas" do + create(:tag_alias, name: "Community violence (gang violence, mass shootings, etc.)", aliasable: domestic_violence) + + translation = described_class.new + expect(translation.resolve("Domestic Violence\nEducation").sector_ids).to contain_exactly(domestic_violence.id, education.id) + expect(translation.resolve("Domestic Violence; Education").sector_ids).to contain_exactly(domestic_violence.id, education.id) + expect(translation.resolve("Community violence (gang violence, mass shootings, etc.)").sector_ids).to eq([ domestic_violence.id ]) + end + + it "drops a clarifying parenthetical when the term alone resolves" do + expect(described_class.new.resolve("Domestic violence (including family violence)").sector_ids).to eq([ domestic_violence.id ]) + end + + it "counts a term nothing answers to instead of guessing at it" do + translation = described_class.new + + expect(translation.resolve("Poverty\nDomestic Violence").sector_ids).to eq([ domestic_violence.id ]) + expect(translation.unmatched).to eq({ "poverty" => 1 }) + end + + it "keeps a term recorded as ignored out of the unmatched list" do + create(:tag_alias, name: "Virtually", aliasable: nil, ignored: true) + translation = described_class.new + + expect(translation.resolve("Virtually\nDomestic Violence").sector_ids).to eq([ domestic_violence.id ]) + expect(translation.unmatched).to be_empty + end + + it "won't reach a retired tag by its own name" do + create(:sector, :unpublished, name: "Poverty") + translation = described_class.new + + expect(translation.resolve("Poverty").sector_ids).to be_empty + expect(translation.unmatched).to eq({ "poverty" => 1 }) + end + + it "reaches a retired tag an alias deliberately points at" do + retired = create(:sector, :unpublished, name: "Poverty") + create(:tag_alias, name: "Poverty", aliasable: retired) + translation = described_class.new + + expect(translation.resolve("Poverty").sector_ids).to eq([ retired.id ]) + expect(translation.unmatched).to be_empty + end + + it "keeps an age-group alias from reaching a category of another kind" do + population_type = create(:category_type, name: "StoryPopulation", published: true) + families = create(:category, :published, category_type: population_type, name: "Families_") + create(:tag_alias, name: "Family windows", aliasable: families) + + expect(described_class.new.resolve("Family windows").age_group_ids).to be_empty + end + + it "answers with an age group for a term that sat in a sector field" do + create(:tag_alias, name: "Elder Abuse", aliasable: elders) + tags = described_class.new.resolve("Elder Abuse") + + expect(tags.sector_ids).to be_empty + expect(tags.age_group_ids).to eq([ elders.id ]) + end + + it "counts a term once, however many vocabularies miss it" do + translation = described_class.new + translation.resolve("Poverty") + translation.resolve("Poverty") + + expect(translation.unmatched).to eq({ "poverty" => 2 }) + end + + it "has nothing to say about a blank value" do + translation = described_class.new + + expect(translation.resolve(nil).sector_ids).to be_empty + expect(translation.resolve(" ").age_group_ids).to be_empty + expect(translation.unmatched).to be_empty + end +end