Skip to content

Import people from the FileMaker Rolodex archive - #2630

Open
maebeale wants to merge 6 commits into
mainfrom
maebeale/rolodex-import-rake-task
Open

maebeale wants to merge 6 commits into
mainfrom
maebeale/rolodex-import-rake-task

Conversation

@maebeale

@maebeale maebeale commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 suggested review level: 5 Inspect 🔬 a one-off import that now writes most of a person's profile, plus a save-time default on two shared models that changes one public-form question's visibility

Adds fm:import_people, which turns FileMaker Rolodex archive rows into people so the old contact list lives in the portal instead of only in the read-only archive.

What is the goal of this PR and why is this important?

  • fm_rolodexes holds ~39k contacts, none of which exist as Person records yet.
  • The archive's contact types don't all map onto ours and plenty of its rows carry none, so the import needed a place to put "we don't know" — which turned out to be a gap everywhere, not just here.

The import

  • PersonServices::ImportFromRolodex does the work; the rake task wraps it in a dry-run transaction.
  • Dry run by default — DRY_RUN=false writes; VERBOSE=true lists every row.
  • Idempotent — a row whose ID a person already carries is left alone, reading the code column as the list it is.
  • Links, doesn't duplicate — matching goes through PersonMatcher, so a person the archive knows by their secondary or login address is linked rather than recreated. Names run through Person.normalize_name first.
  • Flags, doesn't guess — several matches, an email that belongs to someone else, and already-linked people are reported as ambiguous.
  • Skips agencies (isAgency__c), soft-deleted (Delete), and nameless rows.

What an imported person carries

Everything FmRolodex::PORTAL_FIELDS maps onto a person:

  • Identity and profile — names, emails, email type, pronouns, best time to call, ethnicity, DOB, comments → notes
  • FacilitatorSinceMonth + Year → member_since (a year with no month reads as January)
  • Service areas → sectors and PrimaryAgeGroup → age group, through SubmissionTagging — the same seam a submission uses, additively
  • The archive's own DateEntered / DateTimeModified as created_at / updated_at, so a profile dates from when FileMaker first knew the person
  • Every address and phone the row answers to (FmRolodex#addresses / #phone_numbers), the row's primary first
  • A service area no sector answers to is counted in #unmatched_tags and printed by the task — Child Abuse resolves to Child Abuse/Neglect, but a name nothing answers to is a gap for a human, not a guess

Reading the archive's messier fields

  • Dates are matched whole against four formats, so a format that only fits part of the value can't claim it (10-5-1983 used to read as 1910-05-19). A two-digit year gets a century: 83 → 1983, 05 → 2005.
  • Free text is truncated at the 255-character column limit. An over-long address line or note used to fail the whole person.
  • StreetNo is prepended to the first address line unless that line already starts with it.

Unknown as a type (worth the closest look)

  • Address and ContactMethod write "unknown" on save when the type is blank, from any path — no backfill, so rows nothing saves keep their blank.
  • "unknown" is new to ContactMethod::CONTACT_TYPES, so an Unknown option now appears in the contact-method type select. Address already declared it.
  • Address#known_type? is why this isn't a silent regression: person_known_identifiers hides logged_out_only questions a registrant has already answered, and treating "unknown" as an answer would have stopped the public registration form from asking for an address type.
  • AddressDeduper and ReconcileContactMethods read "unknown" and blank as the same type, so a pre-change row still folds into its twin.

Anything else to add?

  • update_columns for the archive dates is deliberate: tagging touches the person, so the dates are written once more after it.
  • PersonMatcher#matches is new — the relation behind #call, for a caller that must tell one match from several. Existing callers unchanged.
  • A row whose email matches nobody is created even when someone shares the name — deliberate. Folding the occasional duplicate with the person deduper beats hand-resolving every name collision in the archive.
  • EnteredBy / ModifiedBy → created_by / updated_by are left unmapped: they're FileMaker usernames, with no reliable path to a portal User.
  • Run the dry run against real data before DRY_RUN=false; db:seed:dev:fm_archive gives dev something to run it against.
  • Affiliations from OrgID are out of scope.

🤖 Generated with Claude Code

@jmilljr24

Copy link
Copy Markdown
Collaborator

Should this also handle worksite address?

rolodex rows                 39,795
  with PrimaryAddrsID         1,480
  with WorksiteAddrsID        1,217

Also I think these are just optional pointers. I think we need to iterate on the fm_address table. FmAddress.where(RolodexID: fm_id) "RolodexID" (26,427 rows → 23,918 people).

Same thoughts on fm_phone_numbers.

Or maybe the intent of the pr is to just import based on the rolodex table and do other tables separately?

@jmilljr24

Copy link
Copy Markdown
Collaborator

A number of "other" email types that would be dropped in the import. Some might be worth mapping to our values.

Preferred       520   
Professional     33   
Business          1
Attn: Carrie      1
Karen's email     1
Carrie            1
Bruce             1

@jmilljr24

Copy link
Copy Markdown
Collaborator

We can pull in address county information from the fm tables.

FmAddress
FmPostalCode  data["PostalCodeLookup"] → FmPostalCode.fm_id (zipcode)
County          FmPostalCode.data["County"]

@maebeale maebeale changed the title Import people from the FileMaker Rolodex archive Import people from the FileMaker Rolodex archive, typing blank address and phone types as unknown Oct 12, 2026
@maebeale
maebeale force-pushed the maebeale/rolodex-import-rake-task branch from da095fc to a2e4424 Compare October 12, 2026 17:41
@maebeale maebeale changed the title Import people from the FileMaker Rolodex archive, typing blank address and phone types as unknown Import people from the FileMaker Rolodex archive Oct 12, 2026
@maebeale
maebeale force-pushed the maebeale/rolodex-import-rake-task branch 2 times, most recently from a2e4424 to 7a70493 Compare October 12, 2026 19:02
@maebeale
maebeale marked this pull request as ready for review October 12, 2026 20:03
@maebeale
maebeale force-pushed the maebeale/rolodex-import-rake-task branch 4 times, most recently from f02b1ab to 3d7d6e6 Compare October 13, 2026 14:23
maebeale and others added 6 commits October 14, 2026 20:26
The FileMaker Rolodex is archived in fm_rolodexes but none of those
contacts exist as people yet. Links on exact name + email so existing
people aren't duplicated, and reports looser collisions for hand review.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A dry run over 39,000 rows is only useful if the rows it reports are the
rows it would write:

- Two-digit years landed in the first century (10/5/83 became 83 AD), and
  because strptime stops where the format ends, 10-5-1983 read as
  1910-05-19 — a date the archive never recorded. Formats now have to
  match the whole value, and a bare year pair gets a century.
- An over-long address line or note failed the entire person on a
  255-character column, dropping a usable record over an incidental field.
  Copied free text is truncated instead.
- The house number has its own field, so addresses imported without it
  unless the first line happened to repeat it.
- Phone types were resolved against the address set, letting a phone claim
  a type the contact-method form can't show. Each model now declares the
  slice it accepts, and a type we can't place reads as unknown rather than
  leaving the record untyped.

Also drops the credentials.yml key that came back in a rebase — #2579
removed it deliberately — and corrects the rake-task count in AGENTS.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every address and contact method now answers "what kind?": a blank type is
written as unknown on save, from any path, so the gap closes as records are
touched rather than needing a backfill. Rows nothing saves keep their blank.

Knowing that a type is unknown isn't the same as knowing the type, so the
public registration form still asks a signed-in registrant for their address
type — Address#known_type? keeps the normalized value from reading as an
answer already on file and hiding the question.

With the models owning the fallback, the Rolodex import drops its own.

Features & tips gains the import and the new Unknown type.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Main's #2632/#2633 made PersonMatcher the one rule for "which person does this
name + email belong to" — it reaches the secondary and login addresses a person
counts as theirs, and compares names with look-alike dashes folded. The import
was a new matching path with its own narrower query, so it would have created a
fresh person for anyone the archive knows by their login address, reopening the
gap that just closed. PersonMatcher gains #matches so a caller that has to tell
one match from several can have the relation.

Archive names go through Person.normalize_name before they're compared, since
that's the form they'll be stored in.

The merge reconcilers read unknown and blank as the same type, so a row written
before an untyped address or number said so still folds into its twin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2647 lets a person carry several codes, so an index built from whole column
values would miss the row whose ID sits inside one and import it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FmRolodex::PORTAL_FIELDS documents which archive field answers to which portal
column; the import only wrote half of them. It now also carries the email type,
the facilitator start month/year as member_since, the service areas and age
group (through SubmissionTagging, the same seam a submission uses, additively),
and the archive's own entry and modification dates — so an imported person
dates from when FileMaker first knew them rather than from the import run.

Addresses and phones come from FmRolodex#addresses / #phone_numbers rather than
the single PrimaryAddrsID/PrimaryPhoneID, so a contact's other addresses arrive
too, the row's primary first.

A service area no sector answers to is counted in #unmatched_tags and printed
by the task, since "Child Abuse" and "Child Abuse/Neglect" are the same sector
but "Homelessness" with no seeded sector is a gap someone has to close.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@maebeale
maebeale force-pushed the maebeale/rolodex-import-rake-task branch from 3d7d6e6 to 872fd1d Compare October 15, 2026 00:34
@maebeale
maebeale force-pushed the maebeale/rolodex-import-rake-task branch from 1eb7308 to 872fd1d Compare October 15, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants