Skip to content

Unspaced Chinese/Korean names: surnames constants + longest-match segmentation #271

Description

@derek73

Unspaced CJK names ("毛泽东", "김민준") can't be parsed today because there's nothing to split into pieces. For Chinese and Korean, segmentation is cheap and fits the library's constants-driven style, because surnames come from small closed sets while given names are open-ended — only a surname dictionary is needed:

  • Korean: 286 surnames total (2015/2019 census); Kim/Lee/Park alone cover ~45% of the population; about a dozen two-syllable surnames. Complete coverage in ~300 entries.
  • Chinese: top 100 surnames cover ~86% of the population, ~6,000 in current use; surnames are 1 character, occasionally 2 (compound: 欧阳, 司马, …). Excellent coverage in ~500 entries.

Proposal:

  1. A new surnames constants set (same SetManager machinery as titles/prefixes), shipping curated Chinese and Korean surname lists.
  2. A pre-processing step (opt-in, like all localization) that longest-matches a surname off the front of an unspaced string and inserts the space — compound/two-syllable surnames matched before single ones (欧阳明 must split 欧阳+明, not 欧+阳明, even though 欧 is itself a surname).
  3. Everything downstream is existing machinery: the segmenter only inserts the space; name_order=('last', 'first') (Support family-first name order via an order-spec config (name_order) #270) does the assignment.

Opt-in is required for a second reason here: a pure-Han string cannot indicate whether it's Chinese or Japanese (林 is Lin or Hayashi), so the caller must select the language. Japanese itself is explicitly out of scope — ~138,500 surnames with genuinely ambiguous split boundaries — and is covered by #272.

Depends on the name_order order-spec (#270). Would resolve the unspaced cases behind #83 and (with ('last','middle','first')) #146-style Vietnamese input; supersedes the analysis in #34.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions