Skip to content

don't split German gender-inclusive colon forms - #14022

Open
gyanu2507 wants to merge 1 commit into
explosion:masterfrom
gyanu2507:fix/de-gender-colon
Open

don't split German gender-inclusive colon forms#14022
gyanu2507 wants to merge 1 commit into
explosion:masterfrom
gyanu2507:fix/de-gender-colon

Conversation

@gyanu2507

Copy link
Copy Markdown

Fixes #14010

German infix [:<>=] splits gender-inclusive colon forms before anything else sees them:

nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")
# Kund / : / innen

Star and interpunct variants already stay whole (Kolleg*innen, Expert路innen). This drops : from that infix class and leaves <>= splitting alone, which is the patch we've been running in production.

eine:n / 脛rzt:in stay one token as well. Comparisons like x<y still split.

Kund:innen and similar were broken on the letter-colon-letter infix.
Keep <>= splitting; drop only the colon from that class.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

German tokenizer splits gender-inclusive colon forms (Kund:innen)

1 participant