Skip to content

FIX: rejoin words with the configured word_split_separator - #2593

Open
Umerkhan Golandaz (UmerkhanGolandaz) wants to merge 1 commit into
microsoft:mainfrom
UmerkhanGolandaz:fix/word-level-converter-preserves-separator
Open

FIX: rejoin words with the configured word_split_separator#2593
Umerkhan Golandaz (UmerkhanGolandaz) wants to merge 1 commit into
microsoft:mainfrom
UmerkhanGolandaz:fix/word-level-converter-preserves-separator

Conversation

@UmerkhanGolandaz

Copy link
Copy Markdown

Description

Fixes #2592.

WordLevelConverter.convert_async splits the prompt on word_split_separator (word_level_converter.py:116), but the default join_words always rejoined with a space (:91). The split and the join disagreed, so a custom separator was silently replaced in the output:

EmojiConverter(seed=1, word_split_separator=",").convert_async(prompt="alpha,beta,gamma")
# before: 🅐🅛🄿🅗🄰 🄱🅴🅣🅰️ 🄶🅐🅜🄼🅰️   <- commas gone
# after:  🅐🅛🄿🅗🄰,🄱🅴🅣🅰️,🄶🅐🅜🄼🅰️   <- preserved

EmojiConverter and BinAsciiConverter both expose word_split_separator in their own constructors, so this is reachable through the public API.

Changes

  • join_words rejoins with the separator the words were split on.
  • A None separator splits on arbitrary whitespace, which has no single representation to restore, so it continues to join with a space.
  • Tests covering custom separators (,, |, -, ::), the unchanged default, the None case, and a custom separator combined with partial word selection.

Why this is safe

The change is a no-op for every caller that does not set a custom separator:

  • word_split_separator=" " — the default — already joined with " "
  • word_split_separator=None — still joins with " "

Only the custom-separator path changes, and it is currently broken. No existing test asserted the old behavior: tests/unit/converter/test_emoji_converter.py:53 sets word_split_separator="|" but asserts only the identifier params, not the output.

Subclasses that override join_words (BinaryConverter, NatoConverter, FirstLetterConverter, UnicodeReplacementConverter, BinAsciiConverter) keep their own joining behavior. BinAsciiConverter delegates to super().join_words outside its all-words encodings, so it picks up the fix on that path.

Tests

  • pytest tests/unit/converter/ — 1445 passed, 34 skipped, including the 5 new tests
  • ruff format --check and ruff check clean on both changed files

WordLevelConverter.convert_async splits the prompt on
word_split_separator but the default join_words always rejoined with a
space, so a custom separator was silently replaced in the output.
EmojiConverter and BinAsciiConverter both expose the parameter, so
"alpha,beta" converted with separator "," came back space-delimited.

Rejoin with the separator the words were split on. A None separator
splits on arbitrary whitespace, which has no single representation to
restore, so it keeps joining with a space.

The default separator is " ", so output is unchanged for every caller
that does not set a custom separator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@UmerkhanGolandaz
Umerkhan Golandaz (UmerkhanGolandaz) force-pushed the fix/word-level-converter-preserves-separator branch from dadf928 to 07bd921 Compare September 7, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: WordLevelConverter rejoins words with a space, discarding a custom word_split_separator

1 participant