Skip to content

textwrap should treat Unicode em-dash like ASCII em-dash #74865

Open
Open
textwrap should treat Unicode em-dash like ASCII em-dash#74865
Labels
stdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-featureA feature request or enhancement

Description

@jonathaneunice
jonathaneunice
mannequin
opened on Jun 15, 2017
BPO 30680
Nosy @bitdancer, @methane, @jonathaneunice
PRs
  • gh-74865: textwrap support for true (Unicode) em-dashes #2224
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2017-06-15.19:09:00.240>
labels = ['type-feature', 'library', '3.10']
title = 'textwrap should treat Unicode em-dash like ASCII em-dash'
updated_at = <Date 2020-10-21.04:33:09.340>
user = 'https://github.com/jonathaneunice'

    bugs.python.org fields:

    activity = <Date 2020-10-21.04:33:09.340>
actor = 'methane'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-06-15.19:09:00.240>
creator = 'jonathaneunice'
dependencies = []
files = []
hgrepos = []
issue_num = 30680
keywords = []
message_count = 4.0
messages = ['296124', '296126', '296127', '379189']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'methane', 'jonathaneunice']
pr_nums = ['2224']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30680'
versions = ['Python 3.10']

    Activity

    1. jonathaneunice commented on Jun 15, 2017

      @jonathaneunice
      jonathaneunicemannequin
      on Jun 15, 2017
      MannequinAuthor

      The textwrap module goes to great lengths to "do the right thing" when it finds the ASCII simulation of an em-dash (two or more consecutive hyphens), but it does nothing to recognize and similarly treat true (Unicode) em-dashes (aka '\N{EM DASH}', '\u2014', or U+2014). Real em-dashes should get at least as good a treatment as simulated em-dashes.

    2. jonathaneunice
      added
      3.7 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Jun 15, 2017

    3. bitdancer commented on Jun 15, 2017

      @bitdancer
      bitdancer
      on Jun 15, 2017
      Member

      This seems sensible to me (I haven't looked at the PR, I'm talking about adding the support). When textwrap was written python was pretty ascii oriented, so it is not too much of a surprise that unicode em dashes were not supported.

    4. jonathaneunice commented on Jun 15, 2017

      @jonathaneunice
      jonathaneunicemannequin
      on Jun 15, 2017
      MannequinAuthor

      Agreed. It makes great sense that textwrap started as highly ASCII-centric. But in the Python 3, Unicode-friendly era, ASCII-biased isn't where we should leave things.

    5. methane commented on Oct 21, 2020

      @methane
      methane
      on Oct 21, 2020
      Member

      Agreed. It makes great sense that textwrap started as highly ASCII-centric. But in the Python 3, Unicode-friendly era, ASCII-biased isn't where we should leave things.

      It needs Unicode experts. If we support Unicode, we should implemente UAX #14.
      http://www.unicode.org/reports/tr14/tr14-45.html

      But I am not sure some core developer love textwrap and Unicode enough to implement it.
      It can be implemented in 3rd party package before adding it in stdlib.

      Then, is U+2014 really important to implement even though we can not implement UAX#14 in foreseeable future?
      It doesn't make sense to me.

    6. methane
      added
      3.10 (EOL)end of life
      and removed
      3.7 (EOL)end of life
      on Oct 21, 2020
    7. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    8. encukou commented on Jan 3, 2025

      @encukou
      encukou
      on Jan 3, 2025
      Member

      I fully agree with @methane.
      IMO, textwrap's niche should be English docstrings: predominantly-ASCII blocks of text intended to be displayed in a monospace font.

      To get out of that niche, it would not only need correct line wrapping, but also for example support for double-width (蠎🐍) or zero-width characters. The scope could be extended practically infinitely. A third-party package on PyPI would be a much better place for this.

      Adding individual bits and pieces, like em-dash support, will either break output compatibility with or grow the already large number of TextWrapper options. I don't think it's worth it.

    9. picnixz
      removed
      3.10 (EOL)end of life
      on Mar 1, 2025
    10. StanFromIreland
      added
      topic-unicode
      on Aug 13, 2025

    11. DanielO commented on Aug 30, 2025

      @DanielO
      DanielO
      on Aug 30, 2025

      I think an escape hatch could be to allow the caller to supply a function to calculate string length.

      This would also fix my use case where I want to wrap strings which contain colour sequences :)

      eg something like so:

      --- /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/textwrap.py	2025-06-08 19:44:57
+++ mytextwrap.py	2025-08-30 13:53:33
@@ -61,6 +61,8 @@
         Truncate wrapped lines.
       placeholder (default: ' [...]')
         Append to the last line of truncated text.
+      lenfunc (default: len)
+        Function used to determine length of a given piece of text.
     """

     unicode_whitespace_trans = {}
@@ -125,7 +127,8 @@
                  tabsize=8,
                  *,
                  max_lines=None,
-                 placeholder=' [...]'):
+                 placeholder=' [...]',
+                 lenfunc = len):
         self.width = width
         self.initial_indent = initial_indent
         self.subsequent_indent = subsequent_indent
@@ -138,6 +141,7 @@
         self.tabsize = tabsize
         self.max_lines = max_lines
         self.placeholder = placeholder
+        self.lenfunc = lenfunc


     # -- Private methods -----------------------------------------------
@@ -217,7 +221,7 @@
         if self.break_long_words:
             end = space_left
             chunk = reversed_chunks[-1]
-            if self.break_on_hyphens and len(chunk) > space_left:
+            if self.break_on_hyphens and self.lenfunc(chunk) > space_left:
                 # break after last hyphen, but only if there are
                 # non-hyphens before it
                 hyphen = chunk.rfind('-', 0, space_left)
@@ -259,7 +263,7 @@
                 indent = self.subsequent_indent
             else:
                 indent = self.initial_indent
-            if len(indent) + len(self.placeholder.lstrip()) > self.width:
+            if self.lenfunc(indent) + self.lenfunc(self.placeholder.lstrip()) > self.width:
                 raise ValueError("placeholder too large for max width")

         # Arrange in reverse order so items can be efficiently popped
@@ -280,7 +284,7 @@
                 indent = self.initial_indent

             # Maximum width for this line.
-            width = self.width - len(indent)
+            width = self.width - self.lenfunc(indent)

             # First chunk on line is whitespace -- drop it, unless this
             # is the very beginning of the text (ie. no lines started yet).
@@ -288,7 +292,7 @@
                 del chunks[-1]

             while chunks:
-                l = len(chunks[-1])
+                l = self.lenfunc(chunks[-1])

                 # Can at least squeeze this chunk onto the current line.
                 if cur_len + l <= width:
@@ -301,13 +305,13 @@

             # The current line is full, and the next chunk is too big to
             # fit on *any* line (not just this one).
-            if chunks and len(chunks[-1]) > width:
+            if chunks and self.lenfunc(chunks[-1]) > width:
                 self._handle_long_word(chunks, cur_line, cur_len, width)
                 cur_len = sum(map(len, cur_line))

             # If the last chunk on this line is all whitespace, drop it.
             if self.drop_whitespace and cur_line and cur_line[-1].strip() == '':
-                cur_len -= len(cur_line[-1])
+                cur_len -= self.lenfunc(cur_line[-1])
                 del cur_line[-1]

             if cur_line:
@@ -323,16 +327,16 @@
                 else:
                     while cur_line:
                         if (cur_line[-1].strip() and
-                            cur_len + len(self.placeholder) <= width):
+                            cur_len + self.lenfunc(self.placeholder) <= width):
                             cur_line.append(self.placeholder)
                             lines.append(indent + ''.join(cur_line))
                             break
-                        cur_len -= len(cur_line[-1])
+                        cur_len -= self.lenfunc(cur_line[-1])
                         del cur_line[-1]
                     else:
                         if lines:
                             prev_line = lines[-1].rstrip()
-                            if (len(prev_line) + len(self.placeholder) <=
+                            if (self.lenfunc(prev_line) + self.lenfunc(self.placeholder) <=
                                     self.width):
                                 lines[-1] = prev_line + self.placeholder
                                 break
    12. StanFromIreland
      marked Textwrap to conform to UAX #14 #86141 as a duplicate of this issue on Oct 31, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions