gh-62222: Fix and extend quoted-printable decoding (GH-152679)

binascii.a2b_qp() and quopri now leave an invalid "=" escape in place and
rescan the following character instead of consuming it, like email.quoprimime
and other decoders (b'==41' now decodes to b'=A').

Add an opt-in strip_ws parameter to binascii.a2b_qp(), quopri.decode(),
quopri.decodestring() and email.quoprimime.decode() to strip trailing
whitespace per RFC 2045.  It is false by default everywhere except
email.quoprimime.decode(), preserving every existing default.

Also document that email's get_payload(decode=True) does not strip trailing
whitespace, contrary to RFC 2045.

Register the new "strip_ws" argument name in the global objects tables
(make regen-global-objects), needed by static builds where binascii is
compiled into the core, and replace an unresolved :func: reference to the
undocumented email.quoprimime.decode with a literal.

* gh-62222: Make strip_ws parameters keyword-only

---------

Co-authored-by: Gregory P. Smith <greg@krypto.org>

Author: serhiy-storchaka

Doc/library/binascii.rst

@@ -283,11 +283,16 @@ The :mod:`!binascii` module defines the following functions:
 
    .. versionadded:: 3.15
 
-.. function:: a2b_qp(data, header=False)
+.. function:: a2b_qp(data, header=False, *, strip_ws=False)
 
    Convert a block of quoted-printable data back to binary and return the binary
    data. More than one line may be passed at a time. If the optional argument
    *header* is present and true, underscores will be decoded as spaces.
+   If the optional argument *strip_ws* is true,
+   trailing whitespace is stripped from each line, as required by :rfc:`2045`.
+
+   .. versionchanged:: next
+      Added the *strip_ws* parameter.
 
 
 .. function:: b2a_qp(data, quotetabs=False, istext=True, header=False)

Doc/library/email.compat32-message.rst

@@ -214,6 +214,14 @@ Here are the methods of the :class:`Message` class:
       defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or
       :class:`~email.errors.InvalidBase64CharactersDefect`, respectively).
 
+      .. note::
+
+         A ``quoted-printable`` payload is decoded without stripping
+         trailing whitespace, contrary to :rfc:`2045` but matching
+         common mail clients.
+         Use :func:`binascii.a2b_qp` with ``strip_ws=True``
+         (or ``email.quoprimime.decode``) for RFC-compliant decoding.
+
       When *decode* is ``False`` (the default) the body is returned as a string
       without decoding the :mailheader:`Content-Transfer-Encoding`.  However,
       for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made

Doc/library/quopri.rst

@@ -20,14 +20,19 @@ few nonprintable characters; the base64 encoding scheme available via the
 :mod:`base64` module is more compact if there are many such characters, as when
 sending a graphics file.
 
-.. function:: decode(input, output, header=False)
+.. function:: decode(input, output, header=False, *, strip_ws=False)
 
    Decode the contents of the *input* file and write the resulting decoded binary
    data to the *output* file. *input* and *output* must be :term:`binary file objects
    <file object>`.  If the optional argument *header* is present and true, underscore
    will be decoded as space. This is used to decode "Q"-encoded headers as
    described in :rfc:`1522`: "MIME (Multipurpose Internet Mail Extensions)
    Part Two: Message Header Extensions for Non-ASCII Text".
+   If the optional argument *strip_ws* is true,
+   trailing whitespace is stripped from each line, as required by :rfc:`2045`.
+
+   .. versionchanged:: next
+      Added the *strip_ws* parameter.
 
 
 .. function:: encode(input, output, quotetabs, header=False)
@@ -43,11 +48,14 @@ sending a graphics file.
    as underscores as per :rfc:`1522`.
 
 
-.. function:: decodestring(s, header=False)
+.. function:: decodestring(s, header=False, *, strip_ws=False)
 
    Like :func:`decode`, except that it accepts a source :class:`bytes` and
    returns the corresponding decoded :class:`bytes`.
 
+   .. versionchanged:: next
+      Added the *strip_ws* parameter.
+
 
 .. function:: encodestring(s, quotetabs=False, header=False)
 

Doc/whatsnew/3.16.rst

@@ -614,6 +614,16 @@ pickle
   :class:`!WeakMethod` and :class:`!WeakSet` can now also be copied.
   (Contributed by Serhiy Storchaka in :gh:`74876`.)
 
+quopri
+------
+
+* :func:`quopri.decode`, :func:`quopri.decodestring` and
+  :func:`binascii.a2b_qp` gained a *strip_ws* parameter.  When true, trailing
+  whitespace is stripped from each line while decoding, as required by
+  :rfc:`2045` for a quoted-printable body.
+  (Contributed by Serhiy Storchaka in :gh:`62222`.)
+
+
 re
 --
 

Include/internal/pycore_global_objects_fini_generated.h

@@ -853,6 +853,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(strict)
         STRUCT_FOR_ID(strict_mode)
         STRUCT_FOR_ID(string)
+        STRUCT_FOR_ID(strip_ws)
         STRUCT_FOR_ID(sub_key)
         STRUCT_FOR_ID(subcalls)
         STRUCT_FOR_ID(symmetric_difference_update)

Include/internal/pycore_global_strings.h

@@ -229,10 +229,14 @@ def body_encode(body, maxlinelen=76, eol=NL):
 
 # BAW: I'm not sure if the intent was for the signature of this function to be
 # the same as base64MIME.decode() or not...
-def decode(encoded, eol=NL):
+def decode(encoded, eol=NL, *, strip_ws=True):
     """Decode a quoted-printable string.
 
     Lines are separated with eol, which defaults to \\n.
+
+    If strip_ws is true (the default), whitespace at the end of a line is
+    stripped, as required by RFC 2045 when decoding a quoted-printable body.
+    Pass strip_ws=False to keep it.
     """
     if not encoded:
         return encoded
@@ -242,7 +246,8 @@ def decode(encoded, eol=NL):
     decoded = ''
 
     for line in encoded.splitlines():
-        line = line.rstrip()
+        if strip_ws:
+            line = line.rstrip()
         if not line:
             decoded += eol
             continue

Include/internal/pycore_runtime_init_generated.h

@@ -109,14 +109,16 @@ def encodestring(s, quotetabs=False, header=False):
 
 
 
-def decode(input, output, header=False):
+def decode(input, output, header=False, *, strip_ws=False):
     """Read 'input', apply quoted-printable decoding, and write to 'output'.
     'input' and 'output' are binary file objects.
-    If 'header' is true, decode underscore as space (per RFC 1522)."""
+    If 'header' is true, decode underscore as space (per RFC 1522).
+    If 'strip_ws' is true, strip whitespace at the end of a line (per
+    RFC 2045)."""
 
     if a2b_qp is not None:
         data = input.read()
-        odata = a2b_qp(data, header=header)
+        odata = a2b_qp(data, header=header, strip_ws=strip_ws)
         output.write(odata)
         return
 
@@ -125,11 +127,19 @@ def decode(input, output, header=False):
         i, n = 0, len(line)
         if n > 0 and line[n-1:n] == b'\n':
             partial = 0; n = n-1
-            # Strip trailing whitespace
-            while n > 0 and line[n-1:n] in b" \t\r":
-                n = n-1
+            # Separate off the line ending (keeping it to re-add after
+            # decoding) so that a trailing "=" -- possibly before the "\r" of
+            # a "\r\n" pair -- is recognized as a soft line break.
+            if n > 0 and line[n-1:n] == b'\r':
+                n = n-1; eol = b'\r\n'
+            else:
+                eol = b'\n'
         else:
-            partial = 1
+            partial = 1; eol = b''
+        if strip_ws:
+            # Strip trailing whitespace (RFC 2045).
+            while n > 0 and line[n-1:n] in b" \t":
+                n = n-1
         while i < n:
             c = line[i:i+1]
             if c == b'_' and header:
@@ -138,25 +148,23 @@ def decode(input, output, header=False):
                 new = new + c; i = i+1
             elif i+1 == n and not partial:
                 partial = 1; break
-            elif i+1 < n and line[i+1:i+2] == ESCAPE:
-                new = new + ESCAPE; i = i+2
             elif i+2 < n and ishex(line[i+1:i+2]) and ishex(line[i+2:i+3]):
                 new = new + bytes((unhex(line[i+1:i+3]),)); i = i+3
             else: # Bad escape sequence -- leave it in
                 new = new + c; i = i+1
         if not partial:
-            output.write(new + b'\n')
+            output.write(new + eol)
             new = b''
     if new:
         output.write(new)
 
-def decodestring(s, header=False):
+def decodestring(s, header=False, *, strip_ws=False):
     if a2b_qp is not None:
-        return a2b_qp(s, header=header)
+        return a2b_qp(s, header=header, strip_ws=strip_ws)
     from io import BytesIO
     infp = BytesIO(s)
     outfp = BytesIO()
-    decode(infp, outfp, header=header)
+    decode(infp, outfp, header=header, strip_ws=strip_ws)
     return outfp.getvalue()