Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,11 @@ jobs:
- name: Install dependencies
run: |
pip install -U codecov tox-gh-actions
pip install -r requirements_dev.txt
if [[ "${{ matrix.py }}" == "3.13t" ]]; then
pip install -r requirements_dev.txt 'mypy<2'
else
pip install -r requirements_dev.txt
fi
- name: Test with tox
run: tox
- name: Check Code Coverage
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
## Unreleased

* [FIX] match Node `qs` 6.16.0 for dotted root keys and dates returned by callable encoding filters
* [FIX] allow `max_depth=0` for root scalar encoding while rejecting nested children
* [FIX] enforce raising comma-group limits inside bracket assignments and spread overflow duplicate values one level

## 1.6.1

* [FIX] match Node `qs` 6.15.3 cumulative list-limit enforcement across duplicate-key combinations and mixed list merges
Expand Down
25 changes: 23 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -536,8 +536,13 @@ default, or raises ``ValueError`` when ``raise_on_limit_exceeded=True``.
) == {'a': {'0': 'x', '1': 'y'}}

With ``comma=True``, a flat comma value is subject to the same limit. A value
assigned through ``[]=`` counts as one outer list element, so its inner
comma-separated group may contain more values than ``list_limit``.
assigned through ``[]=`` counts as one outer list element. Without raising,
its inner comma-separated group may exceed ``list_limit`` and stays nested;
with ``raise_on_limit_exceeded=True``, each oversized inner group raises too.
When duplicate comma values extend an already-overflowed numeric-keyed mapping,
an incoming list or tuple spreads into successive numeric keys. If a later comma
group also exceeds ``list_limit``, it becomes an overflow mapping stored under
one key instead. Bracketed comma groups remain one nested value apiece.

To disable ``list`` parsing entirely, set `parse_lists <https://techouse.github.io/qs_codec/qs_codec.models.html#qs_codec.models.decode_options.DecodeOptions.parse_lists>`__
to ``False``.
Expand Down Expand Up @@ -642,6 +647,13 @@ option. If unset, traversal is unbounded by this option. When set, the provided
except ValueError as e:
assert str(e) == 'Maximum encoding depth exceeded'

``max_depth=0`` permits root scalar values but rejects any nested child:

.. code:: python

assert qs.encode({'a': 'b'}, qs.EncodeOptions(max_depth=0)) == 'a=b'


This encoding can also be replaced by a custom ``Callable`` in the
`encoder <https://techouse.github.io/qs_codec/qs_codec.models.html#qs_codec.models.encode_options.EncodeOptions.encoder>`__ option:

Expand Down Expand Up @@ -814,6 +826,11 @@ You may encode dots in keys of ``dict``\s by setting
),
) == 'name%252Eobj.first=John&name%252Eobj.last=Doe'

assert qs.encode(
{'a.b': 'x'},
qs.EncodeOptions(allow_dots=True, encode_dot_in_keys=True),
) == 'a%252Eb=x'

**Caveat:** When both `encode_values_only <https://techouse.github.io/qs_codec/qs_codec.models.html#qs_codec.models.encode_options.EncodeOptions.encode_values_only>`__
and `encode_dot_in_keys <https://techouse.github.io/qs_codec/qs_codec.models.html#qs_codec.models.encode_options.EncodeOptions.encode_dot_in_keys>`__ are set to
``True``, only dots in keys and nothing else will be encoded!
Expand Down Expand Up @@ -922,6 +939,10 @@ objects, you can provide a ``Callable`` in the
== "a=7"
)

The callable ``filter`` runs before date serialization. A date retained or
returned by the filter still passes through ``serialize_date``.


To affect the order of parameter keys, you can set a ``Callable`` in the
`sort <https://techouse.github.io/qs_codec/qs_codec.models.html#qs_codec.models.encode_options.EncodeOptions.sort>`__ option:

Expand Down
25 changes: 23 additions & 2 deletions docs/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -443,8 +443,13 @@ default, or raises ``ValueError`` when
) == {'a': {'0': 'x', '1': 'y'}}

With ``comma=True``, a flat comma value is subject to the same limit. A value
assigned through ``[]=`` counts as one outer list element, so its inner
comma-separated group may contain more values than ``list_limit``.
assigned through ``[]=`` counts as one outer list element. Without raising,
its inner comma-separated group may exceed ``list_limit`` and stays nested;
with ``raise_on_limit_exceeded=True``, each oversized inner group raises too.
When duplicate comma values extend an already-overflowed numeric-keyed mapping,
an incoming list or tuple spreads into successive numeric keys. If a later comma
group also exceeds ``list_limit``, it becomes an overflow mapping stored under
one key instead. Bracketed comma groups remain one nested value apiece.

To disable ``list`` parsing entirely, set :py:attr:`parse_lists <qs_codec.models.decode_options.DecodeOptions.parse_lists>`
to ``False``.
Expand Down Expand Up @@ -549,6 +554,13 @@ If unset, traversal is unbounded by this option. When set, the provided limit is
except ValueError as e:
assert str(e) == 'Maximum encoding depth exceeded'

``max_depth=0`` permits root scalar values but rejects any nested child:

.. code:: python

assert qs.encode({'a': 'b'}, qs.EncodeOptions(max_depth=0)) == 'a=b'


This encoding can also be replaced by a custom ``Callable`` in the
:py:attr:`encoder <qs_codec.models.encode_options.EncodeOptions.encoder>` option:

Expand Down Expand Up @@ -720,6 +732,11 @@ You may encode dots in keys of ``dict``\s by setting
),
) == 'name%252Eobj.first=John&name%252Eobj.last=Doe'

assert qs.encode(
{'a.b': 'x'},
qs.EncodeOptions(allow_dots=True, encode_dot_in_keys=True),
) == 'a%252Eb=x'

**Caveat:** When both :py:attr:`encode_values_only <qs_codec.models.encode_options.EncodeOptions.encode_values_only>`
and :py:attr:`encode_dot_in_keys <qs_codec.models.encode_options.EncodeOptions.encode_dot_in_keys>` are set to
``True``, only dots in keys and nothing else will be encoded!
Expand Down Expand Up @@ -828,6 +845,10 @@ objects, you can provide a ``Callable`` in the
== "a=7"
)

The callable ``filter`` runs before date serialization. A date retained or
returned by the filter still passes through ``serialize_date``.


To affect the order of parameter keys, you can set a ``Callable`` in the
:py:attr:`sort <qs_codec.models.encode_options.EncodeOptions.sort>` option:

Expand Down
17 changes: 8 additions & 9 deletions src/qs_codec/decode.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,23 +227,22 @@ def _parse_array_value(
Behavior
--------
- If ``comma=True`` and ``value`` is a string that contains commas, split into a list.
When ``enforce_comma_limit`` is ``True``, over-limit comma values raise or degrade to an ``OverflowDict`` here.
Raw query-string parsing and mapping key paths ending in ``[]`` pass ``False`` so the caller can account for
bracket-array key context first.
With ``raise_on_limit_exceeded=True``, an over-limit comma group raises before splitting or decoding,
including values assigned to ``[]``. Otherwise ``enforce_comma_limit`` controls conversion of an
over-limit group to ``CommaOverflowDict``; bracket assignments defer conversion until after wrapping.
- Otherwise, enforce the per-list length limit by comparing ``current_list_length`` to ``options.list_limit``.
When ``raise_on_limit_exceeded=True``, violations raise ``ValueError``.
- When ``list_limit`` is negative, any non-empty comma split exceeds the limit: raising mode raises,
while non-raising mode degrades to an ``OverflowDict``/``CommaOverflowDict``. Raw query-string
parsing temporarily returns the split list when ``enforce_comma_limit=False`` so the caller can
apply bracket-array wrapping before the final limit check.
while non-raising mode degrades to an ``OverflowDict``/``CommaOverflowDict``. Bracket assignments
temporarily return the split list so the caller can apply wrapping before the final limit check.

Returns
-------
Any
Either the original value or a list of values, without decoding (that happens later).
"""
if isinstance(value, str) and value and options.comma and "," in value:
if enforce_comma_limit and options.raise_on_limit_exceeded:
if options.raise_on_limit_exceeded:
comma_count = 0
comma_index = value.find(",")
while comma_index >= 0:
Expand Down Expand Up @@ -401,7 +400,7 @@ def _parse_query_string_values(value: str, options: DecodeOptions) -> t.Dict[str
list_limit_exceeded = len(val) > options.list_limit
if list_limit_exceeded and isinstance(val, (list, tuple)):
if options.raise_on_limit_exceeded:
raise ValueError(_list_limit_exceeded_message(options.list_limit))
raise ValueError(_list_limit_exceeded_message(options.list_limit)) # pragma: no cover - pre-split guard
val = CommaOverflowDict({str(i): item for i, item in enumerate(val)})

existing: bool = key in obj
Expand Down Expand Up @@ -502,7 +501,7 @@ def _parse_object(
leaf = [leaf]
if len(leaf) > options.list_limit:
if options.raise_on_limit_exceeded:
raise ValueError(_list_limit_exceeded_message(options.list_limit))
raise ValueError(_list_limit_exceeded_message(options.list_limit)) # pragma: no cover - pre-split guard
leaf = CommaOverflowDict({str(i): item for i, item in enumerate(leaf)})

# Walk the chain from the leaf to the root, building nested containers on the way out.
Expand Down
17 changes: 8 additions & 9 deletions src/qs_codec/encode.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ def encode(value: t.Any, options: t.Optional[EncodeOptions] = None) -> str:
value=None if key_is_undefined else obj_value,
is_undefined=key_is_undefined,
side_channel=side_channel,
prefix=_key,
prefix=_key.replace(".", "%2E") if opts.encode_dot_in_keys else _key,
generate_array_prefix=list_format.generator,
comma_round_trip=comma_round_trip,
comma_compact_nulls=list_format == ListFormat.COMMA and opts.comma_compact_nulls,
Expand Down Expand Up @@ -809,14 +809,13 @@ def _append_child_result(frame: EncodeFrame, encoded: t.Any) -> None:

if callable(filter_opt):
obj = filter_opt(current_path.materialize(), obj)
else:
if isinstance(obj, datetime):
obj = frame.serialize_date(obj) if callable(frame.serialize_date) else obj.isoformat()
elif frame.generate_array_prefix is _COMMA_GENERATOR and isinstance(obj, (list, tuple)):
if callable(frame.serialize_date):
obj = [frame.serialize_date(x) if isinstance(x, datetime) else x for x in obj]
else:
obj = [x.isoformat() if isinstance(x, datetime) else x for x in obj]
if isinstance(obj, datetime):
obj = frame.serialize_date(obj) if callable(frame.serialize_date) else obj.isoformat()
elif frame.generate_array_prefix is _COMMA_GENERATOR and isinstance(obj, (list, tuple)):
if callable(frame.serialize_date):
obj = [frame.serialize_date(x) if isinstance(x, datetime) else x for x in obj]
else:
obj = [x.isoformat() if isinstance(x, datetime) else x for x in obj]

if not frame.is_undefined and obj is None:
if frame.strict_null_handling:
Expand Down
7 changes: 4 additions & 3 deletions src/qs_codec/models/decode_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ class DecodeOptions:
a list. Above the limit, decoding either uses a numeric-keyed mapping or raises ``ValueError``
when ``raise_on_limit_exceeded=True``.

For bracket-array assignments such as ``foo[]=1,2,3``, the comma-split payload is wrapped
as a single outer list element, so the inner payload may contain more values than
``list_limit`` while still respecting the outer container limit.
For bracket-array assignments such as ``foo[]=1,2,3``, each comma-split payload is wrapped as one
outer list element. With ``raise_on_limit_exceeded=True``, an oversized inner comma group raises
independently of the outer list limit. Otherwise it remains a nested group, while exceeding the
outer list limit changes the result to a numeric-keyed mapping.
"""

charset: Charset = Charset.UTF8
Expand Down
4 changes: 2 additions & 2 deletions src/qs_codec/models/encode_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ def __post_init__(self) -> None:
if not hasattr(self, "_encoder") or self._encoder is None:
self._encoder = EncodeUtils.encode
if self.max_depth is not None:
if not isinstance(self.max_depth, int) or isinstance(self.max_depth, bool) or self.max_depth <= 0:
raise ValueError("max_depth must be a positive integer or None")
if not isinstance(self.max_depth, int) or isinstance(self.max_depth, bool) or self.max_depth < 0:
raise ValueError("max_depth must be a non-negative integer or None")
# Default `encode_dot_in_keys` first, then mirror into `allow_dots` when unspecified.
if self.encode_dot_in_keys is None:
self.encode_dot_in_keys = False
Expand Down
17 changes: 8 additions & 9 deletions src/qs_codec/utils/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -536,12 +536,9 @@ def combine(
Concatenate two values, treating non-sequences as singletons.

Normal list/tuple inputs are flattened into the combined result. When
``a`` is already an :class:`OverflowDict`, however, ``b`` is appended as
one value at the next numeric key, even if ``b`` is a list, tuple, or
another :class:`OverflowDict`. This preserves qs parity for duplicate
values after list-limit overflow: a later comma-split or bracket-array
payload remains a nested value instead of being flattened into the
overflowed container.
``a`` is already an :class:`OverflowDict`, top-level list/tuple elements
from ``b`` are appended at successive numeric keys; a nested list remains
one group, and an incoming overflow mapping remains one copied value.

If `list_limit` is exceeded, converts the list to an `OverflowDict`
(a dict with numeric keys) to prevent memory exhaustion.
Expand All @@ -560,15 +557,17 @@ def combine(
raise ValueError(
f"List limit exceeded: Only {limit} element{'' if limit == 1 else 's'} allowed in a list."
)
# a is already an OverflowDict. Append b as one value at the next numeric index.
# Copy on write; append top-level values after the highest numeric index.
orig_a: OverflowDict = t.cast(OverflowDict, a)
a_copy: OverflowDict = orig_a.__class__({k: v for k, v in orig_a.items() if not isinstance(v, Undefined)})
# Use max key + 1 to handle sparse dicts safely, rather than len(a)
key_pairs: t.List[t.Tuple[int, str]] = _numeric_key_pairs(a_copy)
idx: int = (max(key for key, _ in key_pairs) + 1) if key_pairs else 0

if not isinstance(b, Undefined):
a_copy[str(idx)] = _copy_overflow_append_value(b)
for value in b if isinstance(b, (list, tuple)) else (b,):
if not isinstance(value, Undefined):
a_copy[str(idx)] = _copy_overflow_append_value(value)
idx += 1
return a_copy

# Normal combination: flatten lists/tuples
Expand Down
4 changes: 2 additions & 2 deletions tests/comparison/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
"description": "A comparison of query string parsing libraries",
"author": "Klemen Tusar",
"license": "BSD-3-Clause",
"packageManager": "pnpm@11.9.0",
"packageManager": "pnpm@12.6.0",
"dependencies": {
"qs": "^6.15.3"
"qs": "^6.16.0"
}
}
Loading
Loading