Skip to content

Commit cd42e2f

Browse files
authored
feat(transport): set compression and User-Agent request defaults (#195)
* feat(transport): set compression and User-Agent request defaults The generated client left two server-observable defaults to urllib3. urllib3 puts `Accept-Encoding: identity` on every connection, which is not "no preference" but an explicit request not to compress, so every JSON response came back uncompressed. RESTClientObject.request now advertises urllib3.util.request.ACCEPT_ENCODING -- exactly the codecs the installed urllib3 can transparently decode, so a server can never negotiate an encoding that arrives as undecodable bytes. It is a setdefault, so an operation whose payload is already compressed can still pass `identity`. Requests also identified themselves as OpenAPI-Generator/1.0.0/python, which attributes traffic to neither the SDK nor a release. The generator's httpUserAgent property only bakes a literal at generation time, and regens follow spec changes rather than releases, so a baked version would go stale between them; the string is resolved at import time from installed package metadata instead and lives in the generator-ignored hotdata/_useragent.py. rest.py and api_client.py are generator output, so the edits are re-applied by scripts/patch_request_defaults.py, wired into regenerate.yml alongside the existing patch steps. * fix(transport): match an Accept-Encoding override case-insensitively Header names are case-insensitive, so a caller passing `accept-encoding` left both keys in the dict. urllib3 emits one header line per key, so the server received the opt-out *and* the compressed set and could still compress -- silently losing the documented per-request escape hatch. Compare lowercased, the way urllib3 does before adding its own Accept-Encoding. * perf(arrow): keep Arrow IPC responses uncompressed _call_arrow set only Accept, so the new client-wide compression default applied to Arrow fetches too. IPC record batches are frequently LZ4- or ZSTD-compressed by the writer already, making a gzip pass over the stream CPU on both ends for little size gain. Opt the path out with `Accept-Encoding: identity`, restoring the transfer behavior Arrow had before the default was introduced.
1 parent 6f03dfa commit cd42e2f

10 files changed

Lines changed: 401 additions & 3 deletions

File tree

‎.github/workflows/regenerate.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,9 @@ jobs:
198198
- name: Patch default client exports (enhanced query/results)
199199
run: python3 scripts/patch_query_exports.py
200200

201+
- name: Patch transport request defaults (Accept-Encoding, User-Agent)
202+
run: python3 scripts/patch_request_defaults.py
203+
201204
# The API-token -> JWT key exchange is deprecated: the configured API token
202205
# is the bearer credential and goes on the wire verbatim. This check is the
203206
# inverse of the old "did the exchange survive?" guard — it fails if the

‎.openapi-generator-ignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,14 @@ setup.py
99
# truth for "hand-maintained, don't touch": arrow.py (Arrow IPC result fetch),
1010
# query.py (429 retry + truncation auto-follow, #688), _retry.py (pre-response
1111
# connection-reset retry on all methods, #118), uploads.py (transparent
12-
# presigned direct-to-storage upload flow).
12+
# presigned direct-to-storage upload flow), _useragent.py (runtime-resolved
13+
# User-Agent; the generator's httpUserAgent property can only bake a literal at
14+
# generation time, which goes stale between releases).
1315
hotdata/arrow.py
1416
hotdata/query.py
1517
hotdata/_retry.py
1618
hotdata/uploads.py
19+
hotdata/_useragent.py
1720

1821
# Hand-written test for the patched ApiClient.close()/context-manager behavior
1922
# (re-applied by scripts/patch_api_client_close.py). It lives in the generated

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2828
compatibility. Generated docstrings pick up the new wording on the next client
2929
regeneration from the updated OpenAPI spec.
3030
- feat(query): add dialect parameter to query request
31+
- perf(transport): request compressed responses. urllib3 defaults every
32+
connection to `Accept-Encoding: identity`, which asks the server *not* to
33+
compress, so every JSON response came back uncompressed. The client now
34+
advertises `urllib3.util.request.ACCEPT_ENCODING` — exactly the codecs the
35+
installed urllib3 can transparently decode. Response bodies are unchanged;
36+
a caller or operation can still pass an explicit `Accept-Encoding` (matched
37+
case-insensitively). Arrow IPC fetches opt out and stay uncompressed, since
38+
their record batches are frequently compressed by the writer already.
39+
- feat(transport): send an SDK `User-Agent`. Requests identified themselves as
40+
`OpenAPI-Generator/1.0.0/python`; they now send
41+
`hotdata-python/<version> (Python/<py>; urllib3/<urllib3>)`. Setting
42+
`ApiClient.user_agent` still overrides it.
3143

3244
## [0.10.0] - 2026-08-18
3345

‎hotdata/_useragent.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""The SDK's default ``User-Agent``.
2+
3+
The generator's default is ``OpenAPI-Generator/1.0.0/python``, which identifies
4+
neither the SDK nor its version — server-side telemetry cannot tell a hotdata
5+
client from any other generated client, let alone one release from another.
6+
7+
openapi-generator *does* expose an ``httpUserAgent`` property for this, but it
8+
bakes a literal string in at **generation** time. Regeneration is driven by
9+
OpenAPI spec changes, not by releases, so a baked version string reports
10+
whatever the version happened to be at the last regen — it would go stale
11+
silently and misattribute traffic, which is worse than the generic default. The
12+
version is therefore resolved at import time from installed package metadata,
13+
the same source :mod:`hotdata.__init__` uses for ``__version__``.
14+
15+
This lives outside the generated modules (see ``.openapi-generator-ignore``) so
16+
a regeneration cannot overwrite it; ``scripts/patch_request_defaults.py`` only
17+
has to re-point ``ApiClient`` at the constant.
18+
"""
19+
20+
from __future__ import annotations
21+
22+
import importlib.metadata
23+
import platform
24+
25+
import urllib3
26+
27+
28+
def _default_user_agent() -> str:
29+
"""``hotdata-python/<version> (Python/<py>; urllib3/<urllib3>)``.
30+
31+
The runtime versions ride along because the transport stack is where
32+
client-side failures usually originate: knowing which urllib3 a bug report
33+
came from is the difference between reproducing a problem and guessing.
34+
"""
35+
try:
36+
sdk_version = importlib.metadata.version("hotdata")
37+
except importlib.metadata.PackageNotFoundError:
38+
# Running from a source checkout without an install.
39+
sdk_version = "0.0.0+unknown"
40+
return (
41+
f"hotdata-python/{sdk_version} "
42+
f"(Python/{platform.python_version()}; urllib3/{urllib3.__version__})"
43+
)
44+
45+
46+
#: Sent as ``User-Agent`` on every request. ``ApiClient.user_agent`` still
47+
#: overrides it per client, which is the supported way for an application to
48+
#: identify itself.
49+
USER_AGENT = _default_user_agent()
50+
51+
52+
__all__ = ["USER_AGENT", "_default_user_agent"]

‎hotdata/api_client.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929

3030
from hotdata.configuration import Configuration
3131
from hotdata.api_response import ApiResponse, T as ApiResponseT
32+
from hotdata._useragent import USER_AGENT
3233
import hotdata.models
3334
from hotdata import rest
3435
from hotdata.exceptions import (
@@ -91,7 +92,7 @@ def __init__(
9192
self.default_headers[header_name] = header_value
9293
self.cookie = cookie
9394
# Set default User-Agent.
94-
self.user_agent = 'OpenAPI-Generator/1.0.0/python'
95+
self.user_agent = USER_AGENT
9596
self.client_side_validation = configuration.client_side_validation
9697

9798
def __enter__(self):

‎hotdata/arrow.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,16 @@ def _call_arrow(
174174
# Override only what we need: the Accept header and the format query.
175175
# `GET /v1/results/{id}` is database-scoped, so the required
176176
# X-Database-Id header flows through the generated serializer too.
177-
headers: Dict[str, Any] = {"Accept": ARROW_STREAM_MEDIA_TYPE}
177+
# `Accept-Encoding: identity` opts this path out of the client-wide
178+
# response compression default (hotdata/rest.py). Arrow IPC record
179+
# batches are frequently LZ4/ZSTD-compressed by the writer already, so
180+
# a gzip pass over the stream burns CPU on both ends for little size
181+
# gain. Drop it if the endpoint is measured to serve uncompressed
182+
# batches, where columnar data does compress well.
183+
headers: Dict[str, Any] = {
184+
"Accept": ARROW_STREAM_MEDIA_TYPE,
185+
"Accept-Encoding": "identity",
186+
}
178187
params = self._get_result_serialize(
179188
id=id,
180189
x_database_id=x_database_id,

‎hotdata/rest.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
import ssl
2020

2121
import urllib3
22+
from urllib3.util.request import ACCEPT_ENCODING
2223

2324
from hotdata.exceptions import ApiException, ApiValueError
2425

@@ -164,6 +165,25 @@ def request(
164165
post_params = post_params or {}
165166
headers = headers or {}
166167

168+
# Ask for compressed responses. urllib3 defaults every connection to
169+
# `Accept-Encoding: identity`, which is not "no preference" but an
170+
# explicit request *not* to compress, and a spec-compliant server
171+
# honors it. ACCEPT_ENCODING is built from the codecs the installed
172+
# urllib3 can actually decode (gzip/deflate, plus br/zstd when their
173+
# backends are present), so the server can never negotiate an encoding
174+
# that reaches us as undecodable bytes. urllib3 decodes the body
175+
# transparently, so callers are unaffected.
176+
#
177+
# A default, not an override: an operation whose payload is already
178+
# compressed end-to-end passes `identity` and stays in control. The
179+
# check is case-insensitive because header names are -- a caller
180+
# passing `accept-encoding` would otherwise leave both keys in the
181+
# dict and urllib3 would emit two header lines, so the server would
182+
# see the opt-out *and* the compressed set. urllib3 lowercases names
183+
# the same way before adding its own Accept-Encoding.
184+
if not any(key.lower() == 'accept-encoding' for key in headers):
185+
headers['Accept-Encoding'] = ACCEPT_ENCODING
186+
167187
timeout = None
168188
if _request_timeout:
169189
if isinstance(_request_timeout, (int, float)):

‎scripts/patch_request_defaults.py‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/usr/bin/env python3
2+
"""Re-apply the SDK's transport request defaults after OpenAPI regeneration.
3+
4+
Two defaults the generator leaves on urllib3's (server-observable) behavior:
5+
6+
* ``Accept-Encoding``: urllib3 puts ``identity`` on every connection, which
7+
asks the server *not* to compress. We advertise
8+
``urllib3.util.request.ACCEPT_ENCODING`` instead — exactly the codecs the
9+
installed urllib3 can transparently decode.
10+
* ``User-Agent``: ``OpenAPI-Generator/1.0.0/python`` identifies neither the SDK
11+
nor its version, so server-side telemetry cannot attribute traffic.
12+
"""
13+
14+
from __future__ import annotations
15+
16+
import pathlib
17+
import sys
18+
19+
ROOT = pathlib.Path(__file__).resolve().parents[1]
20+
21+
22+
def patch_accept_encoding() -> None:
23+
"""Advertise compression on every request made through RESTClientObject."""
24+
path = ROOT / "hotdata" / "rest.py"
25+
src = path.read_text()
26+
27+
if "ACCEPT_ENCODING" in src:
28+
return
29+
30+
import_needle = "import urllib3\n"
31+
import_replacement = "import urllib3\nfrom urllib3.util.request import ACCEPT_ENCODING\n"
32+
if import_needle not in src:
33+
sys.exit(f"Failed to patch {path}: urllib3 import anchor not found")
34+
src = src.replace(import_needle, import_replacement, 1)
35+
36+
needle = " post_params = post_params or {}\n headers = headers or {}\n"
37+
replacement = (
38+
" post_params = post_params or {}\n"
39+
" headers = headers or {}\n\n"
40+
" # Ask for compressed responses. urllib3 defaults every connection to\n"
41+
" # `Accept-Encoding: identity`, which is not \"no preference\" but an\n"
42+
" # explicit request *not* to compress, and a spec-compliant server\n"
43+
" # honors it. ACCEPT_ENCODING is built from the codecs the installed\n"
44+
" # urllib3 can actually decode (gzip/deflate, plus br/zstd when their\n"
45+
" # backends are present), so the server can never negotiate an encoding\n"
46+
" # that reaches us as undecodable bytes. urllib3 decodes the body\n"
47+
" # transparently, so callers are unaffected.\n"
48+
" #\n"
49+
" # A default, not an override: an operation whose payload is already\n"
50+
" # compressed end-to-end passes `identity` and stays in control. The\n"
51+
" # check is case-insensitive because header names are -- a caller\n"
52+
" # passing `accept-encoding` would otherwise leave both keys in the\n"
53+
" # dict and urllib3 would emit two header lines, so the server would\n"
54+
" # see the opt-out *and* the compressed set. urllib3 lowercases names\n"
55+
" # the same way before adding its own Accept-Encoding.\n"
56+
" if not any(key.lower() == 'accept-encoding' for key in headers):\n"
57+
" headers['Accept-Encoding'] = ACCEPT_ENCODING\n"
58+
)
59+
if needle not in src:
60+
sys.exit(f"Failed to patch {path}: request() header anchor not found")
61+
src = src.replace(needle, replacement, 1)
62+
63+
path.write_text(src)
64+
65+
66+
def patch_user_agent() -> None:
67+
"""Point ApiClient at the hand-maintained User-Agent constant.
68+
69+
The string itself lives in ``hotdata/_useragent.py`` (generator-ignored), so
70+
this patch is just an import plus the assignment — two anchors instead of
71+
carrying the logic inside generated output.
72+
"""
73+
path = ROOT / "hotdata" / "api_client.py"
74+
src = path.read_text()
75+
76+
if "_useragent" in src:
77+
return
78+
79+
import_needle = "from hotdata.api_response import ApiResponse, T as ApiResponseT\n"
80+
import_replacement = (
81+
"from hotdata.api_response import ApiResponse, T as ApiResponseT\n"
82+
"from hotdata._useragent import USER_AGENT\n"
83+
)
84+
if import_needle not in src:
85+
sys.exit(f"Failed to patch {path}: api_response import anchor not found")
86+
src = src.replace(import_needle, import_replacement, 1)
87+
88+
ua_needle = " self.user_agent = 'OpenAPI-Generator/1.0.0/python'\n"
89+
ua_replacement = " self.user_agent = USER_AGENT\n"
90+
if ua_needle not in src:
91+
sys.exit(f"Failed to patch {path}: default User-Agent anchor not found")
92+
src = src.replace(ua_needle, ua_replacement, 1)
93+
94+
path.write_text(src)
95+
96+
97+
def main() -> None:
98+
patch_accept_encoding()
99+
patch_user_agent()
100+
101+
102+
if __name__ == "__main__":
103+
main()

‎tests/test_arrow.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,10 @@ def test_get_result_arrow_returns_table(monkeypatch: pytest.MonkeyPatch) -> None
163163
assert call["headers"]["Accept"] == ARROW_STREAM_MEDIA_TYPE
164164
# Results are database-scoped: the required X-Database-Id header is sent.
165165
assert call["headers"]["X-Database-Id"] == "db_x"
166+
# Arrow opts out of the client-wide response compression default: IPC
167+
# record batches are frequently LZ4/ZSTD-compressed already, so a gzip
168+
# pass over the stream costs CPU on both ends for little size gain.
169+
assert call["headers"]["Accept-Encoding"] == "identity"
166170

167171

168172
def test_get_result_arrow_forwards_offset_and_limit(

0 commit comments

Comments
 (0)