Skip to content

PERF: track correct UTF-16 code-unit lengths for every string parameter - #759

Draft
Gaurav Sharma (bewithgaurav) wants to merge 1 commit into
bewithgaurav/native-setinputsizesfrom
bewithgaurav/insertmany-utf16len
Draft

PERF: track correct UTF-16 code-unit lengths for every string parameter#759
Gaurav Sharma (bewithgaurav) wants to merge 1 commit into
bewithgaurav/native-setinputsizesfrom
bewithgaurav/insertmany-utf16len

Conversation

@bewithgaurav

Copy link
Copy Markdown
Collaborator

Work Item / Issue Reference

GitHub Issue: #500


Summary

Native parameter detection records the UTF-16 code-unit length of each string parameter in ParamInfo.utf16Len, but until now it did so only on the data-at-execution branch (strings longer than 4000 code units). Inline strings, datetime.time values normalized to their ISO-8601 text, and setinputsizes-formatted values all left the field at its default of 0.

The field is currently unread, so this has no observable effect today. A subsequent change will size a per-batch wide-character (SQL_C_WCHAR) buffer from utf16Len; at that point a stale 0 would hand a string a zero-length slice while the binder writes its full contents, corrupting the heap. This change populates utf16Len correctly at every site where a string is bound as wide characters, computed after any in-place normalization, and counts each astral code point (above U+FFFF) as the two units of its surrogate pair.

This is a no-behaviour-change commit: no existing field is altered, and nothing consumes utf16Len yet. The field is exposed read-only on the ParamInfo binding, and a DetectParamTypesForTesting helper allows the detection contract to be asserted without a live server. The bytes, UUID, and automatic Decimal (bound as SQL_NUMERIC since GH-740) paths bind as narrow or non-string types and are intentionally left untouched.


Correctness

UTF-16 length reported by native detection for each string shape, asserted in the new tests:

Input Bound value utf16Len
"hello" hello 5
"café" / "αβγ" (BMP) unchanged 4 / 3
"😀😀" (astral) unchanged 4
datetime.time(1, 2, 3, 4) 01:02:03.000004 15
Decimal("12.5") via setinputsizes 12.5 4
5000-character string (data-at-execution) streamed 5000

Performance

No throughput change is expected or claimed; the arena that consumes utf16Len lands in a separate change. This commit only has to add no detection overhead, and it does not: the common inline-string path reuses a length the detector already computed for its length gate, and only the rarer normalized-string sites perform one additional short scan.

Native detection of a 2000-parameter batch, median of 270 timed runs, macOS arm64, release build (-O3 -DNDEBUG):

Batch (2000 params) Before After
INT / VARCHAR (issue #500 shape) 320.6 µs 316.5 µs
INT / VARCHAR / TIME / DECIMAL 785.3 µs 795.9 µs

Both differences are within run-to-run noise.


Testing

TestDetectParamTypesUtf16Len adds 12 cases covering ASCII, BMP, astral / surrogate-pair, empty, time, data-at-execution, non-string, and every setinputsizes override shape, and asserts that the caller's parameter list is not mutated. Local run of test_010, test_023, test_020, and test_004 against SQL Server 2022: 746 passed, 8 skipped. black and flake8 are clean.

native detection left utf16Len at 0 for inline strings, time->isoformat strings, and setinputsizes-formatted strings; only the DAE branch set it. a later change will size a per-batch wide-char arena from utf16Len, where a 0 would undersize the slice and corrupt the heap. compute it (surrogate pairs counted as two units) after any in-place normalization at every site a string binds wide. no behaviour change: nothing reads utf16Len yet. expose it read-only and add a DetectParamTypesForTesting hook so the contract is unit-tested without a live server.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the pr-size: medium Moderate update size label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: medium Moderate update size

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants