PERF: track correct UTF-16 code-unit lengths for every string parameter - #759
Draft
Gaurav Sharma (bewithgaurav) wants to merge 1 commit into
Draft
Conversation
native detection left utf16Len at 0 for inline strings, time->isoformat strings, and setinputsizes-formatted strings; only the DAE branch set it. a later change will size a per-batch wide-char arena from utf16Len, where a 0 would undersize the slice and corrupt the heap. compute it (surrogate pairs counted as two units) after any in-place normalization at every site a string binds wide. no behaviour change: nothing reads utf16Len yet. expose it read-only and add a DetectParamTypesForTesting hook so the contract is unit-tested without a live server. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Native parameter detection records the UTF-16 code-unit length of each string parameter in
ParamInfo.utf16Len, but until now it did so only on the data-at-execution branch (strings longer than 4000 code units). Inline strings,datetime.timevalues normalized to their ISO-8601 text, andsetinputsizes-formatted values all left the field at its default of0.The field is currently unread, so this has no observable effect today. A subsequent change will size a per-batch wide-character (
SQL_C_WCHAR) buffer fromutf16Len; at that point a stale0would hand a string a zero-length slice while the binder writes its full contents, corrupting the heap. This change populatesutf16Lencorrectly at every site where a string is bound as wide characters, computed after any in-place normalization, and counts each astral code point (above U+FFFF) as the two units of its surrogate pair.This is a no-behaviour-change commit: no existing field is altered, and nothing consumes
utf16Lenyet. The field is exposed read-only on theParamInfobinding, and aDetectParamTypesForTestinghelper allows the detection contract to be asserted without a live server. The bytes, UUID, and automaticDecimal(bound asSQL_NUMERICsince GH-740) paths bind as narrow or non-string types and are intentionally left untouched.Correctness
UTF-16 length reported by native detection for each string shape, asserted in the new tests:
"hello"hello"café"/"αβγ"(BMP)"😀😀"(astral)datetime.time(1, 2, 3, 4)01:02:03.000004Decimal("12.5")viasetinputsizes12.5Performance
No throughput change is expected or claimed; the arena that consumes
utf16Lenlands in a separate change. This commit only has to add no detection overhead, and it does not: the common inline-string path reuses a length the detector already computed for its length gate, and only the rarer normalized-string sites perform one additional short scan.Native detection of a 2000-parameter batch, median of 270 timed runs, macOS arm64, release build (
-O3 -DNDEBUG):Both differences are within run-to-run noise.
Testing
TestDetectParamTypesUtf16Lenadds 12 cases covering ASCII, BMP, astral / surrogate-pair, empty,time, data-at-execution, non-string, and everysetinputsizesoverride shape, and asserts that the caller's parameter list is not mutated. Local run oftest_010,test_023,test_020, andtest_004against SQL Server 2022: 746 passed, 8 skipped.blackandflake8are clean.