PERF: Optimize checked temporal fetch construction

[Jahnvi Thakkar (jahnvi480)]

Work Item / Issue Reference

GitHub Issue: #554

ADO Task: AB#48255

---

Summary

Reduce Python object-construction overhead when fetching DATE, TIME, and TIMESTAMP values. Previously, native temporal fields were converted into Python arguments and passed through a generic call to an already-cached constructor. Repeated imports were not the bottleneck.

The new helper uses checked CPython construction APIs when the cached constructor is the exact standard type; substituted constructors retain the original call. Six row-wise/batch conversion sites change. Field validation and final object allocation remain. NULLs, precision, timezone/fold, ownership, and exception behavior are preserved; DATETIMEOFFSET, UUID, Decimal, and text are untouched by this PR.

flowchart LR
    A["Native temporal fields after NULL checks"] --> B["Before: Python arguments and generic cached-constructor call"]
    A --> C{"After: exact standard type?"}
    C -->|"Yes"| D["Direct checked CPython construction"]
    C -->|"No: original fallback"| B
    B --> E["Validated Python object in result row"]
    D --> E

Loading

Fresh measurements

Temporal cases contain NULLs every seventh row. Pure cases have eight temporal columns; the row-wise case adds one harmless MAX column. Mixed has DATE/TIME/DATETIME2/DATETIMEOFFSET; narrow is an unchanged int/text/float control.

Workload / path	API / requested batch	Rows × columns	Before → after fetch time	Reduction
DATE / bounded	fetchmany(1000)	4,000 × 8	6.060 → 5.050 ms	16.67%
TIME(7) / bounded	fetchmany(1000)	4,000 × 8	6.815 → 5.479 ms	19.60%
DATETIME2(7) / bounded	fetchmany(1000)	4,000 × 8	8.077 → 5.359 ms	33.65%
DATETIME2(7) / MAX-forced row-wise	fetchall() / all remaining	4,000 × 9	13.906 → 9.562 ms	31.24%
Mixed temporal	Repeated fetchone() / 1	4,000 × 4	234.946 → 231.518 ms	1.46%; inconclusive
Unchanged narrow	fetchmany(1000)	10,000 × 3	6.493 → 6.634 ms	-2.16%

Method/build: September 21 Docker Linux x64; Python 3.13.15, pybind11 3.0.1, GCC 12.2 Release -O3 -DNDEBUG, profiling OFF, SQL Server 16.0.4225.2, ODBC 18.6.2.1. Main c963ee1e versus PR 5aaa6aae: 10 counterbalanced pairs × 5 samples × 14 cases, totaling 1,400 validated drains. Reductions are ratios of medians, excluding execute/validation; no outlier removal or retries. Fallback checks observed three callbacks per temporal type with 3/4/7 positional arguments—not optimized-path or allocation counts.

Limits: identical-build A/A calibration was noisy (speed-ratio interval 0.882×–1.345×). Mixed fetchone and bounded DATE fetchall remain inconclusive; unchanged narrow many/all had negative point estimates with intervals spanning zero. These are scoped bulk-temporal gains, not universal speedups or a no-regression guarantee.

Both builds passed six fresh-process compatibility modes covering boundaries, NULLs, types, substitutions, exceptions, recovery, and both fetch paths. No full-suite or all-OS success is claimed. Complete samples, intervals, provenance, and historical limitations remain in retained local evidence.

[Copilot]

🔵 Needs a closer look

Native extension changes span multiple fetch paths, with limited platform and validation coverage.

Pull request overview

Optimizes native SQL temporal fetch construction while preserving custom constructors and conversion behavior.

Changes:

• Adds checked datetime construction helpers.
• Integrates them into six temporal fetch paths.
• Adds regression tests and changelog documentation.

File summaries

File	Reviewed changes
tests/test_038_fetch_temporal.py	Temporal parity, constructor, exception, and fetch API coverage
mssql_python/pybind/fetch_temporal.hpp	Translation-unit-local checked construction helpers
mssql_python/pybind/ddbc_bindings.cpp	Integration into row-wise and batch temporal fetch paths
CHANGELOG.md	Documents the optimization and preserved behavior

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 92%

🎯 Overall Coverage 84%

📈 Total Lines Covered: 8677 out of 10324 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/ddbc_bindings.cpp (91.7%): Missing lines 4433
• mssql_python/pybind/fetch_temporal.hpp (92.1%): Missing lines 17-19

Summary

• Total: 50 lines
• Missing: 4 lines
• Coverage: 92%

mssql_python/pybind/ddbc_bindings.cpp

Lines 4429-4437

  4429                 case SQL_SS_TIME2: {
  4430                     const SQL_SS_TIME2_STRUCT& t2 = buffers.timeBuffers[col - 1][i];
  4431                     py::object timeObj =
  4432                         FetchTemporal::time(t2.hour, t2.minute, t2.second, t2.fraction / 1000);
! 4433                     PyList_SET_ITEM(row, col - 1, timeObj.release().ptr());
  4434                     break;
  4435                 }
  4436                 case SQL_SS_TIMESTAMPOFFSET: {
  4437                     SQLULEN rowIdx = i;

---

mssql_python/pybind/fetch_temporal.hpp

Lines 13-23

  13 
  14 // datetime.h keeps PyDateTimeAPI per translation unit, so these helpers must too.
  15 static inline void ensure_datetime_api() {
  16     if (PyDateTimeAPI == nullptr) {
! 17         PyDateTime_IMPORT;
! 18         if (PyDateTimeAPI == nullptr) throw py::error_already_set();
! 19     }
  20 }
  21 
  22 static inline py::object date(int year, int month, int day) {
  23     ensure_datetime_api();

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 64.1%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 78.4%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 82.5%
mssql_python.row.py: 83.4%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[github-actions]

PR Performance Report

No consistent slowdowns detected across all 2 environments.

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Environment	Status
Unix / SQL Server 2022	Completed
Unix / SQL Server 2025	Completed

Affected phases and call counts

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task	Before	After	Paired change	Result
Connection opening	11.732 ms	10.981 ms	-6.4%	no signal
SELECT queries	1.180 ms	1.078 ms	-6.9%	no signal
Row insertion	31.369 ms	31.157 ms	-0.4%	no signal
Executemany inserts	137.029 ms	140.707 ms	+2.1%	no signal
Fetch-all queries	165.163 ms	147.266 ms	-10.8%	no signal
Row-by-row fetching	51.945 ms	51.694 ms	+0.1%	no signal
Batched row fetching	155.412 ms	134.781 ms	-13.3%	no signal
Transaction commit and rollback	98.944 ms	102.270 ms	+3.4%	no signal
Arrow row fetching	90.258 ms	93.469 ms	+5.3%	no signal
100,000-row insertion	409.329 ms	399.326 ms	-1.8%	no signal
Row fetching in batches of 100	198.627 ms	178.182 ms	-10.1%	no signal
Row fetching in batches of 10,000	173.071 ms	157.638 ms	-9.1%	no signal
Repeated positional queries	38.498 ms	38.388 ms	-0.4%	no signal
Repeated named-parameter queries	40.887 ms	40.823 ms	-0.3%	no signal
Legacy 100,000-row insertion	313.084 ms	309.240 ms	-1.2%	no signal
Insertion with explicit input sizes	2142.162 ms	2177.272 ms	+2.1%	no signal
Joined aggregation queries	184.754 ms	187.541 ms	+1.9%	no signal
Large joined-result fetching	206.905 ms	202.625 ms	-2.3%	no signal
1.2-million-row fetching	5178.929 ms	5160.806 ms	-0.9%	no signal
Common table expression queries	5.572 ms	5.799 ms	+4.1%	no signal

Unix / SQL Server 2025

Database task	Before	After	Paired change	Result
Connection opening	96.774 ms	96.760 ms	-0.1%	no signal
SELECT queries	5.311 ms	1.159 ms	-78.1%	no signal
Row insertion	38.151 ms	34.301 ms	+0.4%	no signal
Executemany inserts	154.860 ms	149.179 ms	-2.8%	no signal
Fetch-all queries	172.833 ms	153.637 ms	-11.7%	no signal
Row-by-row fetching	62.107 ms	59.459 ms	-3.7%	no signal
Batched row fetching	165.183 ms	144.225 ms	-12.9%	no signal
Transaction commit and rollback	117.069 ms	115.336 ms	-1.8%	no signal
Arrow row fetching	95.445 ms	96.063 ms	+1.2%	no signal
100,000-row insertion	450.680 ms	458.589 ms	-1.7%	no signal
Row fetching in batches of 100	224.399 ms	200.912 ms	-10.6%	no signal
Row fetching in batches of 10,000	186.985 ms	162.534 ms	-13.7%	no signal
Repeated positional queries	42.027 ms	42.094 ms	+0.3%	no signal
Repeated named-parameter queries	45.004 ms	45.006 ms	-0.7%	no signal
Legacy 100,000-row insertion	356.443 ms	345.116 ms	-4.1%	no signal
Insertion with explicit input sizes	2409.250 ms	2377.858 ms	-2.5%	no signal
Joined aggregation queries	162.212 ms	160.620 ms	-0.8%	no signal
Large joined-result fetching	207.514 ms	199.135 ms	-3.9%	no signal
1.2-million-row fetching	5068.403 ms	5121.146 ms	-0.0%	no signal
Common table expression queries	5.215 ms	5.215 ms	+1.6%	no signal

Build, commits and measurement details

ADO build 176916

PR head: bc47daa4be12926321741ac8e20a43d92f03285f
Base: c963ee1ebf11a96b2dc5275b465d2a36f3f55d77
Measured merge: 289e67a0be0b851d4d79bcbbfc50ed65caf038da

• Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4295.3; 5 paired comparisons and 1 warmup.
• Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5005.3; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

[Copilot]

🔵 Needs a closer look

Native C++/CPython changes and stated cross-platform validation limitations warrant final human review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The isolated Linux wheel-validation job fails on a source-tree-relative module path assertion.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Native fetch-path and temporal-construction changes warrant final human review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Copilot review overview

🔵 Needs a closer look

Fix the installed-wheel Linux test path handling before approval.

Review effort: Lite
Findings: 1

Open (1)

• The Linux wheel-validation job copies tests/ into /test_isolated_${PYBIN} and runs the…

[Copilot]

Copilot review overview

🔵 Needs a closer look

Add test-visible coverage proving that standard constructors use the new direct CPython fast path.

Review effort: Lite
Findings: 1

Open (1)

• The Linux wheel-validation job copies tests/ into /test_isolated_${PYBIN} and runs the…

[Copilot]

Copilot review overview

🔵 Needs a closer look

Native fetch-path changes and scoped compatibility/performance validation warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

• The Linux wheel-validation job copies tests/ into /test_isolated_${PYBIN} and runs the…