[kz930]

What changes were proposed in this PR?

Type Casting to a timestamp read text with pd.to_datetime(col, errors="coerce") in the exported script. That is wrong in two ways, and both empty a cell the run itself filled.

pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a java.sql.Timestamp, where 2500-01-01 00:00:00 is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for 1500-06-15 08:30:00 and 9999-12-31 23:59:59.

pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands DateParserUtils one field at a time, so a row states its own format. A column holding 2024-03-05 14:09:07 and March 5, 2024 kept the first and emptied the second.

The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. Text no format reads now ends the run, as the engine's parseField ends its own. The script raises the engine's message, Failed to parse type str to Timestamp, where it used to leave the cell empty. dateutil reads every format the verification hands the engine, so the two refuse the same text.

Review and the verification's edge values found more places where the cast parted from the engine:

• A reading that states an offset, such as Z or +05:30, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same.
• A column that already held timestamps went through the text parse and lost its nanoseconds. parseField returns a java.sql.Timestamp untouched, so the script now hands such a column back at the resolution it arrived in.
• Text with digits past the millisecond kept them. DateParserUtils reads into a java.util.Date, which counts milliseconds, so the script now cuts there too.
• Casting to text wrote a value the way Python's str() does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least .0. A shared _texera_java_text now writes each one as Java's toString does.
• A double column holding a NaN as a value, as an Arrow or Parquet source hands one over, lost it. The casts asked each cell whether it was empty, and pandas hands an empty cell over as NaN too, so a stored NaN came out null where the engine writes NaN as text, 0 as an integer and true as a boolean. The casts now read empty cells from the column's own mask, and the double cast builds a nullable Float64, so a NaN read from text stays one through the next cast. A double past an integer's bounds now saturates as toInt and toLong do, where int() raised on an infinity.
• A moment was read with dateutil's tzlocal(), which knows only today's two offsets, so an instant from before a zone changed its rules came out an hour or more off on every platform. The script now reads the machine's zone by its IANA name and, before 1900, keeps today's standard offset as java.util.Date does.
• On Windows, localtime refuses an instant before 1970, which stopped the cast. The same helper now falls back to the zone's standard offset there.

That Float64 column changed what a Python operator after this cast is handed. pyamber hands a Python operator a DOUBLE column as float64, where a NaN and a null are one value, and writes a NaN out as null. Given Float64 instead, Sort ordered a NaN apart from the nulls, a chart drew points the engine dropped, and a passed-through NaN stayed NaN. StandaloneCodeGenerator gains standaloneInputAdapter(), and for a Python operator the translator feeds the fragment a float64 copy of each input, leaving the upstream frame as it was.

The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where new Timestamp(long) is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it.

Any related issues, documentation, discussions?

Closes #8595. #8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot.

#8341, #8704, #8705 and #8709 call the timestamp and text helpers this adds, so this PR merges before them.

How was this PR tested?

Eight tests in TypeCastingOpDescSpec run the generated Python and compare it cell by cell against AttributeTypeUtils.parseField. They are tagged to run in the integration job, which has pandas.

• The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, 2500-01-01, 1500-06-15, 9999-12-31, readings with Z and +05:30, and readings past the millisecond. It fails on the old generator.
• One test gives a column that already holds timestamps and checks its nanoseconds are kept.
• The cast-to-string test covers a double at 1e7 and 1e-4.
• One test checks that unreadable text ends the script with the engine's message, as the engine refuses it.
• Two tests cast a column holding a stored NaN, both infinities and an empty cell to each type and compare against AttributeTypeUtils, and check that a NaN read from text survives a second cast.
• Two tests read moments from America/Mexico_City in 1899, 1921, 1931 and 2021, at the offset the zone had then, and the same moments with localtime refusing those before 1970.
• Three tests in WorkflowToPythonTranslatorSpec check that a Python operator reads a converted copy of its input and an operator the engine runs on the JVM reads its upstream as it is. Two of them fail without the change.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 8 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main 5264df2 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	563	0.344	16,546/24,475/24,475 us	🔴 +10.1% / 🔴 +100.2%
🔴	bs=100 sw=10 sl=64	1,136	0.693	86,912/103,052/103,052 us	🔴 +15.9% / 🔴 +18.2%
⚪	bs=1000 sw=10 sl=64	1,486	0.907	671,886/745,585/745,585 us	⚪ within ±5% / 🟢 -13.9%

Baseline details

Latest main 5264df2 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	563 tuples/sec	611 tuples/sec	1,019 tuples/sec	-7.9%	-44.8%
bs=10 sw=10 sl=64	MB/s	0.344 MB/s	0.373 MB/s	0.622 MB/s	-7.8%	-44.7%
bs=10 sw=10 sl=64	p50	16,546 us	15,033 us	10,144 us	+10.1%	+63.1%
bs=10 sw=10 sl=64	p95	24,475 us	23,846 us	12,225 us	+2.6%	+100.2%
bs=10 sw=10 sl=64	p99	24,475 us	23,846 us	15,527 us	+2.6%	+57.6%
bs=100 sw=10 sl=64	throughput	1,136 tuples/sec	1,288 tuples/sec	1,371 tuples/sec	-11.8%	-17.2%
bs=100 sw=10 sl=64	MB/s	0.693 MB/s	0.786 MB/s	0.837 MB/s	-11.8%	-17.2%
bs=100 sw=10 sl=64	p50	86,912 us	74,976 us	79,910 us	+15.9%	+8.8%
bs=100 sw=10 sl=64	p95	103,052 us	97,871 us	87,155 us	+5.3%	+18.2%
bs=100 sw=10 sl=64	p99	103,052 us	97,871 us	106,643 us	+5.3%	-3.4%
bs=1000 sw=10 sl=64	throughput	1,486 tuples/sec	1,493 tuples/sec	1,415 tuples/sec	-0.5%	+5.0%
bs=1000 sw=10 sl=64	MB/s	0.907 MB/s	0.911 MB/s	0.864 MB/s	-0.4%	+5.0%
bs=1000 sw=10 sl=64	p50	671,886 us	666,432 us	780,407 us	+0.8%	-13.9%
bs=1000 sw=10 sl=64	p95	745,585 us	717,199 us	817,814 us	+4.0%	-8.8%
bs=1000 sw=10 sl=64	p99	745,585 us	717,199 us	848,683 us	+4.0%	-12.1%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,355.24,200,128000,563,0.344,16545.79,24475.39,24475.39
1,100,10,64,20,1760.82,2000,1280000,1136,0.693,86911.65,103051.73,103051.73
2,1000,10,64,20,13462.98,20000,12800000,1486,0.907,671885.92,745585.11,745585.11

[codecov-commenter]

Codecov Report

❌ Patch coverage is 51.61290% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.73%. Comparing base (2ab8ee0) to head (ddf8d16).
⚠️ Report is 190 commits behind head on main.

Files with missing lines	Patch %	Lines
...exera/amber/operator/StandaloneCodeGenerator.scala	0.00%	8 Missing ⚠️
.../amber/translator/WorkflowToPythonTranslator.scala	72.72%	0 Missing and 6 partials ⚠️
...ache/texera/amber/operator/StandaloneHelpers.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8597      +/-   ##
============================================
- Coverage     92.78%   91.73%   -1.06%     
- Complexity     4898     5077     +179     
============================================
  Files          1236     1249      +13     
  Lines         52121    53864    +1743     
  Branches       6405     6659     +254     
============================================
+ Hits          48359    49410    +1051     
- Misses         2186     2772     +586     
- Partials       1576     1682     +106     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (+5.66%)	⬆️
agent-service	99.32% <ø> (ø)		Carriedforward from bd0de6c
amber	85.94% <0.00%> (-2.67%)	⬇️
computing-unit-managing-service	61.84% <ø> (+6.64%)	⬆️
config-service	87.50% <ø> (+0.12%)	⬆️
file-service	81.92% <ø> (+0.38%)	⬆️
frontend	96.58% <ø> (ø)		Carriedforward from bd0de6c
notebook-migration-service	83.77% <ø> (+0.03%)	⬆️
pyamber	98.48% <ø> (ø)		Carriedforward from bd0de6c
workflow-compiling-service	77.84% <72.72%> (+3.74%)	⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[carloea2]

The new timestamp helper crashes on 2024-03-05T14:09:07Z. I ran the generated helper with pandas 2.2.3. Parsing keeps the timezone, then astype rejects the conversion to a timestamp without a timezone. The previous code accepts this value. Please handle timestamps with a timezone and add a test with Z and an explicit offset.

[kz930]

Fixed in c07493387. The helper now does what the engine does with an offset — DateParserUtils reads it and java.sql.Timestamp keeps no zone, so the moment is held as the wall clock of the machine's own zone (...T14:09:07Z is 06:09:07 eight hours behind UTC) — and both spellings joined the cases the test compares against parseField cell by cell, so the expectation comes from the engine rather than being written down.

[carloea2]

The timezone crash is fixed. I ran the current helper with Z, an explicit offset, year 2500, and null. Those checks passed.

[carloea2]

Reproduced locally with the generated Python code. One correctness issue below. I did not run a frontend workflow or the full Scala suite.

[carloea2]

Retested ca67ce1. Timezone inputs, extended years and nanosecond precision pass. The previous findings are fixed. Looks good.

[kz930]

@carloea2 Since your approval, text read into a timestamp is cut to the millisecond as DateParserUtils does, and a cast to text writes a value as Java's toString does.

[carloea2]

Retested with JDK 17.0.20.1. The helper still writes 1.0E23 where the engine writes 9.999999999999999E22. The JDK 19 form would keep this mismatch. We need maintainer agreement on the supported formatting contract before accepting that difference.

[kz930]

@carloea2 Per Xinyuan's call on the thread, the helper keeps the shortest form and the JDK 17 difference is noted in a comment on _texera_java_text (2b1dcf5). Could you take another look?

[kz930]

@carloea2 Both cases are fixed in 5d46525, with tests for each. Could you take another look?

[kz930]

@carloea2 both cast cases from your last review are fixed in 5d46525. Could you take another look?

[carloea2]

The earlier NaN round trip and pre-1970 crash fixes pass: all 12 author tests and formatting checks are green. A historical timezone case still differs on Windows. At 1931-01-01T00:00:00Z in America/Mexico_City, the generated cast returns 1930-12-31 18:00; the native JVM returns 17:00. Modern, older-summer and null controls pass. The added historical regression fails. No production edits or dependency fixtures used.

sbt -batch 'WorkflowOperator/testOnly *TypeCastingOpDescSpec -- -oDF' 'scalafmtCheckAll'

sbt -batch 'WorkflowOperator/testOnly *TimestampHistoryReviewSpec -- -oDF'

[kz930]

@carloea2 The historical offset case is fixed in 7a3203d. Could you take another look?

[kz930]

@carloea2 I added f0eedd9. The double cast now builds Float64, and a Python operator after it was handed a NaN that pyamber would have turned into null, so the translator now feeds Python operators a float64 copy the way pyamber does. I updated the description to cover this and the earlier rounds; could you take another look?

[carloea2]

The historical timezone finding is fixed. On Windows with JDK 17, all 13 TypeCastingOpDescSpec tests, my 3 historical timezone regressions, and 13 translator tests pass against current upstream main. scalafmtCheckAll passes too. CI for this head is still queued.

Command: sbt -batch "WorkflowOperator/testOnly *TypeCastingOpDescSpec *TimestampHistoryReviewSpec" "WorkflowCompilingService/testOnly *WorkflowToPythonTranslatorSpec" scalafmtCheckAll

[kz930]

@carloea2 Since your approval I pushed two commits. A timestamp cast now raises on text no format reads, as the engine's parseField does, and the spec checks that. Could you take another look?

[carloea2]

Retested the timestamp refusal change on fresh upstream main. All 13 TypeCastingOpDescSpec tests and 13 WorkflowToPythonTranslatorSpec tests pass, including invalid text and the input adapter. scalafmtCheckAll passes. Code looks good. The current CI failures occur before tests because the Ampel action is not allowed.

[kz930]

@carloea2 Since your approval, the script carries only the helpers it calls, one blank line apart, and a holed LONG column cast to text no longer rounds values past 2^53 (test in TypeCastingOpDescSpec). Plain ISO timestamps are also read in bulk and a LONG cast clamps inside the cast itself, with the same answers. Could you take another look?