Found by the pre-commit review of PR #214 while converting the other jest APIs for #215. Pre-existing and not covered by #215 (that issue is about apply().execute throwing; these sites are guarded, but guarded into silent truncation).
The defect
JestScrollApi recovers scroll-page failures into a value that Akka Streams reads as "no more elements":
JestScrollApi.scala:119-123 and 247-250 — .recover { case ex: Exception => None }. unfoldAsync treats None as end-of-stream, so a node dropping or the scroll context expiring after the retries are exhausted ends the stream normally.JestScrollApi.scala:279-282 and 294-297 — a page that fails to parse yields Seq.empty, which does the same.
In both cases the query returns the rows fetched so far and reports success. The caller cannot tell a complete result from a truncated one.
Why it matters more since #209
#209 routes every un-LIMITed SELECT through scroll. On ES6 that means an ordinary SELECT * FROM big_index — not just an explicit scroll — can now come back short and green if the cluster hiccups mid-scroll.
This is the #205 / #207 / #209 silent-truncation family exactly: the answer is wrong, nothing says so, and the only way to notice is to already know the right row count.
Fix
Fail the stream instead of ending it:
.recover { case ex: Exception => None } → let the failure propagate (Future.failed(ex)), so the Source fails and the caller sees an error.- the parse path should rethrow rather than return
Seq.empty.
Distinguish "the scroll is genuinely exhausted" (an empty page from Elasticsearch) from "we could not fetch the next page" — only the first is end-of-stream.
Test
The regression guard has to assert row-count completeness against an oracle, per the #205/#207/#209 lesson: an execution-success assertion cannot see this. A multi-shard index plus a fault injected mid-scroll (a proxy that 503s after N pages) would pin it; ScrollCompletenessSpec is the natural home.
Found by the pre-commit review of PR #214 while converting the other jest APIs for #215. Pre-existing and not covered by #215 (that issue is about
apply().executethrowing; these sites are guarded, but guarded into silent truncation).The defect
JestScrollApirecovers scroll-page failures into a value that Akka Streams reads as "no more elements":JestScrollApi.scala:119-123and247-250—.recover { case ex: Exception => None }.unfoldAsynctreatsNoneas end-of-stream, so a node dropping or the scroll context expiring after the retries are exhausted ends the stream normally.JestScrollApi.scala:279-282and294-297— a page that fails to parse yieldsSeq.empty, which does the same.In both cases the query returns the rows fetched so far and reports success. The caller cannot tell a complete result from a truncated one.
Why it matters more since #209
#209 routes every un-LIMITed
SELECTthroughscroll. On ES6 that means an ordinarySELECT * FROM big_index— not just an explicit scroll — can now come back short and green if the cluster hiccups mid-scroll.This is the #205 / #207 / #209 silent-truncation family exactly: the answer is wrong, nothing says so, and the only way to notice is to already know the right row count.
Fix
Fail the stream instead of ending it:
.recover { case ex: Exception => None }→ let the failure propagate (Future.failed(ex)), so theSourcefails and the caller sees an error.Seq.empty.Distinguish "the scroll is genuinely exhausted" (an empty page from Elasticsearch) from "we could not fetch the next page" — only the first is end-of-stream.
Test
The regression guard has to assert row-count completeness against an oracle, per the #205/#207/#209 lesson: an execution-success assertion cannot see this. A multi-shard index plus a fault injected mid-scroll (a proxy that 503s after N pages) would pin it;
ScrollCompletenessSpecis the natural home.