es6/es7 clients triple-parse every scroll page (serialize→re-parse removed for es8/es9 in #227)

[fupelaqu]

The es6 and es7 clients still process every scroll / search_after page in three passes: the client library parses the HTTP response into its typed form, the bridge re-serializes that typed form to a JSON string, and core parseResponse re-parses the string into the Jackson tree the row parser actually consumes. PR #227 removed this triple pass for es8/es9 (document type ObjectNode + TokenBuffer, single parse), where it was measured at ~19% of Flight sidecar CPU during JOIN leg extraction (softclient4es-arrow#160) — a cost that scales with the number and width of the selected columns, string columns worst.

Evidence

• es7 REST — es7/rest/src/main/scala/app/softnetwork/elastic/client/rest/RestHighLevelClientApi.scala:1672 and :1810: extractHitsOnly(response.toString, …). The RHLC parsed the HTTP response into its typed SearchResponse (pass 1); response.toString re-serializes it via XContent (pass 2); core parseResponse re-parses with Jackson (pass 3).
• es6 REST — es6/rest/src/main/scala/app/softnetwork/elastic/client/rest/RestHighLevelClientApi.scala:1639 plus extractAllResults at :1493/:1528: same shape.
• es6 Jest — es6/jest/src/main/scala/app/softnetwork/elastic/client/jest/JestScrollApi.scala:218 (and the extractAllResults call sites at :83/:112): result.getJsonObject.toString — Gson already parsed the response (pass 1), .toString re-serializes it (pass 2), Jackson re-parses (pass 3).

Why the #227 technique does not transplant directly

The es8/es9 fix re-parents already-parsed Jackson _source trees into the envelope. RHLC holds XContent objects and Jest holds a Gson tree — there is no Jackson node to re-parent. Candidate approaches, per client:

• es7 / es6 REST: issue the paging search/scroll over the RHLC's low-level RestClient and Jackson-parse the raw response entity bytes once → parseSingleSearchResponse (the node-level core entry perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160) #227 already uses). This removes even the typed parse — one pass total. The typed request builders can still build the request body; only the response side changes. Care: shard-failure checks and _scroll_id/sort cursor extraction currently read the typed response and would need to read the tree instead.
• es6 Jest: check whether JestResult.getJsonString retains the raw response body — if so, getJsonString → Jackson readTree drops the Gson re-serialization for a one-line win (two parses instead of three). The full fix is a direct Gson→Jackson tree walk (no string at all), or bypassing Jest's Gson parse for the scroll action.

Notes

• Same behavioural contract as perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160) #227: rows identical for requested fields; the known null-survival delta (Include.NON_NULL dropping null-valued _source entries on re-serialization) applies here too and currently makes es6/es7 row key-sets differ from es8/es9's scroll paths for SELECT *-shaped queries.
• The core node-level entry (parseSingleSearchResponse) and the envelope contract pinned by JavaClientConversionSpec (es8/es9) are reusable; a port of that spec should accompany each client's migration.
• Priority: lower than es8/es9 (the Flight SQL sidecar ships es8), but the REPL and JDBC driver on 6.x/7.x clusters pay this on every no-LIMIT / large-LIMIT query routed through scroll (SELECT without LIMIT returns only 10 rows on the non-scroll search path #209/SELECT with an explicit LIMIT above index.max_result_window fails, while the same query with NO LIMIT succeeds #224).

[fupelaqu]

Implemented in commit 848ff27 on fix/es-response-single-parse (rides PR #227, which now carries Closes #228), with the scope extended per review: single search and multi search (sync and async) are fixed too, not just the scroll paths.

Approach per client

• es6/es7 REST — the issue's first candidate approach: paging and one-shot search/msearch run over the low-level RestClient, and the raw response entity bytes are Jackson-parsed exactly once into the tree core consumes — one pass total (was three). The typed request builders still build request bodies. Shard-failure checks, _scroll_id and the search_after cursor read the tree; endpoints are percent-encoded exactly like RequestConverters; msearch sends UTF-8 bytes as bare application/json — ES 6.8 answers 406 to application/x-ndjson; charset=UTF-8 (found by the es6 suite on first run).
• es6 Jest — confirmed JestResult.getJsonString retains the raw response body (verified in the jest-common 6.3.1 bytecode: createNewElasticSearchResult stores the raw responseBody), so the scroll and search paths now parse that raw body once with Jackson; the Gson re-serialization pass is gone. Jest's own Gson parse is forced by its transport (still used for the scroll cursor), so this client is 2 passes, the floor without bypassing Jest.
• Core contract — executeSingleSearch/executeMultiSearch (sync+async) now return Option[JsonNode]; core gained a node-level parseResponseTree dispatch and a Jackson parseInnerHits. parseResponse(String) remains for external callers.
• es8/es9 — one-shot search/msearch now use document type ObjectNode + searchResponseToTree/msearchResponseToTree (the perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160) #227 re-parenting technique; msearch failure items keep full fidelity so core still sees their error object). The one-shot and scroll paths are now consistent, which also removes the SELECT * null-survival asymmetry between them.

Spec ports: JavaClientConversionSpec (es8+es9) grew searchResponseToTree/msearchResponseToTree contracts; core ElasticConversionSpec pins the parseResponseTree dispatch (single, msearch envelope, per-item error, malformed input → Failure). For es6/es7 there is no envelope-building code to pin — the tree is the raw response parsed once by core-owned code — so coverage is the five real-ES suites.

Hardening found by the adversarial review (all suite-validated): permanent 4xx on paging fails fast instead of burning retries (ResponseException is an IOException); a first scroll page answering 200-with-failed-shards releases the scroll context it created; a hit page without sort values aborts loudly instead of restarting from page one forever; and paging streams now fail on error instead of ending quietly — a mid-scroll failure used to surface a silently truncated result set as a successful result (same class as #209/#224). That last fix also covers the same pre-existing swallow on es8/es9.

Validated on real ES, final code: core 746 unit tests · es6rest 303 · es6jest 290 · es7rest 305 · es8java 316 · es9java 316 — all green; 2.12+2.13 cross-compile, scalafmt, headerCheck clean.

Pre-existing, out of scope, spotted during review: the deprecated multisearchWithInnerHits has never worked on msearch envelopes (its hit walk expects a single-search shape; zero callers in this repo or downstream).