ScrollApi.scroll re-normalizes every row on the stream with O(cols²) ListMap scans (~7% sidecar CPU at arrow#160 scale)

[fupelaqu]

ScrollApi.scroll re-normalizes every row on the stream with per-row collection work that a JFR capture on the arrow#160 benchmark (post-#227 image, J2 = JOIN + GROUP BY over an 11M-row extraction) measures at ~7% of total sidecar CPU — scala.collection.immutable.Set$Set3.contains 3.91% (the top non-Arrow, non-Jackson method), plus StrictOptimizedIterableOps.filterImpl and ListMapBuilder allocations. Verified by stack samples: every Set3.contains sample traces to ScrollApi.$anonfun$scroll$8 at core/src/main/scala/app/softnetwork/elastic/client/ScrollApi.scala:448.

The code

ScrollApi.scala:441-453 (the stream-level normalization after the strategy dispatch):

val normalized = if (fields.nonEmpty) {
  val requestedSet = fields.toSet
  source.map { row =>
    val ordered =
      context match {
        case EntityContext => fields.flatMap(f => row.get(f).map(v => f -> v))
        case _             => fields.map(f => f -> row.getOrElse(f, null))
      }
    val extra = row.filterNot { case (k, _) => requestedSet.contains(k) }
    ListMap(ordered: _*) ++ extra
  }
}

Per row this does:

1. fields.map(f => row.getOrElse(f, null)) — row is a ListMap, whose get is a linear scan with a String.equals per entry ⇒ O(cols²) comparisons per row (ListMap$Node.getInternal also shows in the profile);
2. row.filterNot(...) — a full second walk of the row with a Set.contains per key;
3. ListMap(ordered: _*) ++ extra — a fresh ListMap build plus ++, whose append is O(n) per extra entry.

At 11M rows (the arrow#160 legs) that is hundreds of millions of String.equals and >10M ListMap rebuilds — for rows that in the common case already carry exactly the requested keys. This violates the established hot-path rule (no per-row work for stream-constant decisions).

Suggested fix

Single pass per row with the lookup structure hoisted per stream:

• resolve fields → a positional HashMap[String, Int] once, before source.map;
• walk the row once, splitting entries into a positional array (requested fields) and an ordered extras list;
• build the output ListMap once from those (the output contract — requested fields in SQL SELECT order, extras appended — is unchanged);
• optional fast path: when the walked row's keys already match fields in order and there are no extras, return the row unchanged (measure whether the check pays for itself — rows within one stream are near-homogeneous but not guaranteed identical, so this must stay per-row).

EntityContext keeps its skip-missing semantics (absent fields omitted rather than null-filled) — same single-pass structure, different assembly.

Context

• Found while analyzing the arrow-side residual of softclient4es-arrow#160 (see the analysis comment there with the full post-perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160) #227 profile table); this item plus the arrow-side pair (-Darrow.enable_unsafe_memory_access + ArrowTypeMapping.fillBatch's O(cols²) cell lookup) are what stand between the post-perf(client): parse each Elasticsearch page once on the scroll hits path (softclient4es-arrow#160) #227 marginal (+2.7 s) and fix: REPL JOIN hardening — qualifier round-trip, loud no-extension failure, bare-alias sort rejection (#157, #158, #159) #160's acceptance (J2 ≤ J0 + ~2 s).
• Same defect family as the fillBatch loop: per-cell ListMap lookups treating an O(n) structure as a map on a rows×cols hot path.
• Guard: the scroll completeness suites (ScrollCompletenessSpec, SelectCompletenessSpec, LimitCompletenessSpec) pin row content/order; a fix must keep QueryRows column order and the extras-after-requested contract byte-identical.