From 4751b2c98dbb27ef9721d2ab2a8fb6afb85e8815 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Thu, 13 Aug 2026 08:36:41 +0200 Subject: [PATCH 1/2] feat(install): bootstrap a JDK on Windows, and make install.cmd self-sufficient MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes to the Windows install path, and one consequence of the second that must not ship without it. 1. install.cmd downloads install.ps1 when it is not beside it. Previously it errored and told the user to fetch two files. Now one file is enough. A LOCAL install.ps1 still wins, so a downloaded pair — or a released bundle — stays self-consistent instead of being silently mixed with main. Written as `goto` plus a separate errorlevel test rather than `if ... call ... || exit`: how cmd binds `||` inside an `if` body is ambiguous, and this is not a platform where a subtlety can be settled by running it. The download lives in a subroutine for the same reason the launcher's two FOR loops are on separate lines — cmd expands every %VAR% in a parenthesised block in ONE parse pass, so `set "PS1=..."` followed by a use of %PS1% inside that block would read the OLD value. 2. install.ps1 installs a JDK instead of aborting. Resolution order, and JAVA_HOME is the thing actually tested when it is set — not the PATH `java`, which is frequently a different and older JVM: %JAVA_HOME%\bin\java.exe -> `java` on PATH -> neither, or too old In the last case the installer downloads a portable Temurin 17 JDK (an Adoptium zip, never an MSI, so no administrator rights) and unpacks it to \jdk. 17 satisfies BOTH floors — 11 for ES 6/7/8, 17 for ES 9 — so there is one download to reason about rather than two. The archive unpacks as jdk-17.x.y+z\, a version-stamped directory. It is staged and MOVED one level up so the final JAVA_HOME is the fixed path \jdk: the launcher hard-codes %BASE_DIR%\jdk\bin and must not glob for a name that changes with every Temurin build. Machine state is left alone. JAVA_HOME and PATH are set for the installer's own SESSION only — deliberately not SetEnvironmentVariable(...,"User"), which would silently repoint every other tool on the box. Nothing is needed in later sessions because of (3). Get-JavaMajorVersion is split: Get-JavaMajorFromExe takes an explicit exe path, so JAVA_HOME and PATH are probed by the same code. Resolve-Java runs AFTER the -ListVersions early exit (listing versions must never download a JDK) and BEFORE bundle selection, which reads the resolved major. Check-Prerequisites now asserts that outcome rather than re-deriving it — still exactly ONE parse of `java -version` per run, so a localised or multi-line output cannot be read two different ways. 3. The generated launchers resolve Java the SAME way. This is the half that makes (2) real rather than a one-shot trick: both launchers invoked bare `java`, so a bootstrapped JDK would have been invisible the moment the user opened a new terminal — the install would look broken for exactly the users the feature is for. bin\softclient4es.bat and bin\softclient4es.ps1 now try \jdk, then %JAVA_HOME%, then PATH. They PREPEND to PATH rather than call an absolute exe: every `java` below stays unchanged, and there is no quoting of a path that contains spaces (the default target is under %USERPROFILE%). `setlocal` scopes the edit to the launcher. The summary and VERSION now report which JVM was chosen, and the printed tree shows jdk\ when one was bootstrapped. Verified on macOS with pwsh 7 — there is no Windows host here, so cmd.exe executing install.cmd and Expand-Archive on a real Temurin zip are NOT covered: * install.ps1 parses clean (Language.Parser.ParseFile, 0 errors) * 16/16 checks in a harness that loads the REAL function bodies out of install.ps1's own AST — not a transcription — and exercises Resolve-Java: version parsing (17.0.11 -> 17, 1.8.0_292 -> 8, missing exe -> 0); JAVA_HOME above the floor is used and nothing is bootstrapped; JAVA_HOME BELOW the floor bootstraps exactly once and repoints session JAVA_HOME/PATH; a JAVA_HOME pointing nowhere is not silently trusted; a failed bootstrap returns false rather than passing silently * -ListVersions still exits 0 against live JFrog and never reaches Java resolution; -Help exits 0; -EsVersion 5 still exits 1 * install.cmd stays ASCII + CRLF, and its only goto target exists Co-Authored-By: Claude Opus 5 (1M context) --- install.cmd | 45 ++++++++-- install.ps1 | 240 ++++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 251 insertions(+), 34 deletions(-) diff --git a/install.cmd b/install.cmd index 2fc4aec1..3754490e 100644 --- a/install.cmd +++ b/install.cmd @@ -11,7 +11,9 @@ rem THIS process only, changes nothing on the machine, and needs no elevation. rem rem It is a wrapper and nothing else - every option, default, fallback and rem message lives in install.ps1, so the two entry points can never drift. -rem Pass the same flags you would pass to install.ps1: +rem When install.ps1 is not sitting next to it, it downloads one, so +rem install.cmd on its own is a complete install. Pass the same flags you +rem would pass to install.ps1: rem rem install.cmd rem install.cmd -ListVersions -EsVersion 8 @@ -28,14 +30,43 @@ rem =========================================================================== setlocal +set "PS1_URL=https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/refs/heads/main/install.ps1" set "PS1=%~dp0install.ps1" -if not exist "%PS1%" ( - echo [ERROR] install.ps1 was not found next to install.cmd. 1>&2 - echo [ERROR] Expected: %PS1% 1>&2 - echo [ERROR] Download both files from the same release and keep them together. 1>&2 - exit /b 1 -) +rem A local install.ps1 always wins: a downloaded pair must stay self-consistent, +rem and a released bundle must never be silently mixed with main. +rem Written as goto + a separate errorlevel test rather than `if ... call ... ||`: +rem how cmd binds `||` inside an `if` body is ambiguous, and this is not a +rem platform where a subtlety can be settled by running it. +if exist "%PS1%" goto run +call :fetch_ps1 +if errorlevel 1 exit /b 1 +:run powershell -NoProfile -ExecutionPolicy Bypass -File "%PS1%" %* exit /b %ERRORLEVEL% + +rem --------------------------------------------------------------------------- +rem Each line of a subroutine is parsed when it is reached, so %PS1% below sees +rem the value assigned on the previous line. The same code inside the `if not +rem exist (...)` block above would NOT: cmd expands every %VAR% in a +rem parenthesised block in ONE parse pass, before running any line in it. +rem --------------------------------------------------------------------------- +:fetch_ps1 +where curl.exe >nul 2>&1 +if errorlevel 1 ( + echo [ERROR] install.ps1 is not next to install.cmd and curl.exe is unavailable. 1>&2 + echo [ERROR] curl.exe ships with Windows 10 build 1803+ and Windows Server 2019+. 1>&2 + echo [ERROR] Download install.ps1 manually into the same directory as install.cmd. 1>&2 + exit /b 1 +) +set "PS1=%TEMP%\softclient4es-install.ps1" +echo [INFO] install.ps1 not found next to install.cmd - downloading it... +echo [INFO] URL: %PS1_URL% +curl.exe -fsSL -o "%PS1%" "%PS1_URL%" +if errorlevel 1 ( + echo [ERROR] Could not download install.ps1 from %PS1_URL% 1>&2 + exit /b 1 +) +echo [INFO] Using %PS1% +exit /b 0 diff --git a/install.ps1 b/install.ps1 index 38b0108f..7a43fcdd 100644 --- a/install.ps1 +++ b/install.ps1 @@ -63,7 +63,7 @@ Examples: .\install.ps1 .\install.ps1 -ListVersions -EsVersion 8 .\install.ps1 -Target "C:\tools\softclient4es" -EsVersion 8 -Version 1.0.0 - .\install.ps1 -EsVersion 7 -Version 0.20.3 -NoExtensions + .\install.ps1 -EsVersion 7 -Version 0.20.4 -NoExtensions "@ exit 0 @@ -122,10 +122,17 @@ function Get-RequiredJavaVersion { } } -# Major version of the `java` on PATH, or 0 when it cannot be determined. -function Get-JavaMajorVersion { +# Major version reported by a SPECIFIC java executable, or 0 when it cannot be +# determined. Taking the exe as a parameter is what lets JAVA_HOME and the PATH +# `java` be probed by the same code - they routinely disagree. +function Get-JavaMajorFromExe { + param([string]$Exe) + if (-not $Exe) { return 0 } try { - $out = (& java -version 2>&1 | Select-String -Pattern 'version' | Select-Object -First 1).ToString() + # Select-Object -First 1 matters: with two matching lines (e.g. a + # deprecation notice from _JAVA_OPTIONS) the pipeline yields an array whose + # ToString() is "System.Object[]", no regex matches, and the version reads 0. + $out = (& $Exe -version 2>&1 | Select-String -Pattern 'version' | Select-Object -First 1).ToString() if ($out -match '"1\.(\d+)') { return [int]$Matches[1] } # 1.8.x elseif ($out -match '"(\d+)') { return [int]$Matches[1] } # 11.x, 17.x } @@ -133,8 +140,155 @@ function Get-JavaMajorVersion { return 0 } +# Major version of the `java` on PATH, or 0 when it cannot be determined. +function Get-JavaMajorVersion { + $onPath = Get-Command java -ErrorAction SilentlyContinue + if (-not $onPath) { return 0 } + return (Get-JavaMajorFromExe -Exe $onPath.Source) +} + $REQUIRED_JAVA_VERSION = Get-RequiredJavaVersion -EsVer $EsVersion +# The JDK this installer bootstraps when the host cannot satisfy the floor. 17 +# covers BOTH floors (11 for ES 6/7/8, 17 for ES 9), so there is one download to +# reason about rather than two. +$BOOTSTRAP_JAVA_VERSION = 17 +# Adoptium redirects this to the current GA Temurin 17 JDK *zip* for windows/x64 - +# an archive, deliberately not an MSI: unpacking needs no administrator rights. +$TEMURIN_ZIP_URL = "https://api.adoptium.net/v3/binary/latest/$BOOTSTRAP_JAVA_VERSION/ga/windows/x64/jdk/hotspot/normal/eclipse?project=jdk" +# Lives INSIDE the install tree: `uninstall.ps1` then removes it with everything +# else, and the launcher finds it by relative path with no machine-wide state. +$EMBEDDED_JDK_DIR = Join-Path $Target "jdk" + +# ============================================================================= +# Java resolution — probe, then bootstrap a portable JDK rather than give up +# ============================================================================= +# Resolution order, and it is the SAME order the generated launcher uses, which +# is what keeps "the installer worked" and "the REPL starts" from disagreeing: +# +# \jdk\bin\java.exe (bootstrapped here, if it was needed) +# -> %JAVA_HOME%\bin\java.exe +# -> `java` on PATH +# +# So JAVA_HOME, when it is defined, is the thing actually tested — not the PATH +# `java`, which is frequently a different and older JVM. + +# Set by Resolve-Java; read by Check-Prerequisites, the launcher writer and the +# summary. +$script:JavaMajor = 0 +$script:JavaSource = "not found" +$script:EmbeddedJdkHome = $null + +function Install-EmbeddedJdk { + Write-Info "Installing a portable Temurin $BOOTSTRAP_JAVA_VERSION JDK (zip, no administrator rights)..." + + $zip = Join-Path $env:TEMP "softclient4es-temurin$BOOTSTRAP_JAVA_VERSION.zip" + try { + [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 + # Same reason as the JAR download: Write-Progress per chunk makes a + # ~180 MB download on Windows PowerShell 5.1 look hung. + $previousProgress = $ProgressPreference + $ProgressPreference = 'SilentlyContinue' + try { + Invoke-WebRequest -Uri $TEMURIN_ZIP_URL -OutFile $zip -UseBasicParsing -ErrorAction Stop + } + finally { $ProgressPreference = $previousProgress } + + # The archive unpacks as jdk-17.x.y+z\ — a version-stamped directory. Unpack + # to a staging dir and MOVE that one level up, so the final JAVA_HOME is the + # fixed path \jdk. The launcher hard-codes `%BASE_DIR%\jdk\bin`, and + # it must not have to glob for a name that changes with every Temurin build. + $staging = "$EMBEDDED_JDK_DIR.unpack" + if (Test-Path $staging) { Remove-Item -Recurse -Force $staging } + if (Test-Path $EMBEDDED_JDK_DIR) { Remove-Item -Recurse -Force $EMBEDDED_JDK_DIR } + New-Item -ItemType Directory -Force -Path $staging | Out-Null + + try { + Expand-Archive -Path $zip -DestinationPath $staging -Force + + $inner = Get-ChildItem $staging -Directory | Select-Object -First 1 + if (-not $inner) { + Write-Err "The Temurin archive did not unpack as expected (no directory inside $staging)" + return $null + } + Move-Item -Path $inner.FullName -Destination $EMBEDDED_JDK_DIR + } + finally { + Remove-Item -Recurse -Force $staging -ErrorAction SilentlyContinue + Remove-Item $zip -ErrorAction SilentlyContinue + } + + $exe = Join-Path (Join-Path $EMBEDDED_JDK_DIR "bin") "java.exe" + if (-not (Test-Path $exe)) { + Write-Err "No java.exe under $EMBEDDED_JDK_DIR after unpacking" + return $null + } + + Write-Success "Installed Temurin $BOOTSTRAP_JAVA_VERSION to $EMBEDDED_JDK_DIR" + return $EMBEDDED_JDK_DIR + } + catch { + Write-Err "Could not download or unpack the Temurin JDK: $($_.Exception.Message)" + Write-Err "URL: $TEMURIN_ZIP_URL" + return $null + } +} + +function Resolve-Java { + Write-Info "Resolving Java (ES$EsVersion requires ${REQUIRED_JAVA_VERSION}+)..." + + # JAVA_HOME first, exactly as the launcher will. Probing the PATH `java` when + # JAVA_HOME is set would validate a JVM the REPL is never going to run. + $javaHomeExe = if ($env:JAVA_HOME) { Join-Path (Join-Path $env:JAVA_HOME "bin") "java.exe" } else { "" } + $jhMajor = if ($javaHomeExe -and (Test-Path $javaHomeExe)) { Get-JavaMajorFromExe -Exe $javaHomeExe } else { 0 } + + if ($jhMajor -gt 0) { + $script:JavaMajor = $jhMajor + $script:JavaSource = "JAVA_HOME ($env:JAVA_HOME)" + } + else { + if ($env:JAVA_HOME) { + Write-Warn "JAVA_HOME is set to '$env:JAVA_HOME' but no usable java.exe was found under it" + } + $pathMajor = Get-JavaMajorVersion + if ($pathMajor -gt 0) { + $script:JavaMajor = $pathMajor + $script:JavaSource = "PATH" + } + } + + if ($script:JavaMajor -ge $REQUIRED_JAVA_VERSION) { + Write-Success "Java $($script:JavaMajor) found via $($script:JavaSource) (required: ${REQUIRED_JAVA_VERSION}+)" + return $true + } + + if ($script:JavaMajor -eq 0) { + Write-Warn "No usable Java found" + } else { + Write-Warn "Java $($script:JavaMajor) found via $($script:JavaSource) — below the required ${REQUIRED_JAVA_VERSION}+" + } + + $jdkHome = Install-EmbeddedJdk + if (-not $jdkHome) { + Write-Err "Java $REQUIRED_JAVA_VERSION or higher is required for ES$EsVersion and could not be installed." + Write-Err "Install a JDK ${REQUIRED_JAVA_VERSION}+ manually and re-run, or set JAVA_HOME to one." + return $false + } + + $script:EmbeddedJdkHome = $jdkHome + $script:JavaMajor = Get-JavaMajorFromExe -Exe (Join-Path (Join-Path $jdkHome "bin") "java.exe") + $script:JavaSource = "bundled JDK ($jdkHome)" + + # SESSION scope only — deliberately not [Environment]::SetEnvironmentVariable(...,"User"). + # A machine-wide JAVA_HOME would silently repoint every other tool on the box. + # Future sessions do not need it: the launcher prefers \jdk directly. + $env:JAVA_HOME = $jdkHome + $env:PATH = (Join-Path $jdkHome "bin") + ";" + $env:PATH + Write-Success "Java $($script:JavaMajor) ready — JAVA_HOME and PATH updated for THIS session" + + return $true +} + # ============================================================================= # List Available Versions # ============================================================================= @@ -264,6 +418,13 @@ function Resolve-LatestVersion { return @($versions)[-1] } +# ============================================================================= +# Resolve Java before anything else that depends on it +# ============================================================================= +# Runs AFTER the -ListVersions early exit (listing versions must not download a +# JDK) and BEFORE bundle selection, which reads the resolved major. +if (-not (Resolve-Java)) { exit 1 } + # ============================================================================= # Bundle Selection: default install = ONE self-contained -all assembly # ============================================================================= @@ -296,11 +457,13 @@ if ($WITH_EXTENSIONS) { Write-Warn "No -all bundles published for $BUNDLE_ARTIFACT_NAME - falling back to the plain artifact" } - # The bundle needs Java 11+ (Arrow / logback bytecode). Check-Prerequisites - # aborts below the ES-version floor anyway; this keeps the fallback honest. - $javaMajor = Get-JavaMajorVersion - if ($USE_BUNDLE -and $javaMajor -gt 0 -and $javaMajor -lt 11) { - Write-Warn "Java $javaMajor found - the -all bundle requires Java 11+; falling back to the plain artifact" + # The bundle needs Java 11+ (Arrow / logback bytecode). Resolve-Java has already + # guaranteed >= $REQUIRED_JAVA_VERSION (11 or 17), bootstrapping a JDK if the + # host could not supply one, so this can only fire if that guarantee is ever + # weakened. Kept as a guard rather than deleted: silently shipping the bundle + # to a Java 8 host is a crash at first launch, not a warning. + if ($USE_BUNDLE -and $script:JavaMajor -gt 0 -and $script:JavaMajor -lt 11) { + Write-Warn "Java $($script:JavaMajor) found - the -all bundle requires Java 11+; falling back to the plain artifact" $USE_BUNDLE = $false $Version = $REQUESTED_VERSION } @@ -347,30 +510,19 @@ if ($USE_BUNDLE -and -not (Test-UrlExists -Url $DOWNLOAD_URL)) { # Check Prerequisites # ============================================================================= +# Resolve-Java already probed, and bootstrapped a JDK if it had to. This asserts +# the outcome rather than re-deriving it: ONE parse of `java -version` per run, so +# a localised or multi-line output cannot be read two different ways. function Check-Prerequisites { Write-Info "Checking prerequisites..." - if (-not (Get-Command java -ErrorAction SilentlyContinue)) { - Write-Err "Java is not installed." - Write-Err "ES$EsVersion requires Java $REQUIRED_JAVA_VERSION or higher." - exit 1 - } - - # One parse for the whole installer (Get-JavaMajorVersion), so a localised or - # multi-line `java -version` cannot be read two different ways. - $javaVersion = Get-JavaMajorVersion - - if ($javaVersion -eq 0) { - Write-Warn "Could not determine Java version" - } - elseif ($javaVersion -lt $REQUIRED_JAVA_VERSION) { + if ($script:JavaMajor -lt $REQUIRED_JAVA_VERSION) { Write-Err "Java $REQUIRED_JAVA_VERSION or higher is required for ES$EsVersion." - Write-Err "Found: Java $javaVersion" + Write-Err "Resolved: Java $($script:JavaMajor) via $($script:JavaSource)" exit 1 } - else { - Write-Success "Java $javaVersion found (required: ${REQUIRED_JAVA_VERSION}+)" - } + + Write-Success "Java $($script:JavaMajor) via $($script:JavaSource) (required: ${REQUIRED_JAVA_VERSION}+)" } # ============================================================================= @@ -694,6 +846,19 @@ if not exist "%JAR_FILE%" ( REM Create logs directory if it doesn't exist if not exist "%LOG_DIR%" mkdir "%LOG_DIR%" +REM Java resolution, in the SAME order the installer used: +REM 1. the JDK bundled into this install (present only when the installer had to +REM bootstrap one because the host had no Java, or too old a Java) +REM 2. %JAVA_HOME% +REM 3. whatever `java` is on PATH +REM Prepending to PATH rather than calling an absolute exe keeps every `java` +REM below unchanged and sidesteps quoting a path that contains spaces. `setlocal` +REM at the top means this PATH edit dies with the script. +REM Each `if` is its own line: cmd expands every %%VAR%% in a parenthesised block +REM in ONE parse pass, so a block would not see the value just assigned. +if exist "%BASE_DIR%\jdk\bin\java.exe" set "JAVA_HOME=%BASE_DIR%\jdk" +if defined JAVA_HOME if exist "%JAVA_HOME%\bin\java.exe" set "PATH=%JAVA_HOME%\bin;%PATH%" + if "%JAVA_OPTS%"=="" set JAVA_OPTS=-Xmx512m REM Java major version (1.8.x -> 8, 11.x -> 11), also our Java presence check. @@ -757,6 +922,21 @@ if (-not (Test-Path `$LogDir)) { New-Item -ItemType Directory -Path `$LogDir | Out-Null } +# Java resolution, in the SAME order the installer used: +# 1. the JDK bundled into this install (present only when the installer had to +# bootstrap one because the host had no Java, or too old a Java) +# 2. `$env:JAVA_HOME +# 3. whatever `java` is on PATH +# Prepending to PATH keeps every `java` below unchanged; the assignment is +# process-scoped, so it dies with this script. +`$BundledJdk = Join-Path `$BaseDir "jdk" +if (Test-Path (Join-Path `$BundledJdk "bin\java.exe")) { + `$env:JAVA_HOME = `$BundledJdk +} +if (`$env:JAVA_HOME -and (Test-Path (Join-Path `$env:JAVA_HOME "bin\java.exe"))) { + `$env:PATH = (Join-Path `$env:JAVA_HOME "bin") + ";" + `$env:PATH +} + # Check Java. Select-Object -First 1 matters: with two matching lines (e.g. a # deprecation notice from _JAVA_OPTIONS) the pipeline yields an array whose # ToString() is "System.Object[]", no regex matches, and the version reads 0. @@ -869,6 +1049,7 @@ Elasticsearch: $EsVersion Version: $Version Scala: $ScalaVersion Java Required: ${REQUIRED_JAVA_VERSION}+ +Java In Use: $($script:JavaMajor) via $($script:JavaSource) Artifact: $ARTIFACT_NAME Install type: $installType $licenseLine @@ -893,6 +1074,7 @@ function Print-Summary { Write-Host " Elasticsearch version: $EsVersion" Write-Host " SoftClient4ES version: $Version" Write-Host " Java required: ${REQUIRED_JAVA_VERSION}+" + Write-Host " Java in use: $($script:JavaMajor) via $($script:JavaSource)" if ($USE_BUNDLE) { Write-Host " Install type: -all bundle (engine + extensions, incl. cross-index JOIN)" } else { @@ -911,6 +1093,10 @@ function Print-Summary { Write-Host " | \-- $JAR_NAME" Write-Host " +-- logs\" Write-Host " | \-- (runtime logs)" + if ($script:EmbeddedJdkHome) { + Write-Host " +-- jdk\" + Write-Host " | \-- (bundled Temurin $BOOTSTRAP_JAVA_VERSION - the launcher prefers it)" + } if ($USE_BUNDLE) { Write-Host " +-- licenses\" Write-Host " | \-- (per-component licences)" From 2c4bbb1cea45a8f5d777d93e2e8fb44fae8c1ce5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Thu, 13 Aug 2026 08:37:18 +0200 Subject: [PATCH 2/2] =?UTF-8?q?docs:=200.20.4=20train=20=E2=80=94=20versio?= =?UTF-8?q?ns,=20JOIN=20fan-out=20(#195),=20and=20the=20lifted=20alias=20r?= =?UTF-8?q?estriction=20(arrow#137)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The elasticsql half of the post-release sweep. The softclient4es-web half is a separate PR in that repo. VERSIONS. Resolved against the published JFrog listings, not against commit titles — the currently published set really is engine 0.20.3, drivers / arrow-extensions 0.2.4, community-extensions 0.2.3, bundle line 0.20.1..0.20.3, so every bump here is exactly one release: engine / bundle examples 0.20.3 -> 0.20.4 README, repl.md, install.{sh,ps1} jdbc / adbc / flight-sql 0.2.4 -> 0.2.5 jdbc.md, adbc_driver.md, arrow_flight_sql.md, download_analytics.md, README arrow-extensions 0.2.4 -> 0.2.5 README community-extensions 0.2.3 -> 0.2.4 README Deliberately NOT bumped — dml_statements.md:450 says local JSON / JSON_ARRAY "were additionally moved off Hadoop entirely in 0.20.3". That is a historical fact about which release changed the behaviour, not a current-version reference; bumping it would make the sentence false. The repl.md example listing gains 0.20.4 and its total is corrected 2 -> 4. It claimed "Total: 2 version(s)" under three bullets; a live -ListVersions run prints "Total: 3 version(s)" for the three published bundles, so the count was simply wrong. JOINS — arrow#137, fixed. The "Two ORDER BY gotchas" callout documented a restriction that no longer exists. Rewritten against the shipped planner code (JoinPlanner.processPostJoinIds / isSelectAlias / referencesNoColumn) rather than from the issue text, so the boundaries are the real ones: SELECT aliases and ordinals resolve post-join in GROUP BY, HAVING and ORDER BY, matching is case-insensitive — but an alias is still NOT legal in SELECT, ON or WHERE, and it must be bare, because `ORDER BY d.cnt` qualifies a name no table owns and dies inside DuckDB. Version boundary stated as "since 0.2.5", never "up to and including", which is how this file class rots. The second gotcha — ORDER BY a column present on BOTH sides — is untouched. Nothing in this train changed it, so nothing new is asserted about it. JOINS — #195, fan-out on a non-unique key. New section next to that callout, with the issue's verified 2x3x2 = 12 example. One correction to the issue's own suggested wording, because it matters: it groups "AVG/SUM" together as not visibly breaking. SUM is inflated exactly like COUNT. What is preserved by uniform duplication is AVG, MIN and MAX — which is precisely the trap, since three columns of four then corroborate a count that is 6x off. Also notes that AVG *is* wrong when the fan-out factor varies within a group. Two further corrections against the source rather than the issue: * the issue calls a truncated fan-out "a wrong answer that returns successfully". The engine reports it — SQLWarning 01004, and an x-result-truncated header on Flight SQL, per this file's own Row truncation section, which the new text links to instead of contradicting. * cap figures taken from the meters table in this file: Community 10,000 / Pro 1,000,000. WINDOWS INSTALL. repl.md documents the new Java bootstrap (probe order, Temurin 17 covering both floors, no admin, session-scoped env, uninstall removes it) and the single-file install.cmd path; README's cmd.exe line becomes one command; the install tree gains jdk\. Every SQL example added or touched here was parse-probed through the real Parser.apply — 6/6 parse AND round-trip (the rendered AST re-parses). That covers COUNT(DISTINCT ...), which the new remedy advice depends on and which would have been a guess otherwise. ⚠️ NOT YET SATISFIABLE, and a merge gate: the sweep convention is to verify each version on JFrog BEFORE writing it, because a version with no -all bundle falls back silently to the plain artifact and the docs end up demonstrating the path that has no cross-index JOIN. The 0.20.4 / 0.2.5 artifacts are not published at authoring time. Before merging, HEAD-check: softclient4es{6,7,8,9}-cli-all_2.13 0.20.4, softclient4es{6,7,8,9}-{jdbc,adbc}-driver 0.2.5, softclient4es{6,7,8,9}-arrow-flight-sql 0.2.5, softclient4es-arrow-extensions 0.2.5, softclient4es-community-extensions 0.2.4. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 26 ++++++------- documentation/client/adbc_driver.md | 14 +++---- documentation/client/arrow_flight_sql.md | 10 ++--- documentation/client/download_analytics.md | 2 +- documentation/client/jdbc.md | 14 +++---- documentation/client/repl.md | 41 +++++++++++++++++---- documentation/sql/joins.md | 43 +++++++++++++++++++++- install.sh | 2 +- 8 files changed, 110 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 3f9f2711..43c01b5c 100644 --- a/README.md +++ b/README.md @@ -43,9 +43,9 @@ curl -fsSL https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/ irm https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.ps1 | iex ``` -**Windows (cmd.exe, when `.ps1` files are blocked):** download `install.cmd` and `install.ps1` side by side. It takes the same flags as `install.ps1` and only launches it with `-ExecutionPolicy Bypass`, for that one process. +**Windows (cmd.exe, when `.ps1` files are blocked):** one file — it fetches `install.ps1` if it is not beside it, takes the same flags, and only launches it with `-ExecutionPolicy Bypass`, for that one process. ```bat -install.cmd +curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.cmd && install.cmd ``` ### Connect and Query @@ -216,10 +216,10 @@ Download the self-contained fat JAR for your Elasticsearch version: | Elasticsearch Version | Artifact | |-----------------------|----------------------------------------| -| ES 6.x | `softclient4es6-jdbc-driver-0.2.4.jar` | -| ES 7.x | `softclient4es7-jdbc-driver-0.2.4.jar` | -| ES 8.x | `softclient4es8-jdbc-driver-0.2.4.jar` | -| ES 9.x | `softclient4es9-jdbc-driver-0.2.4.jar` | +| ES 6.x | `softclient4es6-jdbc-driver-0.2.5.jar` | +| ES 7.x | `softclient4es7-jdbc-driver-0.2.5.jar` | +| ES 8.x | `softclient4es8-jdbc-driver-0.2.5.jar` | +| ES 9.x | `softclient4es9-jdbc-driver-0.2.5.jar` | > **Java 11+ recommended** (17+ for ES 9.x): **cross-index JOINs require Java 11+** — the embedded JOIN engine is built on Apache Arrow 18.x, which ships Java-11 bytecode. @@ -236,20 +236,20 @@ Driver class: app.softnetwork.elastic.jdbc.ElasticDriver app.softnetwork.elastic softclient4es8-jdbc-driver - 0.2.4 + 0.2.5 ``` **Gradle:** ```groovy -implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.4' +implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.5' ``` **sbt:** ```scala -libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.4" +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.5" ``` The JDBC driver JARs are Scala-version-independent (no `_2.12` or `_2.13` suffix) and include all required dependencies. @@ -337,13 +337,13 @@ For programmatic access, add SoftClient4ES to your project. resolvers += "Softnetwork" at "https://softnetwork.jfrog.io/artifactory/releases/" // Choose your Elasticsearch version -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.20.3" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.20.4" // Add the community extensions for materialized views (optional) -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.2.3" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.2.4" // Add the arrow extensions for cross-index JOIN (required for JOINs; Java 11+) -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-arrow-extensions" % "0.2.4" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-arrow-extensions" % "0.2.5" // Add the JDBC driver if you want to use it from Scala (optional) -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.2.4" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.2.5" ``` ```scala diff --git a/documentation/client/adbc_driver.md b/documentation/client/adbc_driver.md index e5fc5cf5..f37d2f76 100644 --- a/documentation/client/adbc_driver.md +++ b/documentation/client/adbc_driver.md @@ -23,10 +23,10 @@ Download the self-contained fat JAR for your Elasticsearch version: | Elasticsearch | Artifact | |----------------|-----------------------------------------------------| -| ES 6.x | `softclient4es6-adbc-driver-0.2.4.jar` | -| ES 7.x | `softclient4es7-adbc-driver-0.2.4.jar` | -| ES 8.x | `softclient4es8-adbc-driver-0.2.4.jar` | -| ES 9.x | `softclient4es9-adbc-driver-0.2.4.jar` | +| ES 6.x | `softclient4es6-adbc-driver-0.2.5.jar` | +| ES 7.x | `softclient4es7-adbc-driver-0.2.5.jar` | +| ES 8.x | `softclient4es8-adbc-driver-0.2.5.jar` | +| ES 9.x | `softclient4es9-adbc-driver-0.2.5.jar` | ### Maven / Gradle / sbt @@ -36,20 +36,20 @@ Download the self-contained fat JAR for your Elasticsearch version: app.softnetwork.elastic softclient4es8-adbc-driver - 0.2.4 + 0.2.5 ``` **Gradle:** ```groovy -implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.2.4' +implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.2.5' ``` **sbt:** ```scala -libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.2.4" +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.2.5" ``` --- diff --git a/documentation/client/arrow_flight_sql.md b/documentation/client/arrow_flight_sql.md index d52ed630..f2cc2720 100644 --- a/documentation/client/arrow_flight_sql.md +++ b/documentation/client/arrow_flight_sql.md @@ -49,15 +49,15 @@ Available images per ES version: ### Fat JAR ```bash -java -jar softclient4es8-arrow-flight-sql-0.2.4.jar +java -jar softclient4es8-arrow-flight-sql-0.2.5.jar ``` | Elasticsearch | Artifact | |---------------|----------| -| ES 6.x | `softclient4es6-arrow-flight-sql-0.2.4.jar` | -| ES 7.x | `softclient4es7-arrow-flight-sql-0.2.4.jar` | -| ES 8.x | `softclient4es8-arrow-flight-sql-0.2.4.jar` | -| ES 9.x | `softclient4es9-arrow-flight-sql-0.2.4.jar` | +| ES 6.x | `softclient4es6-arrow-flight-sql-0.2.5.jar` | +| ES 7.x | `softclient4es7-arrow-flight-sql-0.2.5.jar` | +| ES 8.x | `softclient4es8-arrow-flight-sql-0.2.5.jar` | +| ES 9.x | `softclient4es9-arrow-flight-sql-0.2.5.jar` | --- diff --git a/documentation/client/download_analytics.md b/documentation/client/download_analytics.md index bf34903e..1852aaca 100644 --- a/documentation/client/download_analytics.md +++ b/documentation/client/download_analytics.md @@ -15,7 +15,7 @@ beacon to a public endpoint with exactly these fields: |---------------|----------|-----------------------------------------------| | `source` | `portal` | Where the count came from (the docs button) | | `driver` | `jdbc` | Which driver family (`jdbc` or `adbc`) | -| `version` | `0.2.4` | The published artifact version | +| `version` | `0.2.5` | The published artifact version | | `count_delta` | `1` | One download | A timestamp is added on the server. That is the **entire** record. diff --git a/documentation/client/jdbc.md b/documentation/client/jdbc.md index c4a50574..a9b1f33f 100644 --- a/documentation/client/jdbc.md +++ b/documentation/client/jdbc.md @@ -20,10 +20,10 @@ Download the self-contained fat JAR for your Elasticsearch version. The JARs are | Elasticsearch | Artifact | |---------------|----------| -| ES 6.x | `softclient4es6-jdbc-driver-0.2.4.jar` | -| ES 7.x | `softclient4es7-jdbc-driver-0.2.4.jar` | -| ES 8.x | `softclient4es8-jdbc-driver-0.2.4.jar` | -| ES 9.x | `softclient4es9-jdbc-driver-0.2.4.jar` | +| ES 6.x | `softclient4es6-jdbc-driver-0.2.5.jar` | +| ES 7.x | `softclient4es7-jdbc-driver-0.2.5.jar` | +| ES 8.x | `softclient4es8-jdbc-driver-0.2.5.jar` | +| ES 9.x | `softclient4es9-jdbc-driver-0.2.5.jar` | ### Build Tool Integration @@ -33,20 +33,20 @@ Download the self-contained fat JAR for your Elasticsearch version. The JARs are app.softnetwork.elastic softclient4es8-jdbc-driver - 0.2.4 + 0.2.5 ``` **Gradle:** ```groovy -implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.4' +implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.5' ``` **sbt:** ```scala -libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.4" +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.5" ``` --- diff --git a/documentation/client/repl.md b/documentation/client/repl.md index 0ee93738..ec4c1c47 100644 --- a/documentation/client/repl.md +++ b/documentation/client/repl.md @@ -51,6 +51,26 @@ It provides: > 1.5.x and the JOIN engine is built on Apache Arrow 18.x — both ship Java-11 > bytecode. See [Extensions](#extensions-cross-index-joins-materialized-views). +**On Windows you do not have to install Java yourself.** Since `0.20.4`, +`install.ps1` (and therefore `install.cmd`) resolves Java in this order: + +1. `%JAVA_HOME%\bin\java.exe` — when `JAVA_HOME` is set, that is the JVM tested, + not whatever `java` happens to be first on `PATH`; the two frequently differ. +2. the `java` on `PATH`, when `JAVA_HOME` is not set. +3. Neither is present, or the one found is **below the floor** for your ES + version ⇒ the installer downloads a portable **Temurin 17** JDK (a zip from + Adoptium, never an MSI, so it needs **no administrator rights**) and unpacks + it to `\jdk`. + +Java 17 satisfies both floors, so there is only ever one JDK to think about. The +bootstrapped JDK lives **inside the install directory**: `uninstall.ps1` removes +it along with everything else, and nothing machine-wide is modified — the +installer sets `JAVA_HOME` and `PATH` **for its own session only**. Later +sessions do not need them, because the generated launcher applies the same order +and finds `\jdk` by relative path. + +On Linux and macOS `install.sh` still expects a suitable Java to be present. + ### Network Requirements - Network access to JFrog repository (`softnetwork.jfrog.io`) @@ -97,14 +117,18 @@ Windows client default is `Restricted`, and the machine may also be set to `install.ps1` with `-ExecutionPolicy Bypass` **for that one process**, changing nothing on the machine and needing no elevation. -Download **both** files, keep them in the same directory, then: +One file is enough. If `install.ps1` is not sitting next to it, `install.cmd` +downloads one: ```bat curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.cmd -curl -O https://raw.githubusercontent.com/SOFTNETWORK-APP/SoftClient4ES/main/install.ps1 install.cmd ``` +A local `install.ps1` always wins, so a downloaded pair stays self-consistent — +put both files in the same directory when you want a pinned copy rather than +whatever is on `main`. + `install.cmd` accepts exactly the flags `install.ps1` does and forwards them verbatim, so every option, default and fallback documented below applies unchanged — there is no second implementation to drift: @@ -163,8 +187,9 @@ install.cmd -ListVersions -EsVersion 8 • 0.20.1 • 0.20.2 • 0.20.3 + • 0.20.4 - Total: 2 version(s) + Total: 4 version(s) To install a specific version: ./install.sh --es-version 8 --version @@ -192,7 +217,7 @@ install.cmd -ListVersions -EsVersion 8 ./install.sh --list-versions --es-version 8 # Install specific version -./install.sh --es-version 8 --version 0.20.3 +./install.sh --es-version 8 --version 0.20.4 # Install for Elasticsearch 9 (requires Java 17+) ./install.sh --es-version 9 @@ -201,7 +226,7 @@ install.cmd -ListVersions -EsVersion 8 ./install.sh --target /opt/softclient4es # Full custom installation -./install.sh --target ~/tools/softclient4es --es-version 7 --version 0.20.3 +./install.sh --target ~/tools/softclient4es --es-version 7 --version 0.20.4 ``` #### Windows @@ -214,7 +239,7 @@ install.cmd -ListVersions -EsVersion 8 .\install.ps1 -ListVersions -EsVersion 8 # Install specific version -.\install.ps1 -EsVersion 8 -Version 0.20.3 +.\install.ps1 -EsVersion 8 -Version 0.20.4 # Install for Elasticsearch 9 (requires Java 17+) .\install.ps1 -EsVersion 9 @@ -223,7 +248,7 @@ install.cmd -ListVersions -EsVersion 8 .\install.ps1 -Target "C:\tools\softclient4es" # Full custom installation -.\install.ps1 -Target "C:\tools\softclient4es" -EsVersion 7 -Version 0.20.3 +.\install.ps1 -Target "C:\tools\softclient4es" -EsVersion 7 -Version 0.20.4 ``` --- @@ -247,6 +272,8 @@ softclient4es/ │ # + extension jars and dependencies, see Extensions) ├── logs/ # Log files directory │ └── softclient4es.log # (created at runtime) +├── jdk/ # Windows only, and ONLY when the installer had to +│ └── bin/java.exe # bootstrap a JDK — the launcher prefers it ├── LICENSE ├── README.md ├── VERSION diff --git a/documentation/sql/joins.md b/documentation/sql/joins.md index bf2884d2..eedbf3ac 100644 --- a/documentation/sql/joins.md +++ b/documentation/sql/joins.md @@ -111,7 +111,48 @@ ORDER BY COUNT(*) DESC; -- Engineering (3), Marketing (2) survive HAVING ``` -> **Two ORDER BY gotchas:** the JOIN planner has two ordering restrictions — you cannot `ORDER BY` a **SELECT alias** (use `ORDER BY COUNT(*)`, not `ORDER BY cnt`), and you cannot `ORDER BY` a column that exists on **both** sides of the JOIN (order by a column unique to one side, e.g. `d.dept_name`, not the shared join key `d.dept_id`). +> **SELECT aliases and ordinals work here** — since arrow-extensions **0.2.5** (REPL bundle `0.20.4`, JDBC / ADBC / Flight SQL driver `0.2.5`). `ORDER BY cnt`, `HAVING cnt > 1`, `GROUP BY` on an alias and ordinal forms such as `ORDER BY 2` all resolve against the final SELECT list *after* the join, and alias matching is case-insensitive. Two limits remain: an alias is **not** legal in `SELECT`, `ON` or `WHERE` — nothing has been computed at that point — and it must be written **bare**, since `ORDER BY d.cnt` qualifies a name no table owns and fails inside DuckDB. Before 0.2.5 all of these were rejected with `Ambiguous column`. +> +> **The remaining ORDER BY gotcha:** you cannot `ORDER BY` a column that exists on **both** sides of the JOIN — order by a column unique to one side, e.g. `d.dept_name`, not the shared join key `d.dept_id`. + +### JOIN cardinality — fan-out on a non-unique key + +A JOIN on a key that is **not unique** on the other side multiplies rows. That is standard SQL and the engine is doing it correctly, but it is the easiest way to get plausible-looking wrong numbers, because the row multiplication is invisible in the output. + +With one tenant that has **2** EU error rows, **3** US rows and **2** AP rows: + +```sql +SELECT eu.tenant_id, + COUNT(*) AS eu_errors, + AVG(us.latency_ms) AS us_avg_latency, + AVG(ap.latency_ms) AS ap_avg_latency +FROM eu_events AS eu +JOIN us_events AS us ON eu.tenant_id = us.tenant_id +JOIN ap_events AS ap ON eu.tenant_id = ap.tenant_id +WHERE eu.level = 'ERROR' +GROUP BY eu.tenant_id; +``` + +`eu_errors` comes back as **12** — that is 2 × 3 × 2, one row per combination. The answer meant by the query is **2**. + +**Why this is worse than one wrong column.** Within a group whose rows all fan out by the same factor: + +| Aggregate | Under fan-out | +|---|---| +| `COUNT`, `SUM` | **inflated** by the fan-out factor | +| `AVG`, `MIN`, `MAX` | **unchanged** — uniform duplication preserves them | + +So in the example above both averages are exactly right, and only the count is wrong. Three columns out of four corroborate a result that is 6× off, and nothing in the output signals that a fan-out happened. *Do not use "the averages look sensible" as a sanity check on a JOIN.* (If the factor varies across rows within a group — because you grouped by something coarser than the join key — then `AVG` is silently weighted too, and it is wrong as well.) + +**What to do instead:** + +- Count a key from **one** side rather than rows of the joined product: `COUNT(DISTINCT eu.event_id)`. +- Or aggregate **before** joining, so each side contributes one row per key — a materialized view per leg is the durable form of this. +- Sanity-check the row count against the left side alone before adding aggregates. + +**`INNER JOIN` also drops rows.** A key absent from *any* joined table disappears from the result entirely — a tenant running in EU and US but not APAC vanishes from a query whose name says "every region". Use `LEFT JOIN` when the left side is the population you actually mean. + +**In Federation specifically:** each leg is staged and joined coordinator-local, so a fan-out inflates the staged intermediate *and* consumes the joined-output row cap (`maxQueryResults`, Community 10,000 / Pro 1,000,000). Hitting that cap is reported — see [Row truncation at the result cap](#row-truncation-at-the-result-cap) — but a truncated fan-out is still an answer to a question you did not ask. ### ORDER BY … LIMIT (top-N) diff --git a/install.sh b/install.sh index 98e130cf..983c55cc 100755 --- a/install.sh +++ b/install.sh @@ -174,7 +174,7 @@ Examples: $0 $0 --list-versions --es-version 8 $0 --target /opt/softclient4es --es-version 8 --no-extensions - $0 -t ~/tools/softclient4es -e 7 -v 0.20.3 --no-extensions + $0 -t ~/tools/softclient4es -e 7 -v 0.20.4 --no-extensions Detected OS: $OS_TYPE