Buildengine phase1

buildengine/PLAN.md

@@ -0,0 +1,130 @@
+# Local Build Engine — Roadmap (Sketchware-Pro-style, Gradle-free)
+
+## Status: Phase 1 scaffolded (this commit)
+
+`buildengine/engine/` — new module, Gradle-free at *runtime* (it's plain
+Kotlin/JVM code that itself still compiles via the IDE's own Gradle build —
+that's unrelated to what it builds for the user's Android project).
+
+Files added:
+- `model/BuildModels.kt` — BuildError (file+line for editor click-to-line),
+  BuildStage, BuildResult, BuildLogEvent, BuildConfig
+- `tools/BuildToolsLocator.kt` — resolves bundled ecj/d8/aapt2/zipalign/
+  apksigner/android.jar/kotlinc under `<filesDir>/build-tools/`
+- `tools/ProcessRunner.kt` — cancellable subprocess runner w/ line streaming
+- `classpath/ClasspathCollector.kt` — libs/*.jar + libs/*.aar → compile/dex
+  classpath (AAR exploding with content-hash cache)
+- `steps/EcjCompileStep.kt` — Java → .class, parses ECJ diagnostics
+- `steps/KotlincCompileStep.kt` — Kotlin → .class (optional component)
+- `steps/D8DexStep.kt` — .class + libs → classes.dex
+- `steps/Aapt2Step.kt` — res/ + manifest → resources.apk (also emits R.java)
+- `steps/ApkAssembleStep.kt` — merges dex+assets into the apk; ZipAlignStep;
+  ApkSignStep (apksigner, debug keystore)
+- `BuildManager.kt` — async orchestrator (coroutines/Flow), build/clean/
+  rebuild/cancel, streams `BuildLogEvent`s stage-by-stage
+
+This directly maps to requirement #3's pipeline and #7/#8 (async, cancel,
+structured errors with file/line).
+
+## Requirements still open
+
+| # | Requirement | Status |
+|---|---|---|
+| 1 | Disable old Gradle flow, keep editor/explorer intact | Not started — old flow still active, by design (nothing breaks yet) |
+| 2 | Local/offline engine, avoid Gradle daemon | Done for the engine itself |
+| 4 | `build-tools/` dir layout | Defined in `BuildToolsLocator`; actual binaries (ecj.jar, d8.jar, aapt2, zipalign, apksigner.jar, android.jar, kotlin/) still need to be sourced/bundled — see "Bundling tools" below |
+| 5 | Java + Kotlin + XML support | Java/Kotlin done; XML is handled implicitly via aapt2 (layouts/manifests) — XML *editing* already works (existing `xml:*` modules), untouched |
+| 6 | Auto-collect deps | Done for local `libs/*.jar` + `libs/*.aar`. **Not done:** transitive Maven coordinate resolution (`implementation("com.squareup:okhttp3:...")` style) — needs a minimal resolver or a "vendor your jars" convention, Phase 2 |
+| 9 | Don't break existing project format | Not yet validated — needs a real project run through the new engine |
+| 10 | Modular, UI-independent | Done — `BuildManager` has zero UI/Android dependencies |
+| 11 | Remove unused Gradle code safely | Not started — see "Removal list" below, deliberately deferred until Phase 3 is proven working, so nothing is deleted before its replacement works |
+| 12 | No forced huge SDK download | Addressed by design (core tools ~few MB, Kotlin optional) but depends on how tools are bundled |
+| 13 | Fully local, no source upload | Naturally true — no network code anywhere in this module |
+
+## Phase 2 — Local project model: IN PROGRESS
+
+Added `core/projects/.../internal/LocalWorkspaceModelBuilder.kt`. It builds
+the exact same `WorkspaceImpl`/`AndroidModule` shape as `WorkspaceModelBuilder`
+does today, but by scanning the module's `src/main/{java,kotlin,res,assets}`
++ `libs/*.jar`/`*.aar` + `AndroidManifest.xml` directly — no Gradle sync.
+
+Deliberately reuses `AndroidModule`/`GradleProject`/`tooling:builder-model-
+impl`'s `Default*` classes rather than inventing new ones — those are plain
+mutable data holders around real AGP builder-model *interfaces*, not
+Gradle-runtime-coupled, so **`tooling:builder-model-impl` should be kept**,
+not removed in Phase 4 (updating the removal list below).
+
+**Not done yet / honest limitations of this first pass:**
+- Single Android app module only (root dir or `app/` subfolder). Multi-
+  module local projects (`:app`, `:core`, ...) need a second pass that
+  walks subfolders for additional manifests.
+- No transitive Maven dependency resolution — only `libs/*.jar`/`*.aar`,
+  matching what the build engine itself supports (see Phase 1 table, req #6).
+- `ProjectSyncIssues`'s exact interface shape was written from memory —
+  I could not fetch the real `com.android.tools.build:builder-model` jar in
+  this sandbox (no Maven network access here) to confirm the method
+  signature. **This needs a real compile pass against the actual AGP
+  builder-model jar before you trust it** — flagged with a comment in the
+  file itself.
+- **Not wired in yet.** `ProjectManagerImpl.kt` (line ~100) still calls the
+  old `WorkspaceModelBuilder.build(projectDir, CachingProject(project))`.
+  Swapping the call site is deliberately left undone until you can compile
+  and test-run this locally — I don't have an Android SDK/Gradle-connected
+  environment here to verify these ~250 lines of AGP-model-shaped code
+  actually compile. Recommended next step: add a per-project toggle (e.g.
+  a `.acs-local-build` marker file in the project root) so
+  `ProjectManagerImpl` can call `LocalWorkspaceModelBuilder.build(...)`
+  for opted-in projects while everything else keeps using Gradle — safest
+  way to test this without risking every existing project.
+
+## Phase 4 — Removal list (updated)
+
+- `tooling:api`, `tooling:impl`, `tooling:model`, `tooling:events`,
+  `tooling:plugin`, `tooling:plugin-config` — remove once Phase 2/3 proven
+- **`tooling:builder-model-impl` — KEEP.** It's reused by
+  `LocalWorkspaceModelBuilder` (Phase 2); it only depends on AGP's
+  `builder-model` *interfaces*, not the Gradle runtime.
+- `composite-builds/*` (build-logic, build-deps and their Gradle plugin glue)
+- Gradle wrapper usage for *target projects* (the IDE's own build can keep
+  using Gradle — that's a separate concern from what it builds for users)
+
+Files that call into Gradle today and need a `BuildManager`-based
+replacement (found via `grep`, not yet touched):
+
+- `core/app/src/main/java/com/tom/rv2ide/services/builder/GradleBuildService.kt`
+- `core/app/src/main/java/com/tom/rv2ide/services/builder/GradleBuildServiceConnnection.kt`
+- `core/app/src/main/java/com/tom/rv2ide/services/builder/GradleServiceBinder.kt`
+- `core/app/src/main/java/com/tom/rv2ide/services/builder/ToolingServerRunner.kt`
+- `core/app/src/main/java/com/tom/rv2ide/actions/BaseBuildAction.kt`
+- `core/app/src/main/java/com/tom/rv2ide/actions/build/QuickRunWithCancellationAction.kt`
+- `core/app/src/main/java/com/tom/rv2ide/handlers/EditorBuildEventListener.kt` — this is where BuildError→editor click-to-line (requirement #8) plugs in; `BuildLogEvent.Diagnostic` already carries exactly the shape it needs
+- `core/app/src/main/java/com/tom/rv2ide/viewmodel/RunTasksViewModel.kt` + `RunTasksDialogFragment.kt` + `RunTasksListAdapter.kt` — currently list *Gradle tasks*; needs to become a fixed Build/Clean/Rebuild/Cancel action list instead (there are no arbitrary Gradle tasks anymore)
+- `core/app/src/main/java/com/tom/rv2ide/viewmodel/EditorViewModel.kt` — build-trigger entry point
+
+## Bundling the actual tool binaries — DECIDED: first-run download
+
+Added `tools/ToolsProvisioner.kt`. On first build (not on app open — so
+requirement #12 is satisfied), it downloads one small "core" zip (~5-15MB:
+ecj.jar, d8.jar, aapt2, zipalign, apksigner.jar, android.jar), verifies its
+SHA-256 against a manifest, zip-slip-guards the extraction, and unpacks it
+into `build-tools/`. Kotlin support is a second, separate, larger, optional
+package — only fetched the first time a project actually has `.kt` files.
+
+**What you still need to do (I can't do this part for you):**
+1. Build the `core` zip yourself: pull `ecj.jar` from an Eclipse JDT Core
+   release, `d8.jar`/`android.jar` from an Android SDK / R8 distribution,
+   `aapt2` + `zipalign` native binaries per-ABI (arm64-v8a at minimum) from
+   Google's Maven `com.android.tools.build:aapt2`, and `apksigner.jar` from
+   the SDK build-tools. Zip them into the `build-tools/...` layout
+   `BuildToolsLocator` expects.
+2. Host that zip somewhere stable (a GitHub Release asset on this repo is
+   the simplest option — free, versioned, no extra infra) and compute its
+   SHA-256 (`sha256sum core.zip`).
+3. Implement `ToolsProvisioner.ManifestProvider` with that URL + checksum
+   (a one-file, hardcoded implementation is fine to start — no need for a
+   remote manifest server).
+
+**Phase 3 UI hook:** call `provisioner.provision(...)` the first time the
+user taps Build, showing `ProvisionEvent.Progress` as a download progress
+bar before the pipeline's `BuildStage`s start; skip straight to building on
+every run after that since `isProvisioned()` is then true.

buildengine/engine/build.gradle.kts

@@ -0,0 +1,19 @@
+plugins {
+  id("java-library")
+  id("org.jetbrains.kotlin.jvm")
+}
+
+java {
+  sourceCompatibility = JavaVersion.VERSION_17
+  targetCompatibility = JavaVersion.VERSION_17
+}
+
+dependencies {
+  implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.8.1")
+  testImplementation("junit:junit:4.13.2")
+}
+
+// NOTE (Phase 3 wiring): core:app should add
+//   implementation(project(":buildengine:engine"))
+// and GradleBuildService/GradleBuildServiceConnnection get replaced by a thin
+// wrapper around com.tom.rv2ide.buildengine.BuildManager. See buildengine/PLAN.md.

buildengine/engine/src/main/java/com/tom/rv2ide/buildengine/BuildManager.kt

@@ -0,0 +1,217 @@
+package com.tom.rv2ide.buildengine
+
+import com.tom.rv2ide.buildengine.classpath.ClasspathCollector
+import com.tom.rv2ide.buildengine.model.*
+import com.tom.rv2ide.buildengine.steps.*
+import com.tom.rv2ide.buildengine.tools.BuildToolsLocator
+import kotlinx.coroutines.*
+import kotlinx.coroutines.channels.awaitClose
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.callbackFlow
+import java.io.File
+import java.util.concurrent.atomic.AtomicBoolean
+
+/**
+ * Local, offline replacement for GradleBuildService/GradleBuildServiceConnnection.
+ * UI layer (RunTasksViewModel etc.) should depend on this interface only, so
+ * swapping the engine or adding more build steps later doesn't touch the UI
+ * (requirement #10 — modular, UI-independent).
+ *
+ * Wiring note for Phase 3: this class is designed to sit exactly where
+ * GradleBuildService currently sits (same responsibilities: build/clean/
+ * rebuild/cancel + a log/event stream), so EditorBuildEventListener and
+ * RunTasksViewModel can be repointed here with minimal changes.
+ */
+class BuildManager(
+  private val tools: BuildToolsLocator,
+  private val javaBin: File,
+  private val intermediatesRoot: File
+) {
+
+  private val cancelled = AtomicBoolean(false)
+  private var currentJob: Job? = null
+  private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
+
+  fun cancel() {
+    cancelled.set(true)
+    currentJob?.cancel()
+  }
+
+  fun clean(config: BuildConfig): Flow<BuildLogEvent> = callbackFlow {
+    trySend(BuildLogEvent.StageStarted(BuildStage.CLEAN))
+    config.outputDir.deleteRecursively()
+    File(intermediatesRoot, config.moduleDir.name).deleteRecursively()
+    trySend(BuildLogEvent.StageFinished(BuildStage.CLEAN, true))
+    close()
+    awaitClose { }
+  }
+
+  fun build(config: BuildConfig, rebuild: Boolean = false): Flow<BuildLogEvent> = callbackFlow {
+    cancelled.set(false)
+    val startTime = System.currentTimeMillis()
+    val intermediates = File(intermediatesRoot, config.moduleDir.name).apply { mkdirs() }
+    if (rebuild) intermediates.deleteRecursively().also { intermediates.mkdirs() }
+
+    val errors = mutableListOf<BuildError>()
+    val warnings = mutableListOf<BuildError>()
+
+    fun stage(s: BuildStage) = trySend(BuildLogEvent.StageStarted(s))
+    fun stageDone(s: BuildStage, ok: Boolean) = trySend(BuildLogEvent.StageFinished(s, ok))
+    fun log(line: String) = trySend(BuildLogEvent.Output(line))
+    fun diag(e: BuildError) {
+      trySend(BuildLogEvent.Diagnostic(e))
+      if (e.severity == BuildError.Severity.ERROR) errors.add(e) else warnings.add(e)
+    }
+    fun cancelFlag() = cancelled.get()
+
+    currentJob = scope.launch {
+      try {
+        if (!tools.coreToolsPresent()) {
+          val missing = tools.missingCoreTools().joinToString(", ")
+          diag(BuildError(null, null, null, "Missing build tools: $missing. Install the build-tools component first.", BuildError.Severity.ERROR, "engine"))
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close()
+          return@launch
+        }
+
+        // 1. Dependencies
+        stage(BuildStage.COLLECT_DEPENDENCIES)
+        val classpathCollector = ClasspathCollector(File(intermediates, "exploded-aars"))
+        val collected = classpathCollector.collect(config.moduleDir, config.compileSdkJar)
+        stageDone(BuildStage.COLLECT_DEPENDENCIES, true)
+        if (cancelFlag()) { trySend(BuildLogEvent.Cancelled); close(); return@launch }
+
+        // 2. Java compile (ECJ)
+        stage(BuildStage.COMPILE_JAVA)
+        val classesDir = File(intermediates, "classes").apply { mkdirs() }
+        val ecj = EcjCompileStep(tools, javaBin).compile(
+          sources = config.javaSourceDirs.flatMap { it.walkTopDown().filter { f -> f.extension == "java" }.toList() },
+          classpath = collected.compileJars + classesDir,
+          outDir = classesDir,
+          cancelFlag = ::cancelFlag,
+          onOutputLine = ::log
+        )
+        ecj.errors.forEach(::diag); ecj.warnings.forEach(::diag)
+        stageDone(BuildStage.COMPILE_JAVA, ecj.success)
+        if (!ecj.success || cancelFlag()) {
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close(); return@launch
+        }
+
+        // 3. Kotlin compile (only if the module has .kt files)
+        val ktSources = config.kotlinSourceDirs.flatMap { it.walkTopDown().filter { f -> f.extension == "kt" }.toList() }
+        if (ktSources.isNotEmpty()) {
+          stage(BuildStage.COMPILE_KOTLIN)
+          val kt = KotlincCompileStep(tools).compile(
+            sources = ktSources,
+            classpath = collected.compileJars + classesDir,
+            outDir = classesDir,
+            cancelFlag = ::cancelFlag,
+            onOutputLine = ::log
+          )
+          kt.errors.forEach(::diag); kt.warnings.forEach(::diag)
+          stageDone(BuildStage.COMPILE_KOTLIN, kt.success)
+          if (!kt.success || cancelFlag()) {
+            trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+            close(); return@launch
+          }
+        }
+
+        // 4. Resources (aapt2 compile + link)
+        stage(BuildStage.PROCESS_RESOURCES)
+        val resApk = File(intermediates, "resources.apk")
+        val aapt2Result = Aapt2Step(tools).run(
+          resDirs = config.resDirs,
+          manifest = config.manifestFile,
+          androidJar = config.compileSdkJar,
+          extraPackagesFromAars = emptyList(), // Phase 2: read package name from each exploded AAR's manifest
+          intermediatesDir = File(intermediates, "aapt2"),
+          outApk = resApk,
+          cancelFlag = ::cancelFlag,
+          onOutputLine = ::log
+        )
+        aapt2Result.errors.forEach(::diag)
+        stageDone(BuildStage.PROCESS_RESOURCES, aapt2Result.success)
+        if (!aapt2Result.success || cancelFlag()) {
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close(); return@launch
+        }
+
+        // 5. Dex
+        stage(BuildStage.DEX)
+        val dexDir = File(intermediates, "dex").apply { mkdirs() }
+        val d8 = D8DexStep(tools, javaBin).dex(
+          classesDir = classesDir,
+          libraryJars = collected.dexJars,
+          minSdk = config.minSdk,
+          outDir = dexDir,
+          cancelFlag = ::cancelFlag,
+          onOutputLine = ::log
+        )
+        d8.errors.forEach(::diag)
+        stageDone(BuildStage.DEX, d8.success)
+        if (!d8.success || cancelFlag()) {
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close(); return@launch
+        }
+
+        // 6. Assemble
+        stage(BuildStage.ASSEMBLE_APK)
+        val unsignedApk = File(intermediates, "unsigned.apk")
+        val assembled = ApkAssembleStep().assemble(resApk, dexDir, config.assetsDirs, unsignedApk)
+        stageDone(BuildStage.ASSEMBLE_APK, assembled)
+        if (!assembled) {
+          diag(BuildError(null, null, null, "Failed to assemble APK from resources.apk + classes.dex.", BuildError.Severity.ERROR, "engine"))
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close(); return@launch
+        }
+
+        // 7. zipalign
+        stage(BuildStage.ZIPALIGN)
+        val alignedApk = File(intermediates, "aligned.apk")
+        val aligned = ZipAlignStep(tools).align(unsignedApk, alignedApk, ::cancelFlag, ::log)
+        stageDone(BuildStage.ZIPALIGN, aligned)
+        if (!aligned) {
+          diag(BuildError(null, null, null, "zipalign failed.", BuildError.Severity.ERROR, "zipalign"))
+          trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+          close(); return@launch
+        }
+
+        // 8. sign
+        stage(BuildStage.SIGN_APK)
+        config.outputDir.mkdirs()
+        val finalApk = File(config.outputDir, "${config.applicationId}-${if (config.isDebug) "debug" else "release"}.apk")
+        val signed = ApkSignStep(tools, javaBin).sign(
+          inApk = alignedApk,
+          outApk = finalApk,
+          keystore = config.debugKeystore,
+          keystorePass = "android",
+          keyAlias = "androiddebugkey",
+          keyPass = "android",
+          cancelFlag = ::cancelFlag,
+          onOutputLine = ::log
+        )
+        signed.errors.forEach(::diag)
+        stageDone(BuildStage.SIGN_APK, signed.success)
+
+        val result = BuildResult(
+          success = signed.success,
+          outputApk = finalApk.takeIf { signed.success },
+          errors = errors,
+          warnings = warnings,
+          durationMs = System.currentTimeMillis() - startTime
+        )
+        trySend(BuildLogEvent.Finished(result))
+      } catch (ce: CancellationException) {
+        trySend(BuildLogEvent.Cancelled)
+      } catch (t: Throwable) {
+        diag(BuildError(null, null, null, "Build engine crashed: ${t.message}", BuildError.Severity.ERROR, "engine"))
+        trySend(BuildLogEvent.Finished(BuildResult(false, errors = errors, warnings = warnings)))
+      } finally {
+        close()
+      }
+    }
+
+    awaitClose { currentJob?.cancel() }
+  }
+}