diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_index.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_index.md new file mode 100644 index 0000000000..743aa1d5c4 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_index.md @@ -0,0 +1,93 @@ +--- +title: Run ExecuTorch Llama 3.2 1B Instruct on a Vivo X300 Pro with Vulkan + +description: Learn how to export Llama 3.2 1B Instruct with ExecuTorch, build the Android Vulkan runtime, and run the model on a Vivo X300 Pro. + +minutes_to_complete: 120 + +who_is_this_for: This Learning Path is for developers who want to export a Llama 3.2 model with ExecuTorch and run it on an Android phone through the Vulkan backend. + +learning_objectives: + - Set up a Linux host for Android cross-compilation, ADB deployment, and Vulkan-backed ExecuTorch builds. + - Pin the host Python and PyTorch environment required by the ExecuTorch 1.4 release branch. + - Export Meta Llama 3.2 1B Instruct to a Vulkan-enabled `.pte` with the exact quantization settings used in the guide. + - Build, deploy, validate, and troubleshoot the Android `llama_main` runner on a Vivo X300 Pro. + +prerequisites: + - A Linux host with enough disk space for the ExecuTorch source tree, Android SDK and NDK, Vulkan SDK, and the Llama checkpoint. + - A Vivo X300 Pro or a similar Android phone with USB debugging enabled. + - Access to the gated Hugging Face repository `meta-llama/Llama-3.2-1B-Instruct`. + - Working familiarity with the shell, Python virtual environments, and Android ADB workflows. + +author: Ash Naik + +skilllevels: + - Advanced + +subjects: + - ML + +operatingsystems: + - Linux + - Android + +tools_software_languages: + - ExecuTorch + - PyTorch + - Python + - Android SDK + - Android NDK + - ADB + - Vulkan + - glslc + - Hugging Face +armips: + - Cortex-A + - Mali + +further_reading: + - resource: + title: ExecuTorch repository + link: https://github.com/pytorch/executorch + type: documentation + - resource: + title: ExecuTorch Llama documentation + link: https://github.com/pytorch/executorch/blob/main/docs/source/llm/llama.md + type: documentation + - resource: + title: ExecuTorch Android Vulkan documentation + link: https://github.com/pytorch/executorch/blob/main/docs/source/android-vulkan.md + type: documentation + +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- + +## About this Learning Path + +This Learning Path turns the material from `ExecuTorch_Llama32_Vivo_X300Pro_End_to_End_Guide.docx` into an Arm-style walkthrough. It keeps the exact versions, commands, and measured results from the successful run, while reorganizing them into a sequence you can follow from environment setup through validation. + +### Who is this for? + +This topic is for developers targeting Android Arm64 devices who want to run an LLM locally with ExecuTorch and the Vulkan backend instead of CPU-only inference. + +### What will you learn? + +Upon completion of this Learning Path, you will be able to: + +- Prepare the Android SDK, NDK, ADB, and host Vulkan toolchain required for cross-compiling ExecuTorch. +- Fix the PyTorch version mismatch that can break ExecuTorch 1.4 builds on the host. +- Export Llama 3.2 1B Instruct to a Vulkan-ready `.pte` using the same `8da4w` and KV-cache settings as the measured run. +- Build the Android runtime and `llama_main`, deploy them to the phone, and confirm Vulkan-backed execution. + +### Prerequisites + +Before starting, you will need the following: + +- A Linux machine that can build native and Android targets. +- Android Studio or an equivalent Android SDK and NDK installation path. +- A phone connected over USB with developer mode and USB debugging enabled. +- Hugging Face access to download `meta-llama/Llama-3.2-1B-Instruct`. +- Sufficient storage for a multi-gigabyte checkpoint and exported `.pte`. diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_next-steps.md new file mode 100644 index 0000000000..1c50538af4 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/_next-steps.md @@ -0,0 +1,17 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 9 +title: "Next Steps" +layout: "learningpathall" +--- + +Use the metadata in `_index.md` to populate the standard Arm Learning Paths next-steps experience when this content is imported into a Hugo tree. + +If you extend this guide, the most useful follow-on topics are: + +- compare Vulkan against XNNPACK or other Android backends on the same phone +- rebuild with ETDump and capture delegated-region timings +- retune quantization, context length, and prompt format for memory or throughput +- adapt the same flow to a different Android device with a different GPU and driver stack diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/android-device-architecture.svg b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/android-device-architecture.svg new file mode 100644 index 0000000000..f642ab01fd --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/android-device-architecture.svg @@ -0,0 +1,54 @@ + + Android device architecture + Deployed runtime flow on the Vivo X300 Pro showing llama_main, ExecuTorch runtime, Vulkan backend, and Mali GPU, plus deployed artifacts from the host. + + + + + + + + + + + + + + + + + + + + + Android device architecture + Deployed runtime flow on the Vivo X300 Pro + + + + Vivo X300 Pro + arm64-v8a Android device + + + llama_main + + + + + ExecuTorch runtime + + + + + Vulkan backend + + + + + Mali GPU + + Deployed artifacts from host: + • `llama_main` + • Vulkan-enabled `.pte` + • tokenizer files + diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-1-prepare-host-and-device.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-1-prepare-host-and-device.md new file mode 100644 index 0000000000..bd087cd805 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-1-prepare-host-and-device.md @@ -0,0 +1,139 @@ +--- +title: Prepare the host and device +description: Install the Android toolchain, configure ADB access, and verify Vulkan support on the target phone. +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Install Android SDK and NDK + +Use Android Studio or the SDK Manager to install the following components: + +- Android SDK Platform-Tools +- Android SDK Command-line Tools +- CMake +- NDK (Side by side) + +This workflow used Android NDK `r28c`: + +```text +28.2.13676358 +``` + +Typical locations: + +```text +$HOME/Android/Sdk +$HOME/Android/Sdk/ndk/28.2.13676358 +``` + +Validate the NDK layout: + +```bash +ls "$HOME/Android/Sdk/ndk/28.2.13676358" +test -f "$HOME/Android/Sdk/ndk/28.2.13676358/NOTICE" && echo "NDK OK" +test -f "$HOME/Android/Sdk/ndk/28.2.13676358/build/cmake/android.toolchain.cmake" && echo "Toolchain OK" +``` + +## Persist Android environment variables + +Add the SDK and NDK paths to your shell startup file: + +```bash +cat >> ~/.bashrc <<'EOF' +# Android SDK / NDK +export ANDROID_HOME="$HOME/Android/Sdk" +export ANDROID_NDK="$ANDROID_HOME/ndk/28.2.13676358" +export PATH="$ANDROID_HOME/platform-tools:$PATH" +export PATH="$ANDROID_HOME/cmdline-tools/latest/bin:$PATH" +EOF + +source ~/.bashrc +``` + +Verify the configuration: + +```bash +echo "$ANDROID_HOME" +echo "$ANDROID_NDK" +adb --version +test -f "$ANDROID_NDK/NOTICE" && echo "NDK OK" +test -f "$ANDROID_NDK/build/cmake/android.toolchain.cmake" && echo "Toolchain OK" +``` + +## Connect the Vivo over ADB + +If `adb devices` shows `no permissions`, add the user to `plugdev`, install the generic Android udev helpers, and reload the rules: + +```bash +sudo usermod -aG plugdev "$USER" +sudo apt install -y android-sdk-platform-tools-common +sudo udevadm control --reload-rules +sudo udevadm trigger +newgrp plugdev +``` + +The tested device reported this USB ID: + +```text +Bus 002 Device 002: ID 2d95:6001 vivo vivo X300 Pro +``` + +Add a Vivo-specific rule if the default rules are not enough: + +```bash +sudo tee /etc/udev/rules.d/51-vivo-android.rules >/dev/null <<'EOF' +SUBSYSTEM=="usb", ATTR{idVendor}=="2d95", MODE="0660", GROUP="plugdev", TAG+="uaccess" +EOF + +sudo chmod 644 /etc/udev/rules.d/51-vivo-android.rules +sudo udevadm control --reload-rules +sudo udevadm trigger +``` + +Then restart ADB and accept the RSA prompt on the phone: + +```bash +adb kill-server +adb start-server +adb devices +``` + +Expected final state: + +```text +10AFB40J6Q0031C device +``` + +## Verify Vulkan support on the phone + +Check the Vulkan implementation: + +```bash +adb shell getprop ro.hardware.vulkan +``` + +The measured device returned: + +```text +mali +``` + +Then verify the relevant Android features: + +```bash +adb shell pm list features | grep -i vulkan +``` + +Expected features include: + +```text +feature:android.hardware.vulkan.compute +feature:android.hardware.vulkan.level=1 +feature:android.hardware.vulkan.version=4206592 +feature:android.software.vulkan.deqp.level=132711169 +``` + +`vulkan_renderengine: false` from SurfaceFlinger does not block application-side Vulkan compute. The more important signal is that the device advertises `android.hardware.vulkan.compute`. diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-2-install-executorch-and-download-the-model.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-2-install-executorch-and-download-the-model.md new file mode 100644 index 0000000000..7e1a48b94d --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-2-install-executorch-and-download-the-model.md @@ -0,0 +1,93 @@ +--- +title: Install ExecuTorch and download the model +description: Clone the ExecuTorch release branch, create the Python environment, and fetch the gated Llama 3.2 model files. +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Clone ExecuTorch 1.4 + +Keep a source checkout because you will cross-compile the Android runtime and the standalone Llama runner: + +```bash +cd ~ +git clone --branch release/1.4 --recursive https://github.com/pytorch/executorch.git +cd executorch +git submodule update --init --recursive +``` + +Create and activate a Python virtual environment: + +```bash +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip setuptools wheel +``` + +To reduce memory pressure during builds: + +```bash +export CMAKE_BUILD_PARALLEL_LEVEL=2 +``` + +## Fix the PyTorch version mismatch + +On the tested `release/1.4` branch, a host environment with PyTorch `2.14.0+cpu` broke the build with: + +```text +ATen/core/Tensor.h:70:37: error: 'C10_LIFETIMEBOUND' does not name a type +``` + +Pin the expected version explicitly: + +```bash +python -m pip uninstall -y torch +python -m pip install "torch==2.13.0+cpu" --index-url https://download.pytorch.org/whl/cpu +python -c "import torch; print(torch.__version__)" +``` + +Expected output: + +```text +2.13.0+cpu +``` + +Then clean and rebuild the host package: + +```bash +python install_executorch.py --clean +export CMAKE_BUILD_PARALLEL_LEVEL=2 +./install_executorch.sh --use-pt-pinned-commit +``` + +The important lesson is to verify `torch.__version__` before you start a long native build. + +## Download Llama 3.2 1B Instruct + +The model used in this workflow came from the gated Hugging Face repository `meta-llama/Llama-3.2-1B-Instruct`. + +Keep the model outside the source tree: + +```bash +hf auth login +hf download \ + meta-llama/Llama-3.2-1B-Instruct \ + --include "original/*" \ + --local-dir ~/Llama-3.2-1B-Instruct +``` + +You need these files: + +```text +~/Llama-3.2-1B-Instruct/original/consolidated.00.pth +~/Llama-3.2-1B-Instruct/original/params.json +~/Llama-3.2-1B-Instruct/original/tokenizer.model +``` + +The measured run used: + +- `consolidated.00.pth` at about 2.4 GB +- `params.json` +- `tokenizer.model` at about 2.1 MB diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-3-export-the-vulkan-pte.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-3-export-the-vulkan-pte.md new file mode 100644 index 0000000000..2f84c5f6ca --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-3-export-the-vulkan-pte.md @@ -0,0 +1,63 @@ +--- +title: Export the Vulkan PTE +description: Export a Vulkan-enabled ExecuTorch program for Llama 3.2 1B Instruct with the guide's measured settings. +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Configure export variables + +From the ExecuTorch checkout: + +```bash +cd ~/executorch +source .venv/bin/activate + +export MODEL_DIR="$HOME/Llama-3.2-1B-Instruct/original" +export QUANT="8da4w" +export GROUP_SIZE="64" +export CONTEXT_LENGTH="2048" +``` + +## Export Llama 3.2 with Vulkan enabled + +Run the export command used for the successful device run: + +```bash +python -m examples.models.llama.export_llama \ + -c "$MODEL_DIR/consolidated.00.pth" \ + -p "$MODEL_DIR/params.json" \ + -d fp32 \ + --vulkan \ + -qmode "$QUANT" \ + -G "$GROUP_SIZE" \ + --max_seq_length "$CONTEXT_LENGTH" \ + --max_context_length "$CONTEXT_LENGTH" \ + -kv \ + --use_sdpa_with_kv_cache \ + --metadata '{"append_eos_to_prompt": 0, "get_bos_id":128000, "get_eos_ids":[128009, 128001]}' \ + --model "llama3_2" \ + --output_name "$MODEL_DIR/Llama3.2-1B-Instruct_vulkan_8da4w_g64_c2048.pte" +``` + +Expected result: + +```text +Llama3.2-1B-Instruct_vulkan_8da4w_g64_c2048.pte +``` + +The exported file was about `1.8 GB`. + +## Understand the export choices + +These options matter for reproducing the measured run: + +- `8da4w` uses dynamic 8-bit activations and 4-bit weights on the relevant quantized paths. +- `-G 64` sets the weight group size to `64`. +- `-kv` and `--use_sdpa_with_kv_cache` enable the KV cache path used during generation. +- `--max_seq_length` and `--max_context_length` were both set to `2048`. +- The metadata sets the BOS and EOS handling expected by the instruct model. + +If you change the quantization mode, group size, or context length, expect file size, memory usage, and throughput to change. diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-4-build-the-android-vulkan-runtime.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-4-build-the-android-vulkan-runtime.md new file mode 100644 index 0000000000..133d9589d4 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-4-build-the-android-vulkan-runtime.md @@ -0,0 +1,109 @@ +--- +title: Build the Android Vulkan runtime +description: Install the host Vulkan SDK and compile the Android ExecuTorch runtime and Llama runner with Vulkan enabled. +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Install the host Vulkan SDK + +The Android Vulkan build needs a host `glslc` to compile GLSL shaders to SPIR-V. + +Install a recent LunarG Vulkan SDK on the Linux host: + +```bash +sudo apt update +sudo apt install -y curl xz-utils +mkdir -p "$HOME/vulkan" +cd "$HOME/vulkan" + +SDK_VERSION=$(curl -fsSL https://vulkan.lunarg.com/sdk/latest/linux.txt) +curl -fL -o vulkan_sdk.tar.xz \ + "https://sdk.lunarg.com/sdk/download/${SDK_VERSION}/linux/vulkan_sdk.tar.xz" +tar xf vulkan_sdk.tar.xz +source "$HOME/vulkan/$SDK_VERSION/setup-env.sh" +``` + +Validate that the host SDK is active: + +```bash +which glslc +glslc --version +echo "$VULKAN_SDK" +``` + +The important requirement is that `which glslc` resolves to the host SDK, not to an incompatible Android NDK copy. + +## Configure and build ExecuTorch for Android plus Vulkan + +From the ExecuTorch checkout: + +```bash +cd ~/executorch +source .venv/bin/activate +rm -rf cmake-out-android-so + +cmake . \ + -DCMAKE_INSTALL_PREFIX=cmake-out-android-so \ + -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake" \ + -DANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=ON \ + --preset android-arm64-v8a \ + -DANDROID_PLATFORM=android-28 \ + -DPYTHON_EXECUTABLE="$(which python)" \ + -DCMAKE_BUILD_TYPE=Release \ + -DEXECUTORCH_PAL_DEFAULT=posix \ + -DEXECUTORCH_BUILD_LLAMA_JNI=OFF \ + -DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=ON \ + -DEXECUTORCH_BUILD_VULKAN=ON \ + -DEXECUTORCH_BUILD_TESTS=OFF \ + -B cmake-out-android-so + +cmake --build cmake-out-android-so \ + -j4 \ + --target install \ + --config Release +``` + +`EXECUTORCH_BUILD_LLAMA_JNI=OFF` is intentional. The first attempt failed in the JNI target with: + +```text +ld.lld: error: unable to find library -lextension_asr_runner +clang++: error: linker command failed with exit code 1 +``` + +That failure came from `executorch_jni`, not from the Vulkan runtime itself. For the standalone `adb` plus `llama_main` flow, disabling JNI is the simpler fix. + +## Build the Android `llama_main` runner + +Build the example runner against the installed Android runtime: + +```bash +cmake examples/models/llama \ + -DCMAKE_INSTALL_PREFIX=cmake-out-android-so \ + -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake" \ + -DANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=ON \ + -DEXECUTORCH_ENABLE_LOGGING=ON \ + -DANDROID_ABI=arm64-v8a \ + -DANDROID_PLATFORM=android-28 \ + -DCMAKE_BUILD_TYPE=Release \ + -DPYTHON_EXECUTABLE="$(which python)" \ + -B cmake-out-android-so/examples/models/llama + +cmake --build cmake-out-android-so/examples/models/llama \ + -j4 \ + --config Release +``` + +Verify the output binary: + +```bash +file cmake-out-android-so/examples/models/llama/llama_main +``` + +Expected shape: + +```text +ELF 64-bit LSB pie executable, ARM aarch64, ... interpreter /system/bin/linker64 +``` diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-5-deploy-run-and-measure.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-5-deploy-run-and-measure.md new file mode 100644 index 0000000000..1d27945da3 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-5-deploy-run-and-measure.md @@ -0,0 +1,84 @@ +--- +title: Deploy, run, and measure +description: Push the runtime artifacts to the Android phone, run the model, and capture baseline performance output. +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Push the runner, model, and tokenizer + +Create a runtime directory on the phone and push the artifacts: + +```bash +export MODEL_DIR="$HOME/Llama-3.2-1B-Instruct/original" + +adb shell mkdir -p /data/local/tmp/llama + +adb push \ + cmake-out-android-so/examples/models/llama/llama_main \ + /data/local/tmp/llama/llama_main + +adb push \ + "$MODEL_DIR/Llama3.2-1B-Instruct_vulkan_8da4w_g64_c2048.pte" \ + /data/local/tmp/llama/llama32-vulkan.pte + +adb push \ + "$MODEL_DIR/tokenizer.model" \ + /data/local/tmp/llama/tokenizer.model + +adb shell chmod 755 /data/local/tmp/llama/llama_main +adb shell ls -lh /data/local/tmp/llama +``` + +## Run the first successful inference + +The measured run used a plain text prompt and one warmup pass: + +```bash +adb shell 'cd /data/local/tmp/llama && \ +./llama_main \ + --model_path=llama32-vulkan.pte \ + --tokenizer_path=tokenizer.model \ + --prompt="What is the capital of France?" \ + --seq_len=120 \ + --temperature=0 \ + --warmup=1' +``` + +The successful run reported: + +| Metric | Observed value | +|---|---| +| Model load time | 3.033 s | +| Prompt tokens | 7 | +| Generated tokens | 112 | +| Prompt evaluation | 0.157 s / 44.586 tokens/s | +| Decode | 112 tokens in 3.760 s / 29.787 tokens/s | +| Total measured inference | 3.917 s / 28.593 tokens/s overall | +| Time to first generated token | 0.157 s | +| RSS after model load, prefill, and generation | about 2404.8 MiB | +| Sampling time | 0.190 s over 119 tokens | + +The runtime also emitted a `PyTorchObserver` summary with: + +```text +prefill_token_per_sec = 44.586 +decode_token_per_sec = 29.7872 +``` + +## Use the instruct chat template + +The first prompt was useful for validation, but it is not the cleanest prompt shape for an instruct model. A better test uses the chat control tokens and `max_new_tokens`: + +```bash +adb shell /data/local/tmp/llama/llama_main \ + --model_path=/data/local/tmp/llama/llama32-vulkan.pte \ + --tokenizer_path=/data/local/tmp/llama/tokenizer.model \ + --temperature=0 \ + --max_new_tokens=32 \ + --prompt="<|begin_of_text|><|start_header_id|>user<|end_header_id|>What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>" +``` + +If the runner warns that `max_new_tokens` was not provided and it is falling back to `seq_len`, update the command. For instruct models, `max_new_tokens` is the clearer control for generation length. diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-6-validate-and-troubleshoot.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-6-validate-and-troubleshoot.md new file mode 100644 index 0000000000..3475a009fa --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/how-to-6-validate-and-troubleshoot.md @@ -0,0 +1,63 @@ +--- +title: Validate Vulkan execution and troubleshoot +description: Confirm that the model is using Vulkan on-device and resolve common deployment and runtime issues. +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Validate that Vulkan is in use + +You can validate the run at three different levels. + +### Process library check + +During a long generation, capture the process ID and inspect the mapped libraries: + +```bash +PID=$(adb shell pidof llama_main | tr -d '\r') +adb shell "cat /proc/$PID/maps | grep -Ei 'vulkan|mali'" +``` + +Seeing `libvulkan` and the Mali driver libraries confirms that the process loaded the Vulkan stack. It does not prove that every expensive model op ran on the GPU, but it is a fast sanity check. + +### ExecuTorch ETDump and Inspector + +For stronger backend evidence, rebuild with tracing enabled: + +```text +-DEXECUTORCH_BUILD_DEVTOOLS=ON +-DEXECUTORCH_ENABLE_EVENT_TRACER=ON +``` + +Then capture ETDump data from the runner and inspect it with ExecuTorch Inspector to see delegated regions and delegate-call timings. + +### Perfetto or Android Performance Analyzer + +For a graphical view, collect a trace with Perfetto or Android Performance Analyzer and inspect the Vulkan and GPU queue tracks during inference. + +## Known warnings and their meaning + +| Observed warning or behavior | Interpretation | +|---|---| +| Tokenizer JSON parse error | The runner first tried a Hugging Face JSON tokenizer, then logged that it loaded the TikToken tokenizer and continued successfully. | +| Failed to open `/sys/devices/soc0/image_version` | CPU topology probing could not read a vendor-specific path. The runner still selected an 8-thread thread pool and continued. | +| Could not open `/sys/module/mali_kbase/parameters/large_page_conf` | A Mali driver diagnostic path was unavailable to the process. This did not block the measured run. | +| `vulkan_renderengine: false` | SurfaceFlinger was not using Vulkan for display composition. That is separate from application Vulkan compute. | +| Repeated `Paris is the capital...` output | The initial test used a raw completion-style prompt with an instruct model and a long generation limit. Use the chat template plus `max_new_tokens`. | + +## Repeatable checklist + +- Android SDK and NDK `r28c` installed and exported through `ANDROID_HOME` and `ANDROID_NDK` +- ExecuTorch `release/1.4` checkout present with submodules +- Python virtual environment active and `torch.__version__` verified as `2.13.0+cpu` +- Llama original files available under `~/Llama-3.2-1B-Instruct/original` +- `adb devices` shows the Vivo as `device` +- The phone advertises `android.hardware.vulkan.compute` +- Host Vulkan SDK loaded and `which glslc` resolves to the LunarG SDK +- Vulkan `.pte` exported with `8da4w`, group size `64`, and context length `2048` +- Android runtime built with `EXECUTORCH_BUILD_VULKAN=ON` and `EXECUTORCH_BUILD_LLAMA_JNI=OFF` +- `llama_main` built as an Android `aarch64` executable +- Model, tokenizer, and runner pushed to `/data/local/tmp/llama` +- Inference produces text and the expected performance counters diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/linux-host-architecture.svg b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/linux-host-architecture.svg new file mode 100644 index 0000000000..19c2d5ddb6 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/linux-host-architecture.svg @@ -0,0 +1,61 @@ + + Linux host architecture + Export, quantization, and Android build preparation on the host system, producing a Vulkan-partitioned ExecuTorch PTE artifact and staging Android NDK and ADB deployment inputs. + + + + + + + + + + + + + + + + + Linux host architecture + Export, quantization, and Android build preparation on the host system + + + + Linux host + PyTorch + ExecuTorch 1.4 + • Load Llama 3.2 checkpoint and tokenizer + • Apply 8da4w quantization settings + • Lower and partition the graph for Vulkan + + export / quantize / Vulkan partition + + + + + Output artifact + Llama3.2-1B-Instruct + _vulkan_8da4w + _g64_c2048.pte + + + + Host responsibilities + + + + + + + + + + Android NDK cross-build inputs + + + + + + + ADB deployment staging for the phone + diff --git a/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/overview.md b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/overview.md new file mode 100644 index 0000000000..22532ceded --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/executorch-vulkan-learning-path/overview.md @@ -0,0 +1,55 @@ +--- +title: Overview +description: Review the target architecture, host responsibilities, and end-to-end flow for running ExecuTorch with Vulkan on Android. +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Goal + +The goal is to run Meta Llama 3.2 1B Instruct directly on a Vivo X300 Pro with ExecuTorch, using the phone GPU through the Vulkan backend. + +The Linux host is used for: + +- model export +- quantization +- graph lowering and partitioning +- Android cross-compilation +- ADB deployment + +The Linux host does not need CUDA, ROCm, or a working Vulkan GPU. + +## Linux host architecture + +![Diagram showing the Linux host architecture for the ExecuTorch Vulkan workflow, including PyTorch and ExecuTorch 1.4, export and quantization stages, and the generated Vulkan-enabled program artifact#center](linux-host-architecture.svg "Linux host architecture for the ExecuTorch Vulkan workflow") + +## Android device architecture + +![Diagram showing the Android device architecture for the ExecuTorch Vulkan workflow, with llama_main calling the ExecuTorch runtime, which uses the Vulkan backend on the Mali GPU inside the Vivo X300 Pro#center](android-device-architecture.svg "Android device architecture for the ExecuTorch Vulkan workflow") + +## Measured result + +The successful measured run produced the following values: + +| Metric | Observed value | +|---|---| +| Model load time | 3.033 s | +| Prompt tokens | 7 | +| Generated tokens | 112 | +| Prompt evaluation | 0.157 s / 44.586 tokens/s | +| Decode | 112 tokens in 3.760 s / 29.787 tokens/s | +| Total measured inference | 3.917 s / 28.593 tokens/s overall | +| Time to first token | 0.157 s | +| RSS after model load, prefill, and generation | about 2404.8 MiB | + +## What you will build + +By the end of this Learning Path you will have: + +- an ExecuTorch 1.4 host environment with the correct PyTorch pin +- a Vulkan-enabled Llama `.pte` +- an Android `llama_main` binary for `arm64-v8a` +- the model, tokenizer, and runner deployed under `/data/local/tmp/llama` +- a reproducible validation flow for checking that Vulkan is actually in use