Hello,
Problem description
As PR #4459 reveals, the Android CI is using infrastructure and images which have been frozen and deprecated for about a decade, from the Android 7-9 eras (2016-2018), while the crate is used against current Android releases, so regressions against modern bionic are invisible to CI.
Furthermore, the deprecated emulator is slow and flaky (e.g. crashes on simple changes like #4459 ), pinned around a bug and in general not worth fixing.
Solution
The modern mechanism for headless Android testing is Cuttlefish, which is what AOSP's own CI also runs on and has been adopted by other important FOSS projects like Mesa CI (see their cuttlefish-runner script).
So this issue is to track the migration of the CI tests from the deprecated emulator to Cuttlefish.
It's worth noting that Cuttlefish boots a complete current Android system (kernel + userspace), unlike for example the current x86_64 job which just extracts bionic from a 2016 image and runs the test binaries directly on the Ubuntu host kernel.
I already have two prototypes which pass the x86_64 and aarch64 (fully green jobs with 6700/6480 tests in about 7 minutes runtime), heavily inspired from Mesa CI, but I'm opening an issue first for discussion and to track the wider effort, because this will likely require 4 or more PRs.
Challenges
Coverage shift
If we just replace the deprecated emulator with cuttlefish, then the tests move from the Android 7/9-era bionic to the latest release, which means the oldest-supported-API end is no longer covered by CI.
Thankfully cuttlefish images exist for older branches as old as Android 9 / API 28, so maybe we could have runners for both the old and new.
Cuttlefish requires KVM
On x86_64, GitHub runners already expose /dev/kvm, so we can boot and run a full Android virtual device on the CI host, very similar to how Mesa CI does it.
On aarch64 GitHub does not expose /dev/kvm yet, so we need to find a workaround. Fortunately libc-test doesn't need a booted Android, it just needs bionic and the right syscall abi, so we can just download the files for the build and run the test binaries with qemu-aarch64, similar to the existing qemu user pattern. There is no easy fallback (eg TCG mode) because cuttlefish and its backends require KVM.
This above also pave the way to boot and testing with full Android an ARM as well, sometime in the future, after Github Actions enable KVM.
32 bit support
I'd like to find solutions for 32bit support as well, so we can drop the legacy SDK emulator entirely.
Latest Android images are 64bit only.
We have multilib images up to v14 and even pure-x86 images up to v11 which can boot and run just fine under Cuttlefish on the x86_64 GitHub runners with KVM.
32bit arm however needs to run on the multilib arm64 image (no pure 32bit arm image was published) and hits the same problem described above with lack of KVM. We could use the same qemu-user (qemu-arm/x86) approach with a 32-bit linker/lib.
i686-linux-android is currently disabled due to #4297, however by migrating to cuttlefish I think we can re-enable it, maybe even run pure x86 targets with cuttlefish.
Version pinning
We need to decide which specific Cuttlefish Android guest releases to pin and test, as well as the CVD host packages.
In my prototypes, for the host I install from Google's apt repository (latest release, currently 1.55.x). Google's guidance is to keep the guest image and CVD host package from the same build, which cvd fetch handles.
Conclusion and next steps
The above is my current understanding of the problem and how a migration would look like. Likely I missed some things. Ideas are welcome.
I think we can do this transition piece-meal, in small steps, so I'll start preparing some PRs in the meantime.
Hello,
Problem description
As PR #4459 reveals, the Android CI is using infrastructure and images which have been frozen and deprecated for about a decade, from the Android 7-9 eras (2016-2018), while the crate is used against current Android releases, so regressions against modern bionic are invisible to CI.
Furthermore, the deprecated emulator is slow and flaky (e.g. crashes on simple changes like #4459 ), pinned around a bug and in general not worth fixing.
Solution
The modern mechanism for headless Android testing is Cuttlefish, which is what AOSP's own CI also runs on and has been adopted by other important FOSS projects like Mesa CI (see their cuttlefish-runner script).
So this issue is to track the migration of the CI tests from the deprecated emulator to Cuttlefish.
It's worth noting that Cuttlefish boots a complete current Android system (kernel + userspace), unlike for example the current x86_64 job which just extracts bionic from a 2016 image and runs the test binaries directly on the Ubuntu host kernel.
I already have two prototypes which pass the x86_64 and aarch64 (fully green jobs with 6700/6480 tests in about 7 minutes runtime), heavily inspired from Mesa CI, but I'm opening an issue first for discussion and to track the wider effort, because this will likely require 4 or more PRs.
Challenges
Coverage shift
If we just replace the deprecated emulator with cuttlefish, then the tests move from the Android 7/9-era bionic to the latest release, which means the oldest-supported-API end is no longer covered by CI.
Thankfully cuttlefish images exist for older branches as old as Android 9 / API 28, so maybe we could have runners for both the old and new.
Cuttlefish requires KVM
On x86_64, GitHub runners already expose
/dev/kvm, so we can boot and run a full Android virtual device on the CI host, very similar to how Mesa CI does it.On aarch64 GitHub does not expose
/dev/kvmyet, so we need to find a workaround. Fortunately libc-test doesn't need a booted Android, it just needs bionic and the right syscall abi, so we can just download the files for the build and run the test binaries withqemu-aarch64, similar to the existing qemu user pattern. There is no easy fallback (eg TCG mode) because cuttlefish and its backends require KVM.This above also pave the way to boot and testing with full Android an ARM as well, sometime in the future, after Github Actions enable KVM.
32 bit support
I'd like to find solutions for 32bit support as well, so we can drop the legacy SDK emulator entirely.
Latest Android images are 64bit only.
We have multilib images up to v14 and even pure-x86 images up to v11 which can boot and run just fine under Cuttlefish on the x86_64 GitHub runners with KVM.
32bit arm however needs to run on the multilib arm64 image (no pure 32bit arm image was published) and hits the same problem described above with lack of KVM. We could use the same qemu-user (qemu-arm/x86) approach with a 32-bit linker/lib.
i686-linux-androidis currently disabled due to #4297, however by migrating to cuttlefish I think we can re-enable it, maybe even run pure x86 targets with cuttlefish.Version pinning
We need to decide which specific Cuttlefish Android guest releases to pin and test, as well as the CVD host packages.
In my prototypes, for the host I install from Google's apt repository (latest release, currently 1.55.x). Google's guidance is to keep the guest image and CVD host package from the same build, which
cvd fetchhandles.Conclusion and next steps
The above is my current understanding of the problem and how a migration would look like. Likely I missed some things. Ideas are welcome.
I think we can do this transition piece-meal, in small steps, so I'll start preparing some PRs in the meantime.