[Xeicker]

Context:

During a prolonged startup-crash outage on physical Pixel 6 bots, 4 of 7 phones became unusable without manual reflashing (1 with f2fs /data corruption and 3 in adb unauthorized state):

• Android 11+ defaults adb_allowed_connection_time to 7 days, auto-revoking /data/misc/adb/adb_keys if ADB goes idle across an outage. Set it to 0 during bot init and configure_device_settings().
• adb.hard_reset() ran echo b > /proc/sysrq-trigger (an immediate kernel restart without unmounting /data, and without su root when adbd runs as shell) before trying adb reboot. Flush buffers (sync) and try graceful adb reboot first, falling back to root sysrq-trigger only if adb reboot fails or times out.
• adb.factory_reset() broadcasted MASTER_CLEAR on physical devices after boot timeouts, wiping /data/misc/adb/adb_keys. Exit via bad_state_reached() on physical phones (matching Cuttlefish) while keeping MASTER_CLEAR for emulators.

Testing:

• Unit tests (adb_test.py, device_test.py) and butler.py lint.
• Tested on a physical Pixel 6 bot: verified adb_allowed_connection_time went from 'null' to '0' and persisted across reboot, factory_reset() exited via bad_state_reached() without wiping /data, hard_reset() did a clean init reboot (ro.boot.bootreason=reboot), and a simulated adb reboot failure fell back to su root /proc/sysrq-trigger and reset the kernel.

[IvanBM18]

I think i get the issue with the adb-keys but i didn't understood quite well the hard reset issue. Im approving this but i would ask also ask review for someone with more understanding about android

[smolkaj]

Thansk foir the PR!

One question:

Tested on a physical Pixel 6 bot

I take it this is referring to manual testing? Do we have the ability to automate such tests today?

[smolkaj]

How sure are we that we rely on ADB keys at /data/misc/adb/adb_keys rather than keys baked into the build or similar?

I am asking because AI pointed out that the regular flash seems to also wipe that file using fastboot -w, and work fine.

Anyhow, AI suggests that running the following on a phone from the incident should settle the question:

    adb shell getprop ro.adb.secure     # 0 = ADB auth is off, any host is trusted
    adb shell ls -l /adb_keys /product/etc/security/adb_keys   # keys baked into the build

Could we give that a quick go?

[Xeicker]

@smolkaj

Ran those on one of the phones from the incident:

$ adb shell getprop ro.adb.secure
1
$ adb shell ls -l /adb_keys /product/etc/security/adb_keys
lrw-r--r-- 1 root root  30 2009-01-01 00:00 /adb_keys -> /product/etc/security/adb_keys
-rw-r--r-- 1 root root 723 2009-01-01 00:00 /product/etc/security/adb_keys

/product/etc/security/adb_keys contains the OS image's default vendor key, whereas these lab hosts authenticate with their own lab key stored only in /data/misc/adb/adb_keys (confirmed by comparing the public keys on the device vs. host). Syncing the keys would be a whole new work and still needs these changes for stability

Regarding flash.py (fastboot -w): we actually disabled auto-flashing on these bots because it also leaves the phones in an unhealthy state.

[smolkaj]

Thanks for double checking the key location. LGTM

[Xeicker]

Thansk foir the PR!

One question:

Tested on a physical Pixel 6 bot

I take it this is referring to manual testing? Do we have the ability to automate such tests today?

I don't think we can automate on the CI at least because they aren't devices connected. It seems like there is code to be able to test locally but in this case, being a such destructive test is better not to do so

[smolkaj]

Thansk foir the PR!
One question:

Tested on a physical Pixel 6 bot

I take it this is referring to manual testing? Do we have the ability to automate such tests today?

I don't think we can automate on the CI at least because they aren't devices connected. It seems like there is code to be able to test locally but in this case, being a such destructive test is better not to do so

I'm a bit worried that we have to rely on manual testing, since it doesn't prevent future regressions and thus doesn't scale. But this seems like a larger topic that is not in scope for this PR.

[IvanBM18]

Thansk foir the PR!
One question:

Tested on a physical Pixel 6 bot

I take it this is referring to manual testing? Do we have the ability to automate such tests today?

I don't think we can automate on the CI at least because they aren't devices connected. It seems like there is code to be able to test locally but in this case, being a such destructive test is better not to do so

I'm a bit worried that we have to rely on manual testing, since it doesn't prevent future regressions and thus doesn't scale. But this seems like a larger topic that is not in scope for this PR.

Hi Steffen, just chiming in

FYI we have this bug: b/485715892 (bugs under this feature are public but the main feature is not) for that same concern, is a WIP and its currently paused(because of bandwidth) but overall the design is approve and the foundations are already here in the repo.

[smolkaj]

Thanks for the pointers, @IvanBM18, this is useful context.