[Bug] Internal smali assembler silently emits invalid DEX when `.catch` regions are nested/overlapping

[lindongbin]

Describe the bug
When rebuilding with -dex-lib = internal (the default), the smali assembler accepts nested (overlapping) .catch try-regions and emits the overlapping ranges verbatim as try_items. The DEX spec requires try items within a code_item to be sorted, in-range and non-overlapping — so the produced dex is invalid.

Official smali (baksmali/smali, also used by apktool) supports nested .catch by flattening overlaps into disjoint try ranges with merged handler sets. APKEditor's internal assembler does neither: the build succeeds with zero warnings ("Saved to: …") and the invalid dex is shipped. On device, ART then refuses to open the whole dex file, so every class in that dex becomes unresolvable at runtime (NoClassDefFoundError on a perfectly valid, unrelated class). The d command happily disassembles such a file back — nothing in the toolchain ever complains.

To Reproduce
Steps to reproduce the behavior:

1. Used version APKEditor 1.4.9, ARSCLib 1.3.9 and 1.4.0 (both affected), OpenJDK 26.0.1+8-34, device Android 13
2. Operating system Windows 11 x64
3. Command — see attached apkeditor-nestedcatch-repro.zip (includes prebuilt nested.apk / flat.apk / nested_jf.apk):

build_repro.bat <path-to-apkeditor.jar>

It builds two ready-made mini projects (identical semantics, different catch layout) and validates the produced dexes with dexcheck.py (standalone, no deps):

• mini_nested/ — one class, one method, outer .catch region fully containing an inner one (legal smali):

    :try_outer
    invoke-static {p0}, Landroid/graphics/Color;->parseColor(Ljava/lang/String;)I
    ...
    :try_inner
    invoke-static {p1}, Ljava/lang/Float;->parseFloat(Ljava/lang/String;)F
    :try_end_inner
    .catch Ljava/lang/Throwable; {:try_inner .. :try_end_inner} :inner_label
    :inner_label
    ...
    :try_end_outer
    .catch Ljava/lang/Throwable; {:try_outer .. :try_end_outer} :outer_label

• mini_flat/ — same code split into two disjoint regions (the workaround).

Observed output:

=== nested ===
00.465 I: [BUILD] Saved to: nested.apk          (success, no diagnostic)
PROBLEMS (1):
  - LRepro;#0: try overlap/unsorted start=0 prev_end=13

=== flat ===
00.466 I: [BUILD] Saved to: flat.apk
STRUCTURE OK

Log/Stacktrace
Installing the nested variant's real-world counterpart (same construct inside an app's own class) on an Android 13 device:

FATAL EXCEPTION: main
java.lang.NoClassDefFoundError: Failed resolution of: Lf/a/a/a/a;
    at androidx.preference.PreferenceManager.getDefaultSharedPreferencesName(SourceFile:1)
    at androidx.preference.PreferenceManager.getDefaultSharedPreferences(SourceFile:1)
    ...
Caused by: java.lang.ClassNotFoundException: f.a.a.a.a

The class is present in the rebuilt dex's class_defs table (verified by direct dex parsing); the whole dex is simply rejected by ART because of the invalid try-item table. Replacing the nested regions with disjoint ones makes the identical app work.

Used apk file
Not shareable (proprietary, 169 MB). However the issue is fully reproducible without it — see the attached minimal reproduction project apkeditor-nestedcatch-repro.zip (~16 KB: mini projects, prebuilt apks, validator, one-click script, README).

Additional context

• Disjoint side-by-side .catch regions, and multiple handlers over one region, are handled fine — only genuinely nested/overlapping ranges produce the corrupt output.
• The tool's own -dex-lib jf (JesusFreke/smali) backend assembles the identical mini_nested source into 3 correctly flattened, non-overlapping try_items (STRUCTURE OK). So the expected behavior is implemented inside the same codebase; only the default internal assembler emits the overlap silently.
• Suggested fix: at minimum hard-error on overlapping catch regions (naming class/method/labels); ideally flatten them like official smali does.

apkeditor-nestedcatch-repro.zip

[REAndroid]

Thank you for reporting, i will take a look and get back to you soon

[REAndroid]

@lindongbin
As you pointed dex do not allow overlapping exception handlers dex_file_verifier.cc
So logically overlapping handlers can not appear anywhere by android standard tools. Anyways we have to implement self correction since we promised to deliver better features than JF.

Please confirm the overlapping smali generated by you or your script right ?

[lindongbin]

@lindongbin As you pointed dex do not allow overlapping exception handlers dex_file_verifier.cc So logically overlapping handlers can not appear anywhere by android standard tools. Anyways we have to implement self correction since we promised to deliver better features than JF.

Please confirm the overlapping smali generated by you or your script right ?

Yes, hand-written by AI. The original decompiled smali has no overlapping catches, and a plain d + b round-trip never produces them either.

[REAndroid]

Great, i will implement it soon.
Since you mentioned AI, am working on apk agent to manipulate everything wwithout the need to disassemble apk. I would like to ask which model you are using for smali ?

[lindongbin]

Currently using DeepSeek V4 Flash.

[REAndroid]

Here is overlapping fix utility commit.
I tested with your nested scenarios and with some complex and entangled scenarios. Such operations require re-visiting smali code, am afraid it may or may not have significant impact on performance. I am thinking of introducing flag switch -evaluate-smali for end user.

Please test and let us know your results

[lindongbin]

Tested with the CI artifact of run 35763898700 (ARSCLib-1.4.0.jar, commit eef59252 = current main HEAD).

1. The nested .catch case is fixed

backend	tries	overlap	verdict
internal, 1.4.9 (ARSCLib 1.4.0)	2	1	PROBLEMS: LRepro;#0: try overlap/unsorted start=0 prev_end=13
internal, eef59252	3	0	STRUCTURE OK
jf (JesusFreke/smali)	3	0	STRUCTURE OK

mini_flat still builds clean, and the result matches the jf backend exactly.

Performance: full -no-cache rebuild of a 169 MB / 32,972 smali / 4 dex project: 124.8 s (1.4.9) vs 116.2 s (eef59252) — no measurable cost, so I don't think -evaluate-smali is needed for performance reasons. (A flag that hard-errors on any remaining invalid try_item would still be nice.)

2. But current main corrupts try_items on that same project — not caused by this fix

classes2.dex  Lcom/bytedance/apm/s/d;#12755: try out of range 28+65507>123
classes4.dex  Lh/b$b;#22091:            try out of range 19+65516>31

insn_count decodes to -29 / -20, i.e. end < start. ART rejects both dex files, so every class in them becomes unresolvable. Real device/emulator log, app built from those sources with the artifact:

java.lang.NoClassDefFoundError: Failed resolution of: Lcom/bytedance/news/common/service/manager/IService;
    at com.bytedance.android.input.B.j.m(SourceFile:1)
    at com.bytedance.android.doubaoime.ImeApplication.attachBaseContext(SourceFile:7)
Caused by: java.lang.ClassNotFoundException: com.bytedance.news.common.service.manager.IService

IService is defined in classes2.dex and is present in that dex's class_defs (11146, identical to the 1.4.9 build) — the dex is simply rejected as a whole. The 1.4.9 build of the same sources starts normally.

Bisect (APKEditor 1.4.9 + ARSCLib at a given commit):

ARSCLib	mini_nested	that project
1.4.0 (2026-08-25, shipped in 1.4.9)	overlap	OK
b4b7860 (2026-09-15)	overlap	OK
9a43ec6 (2026-09-22, "Improve smali instruction handling")	overlap	out of range
eef59252 (this fix / CI artifact)	OK	out of range

It also reproduces with SmaliOverlappingTryFix.INSTANCE.apply(this) removed, so it is not from this fix.

3. Root cause

SmaliCodeSet.getNextInstruction(int) no longer falls back to the null instruction when there is no following instruction:

// current main
for (int i = index; i < size; i++) { ... }
if (index == size) {
    return getNullInstruction();
}
return null;

At b4b7860 the equivalent code (SmaliLabel.getTargetInstruction() / searchAddress()) used an unconditional fallback to codeSet.getNullInstruction(), whose address is the end of the code.

When .catch is the last line of a method, the code set ends with ..., throw v1, :try_end_0 (label), <try item>; :try_end_0's index is not size, so getTargetAddress() returns -1 and ExceptionHandler.fromSmali feeds it into the try range end:

HandlerOffset.setCatchCodeUnit
ExceptionHandler.setCodeUnit
ExceptionHandler.setAddress
ExceptionHandler$TryEndLabel.setTargetAddress   <- ExceptionHandler.java:246
ExceptionHandler.fromSmali                      <- ExceptionHandler.java:240
CatchTypedHandler.fromSmali -> TryItem.fromSmali -> TryBlock.fromSmali -> CodeItem.fromSmali

setCodeUnit(address - startAddress) becomes -1 - startAddress (negative), and HandlerOffset.setCatchCodeUnit stores it with Block.putShort (u2) -> 65507 / 65516. Both bad values are exactly -(start + 1): -(28 + 1) and -(19 + 1).

That is why only the "handler at loop head" layouts break: there the range end can only come from a label that sits at the very end of the code set.

4. One-line fix (verified)

public SmaliInstruction getNextInstruction(int index) {
    if (index < 0) {
        return null;
    }
    int size = size();
    for (int i = index; i < size; i++) {
        SmaliCode code = get(i);
        if (code instanceof SmaliInstruction) {
            return (SmaliInstruction) code;
        }
    }
    return getNullInstruction();
}

With only this change on top of eef59252:

• mini_nested -> 3 disjoint try_items, STRUCTURE OK (this fix still works)
• mini_flat -> unchanged, STRUCTURE OK
• the single class that reproduces the regression -> STRUCTURE OK
• the 169 MB project -> all 4 dexes clean, and all 4 dexes are byte-identical to the 1.4.9 build (only the signature files differ), i.e. no side effects.

Attached arsclib-tryitem-regression-repro.zip (~9 KB): one class (Lcom/bytedance/apm/s/d;, valid smali, no nested .catch) alone in a build project is enough to reproduce, together with dexcheck.py and a one-click script.

---

This analysis (reproduction, bisect and the suggested one-line fix) was produced with AI assistance — provided for your reference.

arsclib-tryitem-regression-repro.zip

[REAndroid]

Thank you for perfect analysis, your suggested fix is on point and implemented directly here