Function Peekaboo Crafting Self-Masking Functions Using LLVM

[carlospolop]

🤖 Automated Content Update

This PR was automatically generated by the HackTricks News Bot based on a technical blog post.

📝 Source Information

• Blog URL: https://mdsec.co.uk/2025/10/function-peekaboo-crafting-self-masking-functions-using-llvm
• Blog Title: Function Peekaboo: Crafting Self-Masking Functions Using LLVM
• Suggested Section: Windows Hardening > Antivirus (AV) Bypass, as a new page titled "LLVM Compiler-Assisted Self-Masking Functions and Runtime Code Encryption"; it could also be cross-linked from Reversing and Windows C Payloads

🎯 Content Summary

Overview

The post presents a proof-of-concept modification of the LLVM/Clang X86 backend that generates 64-bit Windows PE binaries with selectively self-masking C++ functions. A registered function is XOR-masked in process memory while inactive, decoded immediately before its original body executes, and encoded again before control returns to its caller. This is a compiler-assisted obfuscation and memory-scanner-evasion technique, not a vulnerability or CVE.

The transformation i...

🔧 Technical Details

Compiler-generated self-masking functions: Modify the target backend rather than source code or LLVM IR so selected functions receive a position-independent prologue and epilogue while preserving their normal calling convention. The prologue derives the function address through call/pop, records it in thread-local storage, and calls a symmetric decoder. The epilogue transfers control to the same handler so the body is re-encoded before the handler returns using the original caller's return address.

Machine-return redirection: Appending an epilogue is insufficient because optimized functions can contain several machine-level returns. Find the final return in a pre-emit MachineFunctionPass and erase it so execution falls into the appended epilogue. Replace every earlier return with a jump to the masking handler, ensuring that all normal exit paths re-mask the code. Existing stack/frame teardown instructions...

🤖 Agent Actions

I’ll first map the relevant Windows AV-evasion, PE, reversing, and payload documentation; then compare existing coverage across at least five pages, verify primary sources, implement the smallest non-duplicative update in one content page, and validate links, formatting, and the final diff.
Plan status: repository discovery is in progress. I’m locating candidate pages and checking the working tree before reading and comparing the required set of relevant files.
Repository discovery found an existing src/windows-hardening/av-bypass.md, so the suggested new page is not yet justified. Next I’m identifying related PE parsing, self-modifying-code, shellcode, LLVM/compiler-obfuscation, and reversing coverage before choosing whether to extend that page.
I found no existing self-masking/LLVM backend coverage, but adjacent material exists on LLVM obfuscators, runtime unpacking, and executable-page permission changes. I’m now reading six candidate pages in context, including their references and style, to avoid duplicating packer-analysis content.
The six-file batch was truncated, so I’m narrowing the reads per file. The likely destination remains the existing AV-bypass page, specifically near its obfuscation material, but I still need to confirm reference numbering and whether the reversing/crypto pages already explain the same runtime toggle pattern.
The existing AV page already documents whole-image sleep masking, making a concise comparison valuable: this blog’s novel technique is per-function compiler-assisted masking with machine-return rewriting and pre-CRT size discovery. I’m finishing the page comparisons and gathering exact primary-source/reference details before editing.
Placement decision: update only src/windows-hardening/av-bypass.md; creating a second page would fragment closely related sleep-masking and obfuscation content. The new section will focus on the genuinely new primitives—backend/MC emission, return redirection, pre-CRT boundary discovery, per-function toggling—and concise analysis indicators/limitations.
Source verification is underway against the companion repository rather than relying only on the supplied summary. I’ll also verify the executable-memory API requirement from Microsoft so the limitations section can distinguish PoC behavior from correct self-modifying-code hygiene.
Research is complete. I’m implementing a single, cited section in the existing AV-bypass page, with the companion code cited before the explanatory blog and Microsoft’s cache-coherency requirement cited separately. No cross-file duplicate or new navigation entry is needed.
Initial review found one wording ambiguity around early returns and an important metadata-layout caveat worth preserving. I’m correcting those before running structural and build validation.
Content review is complete and the section now captures the PoC’s QWORD/padding quirk and correct early-return behavior. Final verification will check citation/reference integrity, banned-source exclusion in the diff, single-file scope, Markdown whitespace, and an mdBook build if the toolchain is available.
Plan complete: repository comparison, source verification, implementation, and validation all passed. The only unavailable check was mdbook build because mdbook is not installed in the environment.
Updated src/windows-hardening/av-bypass.md with:

• LLVM backend and MC-layer self-masking workflow
• Machine-level return rewriting
• Pre-CRT metadata and function-boundary discovery
• Runtime XOR masking behavior
• Detection indicators and implementation limitations
• References to the companion repository, MDSec article, and Microsoft documentation

Validation:

• git diff --check passed
• Citation/reference checks passed
• Forbidden-source check passed
• Only one file modified
• mdbook build unavailable because mdbook is not installed

---

This PR was automatically created by the HackTricks Feed Bot. Please review the changes carefully before merging.

[carlospolop]

🔗 Additional Context

Original Blog Post: https://mdsec.co.uk/2025/10/function-peekaboo-crafting-self-masking-functions-using-llvm

Content Categories: Based on the analysis, this content was categorized under "Windows Hardening > Antivirus (AV) Bypass, as a new page titled "LLVM Compiler-Assisted Self-Masking Functions and Runtime Code Encryption"; it could also be cross-linked from Reversing and Windows C Payloads".

Repository Maintenance:

• MD Files Formatting: 1029 files processed

Review Notes:

• This content was automatically processed and may require human review for accuracy
• Check that the placement within the repository structure is appropriate
• Verify that all technical details are correct and up-to-date
• All .md files have been checked for proper formatting (headers, includes, etc.)

Bot Version: HackTricks News Bot v1.0