Describe the bug
Files encoded in UTF-16LE (common default output for many Windows/PowerShell and Microsoft developer tools) are silently treated as binary files and omitted from output. Because UTF-16LE strings alternate text characters with null bytes (0x00), naive binary detection flags them as binary assets.
On the original file from the Microsoft tool, file reports:
HTML document, Unicode text, UTF-16, little-endian text, with CRLF line terminators
To Reproduce
Steps to reproduce the behavior:
-
Create a UTF-16LE file with BOM:
python3 -c "open('sample_utf16.htm', 'wb').write('<!DOCTYPE html>\r\n<html>\r\n<body>\r\n<p>Hello World</p>\r\n</body>\r\n</html>\r\n'.encode('utf-16'))"(Running file sample_utf16.htm confirms: sample_utf16.htm: HTML document, Unicode text, UTF-16, little-endian text, with CRLF line terminators)
-
Run:
code2prompt --tokens --path sample_utf16.htm
-
Output shows no file data:
# Table of Contents
📋 Content copied to clipboard successfully.
✨ Token count: 5
- Clipboard contains only:
Expected behavior
- Preferred: The encoding is automatically detected, opened using that encoding, and treated as a text file.
- Acceptable: A warning is printed to stderr specifying which file was skipped and why (e.g.,
Skipping sample_utf16.htm: detected as binary).
Desktop:
- OS: Microsoft Windows 11 (reproducible cross-platform via the command above)
- code2prompt version 0.8.1
Describe the bug
Files encoded in UTF-16LE (common default output for many Windows/PowerShell and Microsoft developer tools) are silently treated as binary files and omitted from output. Because UTF-16LE strings alternate text characters with null bytes (
0x00), naive binary detection flags them as binary assets.On the original file from the Microsoft tool,
filereports:HTML document, Unicode text, UTF-16, little-endian text, with CRLF line terminatorsTo Reproduce
Steps to reproduce the behavior:
Create a UTF-16LE file with BOM:
python3 -c "open('sample_utf16.htm', 'wb').write('<!DOCTYPE html>\r\n<html>\r\n<body>\r\n<p>Hello World</p>\r\n</body>\r\n</html>\r\n'.encode('utf-16'))"(Running
file sample_utf16.htmconfirms:sample_utf16.htm: HTML document, Unicode text, UTF-16, little-endian text, with CRLF line terminators)Run:
Output shows no file data:
Expected behavior
Skipping sample_utf16.htm: detected as binary).Desktop: