feat: Validate UTF-8 at NANOARROW_VALIDATION_LEVEL_FULL

[rustyconover]

NANOARROW_VALIDATION_LEVEL_FULL is documented as the level for untrusted input, but it never checked that string values are valid UTF-8. Arrow C++'s ValidateFull() does, so a consumer that relies on FULL to reject bad input accepts strings that Arrow C++ rejects.

FULL validation now checks that every non-null string and large_string value is valid UTF-8 (RFC 3629). Overlong encodings, surrogates (U+D800 to U+DFFF), code points above U+10FFFF, stray continuation bytes and truncated sequences are rejected. Binary types and the levels below FULL are unchanged. string_view is left for a follow-up PR, because its out-of-line values can't be read safely until FULL also checks view bounds.

FULL now rejects data it used to accept. That's the point of the change, but a caller that validated non-UTF-8 bytes in a string array at FULL will now get EINVAL.

The behaviour follows Arrow C++ (checked against pyarrow 25.0.1):

• Each value is checked on its own, so a code point split across two values is invalid even though their concatenation is valid.
• Null values are not checked.
• Only the values within a slice are checked, and error indices are relative to the slice.

Errors name the value, e.g. [1] Expected valid UTF-8 string value but found invalid sequence at byte 2.

The validator is portable C with no new dependency. It skips ASCII eight bytes at a time. It first validates the whole referenced byte range in one pass and checks that each value starts on a code point boundary (not a continuation byte). That is equivalent to checking each value, but avoids a call per value for short strings. Only if that fails does it check the non-null values one at a time to report the first invalid one.

Cost of FULL validation over 10 MB, compared with main and with Arrow C++ ValidateFull() (pyarrow 25.0.1) on the same machine (Linux aarch64, Neoverse-N1, Release build):

Array	main	This PR	Arrow C++
string, 1M × 10 B ASCII	0.8 ms	3.1 ms (3.3 GB/s)	7.8 ms
string, 1M × 10 B ASCII, 10% null	0.8 ms	3.0 ms	—
string, 1M × 10 B two-byte	0.8 ms	16.1 ms (0.62 GB/s)	16.6 ms
string, 10k × 1000 B ASCII	0.01 ms	1.1 ms (9.2 GB/s)	1.1 ms
string, 10k × 1000 B mixed ASCII/two-byte	0.01 ms	19.1 ms (0.52 GB/s)	9.0 ms
large_string, 1M × 10 B ASCII	0.8 ms	3.1 ms	—
binary, 1M × 10 B	0.8 ms	0.8 ms	—

Tests: ArrayViewTestValidateUtf8 covers string and large_string. It checks valid one- to four-byte sequences, each class of invalid sequence above (with and without a null in the array), split code points, invalid bytes in null values, slices, that DEFAULT still accepts invalid UTF-8, and that binary types are not checked. ctest passes (Linux aarch64, Debug, 870 tests; NanoarrowIntegrationTestSchema hits the known shared-temp-file collision under ctest -j fixed by #969 and passes when run alone), including under ASan/UBSan with clang (395 tests). The changed files are formatted with clang-format 16.0.6, the pre-commit version.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.38%. Comparing base (10885c5) to head (718d6a6).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #981      +/-   ##
==========================================
+ Coverage   78.30%   78.38%   +0.08%     
==========================================
  Files         106      106              
  Lines       16919    17127     +208     
  Branches     2016     2018       +2     
==========================================
+ Hits        13248    13425     +177     
- Misses       2429     2459      +30     
- Partials     1242     1243       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.