GH-43146: [C++][Python] Validate list value nullability when casting

[1932819331hy-debug]

Rationale for this change

Casting a list value field to non-nullable currently accepts null values, producing an array whose data contradicts its nested field nullability. For example, [[1, null, 3], [], [4, 5]] can be cast to list<item: int64 not null>.

Fixes #43146.

What changes are included in this PR?

• Validate the converted values in list, large-list, and fixed-size-list casts. Checking the output also catches nulls introduced by dictionary decoding.
• Check only values referenced by valid parent lists, respecting both parent and child offsets.
• Propagate ancestor validity before recursive nested casts, including mixed struct/list nesting. Compact hidden map entries with the existing Flatten utility so nested keys retain the map's non-null key constraint.

Are these changes tested?

• Built arrow-compute-scalar-cast-test from the current Arrow 27 source with Apple Clang 17 in Debug mode; all 113 tests passed.
• Five new C++ regression tests cover safe/unsafe casts, all list/fixed-size-list conversion directions, dictionary decoding, parent/child slices, unknown null counts, null and empty lists, NullType, and masked descendants through lists, structs, and maps. List-view inputs also have rejection coverage.
• Added four PyArrow Table.cast regression cases for list/large-list and safe/unsafe modes. All four reproduce the missing exception against released PyArrow 26.0.0. The Python extension was not rebuilt locally; the fixed behavior was validated through the native C++ suite.
• Applied clang-format and checked the diff for whitespace errors.

Are there any user-facing changes?

List casts that would produce null values in a non-nullable value field now raise ArrowInvalid, including with safe=False. The added list-value check respects null ancestors and slice boundaries.

This PR contains a "Critical Fix" for casts that currently produce data inconsistent with the declared nested nullability.

Was AI used for this PR?

Code, regression tests, and this description were generated with AI assistance. The implementation was independently reviewed by AI agents, exercised against the reported failure, and validated with the native test suite above.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [Python] Casting list elements to non-nullable doesn't check for nulls #43146

[github-actions]

⚠️ GitHub issue #43146 has been automatically assigned in GitHub to PR creator.

[raulcd]

Fully automated PRs are not welcome, see https://arrow.apache.org/docs/developers/overview.html#ai-generated-code