Skip to content

Add complete bundled type stubs - #237

Open
heejaechang wants to merge 4 commits into
h2non:masterfrom
heejaechang:heejaechang-complete-bundled-stubs
Open

heejaechang wants to merge 4 commits into
h2non:masterfrom
heejaechang:heejaechang-complete-bundled-stubs

Conversation

@heejaechang

@heejaechang heejaechang commented Jul 27, 2026

Copy link
Copy Markdown

Summary

  • add bundled .pyi coverage for all 13 public authored modules: package reexports, JSONPath AST, lexer/parser, exceptions, extensions, and CLI
  • model fields, mutation returns, callbacks, extension operators, PLY callbacks, aliases, inheritance, and parse/find/update/filter contracts without placeholder Any or checker suppressions
  • ship py.typed and every stub in wheel and sdist, with strict positive and negative Pyright/mypy consumption fixtures

Closes #192.

Completeness and compatibility

  • Pyright VerifyTypes: 0% baseline -> 100% (490/490 exported symbols known; 23/23 referenced non-exported symbols known)
  • current master authored surface: 13 modules, 34 classes, 6 public module functions, 237 class methods, and 709 signature slots
  • inline baseline: 10/709 signature slots annotated (1.4%); the released distribution has no py.typed, bundled stubs, or typeshed package
  • changed files: 13 .pyi files, jsonpath_ng/py.typed, setup.py, and 3 typing fixtures/config files
  • no runtime implementation or private vendored _ply files changed; all 16 runtime .py files are byte-identical to master

Mandatory suppression and uncertainty audit

Scanned all 17 added typing artifacts (13 .pyi files, py.typed, and 3 typing fixtures/config files):

  • type: ignore: 0
  • pyright: ignore: 0
  • noqa: 0
  • typing.cast(...) / cast(...): 0
  • placeholder Any: 0
  • checker-disable config, stubtest allowlist, or equivalent suppression: 0

The runtime has exactly five pre-existing # noqa lines. No precision is inferred solely from them: the 20 itertools wildcard exports are verified by runtime object identity and standard-library contracts; root and extension parse reexports are verified by object identity and concrete source signatures; Fields.fields is independently evidenced by constructor assignment, parser flow, runtime tuple values, and tests.

Stubtest representability audit

Unflagged python -m mypy.stubtest jsonpath_ng cannot reach comparison because private vendored _ply/lex.py references removed Python-2 names types.StringType and types.UnicodeType. A clean master checkout reproduces both errors and additionally reports the existing read-only override in AutoIdForDatum.value and an existing parser assignment error. Making full-package stubtest compile would require modifying or comprehensively stubbing private vendored _ply, contrary to this contribution's public-authored-module boundary.

Running stubtest over every authored leaf module reaches runtime comparison and reports exactly five conflicts:

  1. JSONPath.find(data=...)
  2. JSONPath.find_or_create(data=...)
  3. AutoIdForDatum.value
  4. AutoIdForDatum.path
  5. AutoIdForDatum.context

Each was investigated independently:

  • Runtime base JSONPath.find/find_or_create accepts the data= keyword, while concrete overrides such as This.find rename it to datum= and reject data=. Runtime introspection confirms data=True, datum=False for the base and data=False, datum=True for This, Child, Parent, Descendants, Fields, Index, Slice, and extension overrides. Declaring the base parameter keyword-capable is runtime-faithful for the base but violates static override compatibility; a general positional-plus-keyword overload produces 20 Pyright/mypy override errors across authored subclasses. Receiver-constrained overloads are not an escape hatch: mypy rejects a narrowed self because its erased type is not a supertype of JSONPath. Declaring the common base parameter positional-only is safe for every dynamic receiver but stubtest correctly notes that the base runtime accepts the keyword. Omitting concrete declarations is also not a workaround: omitting only This.find increases stubtest from five to seven conflicts because inherited data disagrees with runtime datum.
  • Runtime DatumInContext.value is a property with a setter and path/context are writable instance fields; direct assignment to all three succeeds. AutoIdForDatum is a real DatumInContext subclass and replaces value, path, and context with properties whose fset is None; direct assignment to each raises AttributeError. Faithful read-only subclass properties are rejected by both Pyright and mypy as overrides of writable attributes (clean master mypy reports this directly). Replacing the base fields/property with a generic writable descriptor and declaring explicit read-only subclass properties still produces three Pyright and four mypy override errors. Inheriting the conservative writable union surface preserves static substitutability, while stubtest correctly reports the runtime descriptors as read-only. Final, explicit property, descriptor, and special-method alternatives either fail checker semantics or falsely promise runtime behavior.

The stubs therefore keep the safest statically substitutable common contracts and expose the concrete runtime methods where representable. The five residuals are reported explicitly, never suppressed, and no unrelated API is weakened.

Annotation follow-up

A second source/runtime audit replaced the parser's object-valued PLY production boundary with a finite invariant grammar-value union. The local protocol now matches vendored YaccProduction integer and slice indexing, plus mutation, while covering token scalars, JSONPath nodes, field/index lists, filter-expression lists, and sort tuples/lists. Base and extended grammar callbacks share this union so the mutable production parameter remains override-compatible. The positive consumer now models the sample document and items with TypedDict and proves concrete Index.update identity without falsely claiming that destructive field filtering preserves required TypedDict keys.

The proposed concrete transform narrowings were checked but intentionally not committed:

  • Runtime proves Len.find values are int, Path.find values are str, Sub/Str values are str, and Split values are str | list[str].
  • DatumInContext[T] is necessarily invariant because value is writable. Therefore DatumInContext[int] and DatumInContext[str] are not subtypes of DatumInContext[object]; narrowing these override returns creates eight Pyright/mypy override errors. Making the datum wrapper covariant would be unsound because callers can replace its value.
  • Narrowing Sub.find input to str | DatumInContext[str] also violates its This.find(object) override. Runtime raises TypeError for non-string values, but the inherited API contradiction cannot be represented by a narrower override without suppression.
  • Split.find is not string-only: the * segment returns a list[str], as both source and runtime tests demonstrate.

These contracts remain conservative rather than using object as a claim of precise transform output, fake covariant wrappers, or checker exceptions.

Validation

  • exact configured python -m tox: 351 passed on CPython 3.13.12 and 3.14.3; CPython 3.12.13 repeatedly completed 350 passed with one test_roundtrip_extended Hypothesis deadline failure, reproduced unchanged on clean master (runtime exceeded 200 ms once and completed below the deadline on replay); configured 3.10/3.11 skipped because unavailable; aggregate coverage 85%
  • exact repo-configured make lint under GNU Make in WSL on equivalent clean trees: branch and clean master each exit 2 with 734 identical lines and identical SHA-256 output; zero attributable differences
  • strict Pyright and mypy positive fixtures: clean
  • negative fixtures: exactly 5 Pyright and 6 mypy diagnostics
  • wheel-installed pyright --verifytypes jsonpath_ng --ignoreexternal: 100%
  • sdist and wheel build successfully; twine check passes both, with only the pre-existing missing long_description_content_type warning
  • both archives contain exactly all 14 expected typing artifacts (13 stubs plus py.typed)
  • authored runtime source: 16 files changed = 0; configured runtime tests above preserve imports, signatures, and representative behavior

GitHub and maintainer policy

The final head has no reviews, conversation comments, inline feedback, check runs, or commit statuses. CI and CodeQL workflow runs are action_required with zero jobs created, which is GitHub's maintainer-approval gate for fork workflows rather than an attributable test failure.

Bundled stubs are the primary fit because the project owns the runtime and packaging, compatibility reexports can ship in lockstep, and users become typed from the normal distribution. A typeshed-only package would separate the contract from implementation and leave the existing distribution unmarked. The remaining maintainer decisions are whether bundled typing is preferred over typeshed and whether the runtime inheritance contradictions should eventually be normalized in a separate behavioral change.

heejaechang and others added 4 commits July 27, 2026 15:01
Ship typed interfaces for the public JSONPath AST, parsers, extensions, and CLI together with strict Pyright and mypy fixtures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Represent runtime inheritance conflicts conservatively and surface the five remaining stubtest mismatches rather than hiding them with checker directives or allowlists.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

missing py.typed marker to allow dependants to consume types

1 participant