ci(deps): narrow the all extra to core addons, add a dev extra (#910)

.github/workflows/cron-conda.yml

@@ -103,12 +103,20 @@ jobs:
         with:
           python-version: '3.14'
 
+      # `.[all,dev]`, not `.[all]` alone: `util/conda-dist.py` below only
+      # *refreshes* the version pin of every package already named in
+      # `conda/requirements.txt` -- via `importlib.metadata.version(name)`,
+      # which raises unless that distribution is installed here -- and that
+      # file names `dpkt`, `scapy`, `pyshark`, `requests` and `beautifulsoup4`
+      # with no marker. #910 narrowed `all` to core addons only, so those five
+      # now come from `dev` instead; without it this step would install
+      # cleanly and the next one would crash on the first missing name.
       - name: Install and Setup
         run: |
           set -x
 
           python -m pip install -U pip setuptools wheel packaging
-          python -m pip install -e .[all]
+          python -m pip install -e .[all,dev]
           #python -m pip install pathlib2 typing_extensions
 
       - name: Setup Conda Distribution

.github/workflows/cron-vendor.yml

@@ -112,13 +112,15 @@ jobs:
           python -m pip install -U pip setuptools wheel packaging
           python -m pip install -U isort
 
-          # `all` does not carry `pycrate` -- see its own comment in
-          # pyproject.toml -- so `vendor` is installed alongside it
-          # specifically for that: `pcapkit-vendor` below regenerates every
-          # crawler with no target given, including pcapkit.vendor.ngap's two
-          # (#880), and both need `pycrate_asn1dir.NGAP` importable. See the
-          # `vendor` extra's own comment in pyproject.toml for the full
-          # reasoning (size, licence) behind installing it here at all.
+          # `all` carries `pycrate` since #910 (a core addon there now, see its
+          # own comment in pyproject.toml), but not `requests`/`beautifulsoup4`
+          # any more -- #910 moved those to the `dev` extra instead, which is
+          # for the toolchain rather than for a vendor crawl. `vendor` is
+          # installed alongside `all` for those two: `pcapkit-vendor` below
+          # regenerates every crawler with no target given, and seven of them
+          # import `bs4` at module scope while `pcapkit.vendor.default` imports
+          # `requests` the same way (#507). See the `vendor` extra's own
+          # comment in pyproject.toml for the full reasoning.
           python -m pip install -e .[all,vendor]
 
       - name: Update Vendor

.github/workflows/deploy-pages.yml

@@ -99,6 +99,10 @@ jobs:
       # so an autodoc import failure is legible in the run log. Both are read at
       # import time, so they are exported before anything imports `pcapkit` --
       # `docs/source/conf.py` sets `PCAPKIT_SPHINX` itself for the same reason.
+      # `.[all,dev]`, not `.[all]` alone: autodoc imports every module, including
+      # the engine wrappers, so it needs the same imports resolvable that
+      # `lint.yml` does and for the same reason (#910's `dev` extra, added when
+      # that issue narrowed `all` to core addons only).
       - name: Install and Build 🔧
         run: |
           set -x
@@ -108,7 +112,7 @@ jobs:
 
           python -m pip install -U pip setuptools wheel
           python -m pip install -r docs/requirements.txt
-          python -m pip install -e .[all]
+          python -m pip install -e .[all,dev]
 
           rm -rf docs/build
           make -C docs html

.github/workflows/lint.yml

@@ -173,17 +173,21 @@ jobs:
         with:
           python-version: '3.14'
 
-      # `.[all]` rather than a bare install: pylint and mypy resolve imports, so
-      # findings depend on what is importable. Note `pypcap` and `pypcapfile` are
-      # marked `python_version < '3.12'` in pyproject.toml and so are absent here
-      # by design -- that is the source of pylint's 8 `import-error` messages,
-      # and they will persist until those engines support a current Python.
+      # `.[all,dev]` rather than a bare install: pylint and mypy resolve imports,
+      # so findings depend on what is importable. #910 narrowed `all` to core
+      # addons only (`cli`, `crypto`, `NGAP`), so the `dev` extra it added is
+      # what puts `dpkt`/`scapy`/`pyshark`/`pypcapfile` back for this job --
+      # without it, narrowing `all` would have grown the count below rather
+      # than left it alone. Note `pypcap` and `pypcapfile` are marked
+      # `python_version < '3.12'` in pyproject.toml and so are absent here by
+      # design -- that is the source of pylint's 8 `import-error` messages, and
+      # they will persist until those engines support a current Python.
       - name: Install package and lint tools
         run: |
           set -x
 
           python -m pip install -U pip setuptools wheel
-          python -m pip install -e .[all]
+          python -m pip install -e .[all,dev]
           python -m pip install -U vermin pylint mypy bandit
 
       # Every step below runs the *Makefile* target rather than spelling the

README.md

@@ -38,12 +38,13 @@ The extraction engines and plug-ins are optional extras:
 pip install pypcapkit[DPKT]         # or Scapy, PyShark, PyPCAPFile, PyPCAP, PCAP_CT
 pip install pypcapkit[crypto]       # ESP payload decryption
 pip install pypcapkit[cli]          # command line interface
-pip install pypcapkit[all]          # every pure-Python extra
+pip install pypcapkit[all]          # core addons only: cli + crypto + NGAP (pycrate)
 ```
 
-Four of the engines need something beyond a `pip install` -- a `tshark` binary, a
-C compiler, `libpcap` headers, or an older interpreter -- and `all` deliberately
-excludes both `pypcap` and `pcap-ct`, which must never be installed together.
+Every engine above is on demand; `all` bundles only the core addons the library
+needs for full functionality. Four of the engines also need something beyond a
+`pip install` -- a `tshark` binary, a C compiler, `libpcap` headers, or an older
+interpreter -- and `pypcap`/`pcap-ct` must never be installed together.
 The [installation guide](https://jarryshaw.github.io/PyPCAPKit/#installation)
 covers every constraint and the reason for it, and `pcapkit` enforces each one in
 code: asking for an engine that cannot run in the current environment warns with

docs/source/contributing/pep.rst

@@ -490,8 +490,10 @@ New Engines
 ``engine='pypcap'`` selects :class:`pcapkit.foundation.engines.pypcap.PyPCAP`;
 each has a matching :mod:`pcapkit.toolkit` module
 (:mod:`pcapkit.toolkit.pypcapfile`, :mod:`pcapkit.toolkit.pypcap`), a
-``pyproject.toml`` extra (``PyPCAPFile``, which ``all`` includes, and
-``PyPCAP``, which it deliberately does not -- see below), docs
+``pyproject.toml`` extra (``PyPCAPFile`` and ``PyPCAP``, neither of which
+``all`` includes -- ``PyPCAP`` for the installability reason below, and
+``PyPCAPFile`` because GitHub issue #910 narrowed ``all`` to core addons
+only), docs
 under :doc:`/pcapkit/foundation/engines/index`, and tests under
 ``tests/foundation/engines/`` and ``tests/toolkit/``. Both were verified
 end-to-end against the sample captures: each agrees with the ``default`` engine
@@ -538,8 +540,10 @@ both of the following are worth knowing before reaching for them:
   build. Since there is no wheel to fall back on, the extra is kept **out of**
   ``all``: otherwise ``pip install pypcapkit[all]`` would demand a compiler and
   the libpcap development files from every user, and it broke the docs, conda
-  and release workflows -- all of which install ``.[all]`` -- on the macOS
-  runner, where :file:`pcap.h` is present but no ``libpcap.dylib`` is.
+  and release workflows -- all of which install ``.[all]``, and some now
+  ``.[all,dev]`` since GitHub issue #910 narrowed ``all`` to core addons only
+  -- on the macOS runner, where :file:`pcap.h` is present but no
+  ``libpcap.dylib`` is.
 
   This is now solved, though not by changing the ``PyPCAP`` extra. `pcap-ct
   <https://pypi.org/project/pcap-ct/>`__ re-implements the ``pypcap`` API in pure

docs/source/index.rst

@@ -313,10 +313,12 @@ plug-in functions, you may want to install the optional ones:
    pip install pypcapkit[PCAP_CT]
    # for ESP payload decryption
    pip install pypcapkit[crypto]
-   # and to install the optional packages -- note this excludes PyPCAP and pcap-ct
+   # and to install every core addon at once -- CLI display, ESP decryption
+   # and NGAP decoding; none of the engines above are included, install each
+   # on demand as shown
    pip install pypcapkit[all]
    # or to do this explicitly
-   pip install pypcapkit dpkt scapy pyshark pypcapfile
+   pip install pypcapkit emoji cryptography pycrate
 
 .. important::
 

pyproject.toml

@@ -111,20 +111,26 @@ pcapkit-vendor = "pcapkit.vendor.__main__:main"
 cli = [ "emoji" ]
 # for ESP payload decryption, c.f. pcapkit.protocols.internet.esp
 crypto = [ "cryptography>=3.4" ]
-# for NGAP decoding, c.f. pcapkit.protocols.application.ngap. Deliberately kept
-# out of ``all``, for two reasons that are each sufficient on their own:
+# for NGAP decoding, c.f. pcapkit.protocols.application.ngap. Counted among
+# ``all``'s *core addons* by the owner's ruling on #910 -- "everything that
+# makes the library itself work at full functionality" -- even though the
+# ``pycrate`` this needs is not a free addition:
 #
 #   * Size. ``pycrate`` ships every specification it has ever compiled in one
 #     distribution: installing it lands ~238 MB of ``pycrate_asn1dir`` (87 spec
-#     modules) to obtain the one 4.9 MB ``NGAP.py`` this needs. Paying that in
-#     ``all`` -- a 50x multiplier over the useful part -- to support one
-#     application protocol is not a trade to make on a user's behalf.
+#     modules) to obtain the one 4.9 MB ``NGAP.py`` this needs. That is a 50x
+#     multiplier over the useful part, to support one application protocol.
 #   * Licence. ``pycrate`` is LGPL-2.1+ where this package is BSD-3-Clause. That
-#     is fine as an optional import a user chooses, and it is not fine as
-#     something ``pip install pypcapkit[all]`` pulls in silently: LGPL carries
-#     obligations on redistribution that a BSD-licensed dependent may not want,
-#     and ``all`` reads as "the rest of the same thing" rather than as a licence
-#     decision.
+#     is fine as an optional import a user chooses, and it is a real thing for
+#     ``pip install pypcapkit[all]`` to pull in without being asked: LGPL
+#     carries obligations on redistribution that a BSD-licensed dependent may
+#     not want.
+#
+# #910's owner accepted that trade deliberately rather than leaving ``all``
+# contradict the reasoning that used to keep ``pycrate`` out of it -- see the
+# ``all`` extra's own comment below for the rest of that ruling. This extra
+# still exists on its own, for installing NGAP support without the ``cli`` and
+# ``crypto`` core addons ``all`` bundles it with.
 #
 # No version floor: NGAP has shipped precompiled in ``pycrate_asn1dir`` for many
 # releases and the two entry points used here (``NGAP_PDU.from_aper`` and
@@ -196,40 +202,65 @@ PCAP_CT = [
 ]
 # pypcapfile 0.12.0's ``linklayer`` module imports :mod:`imp`, removed in Python
 # 3.12, and ``savefile`` imports ``linklayer`` -- so the package installs cleanly
-# and then raises ``ModuleNotFoundError`` the moment the engine is used. Upstream
-# master has fixed it but has not released.
+# and is then unusable there. It does *not* raise when the engine is selected:
+# ``Extractor.run`` consults ``PyPCAPFile.unsupported_reason()`` before the import
+# test and degrades to the default engine with a warning. That guard exists
+# precisely because a bare ``import pcapfile`` succeeds on 3.12+, so an
+# import-only check would be satisfied and let the ``ModuleNotFoundError`` escape
+# from ``__init__`` instead. Upstream master has fixed it but has not released.
 PyPCAPFile = [ "pypcapfile; python_version < '3.12'" ]
 # for developers -- installs pcapkit.vendor's crawlers, run via
 # `pcapkit-vendor` to regenerate pcapkit.const. Includes ``pycrate`` so
-# pcapkit.vendor.ngap's two crawlers (#880) can run: the owner's ruling on
-# this issue is that it belongs here even though it is excluded from
-# ``all`` below, for the same size and licence reasons the ``NGAP`` extra
-# above explains in full (~238 MB of ``pycrate_asn1dir``, LGPL-2.1+ against
-# this package's BSD-3-Clause) -- a developer opting into ``vendor`` has
-# already accepted a far heavier and more permissively-licensed dependency
-# set than an end user installing ``all`` ever would.
+# pcapkit.vendor.ngap's two crawlers (#880) can run, and ``requests``/
+# ``beautifulsoup4`` for the seven crawlers that import them at module scope
+# (c.f. the ``test`` extra's own comment below for which, and #507). #910
+# narrowed ``all`` to core addons only, so this extra now names all three
+# itself rather than leaning on ``all`` to also carry them -- ``pycrate``
+# happens to be a core addon in its own right post-#910 (see the ``all``
+# extra's comment below), but ``requests``/``beautifulsoup4`` are not, having
+# moved to ``dev`` instead. Either way, this extra's own requirements are what
+# it is correct against, not whatever ``all`` happens to carry.
 vendor = [ "requests[socks]", "beautifulsoup4[html5lib]", "pycrate" ]
-# NB: ``pypcap`` is *not* included here. It is an sdist-only C extension, so
-# ``pip install pypcapkit[all]`` would require a compiler and the libpcap
-# development files on every platform, and fails where only the header is
-# present -- as on the macOS runners. Install it explicitly with
-# ``pip install pypcapkit[PyPCAP]`` once libpcap is available.
-#
-# ``pcap-ct`` is left out too, for a different reason: it needs no toolchain at
-# all, but it and ``libpcap`` are published only as pre-releases, and ``all``
-# should not be the way somebody ends up with a beta they did not ask for. It is
-# ``pip install pypcapkit[PCAP_CT]``, which says what it is installing.
+# #910: narrowed to *core addons* only -- the owner's own words are "everything
+# that makes the library itself work at full functionality", not "everything an
+# end user might ever want". An earlier ruling on the same issue read the
+# latter and added ``pycrate`` on that basis; the ruling that stuck narrowed it
+# again, this time by kind: ``cli``, ``crypto`` and ``pycrate`` (NGAP) are core
+# addons and stay, and the four 3rd-party capture engines that used to be here
+# -- ``dpkt``, ``scapy``, ``pyshark``, ``pypcapfile`` -- are "on demand for each
+# one" like ``PyPCAP`` and ``PCAP_CT`` always were, and move out to their own
+# extras below. Install one explicitly: ``pip install pypcapkit[DPKT]`` /
+# ``[Scapy]`` / ``[PyShark]`` / ``[PyPCAPFile]``.
 #
-# ``pycrate`` is left out for the size and licence reasons set out on the
-# ``NGAP`` extra above; ``pip install pypcapkit[NGAP]`` is the way to it.
-all = [
-    "emoji",
-    "cryptography>=3.4",
+# ``pypcap`` and ``pcap-ct``/``libpcap`` stay out for the reason they always
+# have, which #910 leaves untouched: installability, not desirability.
+# ``pypcap`` is an sdist-only C extension that needs a compiler and the libpcap
+# development files, and fails where only the header is present -- as on the
+# macOS runners. ``pcap-ct`` and ``libpcap`` need no compiler, but are published
+# only as pre-releases, and ``all`` should not be how somebody ends up with a
+# beta they did not ask for. Install either explicitly: ``pip install
+# pypcapkit[PyPCAP]`` or ``pip install pypcapkit[PCAP_CT]``.
+all = [ "emoji", "cryptography>=3.4", "pycrate" ]
+# Not for an end user -- for CI, which has a different need entirely: pylint,
+# mypy and Sphinx's autodoc all resolve imports against whatever is installed,
+# so narrowing ``all`` above would otherwise have made every one of
+# ``dpkt``/``scapy``/``pyshark``/``pypcapfile`` newly unresolvable to those
+# tools. This extra is what a workflow installs *alongside* ``all`` --
+# ``.[all,dev]`` -- to see everything ``all`` used to carry, for that purpose
+# only. ``.github/workflows/lint.yml``'s own comment tracks exactly 8
+# ``import-error`` messages, from ``pypcap``/``pcap-ct`` (via ``PyPCAP`` and
+# ``PCAP_CT`` above, both absent on the Python it lints) and from
+# ``pypcapfile`` being excluded there by the same marker this extra repeats
+# below; this is the extra that keeps that count at 8 rather than growing it,
+# and it is not a place to add ``pypcap``/``pcap-ct`` just to silence a
+# finding CI has already decided to carry as advisory.
+dev = [
     "dpkt", "scapy", "pyshark",
-    # same marker as the PyPCAPFile extra: installed on 3.12+ this package is not
-    # merely useless but actively harmful, since `engine='pypcapfile'` then raises
-    # ModuleNotFoundError rather than warning and falling back to the default
-    # engine, which is what happens when it is simply absent
+    # same marker and reasoning as the PyPCAPFile extra above and the old
+    # ``all``: on 3.12+ the package installs cleanly and is then unusable, so
+    # the marker keeps the resolver from fetching something that could never
+    # run. It is not guarding against a hard error -- ``Extractor.run`` already
+    # degrades to the default engine with a warning, installed or not
     "pypcapfile; python_version < '3.12'",
     "requests[socks]", "beautifulsoup4[html5lib]",
 ]

tests/_dependency_gates.py

@@ -122,9 +122,11 @@
 #: :file:`python-compatibility.yml` installs a bare ``.`` and only compiles and
 #: imports, and :file:`codeql-analysis.yml` installs nothing explicitly at all
 #: (CodeQL's own autobuild step). That is exactly why this module keys on *jobs that
-#: run pytest* rather than on install lines anywhere in the workflow tree:
-#: ``.[all]`` carries ``pypcapfile``, ``pyshark`` and ``scapy``, so a guard
-#: reading those lines would satisfy nearly every flag here vacuously.
+#: run pytest* rather than on install lines anywhere in the workflow tree: three
+#: of those five install ``.[all,dev]`` -- carrying ``pypcapfile``, ``pyshark``
+#: and ``scapy`` through the ``dev`` extra #910 added when it narrowed ``all``
+#: to core addons only -- so a guard reading those lines would satisfy nearly
+#: every flag here vacuously.
 WORKFLOW = _tiers.ROOT / '.github' / 'workflows' / 'unit-tests.yml'
 #: Where the extras are declared.
 PYPROJECT = _tiers.ROOT / 'pyproject.toml'
@@ -1404,10 +1406,12 @@ def pytest_jobs(workflow: 'Optional[pathlib.Path]' = None) -> 'tuple[Job, ...]':
     """The jobs of ``workflow`` that run :program:`pytest`.
 
     Keyed on running the suite, not on holding an install line: seven other
-    workflows install ``.[all]`` -- which does carry ``pypcapfile`` -- and never
-    invoke :program:`pytest`, so a guard that looked at install lines anywhere
-    would pass vacuously. Within this workflow the ``changelog`` job is
-    excluded by the same rule; it installs nothing and runs a generator.
+    workflows install ``.[all]`` somewhere, three of them as ``.[all,dev]`` --
+    which does carry ``pypcapfile``, through the ``dev`` extra #910 added when
+    it narrowed ``all`` to core addons only -- and never invoke
+    :program:`pytest`, so a guard that looked at install lines anywhere would
+    pass vacuously. Within this workflow the ``changelog`` job is excluded by
+    the same rule; it installs nothing and runs a generator.
 
     The ``workflow`` argument exists so the guard can be pointed at a doctored
     copy and shown to fail; see