feat(workflows): install custom step types from local dirs and archives

docs/reference/bundles.md

@@ -81,6 +81,8 @@ The source may also be a bundle directory or `.zip` artifact. Refresh uses the s
 
 A local bundle source supplies the manifest, not its component payloads. Components resolved through catalogs still require network access to refresh, even when already installed. Add `--offline` only when the components being installed or refreshed ship with Spec Kit; otherwise the command reports which component needs network access. Re-run without `--offline` to fetch that component through its catalog.
 
+> **Step payloads resolve through the step catalog only.** A bundle's `provides.steps` entries still resolve exclusively through the active step catalogs. Bundle-local `steps/<id>/` payloads and relative `provides.steps[].source` overrides are **not** resolved in this release, so a step declared that way cannot be installed offline. To ship a step with a bundle today, publish it to a step catalog the bundle's users can reach.
+
 ## Update Bundles
 
 ```bash

docs/reference/workflows.md

@@ -543,6 +543,154 @@ specify workflow run speckit -i spec="Build a kanban board with drag-and-drop ta
 
 > **Security note:** a `shell` step runs a local command with **your** privileges. There is no capability sandbox — `requires` is an advisory pre-condition block (spec-kit version, integrations), not a runtime gate, so it does **not** restrict what a step can do. In particular there is no `requires.permissions` capability gate: it is rejected by validation precisely because it would imply a sandbox that does not exist. Review any catalog or downloaded workflow before running it, and use a `gate` step to require explicit approval before sensitive or destructive shell commands.
 
+### Custom step packages
+
+Custom step types are installed with `specify workflow step`. A step is a
+directory package containing metadata and executable Python:
+
+```text
+my-step/
+├── step.yml        # required, at the package root
+├── __init__.py     # required, at the package root
+└── helpers.py      # optional nested modules and data files
+```
+
+`step.yml` declares the step's identity. `step.type_key` must exactly match the
+`<step_id>` passed on the command line — the ID is never inferred from package
+content:
+
+```yaml
+step:
+  type_key: my-step
+  name: My Step
+  version: 0.1.0
+  author: you
+  description: What this step does
+```
+
+`__init__.py` must define a `StepBase` subclass whose `type_key` matches:
+
+```python
+from specify_cli.workflows.base import StepBase, StepResult
+
+
+class MyStep(StepBase):
+    type_key = "my-step"
+
+    def execute(self, config, context):
+        return StepResult(output={"ok": True})
+```
+
+#### Install from a local directory
+
+```bash
+specify workflow step add my-step --dev /path/to/my-step
+```
+
+`--dev` takes a **directory** (not an archive, not a bare `step.yml`) that is a
+complete package. This needs no catalog, server, or network, which makes it the
+supported local-authoring loop:
+
+```bash
+specify workflow step add my-step --dev ./my-step
+specify workflow step list
+specify workflow step info my-step
+# edit ./my-step, then replace the installed copy:
+specify workflow step add my-step --dev ./my-step --force
+specify workflow step remove my-step
+```
+
+#### Install from an archive URL
+
+```bash
+specify workflow step add my-step --from https://example.com/my-step.zip
+```
+
+`--from` accepts a `.zip`, `.tar.gz`, or `.tgz` archive (a bare `step.yml`
+URL is **not** a package). The archive may place `step.yml` and `__init__.py`
+at its root or under exactly one top-level directory; unrelated top-level
+siblings are rejected. Because a step package contains executable Python, a
+direct URL install shows a default-deny trust confirmation before any network
+request; declining cancels with no request and no error. HTTPS is required
+(HTTP is permitted only for loopback hosts), redirects must remain secure, and
+downloads are size-bounded.
+
+#### Install from the catalog
+
+```bash
+specify workflow step add my-step
+```
+
+Catalog installs resolve individual file URLs from the active step catalogs and
+then go through the same validation and commit path as `--dev` and `--from`.
+Discovery-only catalogs cannot be installed from.
+
+#### Replacement and force
+
+```bash
+specify workflow step add my-step --dev ./my-step --force
+specify workflow step add my-step --from https://example.com/my-step.zip --force
+```
+
+`--force` first stages and validates the replacement before touching the
+existing installation, and can replace both a registered install and a leftover
+unregistered directory. Validation and staging failures leave the previous
+package untouched. If removing the old directory fails, the replacement is not
+published. If publishing the replacement or updating the registry fails after
+the old directory has been removed, the installation may be left incomplete:
+rerun the command with the original source and `--force` to reinstall. No
+automatic rollback is attempted.
+
+#### Package validation
+
+Every source is validated identically before anything is committed:
+
+- `step.yml` and `__init__.py` must be regular, non-symlink files at the package
+  root.
+- The package tree is copied recursively (relative imports, nested helper
+  modules, and data files are supported). A symlinked package root, any
+  descendant symlink, and any filesystem object that is not a regular file or
+  directory are rejected.
+- `.git`, `__pycache__`, and `.DS_Store` entries are skipped without being
+  inspected: they are not copied, excluded directories are not entered, and
+  they do not count toward any limit.
+- The installed-package policy permits at most **512 retained entries** (files
+  and directories combined), at most **32 levels** of directory nesting, and
+  **50 MiB** of retained content.
+- Archive URLs also pass transport/extraction safety limits before package
+  validation: at most 512 archive entries, 50 MiB downloaded or extracted, and
+  10 MiB per archive member. Catalog files have a 50 MiB per-response bound.
+- Installation validates and copies the package but does **not** import or
+  execute `__init__.py`. Installed custom step modules are loaded during startup
+  of `workflow add`, `workflow run`, and `workflow resume`, before any particular
+  custom step necessarily executes.
+
+> **Security note:** Loading a custom step runs its Python with **your**
+> privileges. Only install and retain step packages from sources you trust.
+
+#### Listing, running, and removing
+
+Installed custom steps appear in `specify workflow step list` and are loaded
+automatically by `workflow add`, `workflow run`, and `workflow resume`. Remove
+one with:
+
+```bash
+specify workflow step remove my-step
+```
+
+#### Registry provenance
+
+Each installed step records only the *kind* of its source — `catalog`
+(optionally with the catalog name), `local`, or `url`. Local paths and source
+URLs are never persisted. `specify workflow step info <id>` shows the source.
+
+#### Bundle-local limitation
+
+A bundle's `provides.steps` still resolves only through the active step
+catalogs. Bundle-local `steps/<id>/` payloads and relative
+`provides.steps[].source` overrides are **not** resolved in this release, so
+such steps are not installable offline. See the [Bundles reference](bundles.md).
+
 ### Per-Step Integration Configuration
 
 Command steps may pass structured runtime configuration to integrations that

src/specify_cli/shared_infra.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import contextlib
 import hashlib
 import hmac
 import logging
@@ -233,6 +234,60 @@ def _validate_safe_shared_directory(project_path: Path, directory: Path) -> None
             raise ValueError(f"Shared infrastructure directory escapes project root: {label}") from None
 
 
+@contextlib.contextmanager
+def _exclusive_project_lock(project_root: Path, lock_name: str, *, context: str):
+    """Hold an exclusive inter-process lock on ``.specify/<lock_name>``.
+
+    Callers use one lock file per mutable project resource so that its
+    directory and registry changes are serialized across processes. Every
+    failure to acquire the lock is raised as ``OSError``.
+    """
+    project_root = Path(project_root)
+    lock_dir = project_root / ".specify"
+    try:
+        _ensure_safe_shared_directory(
+            project_root, lock_dir, context=f"{context} lock directory"
+        )
+    except ValueError as exc:
+        raise OSError(str(exc)) from exc
+    lock_file = lock_dir / lock_name
+    if lock_file.is_symlink():
+        raise OSError(f"Refusing to use symlinked {context} lock: {lock_file}")
+
+    flags = os.O_RDWR | os.O_CREAT
+    flags |= getattr(os, "O_NOFOLLOW", 0)
+    flags |= getattr(os, "O_CLOEXEC", 0)
+    fd = os.open(lock_file, flags, 0o600)
+    try:
+        if lock_file.is_symlink():
+            raise OSError(f"Refusing to use symlinked {context} lock: {lock_file}")
+        # Call the lock primitives through their modules so tests can observe
+        # contention by patching ``msvcrt.locking`` / ``fcntl.flock``.
+        if os.name == "nt":
+            import errno
+            import msvcrt
+            import time
+
+            if os.fstat(fd).st_size == 0:
+                os.write(fd, b"\0")
+            while True:
+                os.lseek(fd, 0, os.SEEK_SET)
+                try:
+                    msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
+                    break
+                except OSError as exc:
+                    if exc.errno not in (errno.EACCES, errno.EDEADLK):
+                        raise
+                    time.sleep(0.05)
+        else:
+            import fcntl
+
+            fcntl.flock(fd, fcntl.LOCK_EX)
+        yield
+    finally:
+        os.close(fd)
+
+
 def _ensure_safe_shared_destination(
     project_path: Path,
     dest: Path,

src/specify_cli/workflows/__init__.py

@@ -75,11 +75,27 @@ def _register_builtin_steps() -> None:
 
 # The step types Spec Kit ships, snapshotted before any community step can be
 # loaded. ``load_custom_steps`` adds project-installed ids to the process-global
-# ``STEP_REGISTRY`` and never removes them, so ``STEP_REGISTRY`` cannot answer
+# ``STEP_REGISTRY`` and refreshes them for each project, so it cannot answer
 # "is this bundled with Spec Kit?" in a long-lived process: a step loaded for one
 # project would look built-in for the next. Callers that need the immutable set
 # (e.g. the bundler's reference checker) must use this instead.
 BUILTIN_STEP_TYPES: frozenset[str] = frozenset(STEP_REGISTRY)
+_CUSTOM_STEP_MODULES: set[str] = set()
+
+
+def _unload_custom_steps() -> None:
+    """Clear custom registrations and synthetic imports from a prior project."""
+    import sys
+
+    for type_key in tuple(STEP_REGISTRY):
+        if type_key not in BUILTIN_STEP_TYPES:
+            del STEP_REGISTRY[type_key]
+    for module_name in _CUSTOM_STEP_MODULES:
+        sys.modules.pop(module_name, None)
+        prefix = module_name + "."
+        for loaded_name in [name for name in sys.modules if name.startswith(prefix)]:
+            sys.modules.pop(loaded_name, None)
+    _CUSTOM_STEP_MODULES.clear()
 
 
 def load_custom_steps(project_root: Path) -> list[str]:
@@ -97,6 +113,7 @@ def load_custom_steps(project_root: Path) -> list[str]:
     import re as _re
     import sys as _sys
 
+    _unload_custom_steps()
     steps_dir = Path(project_root) / ".specify" / "workflows" / "steps"
 
     # Defense-in-depth: refuse to execute step code from a symlinked
@@ -192,6 +209,7 @@ def load_custom_steps(project_root: Path) -> list[str]:
                 _register_step(step_class())
                 loaded.append(type_key)
                 registered = True
+                _CUSTOM_STEP_MODULES.add(module_name)
             finally:
                 # If the step wasn't successfully registered (failed import,
                 # no matching StepBase subclass, or registration error), remove
@@ -206,7 +224,7 @@ def load_custom_steps(project_root: Path) -> list[str]:
                         k for k in _sys.modules if k.startswith(submodule_prefix)
                     ]:
                         _sys.modules.pop(_mod_key, None)
-        except Exception:  # noqa: BLE001
+        except Exception:  # noqa: BLE001, S112
             # Silently skip broken step packages at load time
             continue
 

src/specify_cli/workflows/_commands.py

@@ -434,51 +434,12 @@ def _stage_workflow_file(
 @contextlib.contextmanager
 def _workflow_install_transaction(project_root: Path):
     """Serialize workflow file swaps with their registry updates."""
-    from ..shared_infra import _ensure_safe_shared_directory
+    from ..shared_infra import _exclusive_project_lock
 
-    lock_dir = project_root / ".specify"
-    try:
-        _ensure_safe_shared_directory(
-            project_root, lock_dir, context="workflow install lock directory"
-        )
-    except ValueError as exc:
-        raise OSError(str(exc)) from exc
-    lock_file = lock_dir / ".workflow-install.lock"
-    if lock_file.is_symlink():
-        raise OSError(f"Refusing to use symlinked workflow install lock: {lock_file}")
-
-    flags = os.O_RDWR | os.O_CREAT
-    flags |= getattr(os, "O_NOFOLLOW", 0)
-    flags |= getattr(os, "O_CLOEXEC", 0)
-    fd = os.open(lock_file, flags, 0o600)
-    try:
-        if lock_file.is_symlink():
-            raise OSError(
-                f"Refusing to use symlinked workflow install lock: {lock_file}"
-            )
-        if os.name == "nt":
-            import errno
-            import msvcrt
-            import time
-
-            if os.fstat(fd).st_size == 0:
-                os.write(fd, b"\0")
-            while True:
-                os.lseek(fd, 0, os.SEEK_SET)
-                try:
-                    msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
-                    break
-                except OSError as exc:
-                    if exc.errno not in (errno.EACCES, errno.EDEADLK):
-                        raise
-                    time.sleep(0.05)
-        else:
-            import fcntl
-
-            fcntl.flock(fd, fcntl.LOCK_EX)
+    with _exclusive_project_lock(
+        project_root, ".workflow-install.lock", context="workflow install"
+    ):
         yield
-    finally:
-        os.close(fd)
 
 
 def _commit_workflow_file(