Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,12 @@ Two YAML lockfiles, same format, two purposes:

Both files are generated by `dev update-deps` and committed to git. Never edit them by hand.

### Host baseline

Alongside per-project dependencies, dev ships a **host baseline** — the org-invariant tools every d3mlabs host converges toward (git, gh, rbenv, shadowenv, and on Macs the Cursor agent CLI). The manifest and its lock live inside dev's distribution (`share/baseline/`), not in any project repo: upgrading dev is what changes the baseline, and hosts never resolve — they install the shipped pin (the lock is regenerated in this repo via `bin/update-baseline.rb` and committed).

`dev up` converges the baseline as its first step, gated by an O(1) digest check against a host-side stamp (`$XDG_DATA_HOME/dev/host-baseline/installed-digest`) — a no-op on warm hosts. `dev up` also works **outside any project**: with no `dev.yml` found it converges the host layer only, which is the fresh-box bootstrap (`brew install dev` → `dev up` → ready). Every other command surfaces a stale baseline as a warn-only nag ("host baseline stale — run `dev up`") — advisory even in CI, never a block. `Dev::Deps::Baseline` owns the layer; it reuses the same resolver → lockfile → installer pipeline and `Staleness` machinery as project deps, pointed at the shipped manifest with a fixed stamp key.

### dependencies.rb

Declare dependencies using a Ruby DSL:
Expand Down Expand Up @@ -335,11 +341,14 @@ All built-in integrations are declared in one place — `lib/dev/deps/registry.r
| `steam()` | SteamIntegration | SteamRepository | deps.lock |
| `xcode()` | XcodeIntegration | XcodeRepository | deps.lock |
| `pip()` | PipIntegration | PipRepository | deps.lock |
| `cursor_agent()` | CursorAgentIntegration | CursorAgentRepository | deps.lock |

`xcode "26.1.1"` pins the Xcode toolchain (macOS only; a no-op on other hosts). dev installs the pin to `/Applications/Xcode-<ver>.app` via the [xcodes](https://github.com/XcodesOrg/xcodes) CLI — declare `brew "xcodes", host: :darwin` in `:build` so it exists first — and publishes `DEVELOPER_DIR` into the project shadowenv. Interactive runs pass any Apple ID/2FA/sudo prompt through to you; headless runs fail fast with remediation instead of hanging (normal practice: pre-install the pin interactively once during machine bring-up, e.g. a CI runner's).

`gem()` declares Ruby gems: dev generates a `Gemfile`/`Gemfile.lock` from your declarations (a top-level `gem` lands in the default group; `group(:test) { gem ... }` scopes it to a bundler group), and `dev install-deps` runs `bundle install`. `brew()` dual-writes — the container build path keeps reading the group structure while `dev install-deps` also installs the formulae on the host (idempotently).

`cursor_agent()` declares the headless Cursor agent CLI (what ai-flow spawns). Cursor publishes no release feed, so resolution fetches the served install script (`https://cursor.com/install`) and pins the version it bakes into its download URL; install downloads that exact pinned package into a version-keyed subdir of `install_dir` with a stable `current` symlink — point `AI_FLOW_AGENT_BIN` at `<install_dir>/current/cursor-agent`. Declared in the host baseline (darwin-gated), not in project manifests.

`python "3.12"` pins the Python toolchain: dev provisions the interpreter (Homebrew `python@3.12`) and a project-local `.venv`, and publishes it into the project shadowenv (`VIRTUAL_ENV` + `.venv/bin` on `PATH`). `pip()` declares packages installed into that venv — like `luarocks()`, you declare only the top-level packages and pip resolves the transitive tree at install time. Gate heavy, platform-specific stacks (e.g. a PyTorch-backed ML tool) with `host:` so only the machines that use them pay the download.

### Custom integrations
Expand Down Expand Up @@ -368,7 +377,7 @@ Custom integrations implement `Dev::Deps::Integration` (with `install_all(pins,

- **`dev update-deps`** — resolve constraints from `dependencies.rb`, write lockfiles (recording the manifest digest for the staleness check). Always available (no need to define in `dev.yml`).
- **`dev install-deps`** — install locked deps handled on the host (gh releases, steam apps) into their version-keyed install dirs, filtered to the detected env and host OS. Finishes by refreshing agent skill links (see [Agent skills & org learnings](#agent-skills--org-learnings)).
- **`dev up`** — auto-installs all deps from lockfiles (build group first), then runs the project's `up:` command from `dev.yml` if defined. On success, stamps the installed lockfile digest (see `dev check`). Finishes by refreshing agent skill links, like `install-deps`.
- **`dev up`** — first converges the host baseline when stale (see [Host baseline](#host-baseline)), then auto-installs all deps from lockfiles (build group first), then runs the project's `up:` command from `dev.yml` if defined. On success, stamps the installed lockfile digest (see `dev check`). Finishes by refreshing agent skill links, like `install-deps`. Also valid outside any project: converges the host baseline only — the fresh-box bootstrap.
- **`dev check`** — report dependency-state staleness explicitly: `dependencies.rb` vs lockfiles (digest recorded by `update-deps`), and lockfiles vs the per-machine installed stamp (`~/.dev/state/<project>/installed-digest`, written after a fully-successful `up`/`install-deps`). The same two O(1) checks run at every command start — warning on workstations, erroring in CI.
- **`dev deps path <integration> <name> <platform>`** — print the absolute path of a locked artifact (e.g. `dev deps path ficsit SML LinuxServer`, or `dev deps path xcode` for the pinned DEVELOPER_DIR) so scripts don't reconstruct cache keys or layout conventions.
- **`dev cred get <namespace> <key>`** — resolve a credential through the provider chain (ENV → keychain → file → prompt) and print it. A non-interactive miss errors with `gh secret set` guidance. Mirrors `dev deps path` for shell consumers (e.g. a staging sync). Global: works without a `dev.yml`.
Expand Down
13 changes: 13 additions & 0 deletions bin/dev
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,19 @@ end
begin
Dev::Runner.new(ui: ui).run(ARGV)
rescue Dev::DevYamlNotFoundError
# `dev up` stays valid outside any project: it converges the host baseline
# (layer 1) only, which IS the fresh-box bootstrap — install dev, `dev up`,
# ready. Everything else still needs a project.
if ARGV.first == "up"
require "dev/deps/baseline"
if Dev::Deps::Baseline.new.converge_if_stale
puts "dev: host baseline converged."
else
puts "dev: host baseline already converged."
end
puts "dev: no dev.yml here — run dev up inside a project to provision it too."
exit 0
end
warn "dev: no dev.yml found in this directory or any parent."
warn "Run dev from inside a project that defines a dev.yml."
exit 1
Expand Down
16 changes: 16 additions & 0 deletions bin/update-baseline.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env ruby
# frozen_string_literal: true

# Regenerate share/baseline/deps.lock from share/baseline/dependencies.rb.
#
# dev-repo maintenance only: run here when the baseline manifest changes,
# commit the lock, release. Consuming hosts never resolve — they install
# the shipped pin (Dev::Deps::Baseline#converge via `dev up`).

lib = File.expand_path("../lib", __dir__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)

require "dev/deps/baseline"

Dev::Deps::Baseline.new.update_lock!
puts "dev: baseline lock regenerated at #{Dev::Deps::Baseline::SHIPPED_DIR / "deps.lock"}"
126 changes: 126 additions & 0 deletions lib/dev/deps/baseline.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# frozen_string_literal: true

require "digest"
require "pathname"
require_relative "../deps"
require_relative "cache"
require_relative "dependency_installer"
require_relative "lockfile"
require_relative "registry"
require_relative "resolver"
require_relative "staleness"

module Dev
module Deps
# The host baseline layer (plans#26): the org-invariant tools every
# d3mlabs host needs (git, gh, rbenv, shadowenv, the Cursor agent CLI),
# declared in a manifest that ships INSIDE dev's distribution
# (share/baseline/) rather than in any project repo — upgrading dev is
# what changes the baseline.
#
# The lock is resolved in dev's own repo (`bin/update-baseline.rb`) and
# committed, so hosts only ever install from the shipped pin — they never
# resolve. Convergence state is one stamp per host under a fixed key
# (Staleness with state_key "host-baseline"), giving the same O(1)
# digest staleness check projects get:
#
# - `dev up` converges a stale baseline as its first step (and is the
# only remediation),
# - every other command surfaces staleness as a warn-only nag via
# DependencyService#guard! — the baseline never blocks, even in CI.
class Baseline
# The shipped manifest+lock, relative to this file (lib/dev/deps/ →
# repo or libexec root) — the installed location under brew, same
# resolution as Plan::Templates::BUNDLE_FILE.
SHIPPED_DIR = Pathname(File.expand_path(File.join(__dir__, "..", "..", "..", "share", "baseline")))

# Fixed stamp key: the baseline is host-singular, so its stamp must not
# vary with the manifest dir's path (which moves on every brew upgrade).
STATE_KEY = "host-baseline"

STALE_MESSAGE = "host baseline stale — run `dev up`"

# @param manifest_dir [Pathname, String] dir holding dependencies.rb +
# deps.lock (the shipped bundle by default)
# @param state_dir [Pathname, String] host state root — XDG data home
# like the learnings cache, NOT ~/.dev/state: project stamps are
# per-checkout working state; this is host-layer state
# @param installer_factory [#call] (lockfile, integrations) → installer;
# the DependencyInstaller seam, injectable so tests never touch the host
def initialize(
manifest_dir: SHIPPED_DIR,
state_dir: default_state_dir,
installer_factory: ->(lockfile, integrations) { DependencyInstaller.new(lockfile:, integrations:) }
)
@manifest_dir = Pathname(manifest_dir)
@state_dir = Pathname(state_dir)
@installer_factory = installer_factory
end

# The warn-only nag for a stale host, nil when converged. A distribution
# without a shipped lock has nothing to converge and is never stale.
#
# @return [String, nil]
def message
staleness.install_message && STALE_MESSAGE
end

# Install the shipped lock (filtered to this env and host OS, like any
# project install) and stamp the host converged. Stamping only happens
# after a fully-successful install, so a crashed run keeps nagging.
#
# @return [void]
def converge
lockfile = Lockfile.new(dir: @manifest_dir)
integrations = Registry.host_integrations(project_root: @manifest_dir, cache: Cache.new)
@installer_factory.call(lockfile, integrations).install(env: Deps.detect_env, host: Deps.detect_host)
staleness.stamp_installed!
end

# The O(1)-guarded converge `dev up` runs first: one digest comparison
# on a warm host, a full converge on a stale one.
#
# @return [Boolean] whether a converge ran
def converge_if_stale
return false if message.nil?

converge
true
end

# Re-resolve the manifest and rewrite the committed lock — dev-repo
# maintenance (bin/update-baseline.rb), never run on consuming hosts.
#
# @param resolver [Resolver] injectable for tests; defaults to the
# registry-wired resolver
# @return [void]
def update_lock!(resolver: Resolver.new(repositories: Registry.repositories(project_root: @manifest_dir)))
manifest = @manifest_dir / "dependencies.rb"
Deps.reset!
Kernel.load(manifest.to_s)
declarations = Deps.last_config&.declarations || []
Lockfile.new(dir: @manifest_dir).lock(
resolver.resolve(declarations),
manifest_digest: Digest::SHA256.file(manifest.to_s).hexdigest,
)
end

private

# The baseline's staleness view: the shipped dir plays the project-root
# role (it holds the lock), while the stamp lives under the fixed
# host-singular key.
#
# @return [Staleness]
def staleness
Staleness.new(project_root: @manifest_dir, state_dir: @state_dir, state_key: STATE_KEY)
end

# @return [String] $XDG_DATA_HOME/dev (the learnings-cache precedent)
def default_state_dir
data_home = ENV.fetch("XDG_DATA_HOME", File.join(Dir.home, ".local", "share"))
File.join(data_home, "dev")
end
end
end
end
123 changes: 123 additions & 0 deletions lib/dev/deps/cursor_agent_integration.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# frozen_string_literal: true

require "fileutils"
require "pathname"
require "rbconfig"
require_relative "integration"
require_relative "../deps"

module Dev
module Deps
# Lifecycle handler for the Cursor agent CLI (cursor_agent integration).
#
# Materializes the locked version into an immutable version-keyed subdir
# of its declared install_dir (install_dir/<version>/, see Integration's
# version-keyed layout) by downloading the same package the official
# install script would — https://downloads.cursor.com/lab/<version>/… —
# but pinned to the lock instead of whatever is latest. A `current`
# symlink gives consumers (AI_FLOW_AGENT_BIN, PATH entries) a stable path
# that survives version bumps.
class CursorAgentIntegration < Integration
class DownloadError < StandardError; end
class ExtractionError < StandardError; end
class UnsupportedArchitectureError < StandardError; end

MARKER_FILE = ".dev-cursor-agent"

# The served package URL scheme, as baked into the official install
# script (CursorAgentRepository resolves the version from the same
# script, so the pair drifts together or not at all).
DOWNLOAD_URL_TEMPLATE = "https://downloads.cursor.com/lab/%{version}/%{os}/%{arch}/agent-cli-package.tar.gz"

# uname-style CPU names → the scheme's arch segment.
ARCHES = { "arm64" => "arm64", "aarch64" => "arm64", "x86_64" => "x64", "amd64" => "x64" }.freeze

# Install all cursor_agent dependencies.
#
# @param dependencies [Array<Dependency>] cursor_agent deps to install
def install_all(dependencies)
dependencies.each { |dep| install(dep) }
end

private

# @param dep [Dependency]
def install(dep)
base_dir = Pathname(File.expand_path(dep.metadata["install_dir"]))
target_dir = versioned_dir(base_dir, dep.version)
if version_published?(target_dir, MARKER_FILE, dep.version)
puts ">>> #{dep.name}@#{dep.version} already installed at #{target_dir}"
publish_current(base_dir, target_dir)
return
end

# Staging lives next to the version dirs so the publish is a cheap
# same-filesystem rename; a crashed run leaves published versions
# intact.
staging_dir = new_staging_dir(base_dir)
package_dir = staging_dir / "package"
archive_path = staging_dir / "agent-cli-package.tar.gz"
FileUtils.mkdir_p(package_dir)

puts ">>> Downloading #{dep.name}@#{dep.version}"
download_package(package_url(dep.version), archive_path)
extract_package(archive_path, package_dir)

# Stamp the marker inside staging so the published dir is atomically
# complete: a reader never sees content without a valid marker.
(package_dir / MARKER_FILE).write(dep.version)
if publish_version(package_dir, target_dir)
puts ">>> Installed #{dep.name}@#{dep.version} to #{target_dir}"
else
puts ">>> #{dep.name}@#{dep.version} published concurrently at #{target_dir}"
end
publish_current(base_dir, target_dir)
ensure
FileUtils.rm_rf(staging_dir) if staging_dir
end

# The pinned package URL for this host.
#
# @param version [String] the locked version
# @return [String]
# @raise [UnsupportedArchitectureError] on a CPU the scheme has no
# package for
def package_url(version)
cpu = RbConfig::CONFIG["host_cpu"]
arch = ARCHES[cpu]
raise UnsupportedArchitectureError, "no cursor-agent package for #{cpu}" unless arch

format(DOWNLOAD_URL_TEMPLATE, version: version, os: Deps.detect_host, arch: arch)
end

# Download the package archive. Isolated so tests can stub the network
# boundary.
#
# @param url [String]
# @param archive_path [Pathname] destination .tar.gz
# @raise [DownloadError] if the download fails
def download_package(url, archive_path)
success = system("curl", "-fsSL", url, "-o", archive_path.to_s)
return if success

raise DownloadError, "cursor-agent download failed: #{url}"
end

# Extract the package, stripping its single top-level directory (the
# official script's --strip-components=1) so cursor-agent lands at the
# version dir root.
#
# @param archive_path [Pathname]
# @param package_dir [Pathname]
# @raise [ExtractionError] if tar fails
def extract_package(archive_path, package_dir)
success = system(
"tar", "--strip-components=1", "-xzf", archive_path.to_s, "-C", package_dir.to_s
)
return if success

raise ExtractionError, "cursor-agent extraction failed for #{archive_path}"
end
end
end
end
Loading
Loading