Skip to content

feat(devtools): open a standalone litellm catalog PR before the version bump - #8799

Merged
Vidit-Ostwal merged 3 commits into
mainfrom
feat/release-litellm-catalog-pr
Nov 1, 2026
Merged

Vidit-Ostwal merged 3 commits into
mainfrom
feat/release-litellm-catalog-pr

Conversation

@Vidit-Ostwal

@Vidit-Ostwal Vidit-Ostwal commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Related issue

None. Internal release-tooling change.

Summary

What changed

  • devtools release gets a new Phase 0 that downloads the LiteLLM model catalog and opens its own PR (chore: update litellm model catalog for vX.Y.Z, branch chore/update-litellm-catalog-vX.Y.Z). It waits for that PR to merge, then pulls main before the version bump starts.
  • If the upstream catalog is identical to the checked-in file, no branch or PR is created, so re-running a failed release is safe.
  • New flag --skip-litellm-catalog-update skips Phase 0. It is carried into the resume hints printed on failure.
  • The catalog download is removed from the [docs-freeze] docs PR (_update_docs_and_create_pr).
  • lib/devtools/README.md, the release docstring and the litellm_catalog_update / scripts/update_litellm_catalog.py docstrings are updated to match.

Why

The catalog JSON (~3 MB, hundreds of changed entries per refresh) was bundled into the docs-freeze PR, which already carries the edge changelogs and a ~770-file docs/vX.Y.Z/ snapshot. That made the catalog change hard to review and tied a model-data update to a docs freeze. Refreshing it was also easy to miss: the last refresh was in the v1.15.31 docs PR, and upstream has since added 214 models, removed 1 and changed 457. A standalone PR makes the diff reviewable, lets it be reverted on its own, and lands the catalog on main before the release commits.

New flow (devtools release X.Y.Z)

  1. Phase 0 (new): catalog PR, skipped if unchanged or with --skip-litellm-catalog-update, then waits for merge.
  2. Phase 1: feat: bump versions to X.Y.Z PR, then waits for merge.
  3. Phase 2: release notes, then the docs PR (edge changelogs, plus the docs/vX.Y.Z snapshot and docs.json for stable releases; no catalog any more), then tag and GitHub release, PyPI publish, and deployment canary repos.
  4. Phase 3: enterprise release (unchanged).

Behavior note: devtools tag used to refresh the catalog via the docs PR; it no longer does. Only release (Phase 0) and scripts/update_litellm_catalog.py refresh it now.

Verification

  • Tests added or updated for the changed behavior
  • Relevant tests and quality checks pass locally

ruff check, ruff format and pytest lib/devtools/tests (100 passed) are clean. devtools release 9.9.9 --dry-run --skip-enterprise --no-edit shows Phase 0 before Phase 1, and with --skip-litellm-catalog-update it prints the skip message and goes straight to Phase 1. The new git/gh steps in Phase 0 were not exercised against a real remote, and I did not add a unit test for them.

Additional context

The first catalog PR after this lands will drop eu.anthropic.claude-haiku-4-5-20251001-v1:0 from the catalog; worth confirming nothing depends on that ID.

🤖 Generated with Claude Code


Note

Medium Risk
Changes release automation and git/gh sequencing; mistakes could block releases or land stale catalog data, but scope is devtools-only with unit tests for Phase 0.

Overview
devtools release now runs Phase 0 before the version bump: it refreshes the checked-in LiteLLM model catalog, opens a dedicated PR (chore: update litellm model catalog for vX.Y.Z on chore/update-litellm-catalog-vX.Y.Z), and waits for merge when the file actually changes. Unchanged catalogs skip branch/PR creation; --skip-litellm-catalog-update skips the phase entirely (including resume hints).

The large catalog JSON is no longer bundled into the [docs-freeze] docs PR—_update_docs_and_create_pr only handles edge changelogs and snapshot/docs.json work. README and module docstrings describe the reordered release flow.

Tests: new test_release_catalog_phase.py covers no-op vs PR creation and catalog restore on branch-reset failure. The Anthropic import test uses monkeypatch to evict the completion module instead of mutating sys.modules directly.

Reviewed by Cursor Bugbot for commit f4d551a. Bugbot is set up for automated code reviews on this repo. Configure here.

@strix-security

strix-security Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 9 commits after the last Strix review (15844f6). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Adds a Phase 0 to `devtools release` that refreshes the LiteLLM model catalog in its own PR before the version bump, adds a `--skip-litellm-catalog-update` flag, and removes the catalog download from the docs-freeze PR.

Review summary

This PR is an internal release-tooling change: devtools release gains a Phase 0 that downloads the LiteLLM model catalog and opens a standalone catalog PR before the version bump, with a --skip-litellm-catalog-update escape hatch, and the catalog download is removed from the [docs-freeze] docs PR.

I reviewed the new _update_catalog_and_create_pr and the edited _update_docs_and_create_pr/release code paths, and traced every value that reaches a git, gh or curl invocation. All subprocess calls use list argv with no shell, the branch name and PR title are prefixed by literals so no leading-dash option injection is possible, and version is a maintainer-supplied CLI argument with no automated caller. The download code is unchanged, uses a fixed HTTPS URL with atomic write, and its output is consumed as inert data. Removal of the catalog step from the docs PR does not weaken the [docs-freeze] title-prefix control, and all staging there remains path-scoped. No security vulnerabilities were found in the changed code.

Ruled out (4)
  • Can the `version` argument reach a shell or inject options into the new git/gh commands in Phase 0?
    `run_command` executes a list argv with no shell, and the branch name and PR title are prefixed by literals so the value can never be parsed as an option; the argument is typed by the maintainer running the tool.
  • Can pre-staged content be published to a remote through the whole-index commit in the docs PR, since `devtools tag` lacks a clean-tree check?
    The commit/push lines and the missing guard in `tag()` are unchanged by this PR, `release` runs `check_git_clean()` before Phase 0, and no attacker-controlled input reaches the index.
  • Can branch-name-only PR resolution plus swallowed `gh pr view` errors let a release proceed on an unattributed or third-party catalog PR?
    `_poll_pr_until_merged` is unchanged and only breaks on `MERGED`, which requires maintainer action in the base repo, so a fork PR can only keep the loop waiting or abort it; the failure mode is a silent wait, not a security boundary crossing.
  • Does opening the catalog PR with an empty body from a floating upstream ref introduce a supply-chain vulnerability?
    The floating URL and `sample_spec`-only validation are pre-existing and unchanged in this diff, and the catalog is consumed as inert JSON data, so the empty PR body is an audit-trail hygiene issue rather than an exploitable weakness.

Review 1 on this PR · 15844f6 · requested Oct 30 05:28 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings

Comment thread lib/devtools/src/crewai_devtools/cli.py Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 15844f6. Configure here.

Comment thread lib/devtools/src/crewai_devtools/cli.py Resolved
Vidit-Ostwal and others added 3 commits November 1, 2026 17:01
@Vidit-Ostwal
Vidit-Ostwal force-pushed the feat/release-litellm-catalog-pr branch from 8e65b3b to f4d551a Compare November 1, 2026 11:32
@Vidit-Ostwal
Vidit-Ostwal enabled auto-merge (squash) November 1, 2026 11:32
@Vidit-Ostwal
Vidit-Ostwal merged commit 5bb6f23 into main Nov 1, 2026
59 checks passed
@Vidit-Ostwal
Vidit-Ostwal deleted the feat/release-litellm-catalog-pr branch November 1, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants