diff --git a/.claude/skills/cli-release/SKILL.md b/.claude/skills/cli-release/SKILL.md new file mode 100644 index 0000000..68c1c9f --- /dev/null +++ b/.claude/skills/cli-release/SKILL.md @@ -0,0 +1,174 @@ +--- +name: cli-release +description: Use before creating any commit intended for release — determines correct version bump, ensures commit prefix is right, and walks through the release flow. Triggers on "release", "bump version", "merge to staging", "merge to main", "create a release", "what version will this be", or any PR from a feature branch to staging or main. +--- + +# CLI Release Skill + +Use this skill **before writing the commit message** for any change headed to production. +Getting the prefix wrong means the wrong version ships — and fixing it requires an amended force-push. + +--- + +## Step 1: Determine the correct version bump + +1. Check the current published tag: + ```bash + git fetch origin --tags + git describe --tags --abbrev=0 + ``` + +2. Examine commits since that tag on your branch: + ```bash + git log ..HEAD --oneline + ``` + +3. Apply the bump rules (highest wins): + | Commit prefix | Bump | + |---|---| + | `feat:` | **minor** — X.(Y+1).0 | + | `fix:` | **patch** — X.Y.(Z+1) | + | `chore:`, `docs:`, other | no bump (don't use for deliverable changes) | + +4. **State the expected version out loud** before any commit: + > "This will produce **1.20.1** because all commits use `fix:` (no `feat:`)." + +--- + +## Step 2: Craft the commit message + +- Use `fix:` for bug fixes, `feat:` for new features. +- If the prefix and bump disagree, add an explicit keyword override: + - `#patch` — force patch regardless of prefix + - `#minor` — force minor regardless of prefix + - `#major` — force major (rare) +- **Only use `feat:`/`fix:` for commits that touch deliverable paths**: `cortexapps_cli/`, `pyproject.toml`, `poetry.lock`, `tests/`, `docker/`. Use `chore:` for anything else. + +Example: +``` +fix: two-pass re-import for catalog entities with x-cortex-relationships #patch +``` + +--- + +## Step 3: Choose the release path + +**Two valid paths — pick based on whether you're batching fixes:** + +### Option A: Feature → main directly (single fix, ship now) +Use this for most fixes. CI runs, tests pass, PR auto-merges, publish fires. + +```bash +gh pr create --base main --head --title "fix: " +``` + +- `test-pr.yml` runs tests and auto-merges when they pass. +- `publish.yml` triggers on the resulting push to `main`. +- The workflow auto-syncs HISTORY.md back to `staging` after publish. + +### Option B: Feature → staging → main (batching multiple fixes) +Use this when you want to collect several fixes before cutting a release. + +```bash +# Step 1: land feature on staging (tests run + automerge) +gh pr create --base staging --head --title "fix: " +# Step 2: when ready to release, merge staging → main (triggers publish) +gh pr create --base main --head staging --title "Release X.Y.Z: " +``` + +--- + +## Human review (when needed) + +By default, PRs auto-merge when tests pass. When you need a human to review first: + +1. Open the PR as a **draft** — tests run for fast feedback, but automerge is skipped. +2. Share for review. Reviewer approves while still in draft. +3. Convert to ready (`gh pr ready `) — tests re-run, automerge fires. + +The reviewer approves *before* you convert, so you control the merge timing. + +```bash +# Open as draft +gh pr create --draft --base main --head --title "fix: " + +# After reviewer approves, convert to ready +gh pr ready +``` + +--- + +## Step 4: Monitor PR CI and auto-merge (no human in the loop) + +CI takes ~5 minutes. Use this polling loop — it waits 5 minutes before the first check, then polls every 60 seconds. This keeps API calls to a minimum. + +```bash +echo "Waiting 5 minutes for CI..."; sleep 300 +while true; do + STATUS=$(gh pr checks 2>/dev/null | awk '{print $2}' | sort -u) + echo "$(date '+%H:%M:%S') checks: $STATUS" + if echo "$STATUS" | grep -q "fail"; then + echo "CI failed — inspect with: gh pr checks "; break + fi + if ! echo "$STATUS" | grep -qE "pending|in_progress|queued"; then + echo "All checks passed — merging" + gh pr merge --merge + break + fi + sleep 60 +done +``` + +**Critically: do NOT add any commits during or after this loop.** The publish workflow auto-commits `chore: update HISTORY.md for main` after merging. That commit is excluded from the `paths:` trigger filter (`HISTORY.md` is not in `cortexapps_cli/**`, `docker/**`, `pyproject.toml`, or `poetry.lock`), so it will **not** trigger a second publish run. Any commit you push that *does* touch those paths will trigger a new, unwanted build. + +--- + +## Step 5: Monitor the publish workflow after merge + +The publish workflow runs 5 parallel jobs under the same `GH_TOKEN` PAT: +`pypi` → `pypi-deploy-event`, `docker`, `docker-deploy-event`, `homebrew` + +Wait 2 minutes for the workflow to start, then poll: + +```bash +echo "Waiting 2 minutes for publish workflow to start..."; sleep 120 +while true; do + gh run list --limit 5 --branch main --json status,conclusion,name,databaseId \ + --jq '.[] | "\(.name) \(.status) \(.conclusion // "running") \(.databaseId)"' + echo "---" + DONE=$(gh run list --limit 1 --branch main --json status --jq '.[0].status') + [ "$DONE" = "completed" ] && break + sleep 60 +done +# Check final result +gh run list --limit 1 --branch main --json conclusion --jq '.[0].conclusion' +``` + +**If a job fails with a rate limit error — do NOT push a new commit.** Re-run only the failed jobs: +```bash +gh run rerun --failed +``` + +**Confirm the tag was cut after a successful run:** +```bash +git fetch origin --tags && git tag --sort=-version:refname | head -3 +``` + +--- + +## Step 5: Homebrew dependency caveat + +`mislav/bump-homebrew-formula-action` **cannot** update `resource` blocks for Python dependencies. +If `pyproject.toml` or `poetry.lock` changed dependency versions, manually update `cortexapps/homebrew-tap/Formula/cortexapps-cli.rb` resource blocks after release. + +--- + +## Quick-reference: what lives where + +| Thing | Location | +|---|---| +| Versioning rules (full reference) | `CLAUDE.md` lines 165–227 | +| Changelog prefixes (`add:`, `change:`, `remove:`) | `CLAUDE.md` lines 219–224 | +| GitHub Actions release workflow | `.github/workflows/publish.yml` | +| Homebrew formula (local copy) | `homebrew/cortexapps-cli.rb` | +| Re-run failed jobs | `gh run rerun --failed ` | diff --git a/.github/workflows/test-pr.yml b/.github/workflows/test-pr.yml index 2362269..7dd933a 100644 --- a/.github/workflows/test-pr.yml +++ b/.github/workflows/test-pr.yml @@ -16,6 +16,7 @@ on: pull_request: branches: - staging + - main paths: - 'cortexapps_cli/**' - 'tests/**' @@ -68,3 +69,18 @@ jobs: - name: Test with pytest run: | just test-all + + automerge: + needs: test + # Only auto-merge on non-draft PRs. For PRs requiring human review, keep as + # draft until approved, then convert to ready — that re-triggers this workflow. + if: github.event_name == 'pull_request' && github.event.pull_request.draft == false + runs-on: ubuntu-latest + permissions: + pull-requests: write + contents: write + steps: + - name: Auto-merge PR + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: gh pr merge ${{ github.event.pull_request.number }} --merge --repo ${{ github.repository }} diff --git a/CLAUDE.md b/CLAUDE.md index 29db243..e9761ba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -143,26 +143,38 @@ Use the GitHub-recommended format: `-` Documentation-only changes (like updates to CLAUDE.md, README.md, STYLE.md) can be committed directly to `main` without going through the staging workflow. ### Release Workflow + +**Default (single-feature release):** Feature branch → PR directly to `main` 1. Create feature branch for changes -2. Create PR to merge feature branch to `staging` for testing +2. Create PR to merge feature branch to `main`: + ```bash + gh pr create --base main --head --title "feat: description" + ``` +3. Merge the PR to trigger release + +**Multi-feature release:** Use `staging` to bundle multiple features +1. Create feature branches for each change +2. Create PRs to merge each feature branch to `staging` 3. Create PR to merge `staging` to `main` to trigger release: ```bash gh pr create --base main --head staging --title "Release X.Y.Z: Description" ``` - Include the expected version number and brief description in title - List all changes in the PR body -4. Version bumping is automatic based on **conventional commit prefixes** in the commit history since the last tag: - - `feat:` prefix → **minor** version bump (new features) - - `fix:` prefix → **patch** version bump (bug fixes) - - If multiple types present, the highest wins (feat > fix) - - Default (no recognized prefix): patch bump -5. Release publishes to: - - PyPI - - Docker Hub (`cortexapp/cli:VERSION` and `cortexapp/cli:latest`) - - Homebrew tap (`cortexapps/homebrew-tap`) + +**Version bumping** is automatic based on **conventional commit prefixes** in the commit history since the last tag: +- `feat:` prefix → **minor** version bump (new features) +- `fix:` prefix → **patch** version bump (bug fixes) +- If multiple types present, the highest wins (feat > fix) +- Default (no recognized prefix): patch bump + +**Release publishes to:** +- PyPI +- Docker Hub (`cortexapp/cli:VERSION` and `cortexapp/cli:latest`) +- Homebrew tap (`cortexapps/homebrew-tap`) ### Determining the Next Version (Claude Instructions) -Before creating a staging-to-main release PR, Claude must: +Before merging to `main`, Claude must: 1. **Check the current version tag**: ```bash @@ -170,9 +182,9 @@ Before creating a staging-to-main release PR, Claude must: git describe --tags --abbrev=0 ``` -2. **Analyze commits since the last tag**: +2. **Analyze commits since the last tag** (use the source branch): ```bash - git log ..origin/staging --oneline + git log .. --oneline ``` 3. **Determine the version bump** by examining commit prefixes: diff --git a/HISTORY.md b/HISTORY.md index be45d37..3313a6c 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [1.20.0](https://github.com/cortexapps/cli/releases/tag/1.20.0) - 2026-06-11 + +[Compare with 1.19.2](https://github.com/cortexapps/cli/compare/1.19.2...1.20.0) + +### Features + +- add integration tests for users roles list ([ef639c6](https://github.com/cortexapps/cli/commit/ef639c6c49a6de44e38c9c72e4d84754af4c5e8a) by Jeff Schnitter). +- add users roles list command ([52013c9](https://github.com/cortexapps/cli/commit/52013c981a38236e8a381b7c069b5f067cbdadba) by Jeff Schnitter). + +### Bug Fixes + +- add explicit pytest import to test_users.py ([517b60a](https://github.com/cortexapps/cli/commit/517b60a48ee1977c575c840a81e90faf92c1f793) by Jeff Schnitter). + ## [1.19.2](https://github.com/cortexapps/cli/releases/tag/1.19.2) - 2026-06-10 [Compare with 1.19.1](https://github.com/cortexapps/cli/compare/1.19.1...1.19.2) diff --git a/cortexapps_cli/commands/backup.py b/cortexapps_cli/commands/backup.py index 2c4fe9d..a0fc377 100644 --- a/cortexapps_cli/commands/backup.py +++ b/cortexapps_cli/commands/backup.py @@ -1,4 +1,5 @@ from datetime import datetime +import time from typing import Optional from typing import List from typing_extensions import Annotated @@ -7,7 +8,6 @@ import os import tempfile import sys -from io import StringIO from contextlib import redirect_stdout, redirect_stderr from rich import print, print_json from rich.console import Console @@ -471,6 +471,18 @@ def import_relationships_file(file_info): return ("entity-relationships", len(results) - failed_count, [(fp, et, em) for rt, fp, et, em in results if et]) +def _has_relationships(file_path): + """Check if a catalog YAML file contains x-cortex-relationships.""" + try: + with open(file_path) as f: + content = yaml.safe_load(f) + info = content.get('info', {}) + relationships = info.get('x-cortex-relationships') + return relationships is not None and len(relationships) > 0 + except Exception: + return False + + def _import_catalog(ctx, directory): results = [] failed_count = 0 @@ -493,7 +505,7 @@ def import_catalog_file(file_info): except Exception as e: return (filename, file_path, type(e).__name__, str(e)) - # Import all files in parallel + # Pass 1: Import all files with ThreadPoolExecutor(max_workers=30) as executor: futures = {executor.submit(import_catalog_file, file_info): file_info[0] for file_info in files} results = [] @@ -506,6 +518,113 @@ def import_catalog_file(file_info): if failed_count > 0: print(f"\n Total catalog import failures: {failed_count}") + # Pass 2: Delete and re-create entities with x-cortex-relationships. + # On first creation (pass 1), the relationship processor runs synchronously but + # the entity hasn't been committed to the DB yet, so it caches a failed state. + # Subsequent updates don't clear this cache. Deleting the entity clears the cache, + # so the fresh re-create triggers proper relationship processing. + # + # Two-wave approach handles multi-tier hierarchies: Wave 1 re-creates all + # relationship entities (e.g., clusters → services, where services are stable from + # pass 1). Wave 2 re-creates entities whose destinations were also recreated in + # wave 1 (e.g., accounts → clusters), ensuring destinations exist before the + # relationship processor runs for those dependent sources. + relationship_files = [(fn, fp) for fn, fp in files if _has_relationships(fp)] + if relationship_files: + print(f"\n Re-importing {len(relationship_files)} entities with relationships...") + + # Collect all tags being recreated in pass 2 + pass2_tags = set() + for fn, fp in relationship_files: + try: + with open(fp) as f: + content = yaml.safe_load(f) + tag = content.get('info', {}).get('x-cortex-tag') + if tag: + pass2_tags.add(tag) + except Exception: + pass + + def delete_entity(file_info): + filename, file_path = file_info + try: + with open(file_path) as f: + content = yaml.safe_load(f) + tag = content.get('info', {}).get('x-cortex-tag') + if tag: + catalog.delete(ctx, tag=tag) + except Exception: + pass # Entity may not exist; that's fine + + def create_entity(file_info): + filename, file_path = file_info + print(f" Re-importing: {filename}") + try: + with open(file_path) as f: + catalog.create(ctx, file_input=f, _print=False) + return (filename, file_path, None, None) + except typer.Exit as e: + return (filename, file_path, "HTTP", "Validation or HTTP error") + except Exception as e: + return (filename, file_path, type(e).__name__, str(e)) + + def has_pass2_destination(file_path): + """Return True if any of this entity's relationship destinations are + also being recreated in pass 2 (i.e., they need to exist first).""" + try: + with open(file_path) as f: + content = yaml.safe_load(f) + info = content.get('info', {}) + for rel in info.get('x-cortex-relationships', []): + for dest in rel.get('destinations', []): + if dest.get('tag') in pass2_tags: + return True + except Exception: + pass + return False + + reprocess_results = [] + + # Wave 1: Delete all, then create all. This establishes every entity + # in the DB and resolves relationships whose destinations are stable + # (i.e., not being recreated in this pass). + with ThreadPoolExecutor(max_workers=30) as executor: + futures = {executor.submit(delete_entity, fi): fi[0] for fi in relationship_files} + for future in as_completed(futures): + future.result() + + time.sleep(2) # Allow deletes to commit before re-creating + + with ThreadPoolExecutor(max_workers=30) as executor: + futures = {executor.submit(create_entity, fi): fi[0] for fi in relationship_files} + for future in as_completed(futures): + reprocess_results.append(future.result()) + + # Wave 2: For entities whose destinations were also recreated in wave 1, + # do another delete+create now that those destinations exist in the DB. + dependent_files = [(fn, fp) for fn, fp in relationship_files if has_pass2_destination(fp)] + if dependent_files: + time.sleep(5) # Let wave 1 entities settle in the DB + + with ThreadPoolExecutor(max_workers=30) as executor: + futures = {executor.submit(delete_entity, fi): fi[0] for fi in dependent_files} + for future in as_completed(futures): + future.result() + + time.sleep(2) + + with ThreadPoolExecutor(max_workers=30) as executor: + futures = {executor.submit(create_entity, fi): fi[0] for fi in dependent_files} + for future in as_completed(futures): + reprocess_results.append(future.result()) + + reprocess_failed = sum(1 for fn, fp, et, em in reprocess_results if et) + if reprocess_failed > 0: + print(f"\n Total catalog re-import failures: {reprocess_failed}") + failed_count += reprocess_failed + else: + print(f" Note: relationships are processed asynchronously and will be visible within 30 seconds.") + return ("catalog", len(results) - failed_count, [(fp, et, em) for fn, fp, et, em in results if et]) def _import_plugins(ctx, directory): diff --git a/tests/test_backup_relationship_reprocess.py b/tests/test_backup_relationship_reprocess.py new file mode 100644 index 0000000..62fb002 --- /dev/null +++ b/tests/test_backup_relationship_reprocess.py @@ -0,0 +1,146 @@ +""" +Tests for the two-pass relationship reprocessing logic in backup import. + +Background: catalog entities are imported in alphabetical order. An entity early +in the alphabet (e.g. aaa-source) may reference a relationship destination +(e.g. zzz-target) that hasn't been created yet when aaa-source is first processed. +Pass 2 detects all entities with x-cortex-relationships and re-creates them after +all entities exist in the DB. + +Wave 2 handles multi-tier hierarchies: if aaa-source depends on bbb-middle, AND +bbb-middle was also recreated in wave 1, then aaa-source needs a second re-create +after bbb-middle is stable. +""" +from tests.helpers.utils import * +import os +import tempfile + + +# Minimal entity YAML with no relationships (alphabetically last — stable destination) +ZZZ_TARGET_YAML = """\ +openapi: 3.0.1 +info: + title: ZZZ Target + x-cortex-tag: zzz-target + x-cortex-type: service +""" + +# Entity with a relationship to zzz-target (alphabetically first — processed before zzz-target in pass 1) +AAA_SOURCE_YAML = """\ +openapi: 3.0.1 +info: + title: AAA Source + x-cortex-tag: aaa-source + x-cortex-type: service + x-cortex-relationships: + - tag: rel-to-zzz + destinations: + - tag: zzz-target +""" + +# Middle entity with relationship to zzz-target (also re-created in wave 1) +BBB_MIDDLE_YAML = """\ +openapi: 3.0.1 +info: + title: BBB Middle + x-cortex-tag: bbb-middle + x-cortex-type: service + x-cortex-relationships: + - tag: rel-to-zzz + destinations: + - tag: zzz-target +""" + +# Entity with relationship to bbb-middle (which is itself a pass-2 entity → triggers wave 2) +AAA_DEPENDENT_YAML = """\ +openapi: 3.0.1 +info: + title: AAA Dependent + x-cortex-tag: aaa-dependent + x-cortex-type: service + x-cortex-relationships: + - tag: rel-to-bbb + destinations: + - tag: bbb-middle +""" + + +def _make_backup_dir(tmpdir, files): + """Create a backup directory structure with the given catalog YAML files.""" + catalog_dir = os.path.join(tmpdir, "catalog") + os.makedirs(catalog_dir) + for filename, content in files.items(): + with open(os.path.join(catalog_dir, filename), "w") as f: + f.write(content) + return tmpdir + + +def test_backup_import_triggers_pass2_for_relationship_entities(monkeypatch): + """ + Pass 2 should run when catalog entities with x-cortex-relationships are present. + aaa-source (has relationships) is processed before zzz-target (no relationships) + in pass 1. Pass 2 must re-import aaa-source so its relationship is resolved. + """ + monkeypatch.setenv("CORTEX_API_KEY", "invalidKey") + + with tempfile.TemporaryDirectory() as tmpdir: + _make_backup_dir(tmpdir, { + "aaa-source.yaml": AAA_SOURCE_YAML, + "zzz-target.yaml": ZZZ_TARGET_YAML, + }) + + result = cli(["backup", "import", "-d", tmpdir], return_type=ReturnType.RAW) + + assert "Re-importing 1 entities with relationships" in result.stdout, ( + "Pass 2 should log that it is re-importing relationship entities" + ) + + +def test_backup_import_no_pass2_when_no_relationships(monkeypatch): + """ + Pass 2 should not run when no catalog entities have x-cortex-relationships. + """ + monkeypatch.setenv("CORTEX_API_KEY", "invalidKey") + + with tempfile.TemporaryDirectory() as tmpdir: + _make_backup_dir(tmpdir, { + "zzz-target.yaml": ZZZ_TARGET_YAML, + }) + + result = cli(["backup", "import", "-d", tmpdir], return_type=ReturnType.RAW) + + assert "Re-importing" not in result.stdout, ( + "Pass 2 should not run when no entities have relationships" + ) + + +def test_backup_import_triggers_wave2_for_dependent_relationships(monkeypatch): + """ + Wave 2 of pass 2 should run when an entity's relationship destination is itself + being re-created in wave 1. + + aaa-dependent → bbb-middle → zzz-target + + bbb-middle is in wave 1 (has relationships). aaa-dependent's destination (bbb-middle) + is also a pass-2 entity, so aaa-dependent must be re-created in wave 2 after + bbb-middle is stable. + """ + monkeypatch.setenv("CORTEX_API_KEY", "invalidKey") + + with tempfile.TemporaryDirectory() as tmpdir: + _make_backup_dir(tmpdir, { + "aaa-dependent.yaml": AAA_DEPENDENT_YAML, + "bbb-middle.yaml": BBB_MIDDLE_YAML, + "zzz-target.yaml": ZZZ_TARGET_YAML, + }) + + result = cli(["backup", "import", "-d", tmpdir], return_type=ReturnType.RAW) + + # Wave 1 re-creates bbb-middle (2 relationship entities total) + assert "Re-importing 2 entities with relationships" in result.stdout, ( + "Wave 1 of pass 2 should re-import all entities with relationships" + ) + # Wave 2 re-creates aaa-dependent (its destination bbb-middle was in wave 1) + assert "Re-importing" in result.stdout, ( + "Wave 2 of pass 2 should re-import entities whose destinations were recreated in wave 1" + )